Skip to content

v1.7.0 — Closed-loop monitoring: ground truth + sliced performance + C/C

Choose a tag to compare

View all tags
@DuqueOM DuqueOM released this 08 Aug 15:13
· 262 commits to main since this release
archive/v1.7.0
01b170a

Note

Historical audit snapshot. This release belongs to the v1.0–v1.12 line that
shipped during the post-audit remediation sprint (2026-04-15 → 2026-04-29).
Per docs/RELEASING.md
these tags are immutable historical snapshots; the active public release line
is now v0.x
hardening, with v1.0.0 reserved for the first verified GKE+EKS cloud E2E rollout.
See CHANGELOG.md for the reclassification rationale.

Release date: 2026-04-23

Closes the largest remaining gap in the template: concept drift went
silent because the system only tracked feature distributions (PSI).
This release wires predictions to their eventual ground-truth labels,
computes sliced performance metrics, and gates promotion on statistical
superiority.

Backfilled release notes (Fase F, produced 2026-04-24). See ADRs
006 (closed-loop monitoring), 007 (sliced analysis), 008
(champion/challenger), 009 (retraining orchestration triggers).

Highlights

Prediction logger (ADR-006)

  • templates/common_utils/prediction_logger.py — async buffered logger
    with 4 pluggable backends (parquet, BigQuery, SQLite, stdout) via
    PREDICTION_LOG_BACKEND env var
  • PredictionEvent frozen dataclass validates prediction_id,
    entity_id, model_version at construction (D-20)
  • Fire-and-forget (D-21), failure-tolerant (D-22)
  • Integrated in fastapi_app.py + main.py lifespan; graceful
    degradation if backend fails

Ground truth ingestion

  • monitoring/ground_truth.py — daily CronJob with user-implemented
    fetch_labels_from_source(); CSV stub for local dev + documented
    BigQuery/Postgres examples
  • Idempotent daily parquet partitions (year=/month=/day=)
  • configs/ground_truth_source.yaml — declarative source config

Sliced performance monitor (ADR-007)

  • monitoring/performance_monitor.py — JOINs predictions+labels on
    entity_id with causality (label_ts >= prediction_ts); AUC/F1/
    precision/recall/Brier globally AND per slice
  • Baseline comparison for concept drift (auc_drop_warning/alert)
  • Tri-state: ok / warning / alert / insufficient_data
  • Pushgateway metrics {slice_name, slice_value, metric} for Grafana
  • configs/slices.yaml — bounded-cardinality declarations

K8s manifests

  • k8s/base/cronjob-performance.yaml — two CronJobs (03:00 ground
    truth, 04:00 performance monitor)
  • k8s/base/performance-prometheusrule.yaml — 5 alerts:
    GlobalAUCBelowAlert, SlicedAUCBelowAlert, F1BelowAlert,
    PerformanceMonitorStale, PredictionLogErrorsHigh

Champion/Challenger statistical gate (ADR-008)

  • evaluation/champion_challenger.py — McNemar exact binomial +
    bootstrap ΔAUC 95% CI → tri-state decision (promote / keep / block)
  • configs/champion_challenger.yaml — alpha, n_bootstrap,
    non_inferiority_margin, superiority_margin
  • cicd/retrain-service.yml — new C/C gate between quality gates and
    promotion; exits 0/1/2

New anti-patterns

  • D-20 — PredictionEvent missing entity_id/prediction_id
  • D-21 — Prediction logging blocking the async event loop
  • D-22 — Logging backend failure propagating to HTTP responses

Agentic system + IDE parity

  • .windsurf/rules/13-closed-loop-monitoring.md
  • .windsurf/skills/concept-drift-analysis/ (new)
  • .windsurf/skills/drift-detection/ (extended)
  • .windsurf/skills/model-retrain/ (new Step 5.5 C/C gate)
  • .windsurf/workflows/performance-review.md
  • .cursor/rules/08-closed-loop.mdc + .claude/rules/08-closed-loop.md

ADRs

  • 006 — closed-loop monitoring
  • 007 — sliced performance analysis
  • 008 — Champion/Challenger statistical gate
  • 009 — retraining orchestration triggers

Tests

  • 50 total passing, 25 new:
    • test_prediction_logger.py — 20
    • test_ground_truth.py — 6
    • test_performance_monitor.py — 14
    • test_champion_challenger.py — 10

Breaking schema changes (services on v1.6.x)

  • PredictionRequest.entity_id now REQUIRED (min_length=1)
  • PredictionResponse.prediction_id now REQUIRED (UUID hex)
  • Optional: PredictionRequest.slice_values: dict[str, str]

New dependencies

  • pyarrow ~=18.0 (parquet backend), pyyaml ~=6.0 (config)

Scope respected

  • No Argo Workflows (ADR-009 documents triggers; GHA default)
  • No Bytewax / streaming (parquet batch covers target audience)
  • No ClickHouse default (future trigger at >100M predictions/day)
  • No Istio shadow mode (ADR-008 future work)

See also

  • CHANGELOG entry [1.7.0]
  • ADR-006, 007, 008, 009
Assets 2
Loading