v1.7.1 — Closed-Loop Hardening: warm-up, PDB, env gates, dynamic Behavior Protocol

Note

Historical audit snapshot. This release belongs to the v1.0–v1.12 line that
shipped during the post-audit remediation sprint (2026-04-15 → 2026-04-29).
Per docs/RELEASING.md
these tags are immutable historical snapshots; the active public release line
is now v0.x
hardening, with v1.0.0 reserved for the first verified GKE+EKS cloud E2E rollout.
See CHANGELOG.md for the reclassification rationale.

---

Release date: 2026-04-24

This is a hardening release for v1.7.0 (closed-loop monitoring).
Seven focused commits close operational gaps that prevented the closed
loop from being production-grade:

1. Model warm-up + readiness gating (D-23, D-24)
2. PodDisruptionBudget + Rego v1 policies (D-27)
3. Champion/Challenger in Argo Rollouts (G-02b)
4. Rollback skill + /rollback workflow (G-05)
5. Environment promotion gates dev→staging→prod (D-26, ADR-011)
6. Dynamic Behavior Protocol via mcp-prometheus (ADR-010)
7. Documentation consolidation (this file + CHANGELOG)

No breaking data changes. No migration required for existing services
EXCEPT environment-promotion adoption (documented below).

---

Highlights

Model warm-up + probe split (D-23, D-24)

Problem: the first inference after pod start took 300–800 ms while
scikit-learn/XGBoost lazily initialized the SHAP explainer. Under a
rolling deploy, pods became Ready on /health while they were still
cold — and the HPA happily sent production traffic their way.

Fix:

• app/fastapi_app.py::warm_up_model() forces a dummy predict_proba
  pass AND builds the SHAP KernelExplainer once during FastAPI
  lifespan before _warmed_up=True
• /ready returns 503 until warm-up completes (separate from /health)
• K8s manifests split: livenessProbe → /health, readinessProbe →
  /ready, new startupProbe → /health with failureThreshold: 24
• The Argo Rollout canary now waits on /ready before routing traffic

PodDisruptionBudget (D-27)

Problem: a single kubectl drain on a node with all replicas evicted
the entire service.

Fix:

• templates/k8s/base/pdb.yaml — minAvailable: 1
• templates/k8s/base/hpa.yaml — minReplicas: 2 (PDB with minAvailable=1
  cannot tolerate voluntary disruption on 1-replica services)
• templates/tests/infra/policies/closed_loop.rego — enforces presence
  • valid selector + explicit annotation escape hatch for minAvailable: 0

Argo Rollouts C/C (G-02b)

Problem: the Champion/Challenger statistical gate (McNemar + bootstrap
ΔAUC, shipped in v1.7.0) ran ONLY offline in retrain-service.yml. The
K8s canary only checked HTTP error rate and latency.

Fix:

• templates/k8s/base/analysistemplate-champion-challenger.yaml
  ships TWO AnalysisTemplates:
  • {service}-cc-online — 4 proxy metrics during canary: prediction
    log error rate, error-rate vs stable, score-distribution shift,
    p95 latency. Auto-rollback on failure.
  • {service}-cc-post-deploy — 3 real business metrics from
    performance_monitor CronJob. Manual trigger or scheduled. Does
    NOT auto-rollback (post-deploy rollback is STOP-class).

Three-layer C/C architecture is now complete:

OFFLINE        →  champion_challenger.py + retrain-service.yml
ONLINE         →  {service}-cc-online during canary
POST-DEPLOY    →  {service}-cc-post-deploy after labels arrive

Rollback skill + /rollback workflow (G-05)

The most critical on-call skill had no dedicated flow. release-checklist
mentioned "rollback plan" but no runbook existed.

New:

• .windsurf/skills/rollback/SKILL.md — 7-step STOP-class procedure:
  triage → identify revision → argo rollouts abort/undo → MLflow
  registry revert → silence alerts → verify → audit issue
• .windsurf/workflows/rollback.md — /rollback slash command with
  15-min triage budget, 5-business-day RCA follow-up

Environment promotion gates (D-26, ADR-011)

Problem: deploy-gcp.yml and deploy-aws.yml were tag-triggered into
a single flat production-{cloud} environment. No staging validation;
no required-reviewer gates.

Fix:

• templates/cicd/deploy-common.yml — reusable workflow_call centralizing
  build/auth/apply/smoke-test logic
• templates/cicd/deploy-{gcp,aws}.yml rewritten as 4-job chain:
  build → deploy-dev → deploy-staging → deploy-prod
• 6 GitHub Environments to configure (gcp/aws × dev/staging/prod) with
  explicit reviewer and wait_timer rules
• docs/environment-promotion.md — operator setup guide with migration
  steps for pre-v1.7.1 services

Dynamic Behavior Protocol via mcp-prometheus (ADR-010)

Problem: AGENTS.md's AUTO/CONSULT/STOP mapping was static. A deploy to
staging that is CONSULT at 10am Tuesday should NOT be CONSULT when a P1
incident is active, drift is severe, the error budget is exhausted, it
is 17:45 Friday, or a rollback happened < 6h ago.

Fix:

• templates/common_utils/risk_context.py — 200-line helper with a
  frozen RiskContext dataclass, escalate(base_mode) applying the
  ADR-010 table (AUTO→CONSULT→STOP, sticky STOP), 60s cache,
  fallback to ops/*.json local files when mcp-prometheus is unavailable
• mcp-prometheus promoted to CORE MCP in AGENTS.md (same tier as
  github/kubectl/terraform)
• .windsurf/rules/01-mlops-conventions.md — Dynamic Behavior Protocol
  section with the table and five signals
• docs/decisions/ADR-010-dynamic-behavior-protocol.md — the trade-off
  analysis, including why we rejected a file-only stub

---

New ADRs

• ADR-010 — Dynamic Behavior Protocol via mcp-prometheus
• ADR-011 — Environment Promotion Gates (dev→staging→prod)

New anti-patterns (D-23..D-27)

ID	Description
D-23	Liveness and readiness probes share a path
D-24	SHAP explainer rebuilt per request
D-25	Pod killed mid-request on deploy / scale-down
D-26	Deploys go directly to prod without staging validation
D-27	Deployment without PodDisruptionBudget

Full corrective actions in AGENTS.md anti-pattern table.

New files

• templates/common_utils/risk_context.py + unit tests
• templates/k8s/base/pdb.yaml
• templates/k8s/base/analysistemplate-champion-challenger.yaml
• templates/cicd/deploy-common.yml
• templates/tests/infra/policies/closed_loop.rego
• .windsurf/skills/rollback/SKILL.md
• .windsurf/workflows/rollback.md
• docs/environment-promotion.md
• docs/decisions/ADR-010-dynamic-behavior-protocol.md
• docs/decisions/ADR-011-environment-promotion-gates.md
• releases/v1.7.1.md (this file)

Changed files

• templates/service/app/fastapi_app.py — warm_up_model() function
• templates/service/app/main.py — /ready endpoint, lifespan warm-up
• templates/k8s/base/deployment.yaml + argo-rollout.yaml — probe
  split, terminationGracePeriodSeconds, startupProbe
• templates/k8s/base/hpa.yaml — minReplicas: 2
• templates/k8s/base/kustomization.yaml — 2 new resources registered
• templates/cicd/deploy-gcp.yml + deploy-aws.yml — 4-job chain
• templates/tests/infra/policies/kubernetes.rego — converted to Rego v1
• AGENTS.md — D-23..D-27 added, mcp-prometheus core MCP, rollback skill
• .windsurf/rules/01-mlops-conventions.md — 6 invariants, dynamic table
• .windsurf/rules/02-kubernetes.md — graceful shutdown, PDB sections
• .windsurf/rules/05-github-actions.md — D-26 enforcement, reusable workflows
• CHANGELOG.md — this release

Test counts

• Unit tests: 75 passing (up from 56 in v1.7.0)
  • test_risk_context.py: 19 new tests
• Rego policies: now parse cleanly on conftest v0.46+

Migration guide (v1.7.0 → v1.7.1)

1. Enable warm-up (service-level)

Update app/fastapi_app.py and app/main.py to the new lifespan
pattern. See diff in commit A1.

2. Add PodDisruptionBudget

Copy templates/k8s/base/pdb.yaml into your service's k8s/base. Add
to kustomization.yaml. Bump HPA minReplicas: 2.

3. Adopt environment promotion

1. Create 6 Environments in repo Settings (gcp-dev, gcp-staging,
   gcp-production, aws-dev, aws-staging, aws-production)
2. Set required_reviewers, wait_timer, deployment_branches per
   the table in docs/environment-promotion.md
3. Replace deploy-{gcp,aws}.yml with the new 4-job chain
4. Add k8s/overlays/{cloud}-{env}/ kustomizations
5. Remove the old flat production-{cloud} environments AFTER first
   successful pipeline

4. Install mcp-prometheus (optional)

Not required for the template to work, but unlocks the Dynamic Behavior
Protocol. Add the mcp-prometheus block shown in AGENTS.md §MCP
Integrations to your ~/.codeium/windsurf/mcp_config.json.

Related documentation

• AGENTS.md — full invariant + mode protocol reference
• docs/environment-promotion.md — operator setup for env gates
• docs/decisions/ADR-010 — dynamic Behavior Protocol
• docs/decisions/ADR-011 — environment promotion gates
• .windsurf/skills/rollback/SKILL.md — emergency revert runbook