Release list

v0.26.0 — The namespace, not the numbers Latest

Latest

github-actions released this 08 Aug 15:26

v0.26.0

b71fb59

v0.26.0 — The namespace, not the numbers

Release date: 2026-08-08

Closes the root cause behind four consecutive releases of pinning.

What was actually wrong

The repository used one mechanism — git tags — for two incompatible
purposes:

Purpose	Requirement
Release-channel markers (`v0.x`)	machine-sortable; only live releases
Frozen audit snapshots (`v1.0.0`–`v1.12.0`)	permanent; never resolved

Version-resolving tooling takes the highest-sorting tag. v1.12.0
outranked every v0.x release, so the April 2026 audit snapshot won every
unpinned resolution. That produced four defects in four releases — a
release that reached nobody, an incomplete docs fix, a destructive
copier update, and a fourth surface the guard's hand-written file list
could not see.

Every fix was a --vcs-ref pin. Pinning is correct and it treats the
symptom. The namespace collision was the disease.

The fix

v1.0.0…v1.12.0 → archive/v1.0.0…archive/v1.12.0. Same commits,
same trees, same signatures — verified 15/15 by SHA, with git diff
confirming byte-identical trees.

It works because Copier filters tags through a PEP 440 check before
sorting (copier/_vcs.py:get_latest_tag):

all_tags = (tag for tag in all_tags if valid_version(tag))
sorted_tags = sorted(all_tags, key=version.parse, reverse=True)

archive/v1.12.0 is not a valid PEP 440 version, so it is discarded before
the sort ever runs. The same property protects sort -V, git describe
heuristics, "latest release" queries and dependency bots — this is not a
Copier-specific patch.

Measured, on the exact bare commands that caused every prior defect

Command	Before	After
`get_latest_tag()`	`v1.12.0`	`v0.25.0`
`copier copy` (no ref)	435 files, no answers file	627 files, `_commit: v0.25.0`
`copier update` (no ref)	627 → 435, 582 deleted, answers file gone	627 → 627, 0 deleted, answers file present

Why not renumber

Jumping the active line past v1.12.0 (to v2.x) was the obvious
alternative and was rejected. RELEASING.md §2 reserves v1.0.0 for the
first cloud E2E validation, and the whole v0.x line exists to signal not
GA yet. Inflating the version to outrank a dead tag would make the version
number misstate the project's maturity in order to satisfy a sorting
algorithm — a worse lie than the bug it fixes.

The immutability rule, reinterpreted rather than quietly broken

ADR-014 and agentic/rules/18 declare the v1.x tags immutable. Renaming
them looks like a violation on the letter, so ADR-045 states the
distinction explicitly instead of leaving the gap for a future reader:

What the rule protects: that the past cannot be made to look better
than it was — a tag must never move to a different commit, and history
must never be re-signed to imply verification that did not happen.
What this change does: preserves every commit, tree, release note and
signature. Only the reference name moved.
Therefore: immutability attaches to the commit and its content, not
to the string used to reach it.

Deleting an archived snapshot outright remains forbidden. Archiving
preserves provenance; deletion destroys it.

Also fixed

agentic/rules/18 still required .gitleaks.toml to mirror the legacy
singular [allowlist] alongside [[allowlists]]. That mirror was removed
in v0.22.0 because gitleaks >= 8.25 refuses to load a config containing
both — the rule was mandating the exact state that breaks the scanner.

Known follow-ons (scoped, not regressions)

External links to /releases/tag/v1.12.0 now 404. The 15 releases
were re-pointed to their archive/ tags and remain public; the old tag
URLs are gone. Acceptable given the template has no adopters beyond the
maintainer, and every body is duplicated in releases/v1.*.md.
Pins and guards are retained deliberately. --vcs-ref remains
correct practice for any Copier template — the default resolves to
whatever sorts highest, which is surprising in general. The trap is gone;
the hygiene stays.
Carried forward: un-rehearsed MIGRATION recovery procedures, copier update across a real version gap, clock-allowlist brittleness keyed by
file:line, ruff UP/B/S, mypy → pyright, shadow-lane precision
data, codecov still linked to the pre-rename repository slug.

Verification

Evidence in VALIDATION_LOG.md Entry 020.

Assets 7

v0.25.0 — Three of four surfaces, and a repo name that no longer exists

github-actions released this 08 Aug 14:47

v0.25.0

9d1d7a3

v0.25.0 — Three of four surfaces, and a repo name that no longer exists

Release date: 2026-08-08

Both defects here were found by an adopter consuming the template, not
by the template's own CI. Both were partially fixed in v0.24.0 — which
is the more useful finding, and the one this note is really about.

`v0.24.0` pinned three of four surfaces

v0.24.0 shipped the fix for a destructive copier update and pinned
--vcs-ref on copier.yml, the scaffold-update skill, and rule
15-template-lifecycle.

It missed agentic/workflows/scaffold-update.md — the one an operator
actually executes as /scaffold-update, and which is vendored into every
generated service. So the destructive downgrade v0.24.0 documented in
detail was still exactly one command away, sitting in the place an operator
would most naturally run it.

The root cause is the guard, not the omission

check_adopter_scaffold_ref.py carried a hand-written list of three
files. It did not miss the workflow because someone was careless; it
missed it because the guard's coverage was a literal list, and a literal
list is only ever as complete as the moment someone last remembered to edit
it.

That is the third instance of the same shape in four releases:

Release	The list that went stale
`v0.22.0`	gitleaks version declared in three places, one drifted
`v0.23.0`	one exclude regex copied into six places
`v0.25.0`	three files enumerated where four needed checking

The guard now scans the tree for executable copier update commands
rather than enumerating files, skipping only historical records. Re-run
against the pre-fix tree it finds all six occurrences — canonical,
vendored, and the .devin adapter — that the enumerated version missed.

A repository name that no longer exists

Live adopter instructions, JSON Schema $id values, and files vendored
into every generated service still carried ML-MLOps-Production-Template.
The repository is ml-service-template.

The audit finding was right to flag it and wrong about the reason. It
was reported as a private-repo reference. It is not: the repository is
public and GitHub 301-redirects the old path — verified by fetching it.
Nothing was broken. It was a stale identifier that every generated service
inherited, which is a real problem for a different reason.

Renamed in live instructions, runbook examples, schema $ids, and the
render root. Verified: zero occurrences in a freshly generated service.

One badge deliberately keeps the old name

Codecov does not follow GitHub's redirect and is keyed on the pre-rename
slug. Fetching both settles it:

Badge URL	Result
`codecov.io/gh/DuqueOM/ML-MLOps-Production-Template`	40%
`codecov.io/gh/DuqueOM/ml-service-template`	unknown

Renaming it to match the other badges would break a working badge. It keeps
the old slug, with a comment in README.md saying why, so the next person
to notice the inconsistency does not "fix" it.

Historical records — CHANGELOG.md, releases/, docs/audit/, ADRs — are
not rewritten. They record what was true when they were written.

Known follow-ons (scoped, not regressions)

The v1.x tag-sort collision is still unresolved — four defects now,
four pins. Moving the frozen snapshots out of the tag namespace, or
advancing the active line past v1.12.0, still needs its own ADR because
agentic/rules/18 and ADR-014 declare those tags immutable.
The codecov project is still linked to the old repository slug.
Re-linking is a codecov-side action outside this repo. Until then the
badge URL and the repo URL disagree by design, not by oversight.
Services scaffolded under v0.24.0 or earlier still carry the unpinned
workflow. They cannot fix themselves; see MIGRATION.md.
Carried forward: un-rehearsed MIGRATION recovery procedures, copier update across a real version gap, clock-allowlist brittleness, ruff
UP/B/S, mypy → pyright, shadow-lane precision data.

Verification

Guard re-run against the pre-fix tree: finds all 6 unpinned occurrences.
Freshly generated service: workflow carries --vcs-ref at both call
sites; zero occurrences of the old repository name.
Old repo URL: 301 → https://github.com/DuqueOM/ml-service-template.

Evidence in VALIDATION_LOG.md Entry 019.

Assets 7

v0.24.0 — A bare `copier update` destroyed the service it was meant to upgrade

github-actions released this 08 Aug 00:42

v0.24.0

a02ea24

v0.24.0 — A bare `copier update` destroyed the service it was meant to upgrade

Release date: 2026-08-07

Hotfix for a destructive defect present in v0.23.0 and every release
before it. Found by running copier update on a freshly generated service
instead of assuming it worked — the same method that produced every finding
in the last two releases.

What was wrong

v0.23.0 pinned --vcs-ref on every documented copier copy. It left
copier update unpinned: in copier.yml's _message_after_copy, in
the scaffold-update skill, and in rule 15-template-lifecycle.

Same root cause — Copier resolves an unpinned git source to the
highest-sorting tag, and the frozen v1.0.0–v1.12.0 audit snapshots sort
above every v0.x tag. Far worse consequence.

copier copy unpinned hands you a stale scaffold. copier update
unpinned rewrites a service you already have, backwards.

Measured on a real v0.23.0 service:

	Bare `copier update`	`--vcs-ref=v0.23.0`
Files after	435	627
Files deleted	582	0
`.copier-answers.yml`	deleted	present

The deleted answers file is the sharp edge. It is the record copier update reads, so once it is gone the service cannot recover on its own —
the operation removes the mechanism that would undo it.

And _message_after_copy was telling every adopter to run exactly that
form, as the last thing they read after scaffolding.

What changed

--vcs-ref pinned on every documented copier update.
_message_after_copy now states the consequence, not just the command.
scripts/check_adopter_scaffold_ref.py extended to cover copier update
as well as copier copy, verified in both directions.

Known follow-ons (scoped, not regressions)

The v1.x tag-sort collision is still unresolved. This is the
third defect it has produced — v0.22.0 reaching nobody, the
copier copy docs, and now copier update — and the third fix that is
a pin. Pinning is correct and lands safely under any long-term option,
but it is mitigation, not resolution. The structural fixes (moving
the frozen snapshots out of the tag namespace, or advancing the active
line past v1.12.0) need their own ADR, because agentic/rules/18 and
ADR-014 both declare those tags immutable. Not decided unilaterally.
The MIGRATION.md recovery procedure for an already-downgraded
service is un-rehearsed. It is derived from the mechanism (the
pre-update commit must exist, because copier update requires a clean
tree), not from a performed recovery.
copier update across a real version gap is still unexercised.
Verified: the destructive unpinned case, and a pinned same-version
update proving non-destructiveness. An update that actually crosses two
releases and three-way-merges local modifications has not been run.
Everything carried forward from v0.23.0: clock-allowlist brittleness,
ruff UP/B/S rulesets, mypy → pyright, shadow-lane precision
data.

Until the structural fix lands, assume any Copier invocation against
this repo needs an explicit ref, and expect the guard to be the thing that
remembers.

Verification

Both directions, on real generated services:

# unpinned  — 627 -> 435 files, 582 deleted, answers file GONE
# pinned    — 627 -> 627 files,   0 deleted, answers file PRESENT

Evidence in VALIDATION_LOG.md Entry 018.

Assets 7

v0.23.0 — The release that reached nobody, and the toolchain consolidation

github-actions released this 08 Aug 00:23

v0.23.0

0a01c40

v0.23.0 — The release that reached nobody, and the toolchain consolidation

Release date: 2026-08-07

v0.x.0 bump carrying full MAJOR paperwork under the newly-added
docs/RELEASING.md §2.1.

Read this first if you skip everything else

v0.22.0 did not reach anyone.

Every adopter-facing copier copy example omitted --vcs-ref. Copier
resolves an unpinned git source to the highest-sorting tag, and this
repo carries the frozen v1.0.0–v1.12.0 audit snapshots (ADR-014)
alongside the active v0.x line:

$ git tag --sort=-v:refname | head -1
v1.12.0

v1.12.0 sorts above every v0.x tag. The documented command served the
April 2026 snapshot:

Command	Result
`copier copy <url> Svc`	435 files, no `.copier-answers.yml`
`copier copy --vcs-ref=v0.22.0 <url> Svc`	626 files, correct answers file

Nothing errored. The scaffold was complete and plausible — just three
months stale. Which is why it survived the entire life of the v0.x line.

Use --vcs-ref=v0.23.0. The docs now say so in all four places, and
scripts/check_adopter_scaffold_ref.py fails the build if they ever drift
from VERSION again.

The v1.x tags were not deleted or renamed. agentic/rules/18 and
ADR-014 both declare them immutable; changing that is a governance
decision needing its own ADR, not a side effect of a docs fix. The pin is
correct under every long-term option, so it lands now regardless.

Breaking for adopters

See MIGRATION.md §"v0.22.0 → v0.23.0" for the actions.

Change	Why it breaks
`black`/`isort`/`flake8` hooks removed, `ruff-check`/`ruff-format` added	§1.3 classifies a pre-commit hook-set change as MAJOR
`ruff format` ≠ `black` output	one-time reflow of your Python tree
scaffold command requires `--vcs-ref`	without it you get a `v1.x` snapshot

A governance gap this release forced open

§1.3 says a pre-commit hook change requires MAJOR. §2 reserves v1.0.0
for cloud E2E evidence. So a MAJOR-class change had nowhere to go —
the policy simply did not cover it.

New §2.1: on the v0.x channel a §1.3 MAJOR-class change ships as
v0.x.0 but carries the full MAJOR paperwork — a Breaking for adopters
block plus a MIGRATION.md section. The bump number is smaller; the
obligations are identical. Nothing about being pre-GA reduces what an
adopter is owed when a contract breaks.

Ruff consolidation (ADR-044)

The speed argument did not survive measurement and is not claimed. The
previous suite already ran --all-files in 2.62 s, inside the config
header's own < 5 s target.

What actually justified it:

Six copies of one exclude list — three tools × two pre-commit
configs, plus [tool.black]/[tool.isort] in two pyprojects. Same
defect class as the gitleaks drift closed in v0.22.0: one truth,
several declarations, silent divergence.
A lint coverage hole. flake8's files: was
^(templates/service/|examples/). scripts/ and templates/tests/
were type-checked and security-linted but never style-linted.

That hole was hiding a real defect: test_different_cache_keys_isolated
created ctx1, never read it, and asserted only on ctx2. The test
named for an isolation property never asserted it — it would have passed
with caching absent entirely. Fixed by adding the missing assertion, not
by deleting the variable as the linter suggested.

Scope is deliberately parity-only (E,W,F,I). Ruff's UP/B/S
rulesets measured at 90 additional findings and are not enabled:
mixing them in would turn a toolchain swap into a code change, and a
reviewer could not tell which line moved for which reason. Ruff does not
replace mypy and only partially overlaps bandit; both retained.

The 55-file formatter reflow is isolated in its own commit and registered
in .git-blame-ignore-revs. Equivalence verified by running the
collectible test suite before and after and confirming an identical result.

Generated services no longer carry unresolvable ADR references

The render root cites 39 template ADRs and vendors 6; the other 33
were dangling, and a consuming repo's reference checker flagged them.

Renaming to template-ADR-NNN is structurally blocked:
check_vendored_runtime_drift.py holds templates/service/agentic, the
shipped ADR files, and the config schemas byte-identical to their root
counterparts. Rewriting identifiers would break that gate or fork the
generated service from upstream — making every future copier update a
conflict. A pointer costs nothing; a fork costs every future update.

Shipped a resolution layer instead:
templates/service/docs/decisions/README.md, enforced by
scripts/check_service_adr_references.py.

Known follow-ons

Clock-isolation allowlist is keyed by file:line and any reformat
invalidates it. It did; entries were remapped 1:1 after verifying same
call count, same APIs, same order. Keying on the enclosing test name
would survive that churn.
Ruff UP/B/S — 90 findings, own ADR.
mypy → pyright — considered in ADR-044, not ruled out.
Shadow-lane precision data — the ADR-019 lane fires now but has
classified nothing. Phase 2 still needs 14 days of real data.

Verification

All contract tests green on main at the tagged commit. Execution
evidence in VALIDATION_LOG.md Entry 017.

Assets 7

v1.9.0 — Batch scoring, GitOps ADR, DORA metrics, devcontainer, secret rotation

DuqueOM released this 08 Aug 15:13

archive/v1.9.0

334126a

Note

Historical audit snapshot. This release belongs to the v1.0–v1.12 line that
shipped during the post-audit remediation sprint (2026-04-15 → 2026-04-29).
Per docs/RELEASING.md
these tags are immutable historical snapshots; the active public release line
is now v0.x
hardening, with v1.0.0 reserved for the first verified GKE+EKS cloud E2E rollout.
See CHANGELOG.md for the reclassification rationale.

Release date: 2026-04-24

Minor release adding scaffolding for scheduled batch inference,
architectural guidance for future GitOps migration, DORA measurement,
a reproducible dev environment, and a scheduled-rotation runbook.

Highlights

Batch inference scaffolding (D1)

New .windsurf/skills/batch-inference/SKILL.md provides the 6-step
workflow for CronJob-based batch scoring that shares the exact same
predict() code as the live API. Prevents training/serving and
training/batch skew — the two classic silent ML failure modes.

Architecture: batch.py uses the same predictor.predict_batch(),
the same Pandera schema from training, and the same log_prediction()
for closed-loop tracking. The CronJob uses PSS restricted + Forbid
concurrency policy + a hard activeDeadlineSeconds cap.

ADR-013 — GitOps strategy (D3)

Codifies the posture: stay on kubectl apply via GitHub Actions
until one of four revisit triggers fires:

5 services in the template
multi-cluster prod (GKE + EKS + on-prem)
≥2 drift incidents from out-of-band kubectl apply
organizational GitOps mandate

When triggered, migrate to ArgoCD (not Flux) — ApplicationSet
matches our dev/staging/prod × services fan-out; shared mental model
with Argo Rollouts already in use.

DORA metrics exporter (D4)

templates/scripts/dora_metrics.py aggregates from GitHub REST API
and ops/audit.jsonl:

Deployment frequency — successful prod deploys per week
Lead time for changes — PR merge → next prod deploy (median, p95)
Change failure rate — rollback-tagged issues / total prod deploys
MTTR — rollback → incident_close (median)

Graceful degradation: without GITHUB_TOKEN, runs in audit-only mode.
Output: ops/dora/{YYYY-MM}-metrics.json for dashboards. 9 unit
tests.

Devcontainer (D7)

.devcontainer/devcontainer.json + post-create.sh give new
contributors a reproducible environment matching the CI runner:

Python 3.11 bookworm base
Features: docker-in-docker, kubectl/helm/minikube, terraform, gh, cosign
Post-create: conftest, syft, gitleaks
Extensions: Python stack + Terraform + K8s + YAML + GH + mermaid

One code . in Codespaces or VS Code Remote Containers gets a
working env without host setup.

Secret rotation runbook (D9)

docs/runbooks/secret-rotation.md — SCHEDULED rotation (complement
to .windsurf/skills/secret-breach-response/SKILL.md which handles
EMERGENCY rotation):

Scope table with cadences per credential class
STOP per credential per environment (even in dev)
Canary in dev → soak 15min → staging → soak 1h → prod
7-day soak on OLD version before retirement (rollback insurance)
Default Q1/Q2/Q3/Q4 calendar

Scope adjustment

v1.9.0 was planned as a 10-item roadmap. This release delivers 5
high-impact items (D1, D3, D4, D7, D9). Deferred:

D2 GPU support path — niche today; revisit when a template
user demands GPU serving
D5 More reusable GHA — deploy-common.yml already done in
v1.7.1; no other workflows are big enough to pay off yet
D6 Terraform tests (terratest-lite) — significant commitment
that deserves its own ADR + staged rollout
D8 Publish as GitHub template repo — operational doc, not code
change; waits until v2.0.0 messaging

New files

.windsurf/skills/batch-inference/SKILL.md
docs/decisions/ADR-013-gitops-strategy.md
templates/scripts/dora_metrics.py
templates/tests/unit/test_dora_metrics.py
.devcontainer/devcontainer.json
.devcontainer/post-create.sh
docs/runbooks/secret-rotation.md
releases/v1.9.0.md (this file)

Tests

127 passing (was 118 in v1.8.1)
- test_dora_metrics.py: 9 new tests

Migration

No breaking changes. All additions opt-in:

Batch — run the batch-inference skill against a service when a
cadence demands scheduled scoring.
GitOps — read ADR-013 before any CD-plane discussion; no action
required today.
DORA — wire dora_metrics.py to a monthly CI job to populate
ops/dora/ for dashboards.
Devcontainer — contributors opt in by opening in Codespaces /
Remote Containers.
Secret rotation — schedule quarterly calendar entry per the
runbook table.

AGENTS.md — Session Init protocol should check for ops/audit.jsonl
and ops/dora/ before proposing operations
ADR-005 — Behavior Protocol (batch deploy is STOP in prod)
ADR-010 — Dynamic Behavior Protocol (DORA mttr informs error_budget)
ADR-011 — Environment promotion gates (rotation reuses the same chain)

Assets 2

v1.8.1 — Hardening: PSS, SBOM, ThreadPool, input quality, dashboards, intersectional fairness

DuqueOM released this 08 Aug 15:13

archive/v1.8.1

a2ac4a1

Note

Release date: 2026-04-24

Patch release hardening the security, observability, and operational
layers of v1.8.0. Six focused commits, no breaking changes.

Highlights

C1 — Pod Security Standards (D-29)

templates/k8s/policies/pod-security-standards.yaml — canonical
Namespace definitions: prod enforce=restricted, dev/staging
enforce=baseline + warn/audit=restricted (shift-left)
deployment.yaml: pod-level and container-level securityContext
compatible with PSS restricted (runAsNonRoot: true,
capabilities.drop: [ALL], seccompProfile: RuntimeDefault)
Rule 02 §Pod Security Standards section with namespace + container
snippets

C2 — SBOM attestation (D-30)

deploy-gcp.yml + deploy-aws.yml now generate a CycloneDX SBOM
(Anchore sbom-action@v0) and attach it as a Cosign attestation
(cosign attest --type cyclonedx)
Full SLSA L3 provenance via slsa-github-generator documented as
ROADMAP with the exact block operators can uncomment once the
Dockerfile is hermetic

C3 — ThreadPoolExecutor sizing

docs/threadpool-sizing.md — operator guide: decision rule,
BLAS/OMP interaction, anti-patterns, sample benchmark table
templates/service/scripts/benchmark_executor.py — executable
sweep script that writes ops/benchmarks/{ts}-executor.json

C4 — Edge-level input quality

common_utils/input_quality.py — opt-in checker against training-time
[p01, p99] quantiles. Emits {service}_input_out_of_range_total
labels (feature, direction) without blocking requests
build_from_env() honors INPUT_QUALITY_ENABLED + INPUT_QUALITY_PATH
14 unit tests covering load/check/disabled paths

C5 — Closed-loop Grafana dashboard

templates/monitoring/grafana/dashboard-closed-loop.json — 10 panels:
SLO availability + burn, per-version AUC, sliced-AUC heatmap,
Champion/Challenger error rate, score-distribution p50, logger error
rate (D-22), input-quality flags (C4), monitor heartbeat, PSI top 10

C6 — Intersectional fairness

fairness.py::compute_intersectional_fairness() — evaluates every
2-way combination of protected attributes so subgroup-only bias
(e.g. race=Black + gender=Female) is no longer masked by per-
attribute DIR
run_fairness_audit(intersectional=False, min_intersectional_samples=30)
— opt-in + min-cell-size guard (avoids noisy DIRs from tiny groups)
4 new tests (1 synthetic biased dataset, 1 single-attr skip, 1
small-cell insufficient, 1 flag-off default)

New anti-patterns

D-29 — Namespace without Pod Security Standards labels
D-30 — Production image without SBOM attestation

Files

New: 8 (policies YAML, benchmark script, docs, dashboard, input_quality
module + tests, intersectional tests, release notes).

Changed: deployment.yaml securityContext, deploy-{gcp,aws}.yml
SBOM steps, fairness.py intersectional, 02-kubernetes.md, AGENTS.md.

Tests

118 passing (was 100 in v1.8.0) — +18:

test_input_quality.py: 14
test_fairness_intersectional.py: 4

Migration from v1.8.0

All changes backward compatible. To adopt:

PSS labels — apply templates/k8s/policies/pod-security-standards.yaml
(per env). Services with privileged pods will see warnings in audit
logs immediately; fix by tightening securityContext.
SBOM attestation — no action; the new steps run automatically on
the next deploy. Verify via cosign verify-attestation post-deploy.
Input quality — set INPUT_QUALITY_ENABLED=true +
INPUT_QUALITY_PATH=artifacts/baseline_quantiles.json. Requires
emitting quantiles at training time (small patch to train.py).
Dashboard — import dashboard-closed-loop.json alongside the
existing dashboard-template.json.
Intersectional fairness — pass intersectional=True to
run_fairness_audit() in your training pipeline quality gate.

AGENTS.md — D-29, D-30 rows added
Rule 02 §Pod Security Standards
Rule 04a — already referenced ThreadPool sizing (docs now flesh it out)
CHANGELOG entry under [1.8.1]

Assets 2

v1.8.0 — Typed handoffs, automated rule audit, API contracts

DuqueOM released this 08 Aug 15:13

archive/v1.8.0

7ba9258

Note

Release date: 2026-04-24

Minor release (v1.8.0) focused on the agentic runtime layer: typed
inter-agent contracts, an append-only audit log, two new operational
skills (rule-audit + performance-degradation-rca), and public API
contract versioning.

No breaking runtime changes. No migration required for existing services.

Highlights

Typed handoffs + AuditLog (B1)

Problem: agents were passing dicts to each other. A missing key or
mistyped value surfaced only during Kubernetes apply, hours downstream.

Fix:

common_utils/agent_context.py — frozen dataclasses already existed
(v1.6) for EDAHandoff, TrainingArtifact, BuildArtifact,
SecurityAuditResult, DeploymentRequest. This release validates
them comprehensively with 25 new unit tests and hardens invariants:
- AuditEntry refuses success in CONSULT/STOP without approver
- AuditEntry now carries risk_signals + base_mode (ADR-010 trail)
- DeploymentRequest re-verified: prod blocks non-STOP mode and
  failing audits at construction
AuditLog class — thread-safe append-only JSONL writer with
record_operation() that extracts signals from a RiskContext
automatically. One function call to persist an auditable op.

rule-audit skill (B2)

Problem: invariants D-01..D-27 lived in documentation. Nothing
automatically CHECKED whether a service complies.

Fix: new .windsurf/skills/rule-audit/SKILL.md — a READ-ONLY
compliance scanner. Per-invariant grep/rg/yq/conftest command.
Emits ops/rule_audit.jsonl with evidence (file:line or metric).
--subset flag for scoped runs (probes, pdb, security, closed-loop,
all). Integrates with AuditLog so scans are part of the ops trail.

performance-degradation-rca skill (B3)

Problem: concept-drift-analysis handles single-slice RCA. Full
incidents need to correlate sliced metrics + drift + deploy history +
upstream data + logger health — and produce a blameless RCA doc.

Fix: new .windsurf/skills/performance-degradation-rca/SKILL.md.
Five-stream evidence correlation. Synthesizes one of R1..R5:

R1 Model regression (deploy-correlated)
R2 Concept drift (gradual, no deploy)
R3 Data-quality / semantic shift (PSI > 2× threshold)
R4 Monitoring failure (logger degraded → verdict inconclusive)
R5 Compound (escalate to architecture review)

Produces docs/incidents/{date}-{service}.md with evidence and
5-business-day corrective actions.

API contract versioning (B4, D-28)

Problem: Pydantic schemas silently drifted. A field rename broke every
client downstream.

Fix:

new rule 14 .windsurf/rules/14-api-contracts.md with semver table
(additive=minor, renames/narrows=major)
templates/service/tests/contract/test_openapi_snapshot.py — fails
if openapi.snapshot.json diverges from the live TestClient response
templates/service/scripts/refresh_contract.py — regenerates the
snapshot (run after any intentional schema edit)
CI guard: PR that changes openapi.snapshot.json without a matching
app.version bump is rejected
D-28 added to AGENTS.md anti-pattern table

Why a snapshot, not full Pact? Engineering Calibration: 1-3 services
with homogeneous clients. The snapshot catches 99% of breaking changes
with 10% of the infra.

New files

.windsurf/skills/rule-audit/SKILL.md
.windsurf/skills/performance-degradation-rca/SKILL.md
.windsurf/rules/14-api-contracts.md
templates/service/tests/contract/__init__.py
templates/service/tests/contract/test_openapi_snapshot.py
templates/service/scripts/refresh_contract.py
templates/tests/unit/test_agent_context.py
releases/v1.8.0.md (this file)

Changed files

templates/common_utils/agent_context.py — AuditEntry validation,
AuditLog writer, risk_signals + base_mode fields
AGENTS.md — D-28 added

Test counts

Unit tests: 100 passing (was 75 in v1.7.1)
- test_agent_context.py: 25 new tests

Migration (v1.7.1 → v1.8.0)

No runtime changes required. Optional adoption:

Contract snapshot — for services with public APIs:
```
cd {service-root}
python scripts/refresh_contract.py
# commit openapi.snapshot.json
```
Add the CI guard block from rule 14 to .github/workflows/ci.yml.

Audit log — instrument critical operations:

from common_utils.agent_context import AuditLog, AgentMode, Environment
from common_utils.risk_context import get_risk_context

ctx = get_risk_context()
final = ctx.escalate(AgentMode.AUTO)
AuditLog().record_operation(
    agent="Agent-MLTrainer",
    operation="train_model",
    environment=Environment.STAGING,
    base_mode=AgentMode.AUTO,
    final_mode=final,
    risk_context=ctx,
    inputs={"dataset_sha": "..."},
    outputs={"mlflow_run_id": "..."},
    approver="alice" if final != AgentMode.AUTO else None,
)

Run rule-audit on the service once to surface existing gaps:
Invoke the rule-audit skill (read-only; no code changes).

Highlights

Model warm-up + probe split (D-23, D-24)

Problem: the first inference after pod start took 300–800 ms while
scikit-learn/XGBoost lazily initialized the SHAP explainer. Under a
rolling deploy, pods became Ready on /health while they were still
cold — and the HPA happily sent production traffic their way.

Fix:

app/fastapi_app.py::warm_up_model() forces a dummy predict_proba
pass AND builds the SHAP KernelExplainer once during FastAPI
lifespan before _warmed_up=True
/ready returns 503 until warm-up completes (separate from /health)
K8s manifests split: livenessProbe → /health, readinessProbe →
/ready, new startupProbe → /health with failureThreshold: 24
The Argo Rollout canary now waits on /ready before routing traffic

PodDisruptionBudget (D-27)

Problem: a single kubectl drain on a node with all replicas evicted
the entire service.

Fix:

templates/k8s/base/pdb.yaml — minAvailable: 1
templates/k8s/base/hpa.yaml — minReplicas: 2 (PDB with minAvailable=1
cannot tolerate voluntary disruption on 1-replica services)
templates/tests/infra/policies/closed_loop.rego — enforces presence
- valid selector + explicit annotation escape hatch for minAvailable: 0

Argo Rollouts C/C (G-02b)

Problem: the Champion/Challenger statistical gate (McNemar + bootstrap
ΔAUC, shipped in v1.7.0) ran ONLY offline in retrain-service.yml. The
K8s canary only checked HTTP error rate and latency.

Fix:

templates/k8s/base/analysistemplate-champion-challenger.yaml
ships TWO AnalysisTemplates:
- {service}-cc-online — 4 proxy metrics during canary: prediction
  log error rate, error-rate vs stable, score-distribution shift,
  p95 latency. Auto-rollback on failure.
- {service}-cc-post-deploy — 3 real business metrics from
  performance_monitor CronJob. Manual trigger or scheduled. Does
  NOT auto-rollback (post-deploy rollback is STOP-class).

Three-layer C/C architecture is now complete:

OFFLINE        →  champion_challenger.py + retrain-service.yml
ONLINE         →  {service}-cc-online during canary
POST-DEPLOY    →  {service}-cc-post-deploy after labels arrive

Rollback skill + /rollback workflow (G-05)

The most critical on-call skill had no dedicated flow. release-checklist
mentioned "rollback plan" but no runbook existed.

New:

.windsurf/skills/rollback/SKILL.md — 7-step STOP-class procedure:
triage → identify revision → argo rollouts abort/undo → MLflow
registry revert → silence alerts → verify → audit issue
.windsurf/workflows/rollback.md — /rollback slash command with
15-min triage budget, 5-business-day RCA follow-up

Environment promotion gates (D-26, ADR-011)

Problem: deploy-gcp.yml and deploy-aws.yml were tag-triggered into
a single flat production-{cloud} environment. No staging validation;
no required-reviewer gates.

Fix:

templates/cicd/deploy-common.yml — reusable workflow_call centralizing
build/auth/apply/smoke-test logic
templates/cicd/deploy-{gcp,aws}.yml rewritten as 4-job chain:
build → deploy-dev → deploy-staging → deploy-prod
6 GitHub Environments to configure (gcp/aws × dev/staging/prod) with
explicit reviewer and wait_timer rules
docs/environment-promotion.md — operator setup guide with migration
steps for pre-v1.7.1 services

Dynamic Behavior Protocol via mcp-prometheus (ADR-010)

Problem: AGENTS.md's AUTO/CONSULT/STOP mapping was static. A deploy to
staging that is CONSULT at 10am Tuesday should NOT be CONSULT when a P1
incident is active, drift is severe, the error budget is exhausted, it
is 17:45 Friday, or a rollback happened < 6h ago.

Fix:

templates/common_utils/risk_context.py — 200-line helper with a
frozen RiskContext dataclass, escalate(base_mode) applying the
ADR-010 table (AUTO→CONSULT→STOP, sticky STOP), 60s cache,
fallback to ops/*.json local files when mcp-prometheus is unavailable
mcp-prometheus promoted to CORE MCP in AGENTS.md (same tier as
github/kubectl/terraform)
.windsurf/rules/01-mlops-conventions.md — Dynamic Behavior Protocol
section with the table and five signals
docs/decisions/ADR-010-dynamic-behavior-protocol.md — the trade-off
analysis, including why we rejected a file-only stub

New ADRs

ADR-010 — Dynamic Behavior Protocol via mcp-prometheus
ADR-011 — Environment Promotion Gates (dev→staging→prod)

New anti-patterns (D-23..D-27)

ID	Description
D-23	Liveness and readiness probes share a path
D-24	SHAP explainer rebuilt per request
D-25	Pod killed mid-request on deploy / scale-down
D-26	Deploys go directly to prod without staging validation
D-27	Deployment without PodDisruptionBudget

Full corrective actions in AGENTS.md anti-pattern table.

New files

templates/common_utils/risk_context.py + unit tests
templates/k8s/base/pdb.yaml
templates/k8s/base/analysistemplate-champion-challenger.yaml
templates/cicd/deploy-common.yml
templates/tests/infra/policies/closed_loop.rego
.windsurf/skills/rollback/SKILL.md
.windsurf/workflows/rollback.md
docs/environment-promotion.md
docs/decisions/ADR-010-dynamic-behavior-protocol.md
docs/decisions/ADR-011-environment-promotion-gates.md
releases/v1.7.1.md (this file)

Changed files

templates/service/app/fastapi_app.py — warm_up_model() function
templates/service/app/main.py — /ready endpoint, lifespan warm-up
templates/k8s/base/deployment.yaml + argo-rollout.yaml — probe
split, terminationGracePeriodSeconds, startupProbe
templates/k8s/base/hpa.yaml — minReplicas: 2
templates/k8s/base/kustomization.yaml — 2 new resources registered
templates/cicd/deploy-gcp.yml + deploy-aws.yml — 4-job chain
templates/tests/infra/policies/kubernetes.rego — converted to Rego v1
AGENTS.md — D-23..D-27 added, mcp-prometheus core MCP, rollback skill
.windsurf/rules/01-mlops-conventions.md — 6 invariants, dynamic table
.windsurf/rules/02-kubernetes.md — graceful shutdown, PDB sections
.windsurf/rules/05-github-actions.md — D-26 enforcement, reusable workflows
CHANGELOG.md — this release

Test counts

Unit tests: 75 passing (up from 56 in v1.7.0)
- test_risk_context.py: 19 new tests
Rego policies: now parse cleanly on conftest v0.46+

Migration guide (v1.7.0 → v1.7.1)

1. Enable warm-up (service-level)

Update app/fastapi_app.py and app/main.py to the new lifespan
pattern. See diff in commit A1.

2. Add PodDisruptionBudget

Copy templates/k8s/base/pdb.yaml into your service's k8s/base. Add
to kustomization.yaml. Bump HPA minReplicas: 2.

3. Adopt environment promotion

Create 6 Environments in repo Settings (gcp-dev, gcp-staging,
gcp-production, aws-dev, aws-staging, aws-production)
Set required_reviewers, wait_timer, deployment_branches per
the table in docs/environment-promotion.md
Replace deploy-{gcp,aws}.yml with the new 4-job chain
Add k8s/overlays/{cloud}-{env}/ kustomizations
Remove the old flat production-{cloud} environments AFTER first
successful pipeline

4. Install mcp-prometheus (optional)

Not required for the template to work, but unlocks the Dynamic Behavior
Protocol. Add the mcp-prometheus block shown in AGENTS.md §MCP
Integrations to your ~/.codeium/windsurf/mcp_config.json.

Highlights

Prediction logger (ADR-006)

templates/common_utils/prediction_logger.py — async buffered logger
with 4 pluggable backends (parquet, BigQuery, SQLite, stdout) via
PREDICTION_LOG_BACKEND env var
PredictionEvent frozen dataclass validates prediction_id,
entity_id, model_version at construction (D-20)
Fire-and-forget (D-21), failure-tolerant (D-22)
Integrated in fastapi_app.py + main.py lifespan; graceful
degradation if backend fails

Ground truth ingestion

monitoring/ground_truth.py — daily CronJob with user-implemented
fetch_labels_from_source(); CSV stub for local dev + documented
BigQuery/Postgres examples
Idempotent daily parquet partitions (year=/month=/day=)
configs/ground_truth_source.yaml — declarative source config

Sliced performance monitor (ADR-007)

monitoring/performance_monitor.py — JOINs predictions+labels on
entity_id with causality (label_ts >= prediction_ts); AUC/F1/
precision/recall/Brier globally AND per slice
Baseline comparison for concept drift (auc_drop_warning/alert)
Tri-state: ok / warning / alert / insufficient_data
Pushgateway metrics {slice_name, slice_value, metric} for Grafana
configs/slices.yaml — bounded-cardinality declarations

K8s manifests

k8s/base/cronjob-performance.yaml — two CronJobs (03:00 ground
truth, 04:00 performance monitor)
k8s/base/performance-prometheusrule.yaml — 5 alerts:
GlobalAUCBelowAlert, SlicedAUCBelowAlert, F1BelowAlert,
PerformanceMonitorStale, PredictionLogErrorsHigh

Champion/Challenger statistical gate (ADR-008)

evaluation/champion_challenger.py — McNemar exact binomial +
bootstrap ΔAUC 95% CI → tri-state decision (promote / keep / block)
configs/champion_challenger.yaml — alpha, n_bootstrap,
non_inferiority_margin, superiority_margin
cicd/retrain-service.yml — new C/C gate between quality gates and
promotion; exits 0/1/2

New anti-patterns

D-20 — PredictionEvent missing entity_id/prediction_id
D-21 — Prediction logging blocking the async event loop
D-22 — Logging backend failure propagating to HTTP responses

Agentic system + IDE parity

.windsurf/rules/13-closed-loop-monitoring.md
.windsurf/skills/concept-drift-analysis/ (new)
.windsurf/skills/drift-detection/ (extended)
.windsurf/skills/model-retrain/ (new Step 5.5 C/C gate)
.windsurf/workflows/performance-review.md
.cursor/rules/08-closed-loop.mdc + .claude/rules/08-closed-loop.md

ADRs

006 — closed-loop monitoring
007 — sliced performance analysis
008 — Champion/Challenger statistical gate
009 — retraining orchestration triggers

Tests

50 total passing, 25 new:
- test_prediction_logger.py — 20
- test_ground_truth.py — 6
- test_performance_monitor.py — 14
- test_champion_challenger.py — 10

Breaking schema changes (services on v1.6.x)

PredictionRequest.entity_id now REQUIRED (min_length=1)
PredictionResponse.prediction_id now REQUIRED (UUID hex)
Optional: PredictionRequest.slice_values: dict[str, str]

New dependencies

pyarrow ~=18.0 (parquet backend), pyyaml ~=6.0 (config)

Scope respected

No Argo Workflows (ADR-009 documents triggers; GHA default)
No Bytewax / streaming (parquet batch covers target audience)
No ClickHouse default (future trigger at >100M predictions/day)
No Istio shadow mode (ADR-008 future work)

Highlights

Agent Behavior Protocol (3 modes)

AGENTS.md — new Agent Behavior Protocol section: AUTO / CONSULT / STOP
Canonical Operation → Mode mapping table (21 operations)
Escalation triggers: automatic STOP even from AUTO/CONSULT (marginal
fairness, drift PSI > 2× threshold, cost > 1.2× budget, credential
detected, etc.)
Structured [AGENT MODE: X] signal format for handoffs

Authorization checkpoints in skills

deploy-gke/SKILL.md, deploy-aws/SKILL.md — authorization_mode
frontmatter (dev=AUTO, staging=CONSULT, prod=STOP)
model-retrain/SKILL.md — train=AUTO, to_staging=CONSULT,
to_production=STOP, plus automatic STOP on D-06 / marginal fairness /
regression > 5%

Agent-SecurityAuditor (Layer 2)

Runs BEFORE Agent-DockerBuilder and Agent-K8sBuilder
Blocks pipeline on findings (never silent)
Chains to /secret-breach on secret leaks
Capability matrix: "Blocked" entries cannot be bypassed by human insistence

Agent Handoff Schema

templates/common_utils/agent_context.py — AgentMode, Environment,
EDAHandoff, TrainingArtifact, BuildArtifact, SecurityAuditResult,
DeploymentRequest, AuditEntry — all frozen=True, fail-fast
DeploymentRequest refuses to construct with env=production +
audit.passed=False

Audit Trail Protocol

Every agentic operation → ops/audit.jsonl (append-only)
Mirrored to GitHub Actions step summary
CONSULT/STOP additionally open a GitHub issue tagged audit
Failures open issue tagged audit + incident

Supply chain security (SLSA L2)

New anti-patterns:

D-17 — Hardcoded credentials / direct os.environ for secrets in prod
D-18 — Static AWS keys or GCP JSON keys in production
D-19 — Unsigned images or missing SBOM in production

Rule + helper + CI + admission policy:

.windsurf/rules/12-security-secrets.md (always_on)
templates/common_utils/secrets.py — cloud-native secret loader;
refuses to fall through to os.environ in staging/prod
ci.yml — security-audit job (gitleaks + grep + IRSA/WI check);
build renamed to "Build, Sign & Attest" with Syft SBOM + Cosign
keyless sign + Cosign attest
templates/k8s/policies/kyverno-image-verification.yaml —
ClusterPolicy rejects unsigned images in prod namespaces; requires
CycloneDX SBOM attestation < 90 days

Incident response

.windsurf/skills/security-audit/SKILL.md — pre-build/pre-deploy scans
.windsurf/skills/secret-breach-response/SKILL.md — 7-phase playbook
.windsurf/workflows/secret-breach.md — /secret-breach slash command

ADR

ADR-005 — Agent Behavior Protocol + security: why 3 modes, keyless
Cosign, Kyverno over Gatekeeper, refuse os.environ in prod, why NOT
Vault (ADR-001 deferred), JSONL audit over GH issues per op, dataclasses
over JSON Schema, revisit triggers

Releases: DuqueOM/ml-service-template

Release list