docs backend evidence cd CD 04

CD-04 — k3s staging executor (R2, R7, R8, R3–R6)

Status: LOCAL_DONE / INTEGRATION_PENDING Validation: code and local harness PASS; Part B RBAC, kubeconfig, host install, baseline and dry-run plan completed; no apply or timer Scope: Phase 3 executor. Part B touched only the authorized staging namespace and host paths; no apply, timer, other namespace or production action.

R2 — captured staging source of truth

infra/k8s/staging/ captures the live abastevo-temp API/web Deployments, Services, DB StatefulSet, ResourceQuota, registry-sync CronJob and Caddy ConfigMap. Images, host paths, cluster IPs, NodePorts and dynamic metadata are placeholders rendered only from an operator file outside Git; secrets use secretRef/configMapRef only. The pre-existing DB PVC is referenced but not templated.

render.py substitutes only declared ${NAME} placeholders, leaving pod shell variables such as $POSTGRES_USER intact. normalize-diff.py is the KUBECTL_EXTERNAL_DIFF helper for the proof; it ignores only server-managed metadata and client-apply bookkeeping (creation timestamp, generation, resource version, UID, managed fields, status, last-applied annotation, deployment revision, manual restart annotation). Spec, labels, desired annotations, data and pull policies are compared.

Proof: the rendered namespace was compared with kubectl diff -n abastevo-temp through the normalizer and exited 0 with empty output. Raw kubectl diff reported only the ignored server-managed fields. No object was applied.

R7 — landing in the signed OCI artifact

The release workflow adds landing-dist.tar.gz as an application/vnd.abastevo.landing.tar+gzip layer, verifies its digest against the signed manifest before push, and checks the published layer after push. The agent verifies the OCI layer digest, the archive SHA-256 and the manifest landing_sha256 before extraction. The dry-run harness now carries the landing layer in its OCI fixture.

R8 — least-privilege RBAC and kubeconfig

infra/k8s/staging/rbac/deploy-agent-rbac.yaml defines a ServiceAccount, Role and RoleBinding restricted to abastevo-temp: deployments get/list/watch/patch, replicasets get/list/watch, jobs create/get/list/watch/delete, pods get/list/watch, pods/log get and resourcequotas get/list. No secrets, ConfigMaps, Services, pods/exec, other namespaces or cluster-scoped resources. It is applied only in abastevo-temp in Part B; no other namespace.

infra/deploy/agent/kubeconfig-from-serviceaccount.sh creates the kubeconfig after the RBAC is applied. The token is read from kubectl create token through stdin, never printed, never placed in argv, and written only to the mode-600 output file. A stubbed run proved the token does not appear in stdout or stderr and the file mode is 600.

Baseline (first-run seed)

abastevo-deploy-agent baseline --sha <sha> fetches the signed abastevo-release:sha-<sha> artifact (not the moving staging tag), verifies the same keyless identity and main ancestry, reads the database maximum with a schema Job, reads the running api/worker images, verifies the running deployment's /health/ready (the running image's migrate.Check validates known checksums), and computes the landing tree digest. It writes mode-600 current.json with baseline: true and an abastevo-deploy-baseline-v1 receipt; it performs no rollout, migration or landing switch.

schema_min_supported and migrations_max are seeded from the real database maximum. The live abastevo.com/schema-version annotation is recorded only as informative data; apply never uses it as a gate. Apply without current.json now refuses with baseline required.

R3–R6 — apply mode

The agent keeps the dry-run verification/plan and adds an opt-in ABASTEVO_APPLY=1 path. The k8s adapter is isolated in infra/deploy/agent/runtime/k8s.sh; orchestration lives in infra/deploy/agent/apply.sh. Release order:

1. schema-floor gate (requires current.json; database query Job; contract floor refused; no live-annotation gate);
2. verified pg_dump backup Job with pg_restore --list, SHA-256 and retention 7;
3. migration Job from the signed migrate digest (backoffLimit: 0, ttlSecondsAfterFinished: 86400);
4. one shared kubectl set image for the api and worker containers, pull policy IfNotPresent, rollout status --timeout;
5. smoke: /health/ready, /, /consultar/, one station-list read, portal manifest and one search shard;
6. landing extraction to releases/<sha>, relative current symlink, retain the previous release and the last 5;
7. current.json written atomically last, then a mode-600 abastevo-deploy-receipt-v1 receipt.

Any failure after the image update runs rollout undo for both containers; a switched landing is restored; failures before the image update write a failure receipt and leave the deployment untouched. Migrations are never rolled back. The agent takes an internal apply.lock in addition to the service flock.

Surge is opt-in: ABASTEVO_ALLOW_SURGE=1 and quota headroom at least ABASTEVO_SURGE_CPU_MILLI produce maxSurge: 1 / maxUnavailable: 0; otherwise the safe 0/1 strategy is used and the receipt records the measured gap. The R5 proof is recorded in the audit below and passes, but surge still ships disabled by default until Part C measures zero unavailable replicas. The current live quota headroom is 2.5 CPU, but that alone does not authorize surge.

R5 worker-loop lock audit

Loop	Lock / fence	Evidence
General dispatcher and fixed safety slots (evidence-sweep, account-erasure-reconcile, privacy-training-reconcile, privacy-training-sweep, payment-inbox)	Queue claim FOR UPDATE SKIP LOCKED plus monotonic lease_token fencing on complete/fail	backend/cmd/worker/dispatchers.go:12-40, backend/internal/platform/jobs/dispatch.go:49-95, backend/internal/platform/jobs/queue.go:158-185, backend/db/queries/platform/jobs.sql:10-47
Native serial slots (enterprise-export-build, enterprise-portal-refresh, quality import/discovery/revalidation/recompute/sanctions/enforcement)	Same queue claim/lease; enterprise handlers also take a filesystem lock	backend/cmd/worker/dispatchers.go:29-40, backend/internal/modules/enterpriseexport/adapters/jobs/build.go:140, backend/internal/modules/enterpriseexport/adapters/jobs/workspace.go:146-165, backend/internal/modules/enterpriseexport/adapters/jobs/portal_refresh.go:508
Main minute loop: subscription reconcile	Per-account row locks and token locks plus job dedupe; parallel scheduling test proves one job	backend/cmd/worker/main.go:1036-1058, backend/internal/modules/subscription/adapters/reconcile.go:14-60, backend/db/queries/subscription/subscription.sql:145-146, backend/internal/modules/subscription/adapters/store_integration_test.go:814-834
Main minute loop: receipt/notification cleanup	FOR UPDATE SKIP LOCKED purge + owner/token row locks	backend/cmd/worker/main.go:1041, backend/db/queries/subscription/subscription.sql:126-137
Main minute loop: scheduler tick	Unique schedule:<name>:<period> dedupe key; concurrent ticks converge on one row	backend/cmd/worker/main.go:1044, backend/internal/platform/jobs/schedule.go:55-75, backend/db/queries/platform/jobs.sql:5-8
Quality/ANP/registry job handlers	Queue lease plus source revision/lease checks; no duplicate effect observed	backend/cmd/worker/quality_native_integration_test.go:117-140, backend/internal/platform/jobs/queue.go:158-185

The overlap integration test backend/cmd/worker/overlap_integration_test.go runs two dispatcher loops with distinct worker IDs over 40 jobs in one disposable PostGIS database. It asserts both workers consumed work, all 40 jobs are done exactly once, every effect row is unique and no job has attempts > 1. It passes. No unlocked loop was found; surge nonetheless stays opt-in and disabled by default until the Part C staging measurement.

Part B staging installation and baseline (2026-10-27)

• RBAC applied only in abastevo-temp; the generated kubeconfig is mode 600 and the ServiceAccount scope checks passed: patch Deployments yes, create Jobs yes, read secrets no, other namespaces no.
• The agent, libraries and agent.env (mode 600) are installed under the authorized host paths; the reviewed unit and ReadWritePaths drop-in are staged under /etc/abastevo-deploy/systemd/. The persistent /etc/systemd/system install and the dedicated service user were completed later the same day (see "Part B completion on staging"). The timer is not enabled. Baseline and plan were run as transient systemd units with the same ProtectSystem/ProtectHome/PrivateTmp/NoNewPrivileges/ReadWritePaths sandbox; their receipts are root-owned and will be chowned to the service user in Part C.
• Baseline receipt baseline-2026-10-27T141650Z-3097961.json: SHA d4dcf7161c94cb444a5676c7fd69c1679e0f9bfd; database max and schema_min_supported 000139; running api/worker docker.io/library/abastevo-{api,worker}:stg-7c7d776f; migrate image sha256:386cea44... from the signed release; landing tree 29a1b17a...; checksum running-readiness; live annotation 000120 recorded as informational only.
• Plan receipt plan-2026-10-27T141737Z-3102578.json: action=plan-only; current and target SHA d4dcf716...; api, worker and landing changed; migrate and migrations unchanged; target migrations_max 000132. The current staging release predates schema_min_supported, so the target floor is null; a later main release with the floor is required before any apply.
• No apply, no timer, no other namespace and no production action. Applying main now is deferred because staging runs the maps branch/schema 000139; an apply today would roll the map layers back.

Review correction (2026-10-27): baseline floor and release match

The first baseline seeded schema_min_supported from the database maximum (000139). That is the database's state, not the running release's floor, and it would let a later contract floor up to 000139 pass as additive. Commit 420d3f48 takes the floor from the release manifest, or from an explicit --floor when the release predates the field, and refuses a floor above the database. It also records release_match: the staging baseline runs hand-deployed stg-7c7d776f images, so release_match is false and the first apply rolls the release digests out. Harness: 166 passed, 0 failed (new cases: pre-floor release refused without --floor, operator floor accepted, floor above the database refused, release mismatch recorded).

The existing staging current.json was written by the earlier code and must be re-seeded after the updated agent is installed: baseline --sha d4dcf7161c94cb444a5676c7fd69c1679e0f9bfd --floor 000132 (000132 is the first floor of the chain, CD-03). The backup directory moves to a dedicated deploy-agent subdirectory so retention pruning never sees other operator backups.

Part B completion on staging (2026-10-27, 14:33 UTC)

Installed from commit 52eebe94 with owner authorization:

• System user abastevo-deploy (no login shell, home = state directory).
• Persistent abastevo-deploy-agent.service and .timer in /etc/systemd/system with a staging drop-in for ExecStart and ReadWritePaths (state, landing root, dedicated backups/deploy-agent). The timer is disabled and inactive.
• Ownership: state, receipts and landing root belong to the service user; the kubeconfig is root:abastevo-deploy 0640; agent.env stays root-only 0600 (read by systemd, not by the service). The backup directory moved to a dedicated subdirectory.
• Re-seeded baseline (baseline-2026-10-27T143324Z-3169345.json) as the service user: database 000139, floor 000132 (operator, release predates the field), release_match: false, readiness verified, landing tree digest captured. current.json is owned by the service user, mode 600.
• One run of the persistent unit as User=abastevo-deploy: Result=success, plan-2026-10-27T143327Z-3170441.json (plan-only; api, worker and landing would change; migrations unchanged). ABASTEVO_APPLY=0.
• Unchanged afterwards: API image stg-7c7d776f, /health/ready 200. No apply, migration, landing switch or other namespace.

Validation

• make test-deploy-agent: 166 passed, 0 failed (157 before the review correction). The harness stubs registry, cosign, GitHub API, kubectl and deployment tools and covers: new-release and no-change planning; tampered/unsigned/foreign manifests; non-ancestor and API failures; invalid schema/current state; successful apply with surge; contract floor refusal; apply without current.json; backup Job failure; backup checksum mismatch; migration failure; broken rollout undo; smoke failure with the previous landing kept; concurrent lock refusal; current.json unchanged on every failure; Job TTL/backoff; one shared set-image; backup retention 7; landing retention 5; and the no-headroom gap receipt.
• make quick-verify: PASS (full selection) with the 157-case harness.
• shellcheck, bash -n, actionlint v1.7.12, scripts/check-release-workflow.sh and git diff --check: PASS.
• RBAC applied in abastevo-temp; live ServiceAccount scope checks passed (patch Deployments/create Jobs yes; secrets/other namespaces no).
• The R2 normalized kubectl diff: exit 0, empty output.
• The real kubeconfig helper produced mode 600 without printing the token; the stubbed run also proved no stdout/stderr leak.

Live staging status and remaining acceptance

The staging database currently runs the maps branch schema at 000139, while the main release chain ends at 000132; the live API annotation 000120 is stale and is no longer read by the gate. Apply without current.json refuses with baseline required. The Part B baseline seeded the real database max (000139), the running api/worker images and the first main SHA (d4dcf716); the plan receipt is recorded above. No apply is performed while staging still runs the maps branch.

OWED to the reviewed acceptance: two consecutive main merges deploying without a human, a deliberately broken image rolled back automatically, a contract-floor release refused, and the landing switch plus rollback drill. The R5 overlap integration test passes, but surge remains disabled by default until the Part C staging run measures zero unavailable replicas.

First real staging applies (2026-10-28)

With ABASTEVO_APPLY=1 set by the owner's request and the timer still off, two supervised runs of the persistent unit stopped before any change; each left current.json, images, database and landing untouched and wrote a failure receipt:

1. Release 8078fae7: landing archive is missing. The workflow pushed the layer as landing-bundle/landing-dist.tar.gz, oras used that path as the title and pulled it into a subdirectory. Fixed in PR #178 (bare layer name, title asserted after push, guard in check-release-workflow.sh).
2. Release d181510a: running floor is above the database schema. The six-digit versions keep leading zeros and bash arithmetic read them as octal, so 000139 was invalid. apply.sh now compares in base 10 like baseline.sh; harness cases with 000139/000143 (decimal window applies, decimal contract floor still refused) took it to 171 passed.

Neither defect was reachable from the stubbed harness with the earlier fixture values (000132, a bare pulled file).