# CD-04 — k3s staging executor (R2, R7, R8, R3–R6)

Status: LOCAL_DONE / INTEGRATION_PENDING
Validation: code and local harness PASS; Part B RBAC, kubeconfig, host install,
baseline and dry-run plan completed; no apply or timer
Scope: Phase 3 executor. Part B touched only the authorized staging namespace
and host paths; no apply, timer, other namespace or production action.

## R2 — captured staging source of truth

`infra/k8s/staging/` captures the live `abastevo-temp` API/web Deployments,
Services, DB StatefulSet, ResourceQuota, registry-sync CronJob and Caddy
ConfigMap. Images, host paths, cluster IPs, NodePorts and dynamic metadata are
placeholders rendered only from an operator file outside Git; secrets use
`secretRef`/`configMapRef` only. The pre-existing DB PVC is referenced but not
templated.

`render.py` substitutes only declared `${NAME}` placeholders, leaving pod shell
variables such as `$POSTGRES_USER` intact. `normalize-diff.py` is the
`KUBECTL_EXTERNAL_DIFF` helper for the proof; it ignores only server-managed
metadata and client-apply bookkeeping (creation timestamp, generation, resource
version, UID, managed fields, status, last-applied annotation, deployment
revision, manual restart annotation). Spec, labels, desired annotations, data
and pull policies are compared.

Proof: the rendered namespace was compared with `kubectl diff -n abastevo-temp`
through the normalizer and exited **0 with empty output**. Raw `kubectl diff`
reported only the ignored server-managed fields. No object was applied.

## R7 — landing in the signed OCI artifact

The release workflow adds `landing-dist.tar.gz` as an
`application/vnd.abastevo.landing.tar+gzip` layer, verifies its digest against
the signed manifest before push, and checks the published layer after push.
The agent verifies the OCI layer digest, the archive SHA-256 and the manifest
`landing_sha256` before extraction. The dry-run harness now carries the landing
layer in its OCI fixture.

## R8 — least-privilege RBAC and kubeconfig

`infra/k8s/staging/rbac/deploy-agent-rbac.yaml` defines a ServiceAccount, Role and
RoleBinding restricted to `abastevo-temp`: deployments get/list/watch/patch,
replicasets get/list/watch, jobs create/get/list/watch/delete, pods
get/list/watch, pods/log get and resourcequotas get/list. No secrets,
ConfigMaps, Services, pods/exec, other namespaces or cluster-scoped resources.
It is applied only in `abastevo-temp` in Part B; no other namespace.

`infra/deploy/agent/kubeconfig-from-serviceaccount.sh` creates the kubeconfig
after the RBAC is applied. The token is read from `kubectl create token`
through stdin, never printed, never placed in argv, and written only to the
mode-600 output file. A stubbed run proved the token does not appear in stdout
or stderr and the file mode is 600.

## Baseline (first-run seed)

`abastevo-deploy-agent baseline --sha <sha>` fetches the signed
`abastevo-release:sha-<sha>` artifact (not the moving `staging` tag), verifies
the same keyless identity and main ancestry, reads the database maximum with a
schema Job, reads the running api/worker images, verifies the running
deployment's `/health/ready` (the running image's `migrate.Check` validates
known checksums), and computes the landing tree digest. It writes mode-600 `current.json` with
`baseline: true` and an `abastevo-deploy-baseline-v1` receipt; it performs no
rollout, migration or landing switch.

`schema_min_supported` and `migrations_max` are seeded from the real database
maximum. The live `abastevo.com/schema-version` annotation is recorded only as
informative data; apply never uses it as a gate. Apply without `current.json`
now refuses with `baseline required`.

## R3–R6 — apply mode

The agent keeps the dry-run verification/plan and adds an opt-in
`ABASTEVO_APPLY=1` path. The k8s adapter is isolated in
`infra/deploy/agent/runtime/k8s.sh`; orchestration lives in
`infra/deploy/agent/apply.sh`. Release order:

1. schema-floor gate (requires `current.json`; database query Job; contract
   floor refused; no live-annotation gate);
2. verified `pg_dump` backup Job with `pg_restore --list`, SHA-256 and
   retention 7;
3. migration Job from the signed migrate digest (`backoffLimit: 0`,
   `ttlSecondsAfterFinished: 86400`);
4. one shared `kubectl set image` for the api and worker containers, pull
   policy `IfNotPresent`, `rollout status --timeout`;
5. smoke: `/health/ready`, `/`, `/consultar/`, one station-list read, portal
   manifest and one search shard;
6. landing extraction to `releases/<sha>`, relative `current` symlink, retain
   the previous release and the last 5;
7. `current.json` written atomically last, then a mode-600
   `abastevo-deploy-receipt-v1` receipt.

Any failure after the image update runs `rollout undo` for both containers; a
switched landing is restored; failures before the image update write a failure
receipt and leave the deployment untouched. Migrations are never rolled back.
The agent takes an internal `apply.lock` in addition to the service `flock`.

Surge is opt-in: `ABASTEVO_ALLOW_SURGE=1` and quota headroom at least
`ABASTEVO_SURGE_CPU_MILLI` produce `maxSurge: 1` / `maxUnavailable: 0`;
otherwise the safe `0/1` strategy is used and the receipt records the measured
gap. The R5 proof is recorded in the audit below and passes, but surge still
ships disabled by default until Part C measures zero unavailable replicas. The
current live quota headroom is 2.5 CPU, but that alone does not authorize
surge.

## R5 worker-loop lock audit

| Loop | Lock / fence | Evidence |
| --- | --- | --- |
| General dispatcher and fixed safety slots (`evidence-sweep`, `account-erasure-reconcile`, `privacy-training-reconcile`, `privacy-training-sweep`, `payment-inbox`) | Queue claim `FOR UPDATE SKIP LOCKED` plus monotonic `lease_token` fencing on complete/fail | `backend/cmd/worker/dispatchers.go:12-40`, `backend/internal/platform/jobs/dispatch.go:49-95`, `backend/internal/platform/jobs/queue.go:158-185`, `backend/db/queries/platform/jobs.sql:10-47` |
| Native serial slots (`enterprise-export-build`, `enterprise-portal-refresh`, quality import/discovery/revalidation/recompute/sanctions/enforcement) | Same queue claim/lease; enterprise handlers also take a filesystem lock | `backend/cmd/worker/dispatchers.go:29-40`, `backend/internal/modules/enterpriseexport/adapters/jobs/build.go:140`, `backend/internal/modules/enterpriseexport/adapters/jobs/workspace.go:146-165`, `backend/internal/modules/enterpriseexport/adapters/jobs/portal_refresh.go:508` |
| Main minute loop: subscription reconcile | Per-account row locks and token locks plus job dedupe; parallel scheduling test proves one job | `backend/cmd/worker/main.go:1036-1058`, `backend/internal/modules/subscription/adapters/reconcile.go:14-60`, `backend/db/queries/subscription/subscription.sql:145-146`, `backend/internal/modules/subscription/adapters/store_integration_test.go:814-834` |
| Main minute loop: receipt/notification cleanup | `FOR UPDATE SKIP LOCKED` purge + owner/token row locks | `backend/cmd/worker/main.go:1041`, `backend/db/queries/subscription/subscription.sql:126-137` |
| Main minute loop: scheduler tick | Unique `schedule:<name>:<period>` dedupe key; concurrent ticks converge on one row | `backend/cmd/worker/main.go:1044`, `backend/internal/platform/jobs/schedule.go:55-75`, `backend/db/queries/platform/jobs.sql:5-8` |
| Quality/ANP/registry job handlers | Queue lease plus source revision/lease checks; no duplicate effect observed | `backend/cmd/worker/quality_native_integration_test.go:117-140`, `backend/internal/platform/jobs/queue.go:158-185` |

The overlap integration test
`backend/cmd/worker/overlap_integration_test.go` runs two dispatcher loops with
distinct worker IDs over 40 jobs in one disposable PostGIS database. It asserts
both workers consumed work, all 40 jobs are `done` exactly once, every effect
row is unique and no job has `attempts > 1`. It passes. No unlocked loop was
found; surge nonetheless stays opt-in and disabled by default until the Part C
staging measurement.

## Part B staging installation and baseline (2026-10-27)

- RBAC applied only in `abastevo-temp`; the generated kubeconfig is mode 600
  and the ServiceAccount scope checks passed: patch Deployments yes, create
  Jobs yes, read secrets no, other namespaces no.
- The agent, libraries and `agent.env` (mode 600) are installed under the
  authorized host paths; the reviewed unit and `ReadWritePaths` drop-in are
  staged under `/etc/abastevo-deploy/systemd/`. The persistent
  `/etc/systemd/system` install and the dedicated service user were completed
  later the same day (see "Part B completion on staging"). The timer is not enabled.
  Baseline and plan were run as transient systemd units with the same
  `ProtectSystem`/`ProtectHome`/`PrivateTmp`/`NoNewPrivileges`/`ReadWritePaths`
  sandbox; their receipts are root-owned and will be chowned to the service
  user in Part C.
- Baseline receipt `baseline-2026-10-27T141650Z-3097961.json`: SHA
  `d4dcf7161c94cb444a5676c7fd69c1679e0f9bfd`; database max and
  `schema_min_supported` `000139`; running api/worker
  `docker.io/library/abastevo-{api,worker}:stg-7c7d776f`; migrate image
  `sha256:386cea44...` from the signed release; landing tree
  `29a1b17a...`; checksum `running-readiness`; live annotation `000120`
  recorded as informational only.
- Plan receipt `plan-2026-10-27T141737Z-3102578.json`: `action=plan-only`;
  current and target SHA `d4dcf716...`; api, worker and landing changed;
  migrate and migrations unchanged; target `migrations_max` `000132`. The
  current staging release predates `schema_min_supported`, so the target floor
  is `null`; a later main release with the floor is required before any apply.
- No apply, no timer, no other namespace and no production action. Applying
  `main` now is deferred because staging runs the maps branch/schema `000139`;
  an apply today would roll the map layers back.

## Review correction (2026-10-27): baseline floor and release match

The first baseline seeded `schema_min_supported` from the database maximum
(`000139`). That is the database's state, not the running release's floor, and
it would let a later contract floor up to `000139` pass as additive. Commit
`420d3f48` takes the floor from the release manifest, or from an explicit
`--floor` when the release predates the field, and refuses a floor above the
database. It also records `release_match`: the staging baseline runs
hand-deployed `stg-7c7d776f` images, so `release_match` is `false` and the first
apply rolls the release digests out. Harness: **166 passed, 0 failed**
(new cases: pre-floor release refused without `--floor`, operator floor
accepted, floor above the database refused, release mismatch recorded).

The existing staging `current.json` was written by the earlier code and must be
re-seeded after the updated agent is installed:
`baseline --sha d4dcf7161c94cb444a5676c7fd69c1679e0f9bfd --floor 000132`
(`000132` is the first floor of the chain, CD-03). The backup directory moves
to a dedicated `deploy-agent` subdirectory so retention pruning never sees
other operator backups.

## Part B completion on staging (2026-10-27, 14:33 UTC)

Installed from commit `52eebe94` with owner authorization:

- System user `abastevo-deploy` (no login shell, home = state directory).
- Persistent `abastevo-deploy-agent.service` and `.timer` in
  `/etc/systemd/system` with a staging drop-in for `ExecStart` and
  `ReadWritePaths` (state, landing root, dedicated `backups/deploy-agent`).
  The timer is **disabled and inactive**.
- Ownership: state, receipts and landing root belong to the service user;
  the kubeconfig is `root:abastevo-deploy` 0640; `agent.env` stays root-only
  0600 (read by systemd, not by the service). The backup directory moved to a
  dedicated subdirectory.
- Re-seeded baseline (`baseline-2026-10-27T143324Z-3169345.json`) as the
  service user: database `000139`, floor `000132` (operator, release predates
  the field), `release_match: false`, readiness verified, landing tree digest
  captured. `current.json` is owned by the service user, mode 600.
- One run of the persistent unit as `User=abastevo-deploy`: `Result=success`,
  `plan-2026-10-27T143327Z-3170441.json` (`plan-only`; api, worker and landing
  would change; migrations unchanged). `ABASTEVO_APPLY=0`.
- Unchanged afterwards: API image `stg-7c7d776f`, `/health/ready` 200. No
  apply, migration, landing switch or other namespace.

## Validation

- `make test-deploy-agent`: **166 passed, 0 failed** (157 before the review correction). The harness stubs registry,
  cosign, GitHub API, kubectl and deployment tools and covers: new-release and
  no-change planning; tampered/unsigned/foreign manifests; non-ancestor and API
  failures; invalid schema/current state; successful apply with surge; contract
  floor refusal; apply without `current.json`; backup Job failure; backup checksum
  mismatch; migration failure; broken rollout undo; smoke failure with the
  previous landing kept; concurrent lock refusal; `current.json` unchanged on
  every failure; Job TTL/backoff; one shared set-image; backup retention 7;
  landing retention 5; and the no-headroom gap receipt.
- `make quick-verify`: PASS (full selection) with the 157-case harness.
- `shellcheck`, `bash -n`, `actionlint v1.7.12`,
  `scripts/check-release-workflow.sh` and `git diff --check`: PASS.
- RBAC applied in `abastevo-temp`; live ServiceAccount scope checks passed (patch Deployments/create Jobs yes; secrets/other namespaces no).
- The R2 normalized `kubectl diff`: exit 0, empty output.
- The real kubeconfig helper produced mode 600 without printing the token; the stubbed run also proved no stdout/stderr leak.

## Live staging status and remaining acceptance

The staging database currently runs the maps branch schema at `000139`, while
the `main` release chain ends at `000132`; the live API annotation `000120` is
stale and is no longer read by the gate. Apply without `current.json` refuses
with `baseline required`. The Part B baseline seeded the real database max
(`000139`), the running api/worker images and the first main SHA (`d4dcf716`);
the plan receipt is recorded above. No apply is performed while staging still
runs the maps branch.

OWED to the reviewed acceptance: two consecutive `main` merges deploying
without a human, a deliberately broken image rolled back automatically, a
contract-floor release refused, and the landing switch plus rollback drill.
The R5 overlap integration test passes, but surge remains disabled by default
until the Part C staging run measures zero unavailable replicas.

## First real staging applies (2026-10-28)

With `ABASTEVO_APPLY=1` set by the owner's request and the timer still off,
two supervised runs of the persistent unit stopped before any change; each
left `current.json`, images, database and landing untouched and wrote a
failure receipt:

1. Release `8078fae7`: `landing archive is missing`. The workflow pushed the
   layer as `landing-bundle/landing-dist.tar.gz`, oras used that path as the
   title and pulled it into a subdirectory. Fixed in PR #178 (bare layer
   name, title asserted after push, guard in `check-release-workflow.sh`).
2. Release `d181510a`: `running floor is above the database schema`. The
   six-digit versions keep leading zeros and bash arithmetic read them as
   octal, so `000139` was invalid. `apply.sh` now compares in base 10 like
   `baseline.sh`; harness cases with `000139`/`000143` (decimal window
   applies, decimal contract floor still refused) took it to 171 passed.

Neither defect was reachable from the stubbed harness with the earlier
fixture values (`000132`, a bare pulled file).