reshard: run each operation in a dedicated pod + stream the catalog backup to S3

[benben]

Problem

During a recent reshard, the control-plane pod (512Mi limit) OOM-crashed while running pg_dump on a ~20k-table DuckLake catalog in the backup_catalog step, orphaning the operation. Two structural problems:

1. Reshards ran inside CP processes, competing with live traffic for the CP's small resource envelope. Catalogs can be far bigger than the one that triggered this.
2. The backup buffered the whole dump (temp file + single PutObject), so memory scaled with catalog size.

What this does

1. Each reshard executes in a dedicated per-operation pod

• New run mode --mode reshard-runner (all-in-one binary + cmd/duckgres-controlplane, -tags kubernetes): connect the config store, claim the ONE op named by DUCKGRES_RESHARD_OP_ID via the existing claim CAS, execute the existing step machine (provisioner.ReshardRunner, moved as-is — RunSingleOperation replaces the scan loop; the takeover claim path still runs reshard: reconstruct rollback progress on takeover so cancel unblocks the warehouse #940's reconstructProgress), exit. The op row is the source of truth for the outcome: failed/cancelled/rolled-back ops exit 0; only infrastructure errors (store unreachable, claim lost/fenced) exit nonzero. SIGTERM cancels the run ctx → rollback within the 600s termination grace.
• The CP spawns duckgres-reshard-op-<id> (labels app=duckgres-reshard, duckgres-op-id=<id>, restartPolicy Never) when a reshard is started. The pod runs the CP's own image + ServiceAccount (no new RBAC) and inherits an allowlist of the CP container's env spec verbatim — a secretKeyRef stays a ref, so the config-store DSN / internal secret are exactly as secret-referenced as on the CP deployment. Arch is pinned to the CP's own (kubernetes.io/arch), nodeSelector/tolerations inherited.
• Resources: env-only DUCKGRES_RESHARD_POD_CPU / DUCKGRES_RESHARD_POD_MEMORY (requests=limits, default 2 CPU / 8Gi), wired through configresolve like the other pod-scheduling knobs.
• Leader-only reconciler (under the janitor lease): respawns the pod of a pending-past-grace or stale-heartbeat op (bounded, 3 attempts, then force-fail with a clear operator-facing error), reaps pods of terminal ops. The old in-CP scan-and-execute loop is gone — the CP never executes reshard steps in-process.

2. Ephemeral ext-target password handoff (replaces claim-on-create)

• The start handler keeps the password in an in-memory stash on the replica that took the POST, creates the op pending, and spawns the pod with DUCKGRES_RESHARD_PASSWORD_URL=http://<cp-pod-ip>:8080/api/v1/reshards/<id>/password.
• New endpoint GET /api/v1/reshards/:id/password: internal-secret identity only (an SSO admin is 403'd — operators never read tenant credentials), 404 for terminal/unknown ops or a stashless replica; the access (never the value) is logged to the op log. GETs are not body-audited.
• The pull URL (never the password) is persisted on the op row (password_url, migration 000022) so a reconciler respawn re-wires the same handoff while the stashing replica lives. A 404 pull → the runner proceeds stashless and the step machine fails the op with the existing clear "password is not available … cancel and re-run" message + rollback.
• CreateReshardOperationClaimed / AdoptClaimedOperation (claim-on-create, Reshard: real claim-on-create for the ext-target password case #936) removed; ClaimReshardOperation stays (the pod uses it).

Invariant preserved: the password is never at rest — not in the op row, log, audit, any pod spec, or a k8s Secret; creating-replica memory → one authenticated pull → runner memory.

3. Streaming catalog backup

PGCatalogBackuper.Backup now streams pg_dump --format=custom stdout straight into an S3 multipart upload (feature/s3/manager, 16MiB parts × concurrency 2) — no temp file, no whole-dump buffer; memory is bounded regardless of catalog size. stderr is captured for diagnostics; a nonzero pg_dump exit or an empty dump deletes the partial uploaded object so it can never masquerade as a valid backup; an upload failure aborts the multipart and kills pg_dump. Key scheme, object tag, PGPASSWORD-not-argv, empty-dump refusal, and Redacted() errors unchanged.

Manifests / RBAC

• k8s/networkpolicy.yaml: new egress-only policy duckgres-reshard-runner-boundaries for app=duckgres-reshard (DNS, 5432 config store/cnpg poolers/external RDS, 443 S3+STS, 8080 CP admin API for the password pull), asserted in tests/manifests/.
• k8s/rbac.yaml needs nothing new: the runner pod uses the CP's own ServiceAccount, which already holds the Duckling-CR patch + config-store access; pod create/delete for the spawner is already in the CP Role.

Charts follow-up (separate repo, required before prod use)

• Production NetworkPolicy equivalent of duckgres-reshard-runner-boundaries.
• Optionally set DUCKGRES_RESHARD_POD_CPU/MEMORY on the CP deployment (defaults 2/8Gi otherwise). POD_IP on the CP deployment is optional — the CP falls back to interface-address detection for the password URL.

Notes

• Rebased on reshard: reconstruct rollback progress on takeover so cancel unblocks the warehouse #940 (merged): reconstructProgress runs inside execute(), so the pod claim path — including reconciler respawn takeovers — inherits it unchanged.
• Migration 000022 covered in both TestConfigStoreRunsVersionedSQLMigrations and the v8 downgrade round-trip (per the tests: cover goose migration 000021 in the v8 downgrade round-trip #941 lesson).

Tests

• go build ./... && go build -tags kubernetes ./... — green
• go vet ./... && go vet -tags kubernetes ./... — green
• go test -tags kubernetes ./controlplane/... ./tests/manifests/... ./tests/mw-dev/ ./configresolve/ . — green, except pre-existing postgres-container tests that need docker-compose (unavailable on the dev machine; they run in CI's configstore-integration-tests job)
• New unit coverage: controlplane/reshard_pod_test.go (pod spec: labels/SA/image, env allowlist + secretKeyRef preservation, knobs, no password env for cnpg targets), reshard_reconciler_test.go (respawn with persisted password URL, healthy-op no-op, retry bound → force-fail, terminal-pod reap matrix), reshard_runner_mode_test.go (password pull: secret header, 404-is-terminal, transient retry), provisioner/catalog_backup_test.go (PATH-shim pg_dump + fake S3: streamed bytes/size, PGPASSWORD-not-argv, stderr in errors, partial-object delete, empty-dump refusal), RunSingleOperation outcome contract in reshard_runner_test.go, password-endpoint auth + spawn-failure paths in admin/reshard_test.go, SetReshardOperationPasswordURL/ListActiveReshardOperations in tests/configstore/.
• e2e (tests/mw-dev/e2e/harness.sh): reshard lanes updated for the pod model (ops start pending; waits widened for pod schedule/pull) and the ext→cnpg positive path now asserts the duckgres-reshard-op-<id> pod appears (with labels) while the op runs and is reaped by the reconciler after it finishes. Mid-op pod-kill respawn stays unit-covered (documented in tests/mw-dev/README.md).

🤖 Generated with Claude Code

[github-actions]

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area	Added	Changed	Deleted
Test files	4	9	0
E2E/journey files	0	0	0
Workflow files	0	0	0

Signals

• Test cases: +22 / -2
• Assertions: +145 / -31
• Skips or known failures added: 0
• Workflow continue-on-error added: 0
• Workflow path filters added: 0
• Test commands removed from justfile: 0
• E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.

[benben]

CI note: the e2e failure (reshard ext→cnpg: pre-flip catalog backup did not complete) turned out to be pre-existing on main, not a pod-model regression — the org duckling IAM roles lack s3:PutObjectTagging, so the tagged backup upload 403s (AccessDenied: not authorized to perform: s3:PutObjectTagging). #939's own merged-commit e2e run failed with the identical error from the old temp-file PutObject path, so this assertion has never passed against the real cluster.

Fixed here (11ed46f): the backup uploads untagged; lifecycle retention keys on the reserved _reshard_catalog_backups/ prefix (already the documented suggested rule — no extra IAM needed). Unit test pins that the upload carries no Tagging; docs updated. 377be7a also added the diagnostics that caught this (reshard-pod logs in run.sh diagnostics, truncation-immune backup-line dump in the harness, AWS-context boot log in the runner).

Everything else in the pod model worked end-to-end in both runs: cancel + rollback lanes completed inside their runner pods and the reconciler reaped the pods afterwards.