Single source of truth for the ghcr pull credential (OpenBao + ExternalSecret)

[devantler]

🤖 Generated by the Daily AI Engineer

Problem: The ghcr pull credential currently has three independent sources of truth: the org-level GHCR_TOKEN Actions secret (materialized into flux-system/ksail-registry-credentials by ksail cluster update on every deploy), OpenBao infrastructure/ghcr/auth (materialized into tenant ghcr-auth secrets by External Secrets), and the Talos machine.registries auth (env-expanded at cluster create). A rotation therefore requires updating several places in the right order, and missing one leaves a stale credential live — fragile and easy to get wrong.

Proposed direction: Make OpenBao infrastructure/ghcr/auth the single source of truth for the pull credential: add a flux-system ExternalSecret deriving ksail-registry-credentials from it, and stop the deploy path from upserting that secret once devantler-tech/ksail#6083 (pull/push token split) lands. The org GHCR_TOKEN secret then has exactly one job (the deploy's OCI push). Document the two-write rotation runbook (OpenBao pull + org push) in the platform docs.

Rough size: M — one ExternalSecret manifest + ksail config change (gated on ksail#6083) + runbook docs. Sequencing: ES manifest can land first; until #6083, the deploy overwriting the ES-managed secret is a known benign conflict.

[coderabbitai]

🔗 Related PRs

#2319 - feat(security): cosign-verify the infrastructure Flux OCI artifact [merged]
#2331 - feat(actual-budget): seed Enable Banking bank-sync via OpenBao/ESO + reconcile sidecar [merged]
#2343 - fix(openbao): grant external-secrets role write/read on apps/actual-budget path [merged]
#2345 - fix(openbao): grant actual-budget the missing vault ACL paths [merged]
#2433 - fix(dex): source GitHub OAuth client secret from a Secret, not plaintext env [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[devantler]

🤖 Generated by the Daily AI Engineer

Live evidence for this consolidation (2026-07-13): the fragmented copies now actively fight each other. A Git-declared seed (bootstrap SOPS secret → PushSecret → secret store → ExternalSecrets) re-asserted a rotated-out registry credential over a manual store-side fix within one refresh interval — re-freezing tenant OCI delivery and image-verify admission until the Git source itself is updated. A single authoritative source (this issue) removes that regression-loop class entirely. Suggest treating this as priority evidence when picking it up.

[devantler]

🤖 Generated by Codex Daily AI Engineer (scheduled automation).

Confirmed production impact and a concrete preflight opportunity from merge-group run https://github.com/devantler-tech/platform/actions/runs/29236522365:

• Kyverno imagevalidatingpolicy/verify-ksail-images rejected every new KSail Operator 7.168 Pod before admission because GHCR token exchange returned DENIED for devantler-tech/ksail:pull.
• The operator binary never started; Helm observed the Deployment stuck InProgress, exhausted its timeout/remediation budget, and rolled back to the already-admitted 7.167 ReplicaSet.
• Wedding-app and ascoachingogvaner OCIRepositories simultaneously returned the same GHCR pull denial. Their ghcr-auth ExternalSecrets report SecretSynced=True, which narrows this to an accepted-but-rejected credential value rather than a missing Secret object.
• KSail 7.168 release/image checks found no operator regression; the misattributed KSail tracker was closed with this evidence: KSail Operator 7.168.0 times out during Helm rollout ksail#6092 (comment).

Recovery remains: rotate/reconcile infrastructure/ghcr/auth to a valid read:packages credential, wait for the cluster ghcr-auth ExternalSecrets to refresh, verify Kyverno admits the image, then requeue the unchanged Platform update.

Suggested prevention acceptance criterion: before Flux reconcile, perform a server-side dry-run Pod CREATE using ghcr.io/devantler-tech/ksail:v${KSAIL_VERSION} under verify-ksail-images. It persists no Pod but fails immediately with the exact GHCR auth error, turning the current long Helm timeout into an actionable credential preflight.

[devantler]

🤖 Generated by the Codex Daily AI Engineer instance.

Production root-cause update — run 29265027478 failed after the store-side rotation, and the credential paths are now traced end to end.

• The root flux-system/ksail-registry-credentials Secret is written directly by KSail. It is not an ExternalSecret and has no owner reference to the OpenBao path.
• Updating OpenBao repaired the tenant credential path, but it could not refresh the root Flux source. The heal job pushed the main artifact, then timed out while OCIRepository/flux-system continued returning GHCR DENIED; apps remained Progressing.
• The workflow reconciles the workload before cluster update. That ordering deadlocks on an expired root credential.
• Even if the update step were reached, KSail currently redacts credential values from its structural diff and does not run the root-secret upsert for a credential-only change. That separate production-proven bug is now tracked in Refresh root registry credentials on credential-only changes ksail#6107.
• The Git/SOPS-backed seed-ghcr PushSecret refreshed OpenBao again at 17:08Z. A store-only edit is therefore not durable unless the upstream SOPS source is rotated too.

Do not rerun the Platform deploy yet. Safe recovery is to update the authoritative Git/SOPS ghcr_dockerconfigjson, let it propagate through OpenBao, then perform the one-time root-secret replacement from the same value before reconciling the root OCIRepository.

For the durable fix in this issue, the root ExternalSecret is now confirmed necessary rather than optional. PR devantler-tech/ksail#6090 supplies the pull/push split, but its current scope alone does not fix credential-only refresh or the reconcile-before-refresh deadlock; #6107 records that boundary.