Refresh Talos GHCR auth from Git/SOPS pull credential

[devantler]

🤖 Generated by the Codex Daily AI Engineer instance.

Description

Production Talos nodes still take their host-level ghcr.io registry credential from the separate Actions GHCR_TOKEN, even though k8s/bases/bootstrap/secret.enc.yaml now contains the validated authoritative pull credential. A newly autoscaled node therefore received stale host auth and cannot verify or pull a private first-party image.

The live ksail-operator/ksail-operator-5f6c87b8cb-4lx8b pod on autoscale-cx43-293a3c3e6986bf96 fails while Talos resolves ghcr.io/devantler-tech/ksail:v7.169.0: GHCR's token endpoint returns HTTP 403 for repository pull scope. The Git/SOPS credential independently completes authenticated manifest reads for the KSail package, so package availability is not the failure.

Steps to Reproduce

1. Rotate the Git/SOPS GHCR pull credential without rotating the separate Actions GHCR_TOKEN to the same pull token.
2. Create or autoscale a Talos node from the current production machine configuration.
3. Schedule an uncached private first-party image on that node.
4. Observe Talos image verification fail at the GHCR token exchange with HTTP 403 and the pod enter ImagePullBackOff.

Expected Behavior

Talos host registry authentication is derived from the same Git/SOPS Docker config that the deploy validates and distributes to Flux, OpenBao, tenant workloads, and Kyverno. Existing and newly created/autoscaled nodes can verify and pull every required private first-party package after a single pull-credential rotation.

Actual Behavior

talos/cluster/authenticate-ghcr-pulls.yaml expands ${GHCR_TOKEN}, while the deploy's credential bridge never updates Talos nodes. The stale node credential gets HTTP 403 even though the authoritative SOPS pull token is healthy.

Environment

• Production Talos v1.13.5 on Hetzner
• KSail deployment path using ksail.prod.yaml
• Private image ghcr.io/devantler-tech/ksail:v7.169.0
• Parent credential-convergence issue: Single source of truth for the ghcr pull credential (OpenBao + ExternalSecret) #2613

Acceptance Criteria

• Derive Talos ghcr.io registry auth from stringData.ghcr_dockerconfigjson without placing plaintext, encoded credentials, or tokens in argv/logs.
• Keep Actions GHCR_TOKEN scoped to OCI publication/signing compatibility; it must no longer be the Talos pull source.
• Apply the Git/SOPS credential to existing Talos nodes and to the declared config used by new/recreated/autoscaled nodes.
• Fail closed if any node cannot accept or use the credential; avoid an uncontrolled simultaneous cluster restart.
• Validate the exact deployed KSail image reference, not only the mutable repository tag.
• Add secret-free regressions for deploy and DR ordering, node discovery, partial failure, and temporary-material cleanup.
• Recover the live ksail-operator rollout and verify no pods remain in ErrImagePull or ImagePullBackOff.

Proposed Direction and Size

Extract the Docker-config username/password into mode-restricted temporary files, render the supported Talos registry-auth configuration, and apply it through the Talos API. Reuse the same extraction when KSail renders production machine configuration so autoscaler templates cannot retain the push credential. Synchronize existing nodes in a controlled, observable sequence and verify an authenticated host pull before declaring recovery.

Rough size: M. This is the live Talos child of #2613 and extends the incident recovery in #2624 / PR #2625.

[coderabbitai]

🔗 Related PRs

#2352 - chore(talos): drop redundant install-image.yaml; KSail derives it from extensions [merged]
#2416 - fix(talos): trust provider-upjet-* publish-provider-package.yml signing identity [merged]
#2457 - fix(talos): accept the actions publish-app identity at node image verification [merged]
#2474 - refactor: name patch fragments by intent and codify patch naming rules [merged]
#2491 - ci: validate rendered talos patches on the PR event [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!