Prod GKE backend ExternalSecret still references missing SERVICE_ACCOUNT_JSON

[Git-on-my-level]

Summary

Prod GKE prod-omi-backend-external-secret is currently unhealthy because it still references the missing Secret Manager key SERVICE_ACCOUNT_JSON.

This appears separate from the Cloud Run deploy-workflow cleanup in #9164/#9169: Cloud Run stale secret bindings were removed, but the GKE ExternalSecret still tries to sync the deleted/absent key.

Evidence

Read-only prod checks on 2026-07-07 UTC:

Warning UpdateFailed externalsecret/prod-omi-backend-external-secret
error processing spec.data[15] (key: SERVICE_ACCOUNT_JSON), err: unable to access Secret from SecretManager Client: Secret does not exist

kubectl get externalsecret -n prod-omi-backend prod-omi-backend-external-secret -o json showed:

condition Ready False SecretSyncedError could not get secret data from provider
mapping SERVICE_ACCOUNT_JSON remote SERVICE_ACCOUNT_JSON

Current workloads may continue running from already-synced Kubernetes Secret material, but the failed sync reduces confidence in future restarts/rollouts.

Expected behavior

Prod backend ExternalSecret resources should only reference live Secret Manager keys, and the ExternalSecret should report Ready=True.

Acceptance criteria

• Remove or replace the stale SERVICE_ACCOUNT_JSON mapping from the prod backend ExternalSecret configuration.
• prod-omi-backend-external-secret reports Ready=True / no SecretSyncedError.
• Backend/listen/pusher rollout checks confirm pods do not depend on a deleted Secret Manager key.
• Add a preflight check or runbook note so deleted Secret Manager keys do not silently leave ExternalSecrets unhealthy.

Related

• Harden prod backend deploy workflow against failed Cloud Run revisions #9164 / Harden prod backend Cloud Run deploy workflow (#9164) #9169 fixed Cloud Run deploy workflow secret drift, but this issue is about the GKE ExternalSecret state.