feat(gr-foundry): migrate sentinel from gr-prod (Vault-backed config) - #129
Merged
Conversation
… Vault Same migration recipe as vault (#123): copy manifests with Traefik + public-hostname adjustments, add ArgoCD Application on the gr-foundry tree, remove from gr-prod. Same shared gr-postgres DB — sentinel state (users, OAuth apps, SAML signing keys, JWKS) persists across the move. Difference from vault: every sensitive AND non-sensitive config env value comes from a VaultSecretSync-materialized k8s Secret rather than being baked into the Deployment YAMLs. Rationale: single source of truth for both secret rotations and hostname/ID changes, with vault-k8s-operator auto-rolling the Deployments listed in rolloutTargets on any value change. What's in Vault (`sentinel` app-secret, 19 fields): Sensitive - postgres_password - internal_bootstrap_secret - kerbecs_password - discord_token - discord_client_secret - google_service_account Config (previously plain env) - database_host / database_user / database_name - issuer - kerbecs_endpoint / kerbecs_user - discord_client_id / discord_redirect_uri / discord_guild / discord_prefix - web_base_url - team_google_client_id - google_admin_subject What stays plain in the Deployment manifests (truly structural): - ENV=PROD - PORT=<service-specific, must match containerPort> - DATABASE_PORT=5432 - KERBECS_CONFIG=/etc/kerbecs/kerbecs.yaml Pre-merge bootstrap: populate the `sentinel` app-secret in the Vault UI at https://vault.gauchoracing.com with all 19 fields. Values come from the gr-prod sentinel-secrets Secret + the current plain env values on the gr-prod Deployments. Snippet to extract the sensitive half: kubectl --context gr-prod -n sentinel get secret sentinel-secrets -o json \ | jq -r '.data | to_entries[] | "\(.key | ascii_downcase) = \(.value|@base64d)"' Post-merge: 1. gr-prod ArgoCD prunes sentinel Application → workloads down on gr-prod. 2. gr-foundry ArgoCD creates the Application + VaultSecretSync. 3. Operator fetches from Vault → materializes sentinel-secrets → pods start. 4. External-dns updates sentinel-v5.gauchoracing.com CNAME → tunnel. 5. Traffic on gr-foundry.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Include VaultSecretSync in kustomization resources
Because this directory has a Kustomization, ArgoCD builds only the files listed under resources; the newly added vaultsecretsync.yaml is not included here. On a fresh gr-foundry sync the VaultSecretSync CR is never applied, so sentinel-secrets is never materialized and every Deployment using envFrom: sentinel-secrets will fail to create pods unless someone manually creates that Secret. Add vaultsecretsync.yaml to this resources list.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codex caught this on PR #129: the kustomization.yaml resources list was missing vaultsecretsync.yaml, so kustomize would silently skip it on ArgoCD sync. Without the VaultSecretSync applied, sentinel-secrets never materializes and every Deployment's `envFrom: sentinel-secrets` fails at pod creation. Root cause: I `cp`'d the file from gr-prod (which never had a VaultSecretSync) and my subsequent Write to update the resources list silently errored out mid-batch — Write requires a Read on existing files first, and I hadn't Read this one post-cp. The Write's failure got buried in a stream of successful ones.
Contributor
Author
|
Good catch @chatgpt-codex-connector — confirmed the bug and pushed a fix in the next commit. The kustomization.yaml was a leftover from the gr-prod |
|
To use Codex here, create an environment for this repo. |
Two coupled changes that grew out of populating the Vault app-secret:
1. Public hostname changes from sentinel-v5.gauchoracing.com to
sso.gauchoracing.com. The Ingress on gr-foundry now serves sso, and
every downstream OIDC RP has its SENTINEL_URL updated to match:
- gr-foundry/manifests/vault/server.yaml
- gr-prod/manifests/mapache/{auth,query}.yaml
ISSUER itself is now Vault-managed (see the vaultsecretsync); the
Vault value has already been set to https://sso.gauchoracing.com,
which is what future tokens will claim in `iss`. Existing tokens
with iss=https://sentinel-v5.gauchoracing.com become invalid on
cutover — every user re-authenticates once.
2. VaultSecretSync references updated from `sentinel.<field>` to
`sentinel-prod.<field>` to match the actual Vault app-secret name
we're using. Convention matches other app-secrets and leaves room
for future sentinel-dev / sentinel-staging alongside prod.
Contributor
Author
|
Scope expanded a bit — the Vault values populated by @BK1031 point at
Two things to know for cutover:
|
…TABASE_NAME Two shifts to the config sourcing: 1. DATABASE_HOST/PORT/USER + POSTGRES_PASSWORD now come from the shared `gr-postgres` Vault app-secret rather than being duplicated in `sentinel-prod`. One place to rotate the postgres password — every consumer's Deployment restarts automatically via vault-k8s-operator's rolloutTargets. Follow-up: mapache-secrets and vault-secrets on gr-foundry could move to the same pattern in later PRs. 2. DATABASE_NAME is not in Vault anymore. It's a per-app structural constant that matches the app name and never rotates — hardcoded as plain env on each Deployment, matching how vault-server (`DATABASE_NAME: vault`) and mapache (`DATABASE_NAME: mapache`) already do it. Frees up one Vault field per service. Also drop the plain `DATABASE_PORT: "5432"` overrides on every sentinel Deployment now that the port comes from gr-postgres via envFrom. Fields deleted from `sentinel-prod` in Vault after this lands (optional cleanup): database_host, database_user, database_name, postgres_password.
The postgres.sentinel.svc.cluster.local ExternalName pointed at gr-postgres.gauchoracing.com so Deployments could dial a namespace-local DNS name. With DATABASE_HOST now sourced from the gr-postgres Vault app-secret (currently gr-hamilton.gauchoracing.com), Deployments dial the postgres server directly and the Service shim is unused. Removing it. Also update kustomization.yaml to drop postgres.yaml from resources.
Forgot to include this in the previous commit — the file was removed from disk but kustomize still tried to include it, which would break the ArgoCD sync.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates sentinel from gr-prod (EKS) to gr-foundry (on-prem k3s). Follows the same pattern as the vault migration (#123), with one big shift: every sensitive AND non-sensitive config env value now comes from a `VaultSecretSync`-materialized k8s Secret, not baked into the Deployment YAMLs.
Why route config through Vault
What lives where
In Vault (`sentinel` app-secret, 19 fields, snake_case):
Sensitive:
Config (previously plain env):
Plain in the Deployment manifests (truly structural, never changes without a code change):
Data persistence
Sentinel state (users, OAuth apps, SAML signing key, JWKS row) lives in the `sentinel` DB on gr-postgres. Same DB, same signing key → tokens minted by the new pod validate against the same JWKS. Every downstream RP that trusts `https://sentinel-v5.gauchoracing.com\` keeps working.
Pre-merge bootstrap
Populate the `sentinel` app-secret in Vault UI (https://vault.gauchoracing.com) with all 19 fields. Sensitive values → extract from gr-prod's Secret:
```bash
kubectl --context gr-prod -n sentinel get secret sentinel-secrets -o json \
| jq -r '.data | to_entries[] | "\(.key | ascii_downcase) = \(.value|@base64d)"'
```
Config values → read straight off the current gr-prod Deployments (or just enter the known constants: `issuer=https://sentinel-v5.gauchoracing.com\`, `database_host=postgres.sentinel.svc.cluster.local`, etc.).
Cutover order
Total downtime for `sentinel-v5.gauchoracing.com`: ~2-4 min from merge.
Test plan