ci(06): verify kubectl-native CD — WG handshake + SA-auth + server-side dry-run by Afgan0r · Pull Request #1 · solid-stats/infrastructure

Afgan0r · 2026-06-13T03:53:32Z

Deferred Phase 6 verification: prove the kubectl-native CD pipeline from a real runner.

Exercises (non-mutating)

validate — manifest/script shape + validate-staging.py
dry-run — WireGuard handshake from runner -> ci-deployer SA-token auth -> kubectl apply --dry-run=server of all k8s/staging/*.yaml (excl. operator-managed 01-ci-rbac.yaml)

Change

Pass WG_LOCAL_IP=10.8.0.3/32 to the WG step (both jobs) so CI uses a dedicated WG peer, distinct from the operator laptop 10.8.0.2.

Operator prereqs (done out-of-band)

01-ci-rbac.yaml applied -> ci-deployer SA + token + Role/Binding
Dedicated CI [Peer] at 10.8.0.3/32 on staging VPS
staging env secrets set: WG_, K8S_TOKEN, K8S_CA_CERT, GHCR_

Real deploy job runs only on master push / dispatch — not on this PR.

Generated with Claude Code

…8.0.2 collision) The operator laptop already occupies 10.8.0.2 on the WG network; the CI runner must use a distinct peer IP so both tunnels can coexist. Pass WG_LOCAL_IP to the wg-tunnel-up.sh step in both dry-run and deploy jobs. Pairs with a dedicated [Peer] at 10.8.0.3/32 added on the staging VPS WG server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44aed3d6c2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-13T03:55:37Z

          WG_PRIVATE_KEY: ${{ secrets.WG_PRIVATE_KEY }}
          WG_PEER_PUBLIC_KEY: ${{ secrets.WG_PEER_PUBLIC_KEY }}
          WG_ENDPOINT: ${{ secrets.WG_ENDPOINT }}
+          WG_LOCAL_IP: 10.8.0.3/32


Keep the CI peer IP consistent with the bootstrap runbook

With this workflow now assigning the runner 10.8.0.3/32, a fresh or rotated staging setup that follows the checked-in runbook still configures the VPS peer with AllowedIPs = 10.8.0.2/32 (docs/operator-bootstrap.md:154-168). WireGuard treats AllowedIPs as the tunnel source IPs a peer may send from, so that documented server config will drop traffic from this runner even though the job is using the right keys; update the runbook/server-side peer IP at the same time or keep the workflow on the documented address.

Useful? React with 👍 / 👎.

…substitution First live CI run exposed `fopen: No such file or directory` at `wg set private-key <(...)`: sudo closes inherited FDs (closefrom=3), so the process-substitution /dev/fd/NN path vanishes in the wg process. /dev/stdin (FD 0) is preserved by sudo, keeping the key off disk while fixing the handshake step. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… just wait Second live run: interface showed no `transfer:` line — zero bytes sent, so the handshake was never initiated. WireGuard is lazy: it only sends a handshake when an outbound packet is destined for the peer. The wait loop polled latest-handshakes without generating any traffic, so it always timed out. Fix: set `persistent-keepalive 25` on the peer and poke the API server (/dev/tcp/10.8.0.1:6443) at the top of the wait loop to force immediate handshake initiation. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… none) Third live run: handshake completed but the API reachability check timed out. Root cause: `wg set allowed-ips` only sets WireGuard cryptokey routing, not a kernel route, and the /32 local address adds no subnet route — so 10.8.0.1 had no route via wg0 (the handshake still succeeded because keepalive/handshake go endpoint-to-endpoint over the public net, bypassing the tunnel IP). Add an explicit `ip route replace <allowed-ip> dev wg0` after link-up. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e files) Fourth live run: `xargs kubectl apply ... -f` placed all filenames after a single -f, so kubectl took only the first and rejected the rest ("Unexpected args"). Prefix each path with `-f ` via sed so every manifest gets its own flag. Same fix in both the dry-run and deploy jobs. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Fifth live run: the dry-run apply failed with "unable to read certificate-authority /tmp/tmp.XXX: no such file or directory". set-cluster stored the CA *path*, but the temp CA file is rm'd by the script's EXIT trap; the in-script `auth whoami` passed only because the trap had not fired yet. Add --embed-certs=true so the CA is inlined into the kubeconfig and later steps work after cleanup. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ped) Sixth live run: every workload dry-ran cleanly, but 00-namespace.yaml was Forbidden — the namespace-scoped ci-deployer Role cannot get/create the cluster-scoped Namespace (by design: "CI never creates the namespace"). The glob already excluded 01-ci-rbac.yaml; add 00-namespace.yaml to the same operator-managed exclusion in both jobs. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…substitution First live CI run exposed `fopen: No such file or directory` at `wg set private-key <(...)`: sudo closes inherited FDs (closefrom=3), so the process-substitution /dev/fd/NN path vanishes in the wg process. /dev/stdin (FD 0) is preserved by sudo, keeping the key off disk while fixing the handshake step. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… just wait Second live run: interface showed no `transfer:` line — zero bytes sent, so the handshake was never initiated. WireGuard is lazy: it only sends a handshake when an outbound packet is destined for the peer. The wait loop polled latest-handshakes without generating any traffic, so it always timed out. Fix: set `persistent-keepalive 25` on the peer and poke the API server (/dev/tcp/10.8.0.1:6443) at the top of the wait loop to force immediate handshake initiation. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… none) Third live run: handshake completed but the API reachability check timed out. Root cause: `wg set allowed-ips` only sets WireGuard cryptokey routing, not a kernel route, and the /32 local address adds no subnet route — so 10.8.0.1 had no route via wg0 (the handshake still succeeded because keepalive/handshake go endpoint-to-endpoint over the public net, bypassing the tunnel IP). Add an explicit `ip route replace <allowed-ip> dev wg0` after link-up. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e files) Fourth live run: `xargs kubectl apply ... -f` placed all filenames after a single -f, so kubectl took only the first and rejected the rest ("Unexpected args"). Prefix each path with `-f ` via sed so every manifest gets its own flag. Same fix in both the dry-run and deploy jobs. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Fifth live run: the dry-run apply failed with "unable to read certificate-authority /tmp/tmp.XXX: no such file or directory". set-cluster stored the CA *path*, but the temp CA file is rm'd by the script's EXIT trap; the in-script `auth whoami` passed only because the trap had not fired yet. Add --embed-certs=true so the CA is inlined into the kubeconfig and later steps work after cleanup. Found by the dry-run job in PR #1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The deferred live-CI human-verification item was executed end-to-end on 2026-06-13: PR #1 dry-run green (WG handshake + ci-deployer SA auth + server-side apply) and master-push deploy green (render+apply secrets, apply manifests, rollout of 5 workloads). Six latent script/workflow bugs surfaced by the first live runs were fixed (private-key /dev/stdin, handshake init, kernel route, per-file -f, kubeconfig --embed-certs, exclude 00-namespace). Updates 06-VERIFICATION.md (status human_needed → verified, Live CI Verification section) and STATE.md (current position, pending todos, blockers resolved). [skip ci] — docs-only; does not need to re-trigger the deploy workflow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Activate the merged errors-only Sentry SDK on staging by pinning each app to a post-Sentry-merge image: - server-2: 3866f6b → e31b129 (sentry #19 + parity #18/#21) - replay-parser-2: 5e09d33c → 72116db (sentry #1) - replays-fetcher: 7e5ca97 → 205b4cd (sentry #1 + shared-clients #2) replays-fetcher: 7e5ca97 (F1 conditional-writes baseline) is an ancestor of 205b4cd, so the F1 fix is preserved. PARITY-COORDINATION.md is archived. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Afgan0r had a problem deploying to staging June 13, 2026 03:53 — with GitHub Actions Failure

chatgpt-codex-connector Bot reviewed Jun 13, 2026

View reviewed changes

Afgan0r had a problem deploying to staging June 13, 2026 03:56 — with GitHub Actions Failure

Afgan0r had a problem deploying to staging June 13, 2026 04:00 — with GitHub Actions Failure

Afgan0r had a problem deploying to staging June 13, 2026 04:03 — with GitHub Actions Failure

Afgan0r had a problem deploying to staging June 13, 2026 04:06 — with GitHub Actions Failure

Afgan0r had a problem deploying to staging June 13, 2026 04:08 — with GitHub Actions Failure

Afgan0r temporarily deployed to staging June 13, 2026 04:11 — with GitHub Actions Inactive

Afgan0r merged commit c95daf4 into master Jun 13, 2026
3 checks passed

Afgan0r mentioned this pull request Jun 15, 2026

chore(staging): bump app images to post-Sentry builds #6

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci(06): verify kubectl-native CD — WG handshake + SA-auth + server-side dry-run - #1

ci(06): verify kubectl-native CD — WG handshake + SA-auth + server-side dry-run#1
Afgan0r merged 7 commits into
masterfrom
ci/verify-phase6-cd

Afgan0r commented Jun 13, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Afgan0r commented Jun 13, 2026

Exercises (non-mutating)

Change

Operator prereqs (done out-of-band)

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant