Incident 2026-08-25: helm apply with public values half wiped the GitHub App identity from both prod ConfigMaps — GitSync + registry pulls down ~15 min

[rbuergi]

Impact

2026-08-25 ~11:13–11:30 UTC, both prod portals (memex.systemorph.com and memex.meshweaver.cloud): every GitHub sync failed with "Connect your GitHub account first … or configure the GitHub App identity". Because memex-cloud is the plugin registry and its GitHub App credential is what pulls Systemorph/MeshWeaver.Plugins@main, this also stalled catalog auto-updates and the in-flight _App migration. Restored at ~11:24Z; both portals verified (git_hub_sync op=check space=Store → "Branch 'main' is at b8f3523a", Succeeded).

Root cause

The GitHub App identity lived only as live drift on the ConfigMap, and a helm apply rendered from the public values half clobbered it.

• GitHub__App__{ClientId,InstallationId,InstallationOwner} were never in any values source: I decoded the last-good helm release secrets (sh.helm.release.v1.memex.v17, sh.helm.release.v1.memexcloud.v3) and their rendered memex-portal-config has all three empty — the chart templates them as default "". The real values had been kubectl patched onto the live ConfigMap (2026-07-09, the GitSync-as-App rollout) and survived every apply since only because those applies didn't touch the keys.
• At 10:54:08Z (ns memex, helm rev v18) and 11:06:06Z (ns memex-cloud, rev v4) — four minutes after helm: render OpenRouter__Models — twelve declared keys reached no container #2203 ("render OpenRouter__Models") merged — helm upgrades were applied with the public halves only. The CM re-render emptied the three keys; the failure began when the new ReplicaSets came Ready (~11:13Z).
• The applier is unidentified (no session claimed it at the time of writing); the mechanism does not depend on who — any apply without the identity in a values source reproduces it.
• Not related to RCA #2194: stale-build retention docs + fleet-wide Modules:AutoRecycleOnStaleBuild opt-in #2197's enablement earlier this morning: that was a single-key kubectl patch --type merge (cannot remove keys), and sync still worked at 11:04Z, after its rollout.

Remediation (done)

Three-key merge patch restored in both namespaces (values confirmed from two independent sources: the 2026-07-09 rollout record, and gh api /orgs/Systemorph/installations → installation 144517285, app meshweaver-cloud, app_id 4220566) + rolling restart; newest pods' env verified; sync check green on both portals. Full-CM diff against both last-good renders: zero other chart-managed keys were emptied — the GitHub App trio was the complete loss set. The PEM half (KV github-app-privatekey → SPC → Secret) was never affected.

Durable fixes

1. The identity enters the per-env values source (config half, beside the public per-env values) so any future render carries it — a live-drift-only value is one apply away from deletion, permanently. (Applied on the operator machine's per-env values; see below.)
2. Pre-apply guard: the sanctioned apply path must diff the rendered memex-portal-config against the LIVE ConfigMap and refuse when the render empties a key that is non-empty live (override flag for intentional removals). A chart whose defaults are "" makes "the private half was missing from this render" indistinguishable from "this key was never set" — the guard is what makes that loud. This is the same invariant class as the existing check-chart-drift.sh, but enforced before the write instead of observed after.
3. Standing rule already in AGENTS/runbook, re-affirmed: deploy.sh/full-chart applies are env-setup, not the code/config-update path; a config fix ships as a values change + targeted apply with the guard, never a public-halves-only helm upgrade.

Cross-refs: #2194 (this morning's unrelated stale-build outage — same day, different mechanism), #2203 (the chart change whose deploy carried the bad render), 2026-07-09 GitSync-as-App rollout (where the drift was introduced).

[rbuergi]

Root cause sharpened + durable fixes landed (Systemorph/Memex#112, merged):

Why it recurred: this exact incident already happened on 2026-08-24, and its fix committed the GitHub App identity into the overlay one section off — under config.memex_migration, where the portal ConfigMap template (config.memex_portal.GitHub__App__*) never reads it. The overlay looked safe (key + reassuring comment visibly present), while every helm-release deploy still rendered the keys empty. Today's 10:53Z/11:04Z deploys (helm-release workflow_dispatch, actor = the shared rbuergi identity) applied that blanking to both instances.

Landed in Systemorph/Memex#112:

1. Identity block moved under memex_portal in both overlays; InstallationId/Owner committed too so render == live exactly (ClientId + PrivateKey remains the verified minimum pair).
2. Modules__AutoRecycleOnStaleBuild: "true" in all three overlays — the helm-release pipeline layers only vault + overlay, so the platform repo's values.aks.yaml (RCA #2194: stale-build retention docs + fleet-wide Modules:AutoRecycleOnStaleBuild opt-in #2197) never reaches these deploys; the fleet policy must live in the overlays.
3. Blank-guard in helm-release.yml (deploy only): renders vault + overlay on the runner and refuses when the render carries as "" a key that is non-empty on the live memex-portal-config, naming each key; allow_blank input confirms intentional removals. Keys absent from the render are not flagged (helm's three-way merge leaves them alone). Verified by local render with the live chart + real vault halves; the guard itself gets its first live exercise on the next real deploy.

Live state, the overlays, and the chart now agree; the drift that carried the identity for six weeks is retired.

[rbuergi]

Durable fix 2 ("pre-apply guard") shipped in Systemorph/Memex (helm-release.yml blank-guard). This adds the half it cannot cover: the guard runs at DEPLOY time, against the live cluster, so a mis-nested key is invisible until someone deploys. #2278 asserts the same invariant at authoring time, from the files alone.

deploy/aks/scripts/check-values-are-read.{sh,py} — every config.<component>.<KEY> a values file sets must be read by that component's templates. It takes values paths, so the private repo runs the identical check on its overlays.

Rendered proof of the mechanism, since a template must never be verified by reading it

helm template over values.yaml + values.aks.yaml, correct nesting vs. the exact 08-24 mis-nesting:

--- render-correct.yaml
+++ render-misnested.yaml
@@ -533,9 +533,9 @@
-  GitHub__App__ClientId: "Iv23liBtPCOxYjqyK3Yx"
-  GitHub__App__InstallationOwner: "Systemorph"
-  GitHub__App__InstallationId: "144517285"
+  GitHub__App__ClientId: ""
+  GitHub__App__InstallationOwner: ""
+  GitHub__App__InstallationId: ""

The new check refuses that input, naming the fix:

::error::`config.memex_migration.GitHub__App__ClientId` is read by NO template — but
  `config.memex_portal.GitHub__App__ClientId` IS read. This key is MIS-NESTED … the ConfigMap
  renders the key EMPTY (MeshWeaver#2210). Move it to `config.memex_portal`.

🚨 It found two LIVE defects of this exact class on its first contact with the production overlays

Run against the three committed values.<release>.public.yaml from Systemorph/Memex — 184 keys examined, 2 orphaned. Both are the "looks configured, reaches no container" shape, and both are currently live:

1. memex — a dead Hosting__Operator__Enabled, still carrying its reassuring comment

values.memex.public.yaml:81 sets config.memex_portal.Hosting__Operator__Enabled: "true", under a comment explaining the safety model. That path reaches nothing: the chart renders the key from .Values.hostingOperator.enabled, not from config.memex_portal.

This is the same key that already caused a two-day outage of the feature, documented in three places in this repo — INSTANCE-LIFECYCLE.md:95 ("memex declared Hosting__Operator__Enabled: "true" for two days while its live ConfigMap carried no Hosting__* key at all, and every action refused as 'disabled on this installation'") and again in config.yaml's own block comment. The chart was fixed; the overlay's wrong-path line was never removed.

Harmless today — hostingOperator.enabled: true is also set correctly at line 334, so the operator works — which is precisely why nobody noticed. It is a landmine with a comment beside it saying it is load-bearing.

2. atioz — email is configured under a key that no longer binds

values.atioz.public.yaml:65 sets config.memex_portal.Email__NoReplyAddress. The chart says what happened to it, at config.yaml:570:

# NOTE: binds EmailOptions.MailboxAddress — the old key Email__NoReplyAddress no longer binds.

atioz sets Email__Enabled: "true" and no Email__MailboxAddress at all; EmailOptions.MailboxAddress defaults to "". So a customer production portal has email switched on with no sender address, while its overlay reads as fully configured. This one is not harmless — it is a renamed key that took its value with it, the class in renamed-config-key-is-a-silent-deletion.

Neither would be caught by the deploy-time blank-guard: it only flags a key the render carries as "" while live is non-empty, and both of these are keys the render never carries at all.

Follow-up

A PR in Systemorph/Memex wires this check into build.yml (pull_request — cluster-free, no secret, so no skip-trapdoor, alongside the existing docs job) and fixes both overlays. Extracting is not using; the mis-nesting happened in the overlays, so the check has to run there.

🚨 The atioz fix changes behaviour — renaming the key to Email__MailboxAddress makes outbound email actually resolve a sender on a customer portal. The values change is unambiguous (the overlay's intent is plain), but the deploy is a separate decision and I am not making it.

I have not changed any live cluster state, and no portal has been rolled.

[rbuergi]

Closing — root cause identified and every named durable fix has shipped and merged, verified against origin/main and Systemorph/Memex today (2026-08-26).

Incident remediation (done, per the issue body): GitHub App identity restored on both memex and memex-cloud ConfigMaps; sync verified green on both portals.

Root cause (found in this thread): the 2026-08-24 fix for the same recurring problem put the identity block under config.memex_migration in the overlay — a section the portal ConfigMap template never reads (only config.memex_portal.GitHub__App__* is rendered). A helm-release deploy from that overlay therefore always rendered the three keys empty; the 2026-08-25 10:54Z/11:06Z deploys just happened to be the ones that triggered it.

Durable fix 1 — identity moved into the correct values path: Systemorph/Memex#112 ("GitHub App identity under memex_portal + blank-guard on deploy (MeshWeaver#2210)") — merged 2026-08-25T11:31:29Z. Moves the block from config.memex_migration to config.memex_portal in both overlays and commits InstallationId/InstallationOwner so the render matches live exactly.

Durable fix 2 — pre-apply guard shipped in the same PR: helm-release.yml now renders vault+overlay on the runner and refuses when the render would empty a key that is non-empty on the live ConfigMap, naming the key (allow_blank for intentional removals). This is exactly the class of failure that caused the incident — a mis-nested/incomplete values render silently wiping live config — and it is now a refused deploy, not a silent apply.

Durable fix 3 was "re-affirm the standing rule" (deploy.sh/full-chart applies are env-setup, not the code/config-update path) — already in AGENTS.md, no code owed.

Why the symptom can't recur the same way: the identity now lives in a section every render path actually reads, and even if a future overlay change reintroduces a mis-nesting of this or any other previously-set key, the blank-guard refuses the deploy before it reaches the cluster.

Not part of this close, tracked separately: #2278 ("Close three gates that render a tick without checking anything (#2210, #1786, #1709)") is a further, forward-looking hardening pass (an authoring-time check, catching this class of mistake before deploy rather than at deploy) — it's additional armor, not a precondition for closing this incident, and it remains open on its own.