Chart Drift's first completing run found 91 divergences on memex/memex-cloud — 8 DIFFERS a deploy would overwrite, and nobody owns the backlog

[rbuergi]

Chart Drift completed for the first time ever on 2026-08-26 (run 32939313904) after #1709 unblocked it — it had failed at preflight on all ten prior attempts. The backlog it found is the finding, and right now nobody owns it.

Filing so the daily red is a worklist rather than weather. #1709's own header is explicit about the risk: a gate whose red is permanent is a gate people learn to ignore.

What it found — 91 divergences across memex and memex-cloud

class	count	what it means	what a helm upgrade does
CLUSTER-ONLY	81 (64 of them inline env:)	live setting the chart has never heard of	deletes it
DIFFERS	8	chart and cluster disagree	overwrites live with the chart's value
CHART-ONLY	2	chart has it, cluster does not	nobody is getting it today

The 8 DIFFERS are the ones to look at first

These are not "missing config" — they are values where a deploy actively changes what is running:

• memex-cloud: ModelTier__Heavy, ModelTier__Light, ModelTier__Standard, ModelTier__Utility — AI model routing. A deploy would re-point tiers to whatever the chart says.
• memex: livenessProbe, readinessProbe — health-check timing. Worth pairing with the known rollout-stall behaviour on slow startup; changing probe budgets by accident is how a deploy turns into an outage.

This is not hypothetical — one entry has already been cashed in

##[error]CLUSTER-ONLY inline env WebhookInbox__Targets__0 (memex-cloud) is the root cause of #2235, the release broadcast that had not delivered since 08-22. An explicit env: on the Deployment (hand-set via kubectl set env, predating the chart key) takes priority over the same-named var from envFrom, so the ConfigMap said Hosting/PlatformBuilds while the pod used Store/Payments. Every signal was green: helm said "Upgrade complete", the ConfigMap was correct, the endpoint was wired — and the pod answered 404. Fixed in #2352 + Systemorph/Memex#121.

That was found independently by exec'ing into the pod, hours after this gate had already named the exact key. The other 80 entries have not been read.

Correcting a stale warning in the workflow header

chart-drift.yml's header warns that Authentication__Microsoft__ClientId is still CHANGE_ME_aad_application_client_id in the chart, so a deploy would overwrite working auth with the placeholder. That does not reproduce in this run — zero Authentication__ findings. Either it was fixed since the header was written or the key is not compared. Worth confirming which before anyone treats it as live, and updating the header either way.

Suggested shape

Triage by class rather than one-by-one: the 64 inline env: entries are almost certainly one story (hand-applied drift accumulated over time, each needing either a chart key or deletion), while the 8 DIFFERS each need an explicit "which side is authoritative" decision. #2210 is the cautionary tale for getting that wrong — a key committed one section off rendered "" and helm applied it as a silent deletion.

[rbuergi]

The 8 DIFFERS in detail — two separate stories, and one of them is a loaded gun

Pulling the actual values, because "decide which is authoritative" is impossible without them.

memex-cloud — 4 keys where the CHART RENDERS EMPTY

DIFFERS  ConfigMap ModelTier__Heavy     chart '' vs live 'claude-opus-4-7'
DIFFERS  ConfigMap ModelTier__Light     chart '' vs live 'claude-haiku-4-5'
DIFFERS  ConfigMap ModelTier__Standard  chart '' vs live 'claude-sonnet-4-6'
DIFFERS  ConfigMap ModelTier__Utility   chart '' vs live 'claude-haiku-4-5'

This is the #2210 shape exactly: a key rendered as "" clobbers live; a key ABSENT from the render is left alone. A helm-release.yml deploy of memexcloud would render these empty and blank the portal's AI model-tier routing.

The blank-guard should catch it — Memex#112 added a step that refuses a deploy when the render carries "" for a key that is non-empty live, naming each key. So the expected outcome is a REFUSED deploy, not a silent blanking. That is the guard working as designed, but it has a consequence worth stating plainly:

memex-cloud is currently undeployable via helm-release.yml until its overlay carries the four ModelTier__* keys.

That matters beyond tidiness: #2352's release-broadcast fix requires a memexcloud deploy to take effect. So that fix is blocked on this drift as well as on #2293.

memex — 2 probes where the chart is MISSING a field the live pod has

DIFFERS  livenessProbe
  chart {..., "periodSeconds": 15, "timeoutSeconds": 10, "failureThreshold": 6}
  live  {..., "initialDelaySeconds": 60, "periodSeconds": 15, "timeoutSeconds": 10, "failureThreshold": 6}

DIFFERS  readinessProbe
  chart {..., "periodSeconds": 10, "timeoutSeconds": 5, "failureThreshold": 3}
  live  {..., "initialDelaySeconds": 20, "periodSeconds": 10, "timeoutSeconds": 5, "failureThreshold": 3}

The chart declares no initialDelaySeconds; the live pod has 60 s (liveness) and 20 s (readiness).

Verified live just now — memex-portal-deployment, revision 573: liveness: 60 readiness: 20. So the recent roll onto 2699e4a05 did NOT lose them, which means it was a kubectl set image roll rather than a helm upgrade. Good — but it makes this a loaded gun rather than a past incident.

A helm upgrade of memex would remove both delays, so liveness starts probing at t=0 with a 6×15 s budget and readiness at t=0. On a portal whose startup does NodeType prewarming and schema work, losing a 60 s liveness head start is how a routine deploy becomes a CrashLoop.

🚨 And I do not think the blank-guard covers this one. It is described as refusing when the render carries "" for a key that is non-empty live — a blanked string key. Here nothing is blanked: a structured probe object simply lacks a sub-field. Worth confirming against the guard's implementation before anyone relies on it for memex.

Suggested resolution

• ModelTier__* — live is authoritative (those are the models actually serving). Commit the four keys into deployments/aks/memex-cloud/values.memexcloud.public.yaml.
• probes — live is authoritative. Add initialDelaySeconds to the chart's probe templates (or the memex overlay), so the chart stops being the more dangerous of the two.

In both cases the chart is the one that is wrong, and in both cases the drift only bites on the next deploy — which is precisely why it has gone unnoticed.

[rbuergi]

Re-measured 2026-08-29: 91 → 36 divergences, and the remaining set has a shape

Newest completing run (workflow "Chart Drift", scheduled 2026-08-28 09:22Z). It has been red on
every one of its last 17 runs — I found it by sweeping the fleet for workflows that have never
gone green, which is worth saying out loud: a detector that is permanently red stops being read.

The 36 split into three groups, and they do not have the same fix:

1. The AI providers — hand-set during the 08-27 restore, never moved into the chart

CLUSTER-ONLY ConfigMap  Features__Ai__Providers__AzureOpenAI
CLUSTER-ONLY ConfigMap  Features__Ai__Providers__OpenAI
CLUSTER-ONLY ConfigMap  Features__Ai__Providers__OpenAICompatible
CLUSTER-ONLY ConfigMap  Features__Ai__Providers__OpenRouter

These are the keys the models were brought back with by env activation. The follow-up — move them
into the Memex overlays + chart — is exactly what this detector is reporting as still outstanding.

2. The plugin-catalog and Kestrel block — inline env: on the Deployment

PluginCatalog__RegistryToken · RegistryUrl · RequireInstanceKey · DefaultGrants__0
PluginCatalog__Sources__{0,1,2}__{Format,Name,Ref,RepoPath}
ASPNETCORE_Kestrel__Endpoints__{Grpc,GrpcTrusted,Http}__{Url,Protocols}
Grpc__TrustedPort · Commerce__BaseUrl · Portal__ReactAppUrl · PreWarm__BatchBake · PreWarm__GateReadiness

🚨 These cannot be cleared by re-rendering the chart. An inline env: entry on the Deployment
shadows envFrom permanently — the same mechanism that was the root cause in #2494. Adding the
key to the chart does not remove the shadow; the entry has to be deleted explicitly
(kubectl set env deploy/… VAR-, through the pipeline). So any plan that reads "add these to the
chart and the drift clears" will leave the cluster on the hand-set value and the detector red.

3. PluginCatalog__RegistryToken is a credential sitting in a Deployment spec. Worth separating
from the rest of the cleanup: anything that can read the Deployment can read it, and it belongs in
the secret half of the values split, not inline. Flagging rather than touching — this is a live
cluster and the roll policy is that CD owns it.

Suggestion

Split this issue by group. (1) is a known outstanding follow-up with an owner. (2) needs the
shadow-removal step spelled out or it will be "fixed" and stay red. (3) is a security cleanup that
should not wait on either.

And while the count is high, consider whether this lane should be a required context somewhere,
or at least alert — 17 consecutive reds on a drift detector is indistinguishable from the detector
being switched off.

[rbuergi]

Cross-link: the divergence backlog now has an owner-shape. Systemorph/Memex#148 inventories everything that lives only on AKS (env patches, live-edited ConfigMap keys, unmanaged objects — measured against helm get manifest on 2026-08-30), the records render the values since Memex#146/#147, and the Hosting Audit action (#2734 merged, Plugins#955 in flight) reports live-vs-record drift per instance. Chart Drift's remaining 36 should shrink to zero as those rows move onto the records; when it does, this workflow's red is meaningful again.

[rbuergi]

The CLUSTER-ONLY class is not a latent risk — one of its members took memex down at boot four days later

This issue frames the 81 CLUSTER-ONLY settings as "live setting the chart has never heard of → a helm upgrade deletes it", and ranks the 8 DIFFERS first. I would re-rank that, on the strength of an incident on 2026-08-30.

What happened

The self-updater rolled ci.6576 onto memex. One pod died at boot with SIGABRT + core dump on EmailConfigurationGuard — "Email:Enabled=true but Email:ClientId is not set". The restart on identical config came up fine.

The cause is precisely the CLUSTER-ONLY shape this issue counts:

• the chart's ConfigMap memex-portal-config rendered the chart defaults (Email__Enabled=false, Email__ClientId=);
• a hand-made secret memex-email-secret had been patched onto the live Deployment as explicit env entries EMAIL__ENABLED, EMAIL__CLIENTID, … — absent from helm get manifest, i.e. cluster-only, i.e. exactly the 64 inline env: entries this issue found.

Linux env is case-sensitive, so both sets exist on the pod. .NET's environment provider is case-insensitive, and the last-enumerated duplicate wins — a coin toss per process start. A mixed draw (Enabled from the secret, ClientId from the ConfigMap) is exactly the refusal shape the guard reported.

Why this changes the ranking

The issue's model of the risk is "a deploy will delete these" — a future, deploy-triggered loss. That understates it. A cluster-only setting that collides with a chart key is not waiting for a deploy at all: it is already non-deterministic on every pod start, and it fails at random. "Works after a restart, on unchanged config" is the signature.

So the CLUSTER-ONLY column contains two different hazards, and only one of them is about helm upgrade:

	hazard	when it fires
cluster-only, no chart counterpart	deleted by the next helm upgrade	on deploy
cluster-only, colliding with a chart key	resolves at random per pod start	already, on every start

The second is worth extracting from the 81 as its own worklist, and it is mechanically findable: for each cluster-only env: key, check whether the chart ConfigMap defines the same key under case-insensitive comparison. Any hit is live non-determinism, not future drift.

Stated precisely: I have not verified that the EMAIL__* keys were in the 2026-08-26 snapshot this issue reports (the drift run predates the incident by four days). What is verified is that the mechanism is the class this issue counts, on one of the two clusters it covers.

The policy question this issue asks is already answered

"nobody owns the backlog"

The maintainer gave the directive on the morning of the incident: "get rid of secret places and make all transparent in Systemorph/Memex and Hosting as data types" — every non-secret value on the Deployment RECORD (mesh/Deployments/<env>.json, typed Hosting/Deployment) → HelmValues.Render → values.<release>.public.yaml → chart ConfigMap. Nothing may live only on the cluster.

That converts this issue from "who owns this?" into a defined migration: the 81 cluster-only settings are the backlog of values that have not yet moved onto the Deployment record. Suggest re-titling around that, so the daily red reads as progress against a known target rather than as weather — which is the exact failure mode #1709's header warned about and this issue was filed to prevent.

[rbuergi]

Re-measured on the latest completing run: the drift has GROWN, and the dangerous class is not DIFFERS — it is CLUSTER-ONLY

Re-ran this issue's own measurement rather than trusting the filing (run 33297180432, 2026-08-30). Two things changed.

1. Chart Drift is not broken — it has failed every day and each red is real

08-27  failure      08-28  failure      08-29  failure      08-30  failure

Four consecutive daily reds, and the gate is working as designed: Assert the inputs exist and Read the environment list both pass, then the per-environment jobs report actual divergence. This is the honest failure, not a stuck job — worth stating, because four identical daily reds is exactly the shape people learn to stop reading.

2. The count is up, and the class is worse than reported

	this issue (filed)	now
divergences	91	112 (memex 75, memex-cloud 37)
of which CLUSTER-ONLY	—	69

This issue's headline is "8 DIFFERS a deploy would overwrite". CLUSTER-ONLY is the sharper problem: a helm upgrade does not overwrite those values, it DROPS them, because they exist on the live cluster and in no committed chart.

What is currently cluster-only (key names only — values deliberately not read):

ConfigMap   Features__Ai__Providers__{AzureOpenAI,OpenAI,OpenAICompatible,OpenRouter}
            LogWatch__{DefaultRepository,Routes__0__*,Routes__1__*}
            Portal__ReactAppUrl
inline env  EMAIL__{CLIENTID,CLIENTSECRET,TENANTID,ENABLED,INBOUNDENABLED,
                    MAILBOXADDRESS,SUBSCRIPTIONCLIENTSTATE,WEBHOOKBASEURL}
            ASPNETCORE_Kestrel__Endpoints__{Grpc,GrpcTrusted,Http}__*
            Features__Ai__{Clis__{ClaudeCode,Copilot},Providers__AzureFoundry}
            Commerce__BaseUrl · Grpc__TrustedPort · PluginCatalog__AutoUpdateByDefault

Read that list as "what breaks on the next helm upgrade": email delivery and inbound webhooks, every AI provider, the gRPC endpoints, LogWatch routing, the React app URL, plugin auto-update. That is not a backlog of cosmetic divergence — it is a live configuration set that exists in exactly one place and is one deploy away from being gone.

It also directly contradicts the fleet rule that nothing may live only on AKS, which was learned from the memex boot crash where a chart ConfigMap was blank and the real value existed only as a cluster patch.

Ownership, which is what this issue is actually about

The issue's own words are "nobody owns the backlog". That is still true and is now the binding constraint: the measurement is automated, correct, running daily, and read by no one. 69 entries need triaging into "commit to the chart" or "delete from the cluster", and until someone does, the daily red is load-bearing information going nowhere.

🚨 EMAIL__CLIENTSECRET appears as a cluster-only inline env. I have deliberately not read any value — only key names — but a secret held that way is worth a separate look independent of this issue.

(No cluster access was used: this is entirely from the workflow's own log. Note the log is dominated by the workflow echoing its own run: blocks — filtering to real ##[error] output is what makes the counts trustworthy.)

[rbuergi]

Re-measured on the newest completing run (2026-09-01) — 112 → 63, and the credential is gone

Re-ran this issue's own measurement rather than trusting the last comment's numbers, per the method that has already caught drift on this issue twice. Run 33478523371, scheduled 06:39Z today.

The count, tracked across every re-measurement

	filed 08-26	08-29	08-30	09-01
total divergences	91	36	112	63
CLUSTER-ONLY	81	—	69	61
DIFFERS	8	—	—	2
CHART-ONLY	2	—	—	0

Split by namespace today: memex-cloud 35 (all CLUSTER-ONLY, 0 DIFFERS), memex 28 (26 CLUSTER-ONLY + the 2 DIFFERS).

Two things genuinely got fixed, and they are the two that mattered most

1 · 🚨 EMAIL__CLIENTSECRET is gone from the Deployment. The 08-30 comment flagged eight cluster-only inline EMAIL__* entries including a client secret, and asked for it to be looked at independently of this issue. All eight are absent from today's run, and I confirmed it directly on the live cluster rather than trusting the report:

kubectl get deploy memex-portal-deployment -n memex       -o json | grep -o '"name": "EMAIL__[A-Z]*"'  → (none)
kubectl get deploy memex-portal-deployment -n memex-cloud -o json | grep -o '"name": "EMAIL__[A-Z]*"'  → (none)

(Key names only — no value was read. Auditing for a plaintext secret by selecting its value is how the audit becomes the leak.)

2 · The ModelTier__* blank-render is gone. The four memex-cloud DIFFERS where the chart rendered '' against live claude-opus-4-7 / claude-haiku-4-5 / claude-sonnet-4-6 — the #2210 shape, and the reason the comment concluded "memex-cloud is currently undeployable via helm-release.yml" — do not appear in today's run. memex-cloud now reports zero DIFFERS.

What is still loaded, and the ranking still holds

The 2 remaining DIFFERS are the probes on memex, exactly as detailed on 08-26 and still not fixed:

DIFFERS  livenessProbe    chart has no initialDelaySeconds; live has 60
DIFFERS  readinessProbe   chart has no initialDelaySeconds; live has 20

A helm upgrade of memex still removes both delays, so liveness would start probing at t=0 on a portal that does NodeType prewarming at startup. The 08-26 comment's doubt about whether the blank-guard covers a structured probe object missing a sub-field (as opposed to a blanked string key) was never resolved, so this remains the loaded gun it was called.

And the CLUSTER-ONLY class is still what a helm upgrade would drop — the 08-30 re-ranking stands. Today's set, key names only:

memex-cloud (35)  Features__Ai__Providers__{AzureOpenAI,OpenAI,OpenAICompatible,OpenRouter}
                  Portal__ReactAppUrl · PluginCatalog__{RegistryUrl,RegistryToken,RequireInstanceKey,
                  DefaultGrants__0,Sources__{0,1,2}__{Format,Name,Ref,RepoPath}}
                  ASPNETCORE_Kestrel__Endpoints__{Grpc,GrpcTrusted,Http}__* · Grpc__TrustedPort
                  PreWarm__{BatchBake,GateReadiness,PrebuiltBundleRoot} · Commerce__BaseUrl
                  Speech__{Enabled,Endpoint,Language} · WebhookInbox__Targets__0

memex (26)        the same AI-provider and Kestrel families, plus
                  LogWatch__{DefaultRepository,Routes__{0,1}__{Prefix,Repository}}
                  Features__Ai__{Clis__{ClaudeCode,Copilot},Providers__AzureFoundry}
                  PluginCatalog__AutoUpdateByDefault

PluginCatalog__RegistryToken is still a credential inline on both Deployments — the 08-29 comment separated that from the rest of the cleanup and it has not moved, while the EMAIL__* set did. Worth carrying as the remaining half of that security item.

Note WebhookInbox__Targets__0 is still CLUSTER-ONLY on memex-cloud — the same key whose shadowing was the root cause of #2235. It was fixed by adding the chart key; the inline env that shadows it was never deleted, which is exactly the trap the 08-29 comment predicted ("any plan that reads 'add these to the chart and the drift clears' will leave the cluster on the hand-set value and the detector red").

Verdict: keep, but the framing has earned an update

The issue's headline — "8 DIFFERS a deploy would overwrite, and nobody owns the backlog" — is now wrong in both halves. There are 2 DIFFERS, not 8, and the backlog demonstrably has an owner: 49 entries have been cleared in six days, including the credential and the four blanked model tiers. The 08-30 comment's suggestion to re-title around the Deployment-record migration is the right move, and today's numbers are the evidence that the migration is working rather than stalled.

What would close it: Chart Drift going green — i.e. initialDelaySeconds committed to the memex probe templates, and the 61 cluster-only settings either on the Deployment record or deleted from the cluster (deleted, for the inline env: ones — adding a chart key does not lift an env: shadow).

(Read-only: the workflow's own log, plus two kubectl get deploy reads projected to key names.)

[rbuergi]

Context that applies to this issue and is recorded nowhere: both portals are ~226 commits behind, and not self-updating

Measured 2026-09-02:

$ curl -s https://memex.meshweaver.cloud/api/version
{"version":"3.0.0-rc9+0a1eabdc…","commit":"0a1eabdc028a6f6cf09b1f9927b1dbaf1dbb30c4"}
$ curl -s https://memex.systemorph.com/api/version      # same commit

0a1eabdc = 2026-09-01 05:41
$ git rev-list --count 0a1eabdc..origin/main → 226

Why they are stuck is a chicken-and-egg worth knowing before diagnosing anything against these portals. The self-update fix #2935 (fcfcca231, "establishing the policy is not the same as re-establishing it") merged at 07:19 on 09-01 — 1h38m AFTER the build they are running. So they carry the defect it fixes: the first Admin/UpdatePolicy read faults, the retry is paced at the 6 h RetryInterval rather than 30 s, and the install is inert, not slow — the safety net cannot bound it because it sits behind the same gate. The fix for the stuck self-update is in a build the stuck self-update will not take.

What this means for this issue specifically:

• Any behaviour measured on memex or memex-cloud right now is measured on 09-01 05:41 code, not on main. A symptom that reproduces there may already be fixed; one that does not may still be live.
• Conversely, none of the fixes merged since then are in production — so "still happening on prod" is not evidence that a merged fix did not work.

The recovery order matters and is not obvious. Rolling now would fix the self-update but land on a build whose content bake is inert (#3022, reopened — the bake identity still does not match the promoted portal image), so every pod would recompile the whole shipped mesh at boot: the #2194 wave that took two silos to 20 GB. The sequence is:

1. land main-cd: the bake's framework identity does not match the promoted portal image, so the publication is never sealed and every satellite is starved #3022 so the bake identity matches and bundles are adoptable;
2. verify that build's framework-identity dirs exist in that instance's storage (/data/prebuilt-bundles/_releases/<version>);
3. then roll — which fixes the self-update in the same move.

Posting this on each affected issue rather than only in one place, because the staleness silently changes what a prod measurement means, and the next person to measure will not otherwise know.

[rbuergi]

Re-measured 2026-09-02: the backlog is 63, not 91 — and every remaining DIFFERS is the same thing

The issue's own numbers are a week old, and this issue exists to stop a red becoming weather, so the first useful act is to re-read it. Latest completing run: 33600045101 (2026-09-02 06:42Z).

class	2026-08-26	2026-09-02
CLUSTER-ONLY	81	63	−18
DIFFERS	8	4	−4
CHART-ONLY	2	2	—
total	91	63	−28

Split by namespace: memex-cloud 35 divergences across 204 compared fields, memex 28 across 188.

So "nobody owns the backlog" is no longer true — roughly a third of it has been worked off in a week. Worth saying plainly, because the premise of this issue is that the red would be ignored, and the measurement says it has not been.

🚨 The four remaining DIFFERS are all one story, and it is the dangerous one

##[error]DIFFERS      livenessProbe
##[error]DIFFERS      readinessProbe

…once per namespace. The four ModelTier__* entries are gone — Heavy, Light, Standard, Utility on memex-cloud no longer diverge, so AI model routing is now consistent between chart and cluster.

What is left is exactly the pair this issue singled out as most worth fearing:

memex: livenessProbe, readinessProbe — health-check timing. Worth pairing with the known rollout-stall behaviour on slow startup; changing probe budgets by accident is how a deploy turns into an outage.

Two things have changed about them since:

1. They have SPREAD. In August the probes diverged on memex only. They now diverge on both namespaces. Whatever set them by hand has been applied twice, or the chart moved under both.
2. The stakes are now documented. Doc/Architecture/DeploymentAKS and the rollout-stall record describe a portal that fails its probes while a startup DDL deadlock clears — so a helm upgrade overwriting live probe budgets with the chart's is not a theoretical risk on this deployment, it is the mechanism of a known incident.

That makes the remaining DIFFERS list unusually tractable: it is not eight unrelated decisions any more, it is one — which side owns probe timing? — and answering it clears the whole DIFFERS class.

Correcting the header, as the issue asked

The issue flagged that chart-drift.yml's header warns about Authentication__Microsoft__ClientId still being CHANGE_ME_…, and that it did not reproduce. It still does not: zero Authentication__ findings in this run either. Two clean runs a week apart is enough to call the header stale rather than lucky.

Disposition

Not closing — the 63 CLUSTER-ONLY entries are still a real backlog, and every one of them is something a helm upgrade deletes. But this issue's worklist should be re-cut around the numbers above rather than the August ones, and the single probe-ownership decision is the highest-value thing on it.

_{Re-measured 2026-09-02 from run 33600045101. Re-read before acting — this drifts daily, which is the point of the gate.}

[rbuergi]

Re-measured 2026-09-03 — and the premise this issue ranks by does not hold

Re-ran this issue's own measurement, per the method that has already caught drift here three times. But the useful finding this round is not the count: it is that "what a helm upgrade does" — the column this issue's table is built on, and every re-ranking since — is wrong.

The claim, and the measurement

The report's legend, quoted in the opening table:

class	claimed consequence
CLUSTER-ONLY	"a helm upgrade deletes it"
DIFFERS	"overwrites live with the chart's value"

Measured on helm v3.21.1 — the exact binary az aks command invoke runs — and repeated on v4.2.4. A throwaway release was installed, drifted in all four shapes this detector reports, then upgraded with a chart that genuinely changed: new image, new ConfigMap value, and a new periodSeconds on the same probe object the drift had touched.

drift applied to the live objects	after helm upgrade
live-only ConfigMap key	survives
live-only inline env: entry	survives
inline env: duplicating a chart ConfigMap key	survives
initialDelaySeconds added to a live probe	survives

The chart's own three changes all landed — the positive control, without which the test could not have failed. Helm's three-way merge removes only what Helm previously owned; anything applied out-of-band is in neither the old nor the new manifest, so no patch touches it.

Systemorph/Memex#148 already recorded this, in its section B: "a helm upgrade PRESERVES them". That inventory and this detector's legend have been contradicting each other for a week, and the inventory was right.

Both halves matter. The urgency this issue created — "one deploy away from being gone" — was false. So is the reassurance that the next deploy tidies it up. Drift is only ever cleared deliberately.

What the false ranking hid

An inline env: entry overrides envFrom. So an inline entry matching a ConfigMap key does not merely duplicate it — it makes the chart's value dead, while it still renders, still reads correctly in values.*.yaml, and still shows in kubectl get configmap. That is #2235 exactly (ConfigMap said Hosting/PlatformBuilds, pod ran Store/Payments, every signal green, 404 for eleven days), and the 2026-08-30 memex boot crash (EMAIL__* beside Email__*; Linux env case-sensitive, .NET's provider not, last enumerated wins ⇒ coin toss per pod start).

check-chart-drift.sh's header had described that override since the day it was written, and nothing in it ever checked for it. Every such entry was reported as a plain CLUSTER-ONLY, sorted in among dozens of harmless ones — which is the mechanical reason this issue reads as an undifferentiated backlog nobody owns. It was not a lack of ownership; it was a report that could not tell the dangerous entries from the inert ones.

Re-cut, same findings, ranked by what is actually wrong

#3168 adds two classes above the existing three and ranks the report worst-first. Totals and compared-field counts are byte-identical to the current script on both namespaces (A/B on the same inputs) — nothing invented, nothing lost:

	today	of which
memex	28	13 SHADOWS (5 disagreeing), 13 CLUSTER-ONLY, 2 DIFFERS
memex-cloud	44	24 SHADOWS (3 disagreeing), 11 CLUSTER-ONLY, 9 DIFFERS

(memex-cloud is 44, not the 35 of 09-02, because its committed overlay changed since — visible under the old script too.)

🚨 The disagreeing ones are wrong on the cluster right now, and they are on the bake path:

both portals   PreWarm__BatchBake            ConfigMap ''      → pod runs 'true'
both portals   PreWarm__PrebuiltBundleRoot   ConfigMap ''      → pod runs '/data/prebuilt-bundles'
both portals   PluginCatalog__RegistryUrl    ConfigMap ''      → pod runs 'https://memex.meshweaver.cloud'
memex          PreWarm__GateReadiness        ConfigMap 'false' → pod runs 'true'
memex          Features__Ai__Providers__AzureOpenAI  ConfigMap 'false' → pod runs 'true'

PreWarm__GateReadiness=true with PreWarm__DynamicTypes=false is the combination the chart documents as making the readiness gate permanently green — armed, reporting green, verifying nothing. Filed as Systemorph/Memex#171. Anyone reasoning about #3022 or #2194 from the chart or the ConfigMap is reasoning about a configuration no pod runs.

The probes: this issue's top-ranked item is inert, and the recommendation was backwards

Called "a loaded gun" on 08-26 and "the highest-value thing on the list" on 09-02, with the fix stated three times as "live is authoritative — add initialDelaySeconds to the chart". It is the opposite, and two measurements settle it:

1. The live pod carries a startupProbe, byte-identical to the chart (/health, periodSeconds: 5, failureThreshold: 60 — a 300 s window). Kubernetes suspends liveness and readiness until a startup probe succeeds, so a delay measured from container start can never be what protects a slow boot. The startup probe already did.
2. memex-cloud runs this exact probe block with no initialDelaySeconds at all, and is healthy. The "loaded gun" already fired there, harmlessly. (Correcting 09-02's "they have SPREAD… they now diverge on both namespaces" — the run it cites shows the probes diverging on memex only, and memex-cloud reporting zero DIFFERS. Same run also shows 0 CHART-ONLY, not 2.)

So adding them would enshrine dead config and make the chart the more dangerous side. #3168 puts that reasoning beside the probes so it is not re-derived wrongly a fourth time.

Disposition

Not closing. The backlog is real — but it should be worked in this order, and only the first two are live-wrong:

1. SHADOWS / COLLIDES — wrong now. Clearing one takes both steps in order: chart value first, then delete the inline entry (kubectl set env deploy/… KEY-). Either step alone leaves the pod on the inline value, so any plan reading "add these to the chart and the drift clears" leaves the cluster exactly where it is. (08-29's comment made this point about the shadow; it was right and got lost.)
2. CLUSTER-ONLY — not deploy-urgent, but in no committed source. This is the Systemorph/Memex#148 migration.
3. DIFFERS — a decision, never a deploy.

_{Re-measured 2026-09-03 from the live cluster and from a controlled helm experiment. Re-read before acting — this drifts daily, which is the point of the gate.}

[rbuergi]

The 76 findings, classified — and two more things the detector could not see

Re-measured first, per this issue's own method: I dispatched the workflow rather than reading a stale log, so this is run 33755512452 (2026-09-03 12:30Z) — the first run with #3168's and #3204's classes, since yesterday's scheduled run started at 06:38Z, four minutes before #3168 merged. No cluster access was used: the workflow's own log, plus the committed overlays read from Systemorph/Memex.

memex        32 divergences across 188 compared fields:  14 SHADOWS, 16 CLUSTER-ONLY,  2 DIFFERS
memex-cloud  44 divergences across 204 compared fields:  24 SHADOWS, 15 CLUSTER-ONLY,  5 DIFFERS

76 total. Zero COLLIDES, zero CHART-ONLY.

The classification — seven groups, not seventy-six decisions

Each finding lands in exactly one group; the counts sum to 76.

#	group	n	what it actually is	remedy
1	🚨 Disagreeing SHADOWS	8	the pod runs a value the ConfigMap contradicts. PreWarm__BatchBake + PreWarm__PrebuiltBundleRoot + PluginCatalog__RegistryUrl (both namespaces), PreWarm__GateReadiness + Features__Ai__Providers__AzureOpenAI (memex)	chart value first, then delete the inline entry. Either step alone changes nothing
2	🔑 The registry credential	2	PluginCatalog__RegistryToken. On memex a SHADOWS over secret/memex-portal-secrets — the two values differ. On memex-cloud the inline entry is the only copy	#3201, in its stated order. Not a cleanup — a rotation
3	Agreeing SHADOWS	29	not wrong today. The ConfigMap simply is not what the pod reads, so the next chart change to any of them silently fails to take effect — Kestrel endpoints, PluginCatalog__Sources__*, WebhookInbox__Targets__0, Grpc__TrustedPort	same two steps, no urgency
4	Chart-retired, inert	8	FrameworkBroadcast__Subscribers__0..3 × 2 namespaces. The chart stopped rendering these today — the subscriber set is mesh data now — so the live keys feed nothing	delete the keys; and drop them from memex-cloud's overlay, which still sets them
5	Cluster-only, now expressible	22	live-edited settings the chart had no key for until #3199 — AI providers ×8, LogWatch__* ×5, Portal__ReactAppUrl ×2, Features__Ai__Clis__* ×2, Speech__* ×3, Commerce__BaseUrl, PluginCatalog__DefaultGrants__0	onto the Hosting/Deployment record — Memex#148. #3199 is what made this possible at all
6	Committed, never deployed	5	memex-cloud's overlay carries PreWarm__{BatchBake,BuildProtocol,DynamicTypes,GateReadiness}: "true" and probes.startup.failureThreshold: 1080; live runs the shipped defaults. This is not cluster drift — it is an un-run deploy surfacing here	a deploy. It is deploy-drift's class, visible in this report by coincidence
7	Ruled inert	2	the memex liveness/readiness initialDelaySeconds	nothing. #3168 settled it; the reasoning now sits beside the probes so it is not re-derived a fifth time

Group 4 is why the counts moved rather than the drift. memex went 28 → 32 and memex-cloud's DIFFERS 9 → 5 because those four keys per namespace changed class today when the chart retired them. Reading that as "the backlog grew" would be wrong.

Is the detector trustworthy? Mostly — with two holes I closed and one boundary that is real

This issue's premise was ranked by a hazard #3168 showed does not exist, so the question is fair. I audited the comparator rather than the output. Three findings.

🚨 A · A branch that counted a comparison and made none — FIXED

An inline env: entry present on both sides was matched by NAME and continued, after comparisons += 1. So the run counted it as a field it had compared and compared nothing.

The chart renders five such entries — the DOTNET_Dbg* crash-dump block and the self-updater's AZURE_CLIENT_ID. A kubectl set env over any of them produced no finding at all, and a wrong AZURE_CLIENT_ID breaks the ACR credential the self-updater pulls with, silently, while this report said the inline env was clean. Same family as "an aggregate that only matches skipped": the check ran, rendered a verdict, and could not have failed.

Fixed in #3214 — values compared, never printed. Reported as DIFFERS, carrying the one exception to #3168's measurement: helm owns an entry it renders, so an upgrade DOES reset that one.

🚨 B · The shadow check saw Secrets and not ConfigMaps — FIXED

#3204 taught it to see an inline env shadowing an envFrom Secret. But .Values.extraEnvFrom takes verbatim EnvFromSource objects and the chart documents both {secretRef: …} and {configMapRef: …}; an inline env overrides both identically. Only secretRef sources were enumerated, so a shadow over a second envFrom ConfigMap would have been filed as a plain CLUSTER-ONLY — #3201's blind spot, one source-kind over.

No live environment uses a second envFrom ConfigMap today, so this changes none of the 76. That is the point: found by reading the chart's documented contract, not by waiting for it to fire.

⚠️ C · The left-hand side is not the chart a deploy renders — NOT fixed, and it bounds group 6 and the DIFFERS column

This one is structural and it is worth knowing before acting on any DIFFERS:

	renders with
chart-drift.yml	deploy/helm/values.yaml + values.<release>.public.yaml
helm-release.yml (deploy)	the Key Vault capture + values.<release>.public.yaml

The capture is helm get values <release> — the whole user-supplied values document, config: included — and it is layered first, the committed overlay last. So:

• for a key the committed overlay carries, both renders agree and the finding is sound;
• for a key it does not carry, the "chart" column is a chart default, and a deploy may render something else.

That confines the doubt rather than removing it, and it is checkable per finding — look the key up in the overlay. I did that for the groups above: groups 1–5 all concern keys with no chart key at all before #3199 or a key the overlay explicitly carries, so they are sound. The residue is at most Portal__ReactAppUrl ×2 (gated behind portalNext.enabled, which the capture could set) and memex's FrameworkBroadcast ×4 — 6 of 76 whose group cannot be settled from outside the cluster.

Note the asymmetry that makes this a real defect rather than a nuance: the deploy's own blank-guard renders guard-vault.yaml + overlay, i.e. it uses the right inputs. The daily detector does not. The durable fix is to assert the capture holds no config: section at all — the overlay is already generated from the Hosting/Deployment record by HelmValues.Render, so the vault half only needs the secret sections it was created for. That assertion belongs beside the capture, in Systemorph/Memex's helm-release.yml; I have not written it, because it is that repo's and because verifying it means reading a values blob containing pgbackrest.azure.accountKey.

And the header warning this issue asked about, settled

chart-drift.yml's header warned that Authentication__Microsoft__ClientId was still CHANGE_ME_… so a deploy would overwrite working auth. Both halves are false. No run has ever reported an Authentication__ finding (08-26, 09-02, 09-03) — every environment sets a real client id in its own overlay, layered over the chart default, so the placeholder never renders; and a deploy does not overwrite a live value anyway (#3168). Corrected in place in #3214 rather than deleted, because that warning is what this issue was triaged against for a week.

What is blocked, plainly

Groups 1–5 all require editing the Deployment spec — kubectl set env deploy/… KEY-, or a values change that re-renders the pod template. Both spawn a new ReplicaSet, and #3207 is explicit that any new ReplicaSet on memex lands on DbVersionGate and crash-loops, because the image is ahead of its schema and only helm upgrade runs the migration Job. A fleet freeze is also active with both portals pinned. So:

• nothing in groups 1–5 is urgent enough to justify a pod restart today. Group 1 is live-wrong, but it has been live-wrong for weeks and the affected settings (PreWarm__*, PluginCatalog__RegistryUrl) are not causing an outage;
• group 2 (memex authenticates to the plugin registry with a plaintext inline token that SHADOWS the Key Vault one — and the two differ #3201) has steps 1–2 that are safe now and a step 3 that waits, exactly as that issue says;
• group 6 needs a deploy, which is the same blocked thing;
• group 7 needs nothing.

Order when the freeze lifts and the schema is current: #3207 → group 2 (credential, with rotation) → group 1 → group 6's deploy → groups 4 and 5 as one Memex#148 pass.

Disposition

Not closing. But "91 divergences, and nobody owns the backlog" is no longer the state: it is 76, in seven groups, six of which have a named owner and a stated remedy, and the seventh needs nothing. The classification is committed to Doc/Architecture/ChartDriftSemantics in #3214 so the next reader inherits it instead of re-deriving it — which is the fourth time this issue has been re-derived, and the point at which it should stop being.

_{Measured 2026-09-03 from a dispatched run of the workflow plus the committed overlays. No cluster mutation, no value read from any secret. Re-read before acting — this drifts daily, which is the point of the gate.}

[rbuergi]

Correcting my own comment above: I overstated the doubt in finding C

I wrote that the values-layer gap leaves "at most Portal__ReactAppUrl ×2 and memex's FrameworkBroadcast ×4 — 6 of 76 whose group cannot be settled from outside the cluster". The FrameworkBroadcast half is wrong, and the bound is tighter than I gave it.

COLLIDES and SHADOWS cannot be affected by that gap at all. Their verdict is computed against the live ConfigMap — the object the pod actually mounts — and falls back to the render only for a key that is rendered but not yet live. "The pod runs one value and the ConfigMap holds another" is a statement about two live objects; which values file CI rendered with is irrelevant to it. That is 38 of the 76 findings, including all 8 of the live-wrong list — i.e. group 1, the one that ranks first, is not in doubt.

And a key with no chart key at all is immune from the other direction: nothing can render it whatever the capture holds. That covers the retired FrameworkBroadcast__Subscribers__* (group 4) and every group-5 key that #3199 only just made expressible.

So the residue is Portal__ReactAppUrl ×2 — rendered only when portalNext.enabled is set, which the committed overlay does not set but the Key Vault capture could. Two findings, not six.

The general rule still stands and is worth keeping: for a key the committed overlay does not carry, the "chart" column is a chart default and not necessarily what a deploy would render. It is just narrower in practice than I made it sound, and the narrowing is now recorded beside the rule in Doc/Architecture/ChartDriftSemantics (#3214) rather than only here.

[rbuergi]

Re-measured 2026-09-04: 81 divergences — and the class the issue ranked first is the one to leave alone

Dispatched this issue's own gate on main rather than reading any prior number
(run 33843264982, 06:10–06:12Z
today), then cross-checked every load-bearing claim against the live cluster read-only through
az aks command invoke (probes, ConfigMap key presence and value lengths, managedFields,
helm history, helm get manifest, helm get values, the Deployment inventory) plus an ACR tag
listing. No cluster object was mutated.

Current count by class

namespace	compared fields	divergences	COLLIDES	SHADOWS	CLUSTER-ONLY	CHART-ONLY	DIFFERS
memex	188	35	0	14	19	0	2
memex-cloud	204	46	0	24	17	0	5
total	392	81	0	38	36	0	7

🚨 These columns are not this issue's columns. COLLIDES and SHADOWS were carved out of what
used to be reported as CLUSTER-ONLY (#3168, 2026-09-03), so "81 → 36" is a re-partition, not a
reduction. Comparing the totals across the change is the only comparison that holds: 91 (08-26) →
76 (09-03) → 81 (09-04).

And the header of this issue is measured false. helm upgrade does not delete
CLUSTER-ONLY settings and does not overwrite a DIFFERS value — measured on helm v3.21.1 (the
binary az aks command invoke runs) and v4.2.4, with a positive control. Ranking this backlog by
"what a deploy would destroy" ranked it by a hazard that does not exist. The live-wrong class is
SHADOWS (an inline env: overrides envFrom, so the chart's value is dead while still rendering).

76 → 81 in a day, and nothing drifted. The five new findings are CLUSTER-ONLY ConfigMap Authentication__DevAdminUsers and Authentication__Google__ClientId on both namespaces plus
Authentication__LinkedIn__ClientId on memex. Cause measured, not guessed: the first-run-setup
change made those keys conditional (if) where they had been emitted unconditionally with
| default "", so the render stopped carrying a key the cluster still holds — at zero length.
The live ConfigMaps have not been written since 2026-08-30 (memex) / 2026-08-29 (memex-cloud)
(managedFields; helm revisions 28 and 21). A count that rises is not automatically drift.

---

The 7 DIFFERS — which side is authoritative

memex (2): the probes. The CHART is authoritative. Do not "fix" these.

DIFFERS livenessProbe   chart: no initialDelaySeconds   live: 60
DIFFERS readinessProbe  chart: no initialDelaySeconds   live: 20

This issue and three re-measurements read these as "live is authoritative, add them to the chart"
and called them a loaded gun. Measured today, that is backwards on both counts:

• The delay is unreachable. Both live pods carry a startupProbe (/health, periodSeconds: 5,
  failureThreshold: 60 — a 300 s window), and Kubernetes suspends liveness and readiness entirely
  until the startup probe succeeds. A delay measured from container start can never be what
  protects a slow boot; the startup probe already did.
• The control arm exists and is green. memex-cloud runs the byte-identical probe block with
  no initialDelaySeconds at all — verified live today — on the same image
  (memex-portal-ai:3.0.0-rc9.ci.7693), 4/4 ready. The "loaded gun" already fired there, harmlessly.
• There is no deploy hazard either, because a helm upgrade does not remove a live probe field
  (three-way merge; that exact shape was one of the four in the measurement).

Recommendation: close this thread. Adding initialDelaySeconds to the chart would enshrine dead
config and make the chart the more dangerous side of the comparison. The reasoning now lives beside
the probes in deploy/helm/templates/memex-portal/deployment.yaml, naming this issue, so it stops
being re-derived wrongly a fourth time.

memex-cloud (5): one item, and it is a deploy, not an edit

DIFFERS ConfigMap PreWarm__BatchBake      chart 'true'  vs live ''
DIFFERS ConfigMap PreWarm__BuildProtocol  chart 'true'  vs live ''
DIFFERS ConfigMap PreWarm__DynamicTypes   chart 'true'  vs live 'false'
DIFFERS ConfigMap PreWarm__GateReadiness  chart 'true'  vs live 'false'
DIFFERS startupProbe   chart {periodSeconds 10, failureThreshold 1080} vs live {periodSeconds 5, failureThreshold 60}

All five come from one committed intent: values.memexcloud.public.yaml states a gated cold bake
(PreWarm__{BatchBake,BuildProtocol,DynamicTypes,GateReadiness}: "true" at lines 90–93, and
probes.startup.failureThreshold: 1080 at line 192), and the namespace has never been deployed with
it — helm revision 21 is from 2026-08-30. This is deploy-drift's class surfacing here, not cluster
drift. The overlay is authoritative by intent, and the finding is sound (the overlay carries
these keys, so the "chart" column is not a chart default).

🚨 But the deploy as it stands buys the cost of the gate without the gate.
PreWarm__GateReadiness also exists as an inline env: on the Deployment, which SHADOWS the
ConfigMap (agreeing today at false). A helm upgrade would set the ConfigMap to true and
raise the startup budget to 1080 × 10 s — a pod-template field helm owns, so that half lands — while
the pod keeps reading false from the inline entry. progressDeadlineSeconds is derived as
periodSeconds × failureThreshold + 600, so a genuinely failed rollout would be declared failed
after 3 h 10 m instead of ~16 minutes, with no gate in exchange.

Recommendation: delete the inline PreWarm__GateReadiness entry in the SAME change as the deploy,
or revert the overlay to match live. Not two changes.

---

The CLUSTER-ONLY 36, grouped into stories

First, a distinction this issue did not have and that decides three of the four groups. CLUSTER-ONLY
splits in two, and the test is one command — helm get manifest <release> -n <ns>, the original
side of helm's merge:

sub-class	test	next helm upgrade
owned-but-retired	the key IS in the release manifest	deletes it — helm owned it
never-owned	the key is NOT in the manifest	leaves it — the migration backlog

Measured over all 36: 13 owned-but-retired (memex 7, memex-cloud 6) and 23 never-owned
(memex 12, memex-cloud 11). So the issue's original "a deploy deletes it" is right for 13 of 36 —
for the opposite reason it gave, and only because a chart retirement is landing.

A · Chart-retired, helm-owned — 13 — deletion is a no-op, but for a reason nobody has stated

FrameworkBroadcast__Subscribers__0..3 (both namespaces, 8) + the five Authentication__* keys above.

🚨 All eight FrameworkBroadcast slots are "" on BOTH namespaces. Verified by three
independent methods — live ConfigMap jsonpath (len=0 ×8), live ConfigMap YAML
(FrameworkBroadcast__Subscribers__0: ""), and the deployed release manifest — and helm get values
shows the deployed release sets no value for any of them. The manifest's own comment says as
much: "Empty on every deployment EXCEPT the control instance … the broadcast resolved to an EMPTY
set."

That contradicts the reading posted on Systemorph/Memex#152 this morning — "they are functional
today", "next helm upgrade on memex-cloud ⇒ subscriber list goes from 4 to 0". It is already
0, on both portals. The pending deletion removes four empty strings. The real item is that the
release wave has zero subscribers today, which is a live condition to fix on the record
(Memex#152 / #173), not a deploy hazard to schedule around. The five Authentication__* keys are
likewise zero-length; deleting them is a no-op too.

Recommendation: let the next upgrade delete all 13 — after re-reading each value, because "empty"
is a property of these keys on this day, not of the sub-class.

B · The registry credential — 2 — 🚨 dangerous to delete, and tracked as #3201

• memex: inline PluginCatalog__RegistryToken SHADOWS secret/memex-portal-secrets. The checker
  reads only KEY NAMES from secret sources (deliberately — a drift checker must not copy the
  credentials it audits onto a CI runner), so it reports no agree/disagree verdict. memex authenticates to the plugin registry with a plaintext inline token that SHADOWS the Key Vault one — and the two differ #3201
  established the two differ: the portal authenticates with the plaintext inline copy while the
  Key Vault credential goes unused. (I did not read either value.)
• memex-cloud: it is CLUSTER-ONLY, i.e. the inline entry is the only copy — there is no
  secret key of that name to fall back to.

Deleting the inline entry is wrong on both. On memex it switches the portal onto a different
token; on memex-cloud it removes the catalog credential outright. The sequence is #3201's:
establish the live token → vault it on both → delete both inline entries → rotate. One change.

C · Live-edited settings the chart can now express — 22 — the Memex#148 migration

Never-owned, so a deploy leaves them; they want a Hosting/Deployment record entry + a chart key,
then the inline shadow deleted where one exists.

both ns   Features__Ai__Providers__{AzureOpenAI,OpenAI,OpenAICompatible,OpenRouter}   (8)
          Portal__ReactAppUrl                                                        (2)
memex     LogWatch__{DefaultRepository,Routes__{0,1}__{Prefix,Repository}}           (5)
          inline env Features__Ai__Clis__{ClaudeCode,Copilot}                        (2)
cloud     inline env Commerce__BaseUrl · PluginCatalog__DefaultGrants__0
                     Speech__{Enabled,Endpoint,Language}                             (5)

This group already has an owner and an open PR: Systemorph/Memex#175, "every cluster-only
setting on memex/memex-cloud becomes committed config (apply NOT included)". It was blocked on
#3207 (closed today) and on Config key coverage, which is green on Memex main as of 06:08Z
today. So the stated blocker is gone.

D · One of them points at a workload that has been down for weeks

Portal__ReactAppUrl=/next is live and effective on both portals (ConfigMap only, no inline
shadow — verified). Meanwhile portal-next-deployment is 0/1 ready on both namespaces, pinned to
memex-portal-next:3.0.0-ci.2054 — a tag ACR does not have (az acr repository show → "the
specified tag does not exist"; newest published is 3.0.0-next.31). Systemorph/Memex#172.
Honest limit: both /next URLs answer HTTP 200, and I did not determine which workload answers
them, so the evidence stops at "the config names a route whose dedicated workload is not running".

---

What these 81 do not cover

The checker compares memex-portal-config, memex-portal-deployment, the PodDisruptionBudget and
the ScaledObject. Everything else in these namespaces is outside its scope — and two things there are
drifted right now:

• portal-next-deployment (both namespaces) — kubectl-client-side-apply 2026-07-05, then
  kubectl set 2026-08-07; no meta.helm.sh/release-*, no app.kubernetes.io/managed-by: Helm (the portal Deployment beside it has both). Pure hand-applied cluster-only drift, and Chart
  Drift has never been able to see it.
• k8s-dashboard-deployment (both), memex-website and whisper-swiss-german (memex-cloud).

Also worth recording: deployments/aks/pearl/ exists in the deploy repo but there is no pearl
namespace on the cluster and it is correctly absent from envs.json — so it is out of scope by
intent, not by oversight.

---

The Authentication__Microsoft__ClientId warning (this issue's last open question)

Fixed, and now positively verified as compared and equal — not "not compared". Both live
ConfigMaps carry a 36-character GUID; each overlay states its own real client id
(memex eed68df1-…, memex-cloud 77bc55bb-…), layered over the chart default, so the
35-character CHANGE_ME_aad_application_client_id placeholder reaches neither namespace and nothing
renders it. The claim could have failed: the same run emitted four DIFFERS ConfigMap findings, so
the comparator was live on that object.

The workflow header's stronger claim — "NO run has EVER reported an Authentication__ finding
(2026-08-26, 09-02, 09-03)" — went false today, five findings' worth. A "no run has ever" clause
in a gate's header is a standing claim about future runs and dates the moment the chart changes.
Corrected, with the mechanism rather than a tally, in the PR below.

---

Committed work product

Systemorph/MeshWeaver #3275 (documentation + gate-header only, nothing production
runs): the 09-04 re-measurement and the owned/never-owned split in
Doc/Architecture/ChartDriftSemantics, the corrected Authentication__ paragraph in
chart-drift.yml, and the CLUSTER-ONLY legend in check-chart-drift.sh qualified so it stops
claiming a retired key survives.

Nothing was applied to the cluster or to any deploy overlay. The three actionable groups all have
open PRs on Systemorph/Memex (#173 draft, #174, #175) and re-doing any part of them here would
conflict.

---

Owner decisions — each answerable yes/no

1. Probes: confirm the chart is authoritative and this thread is closed — we do not add
   initialDelaySeconds to the chart, now or later. Yes/no?
2. memex-cloud gated bake: deploy the committed PreWarm__* + probes.startup.failureThreshold: 1080 overlay, with the inline PreWarm__GateReadiness entry deleted in the same change?
   Yes/no?
3. If no to (2): revert those five overlay values to match live, so the committed file stops
   describing a portal nobody runs? Yes/no?
4. The 13 chart-retired keys: accept that the next helm upgrade deletes them (all measured
   empty today, so a no-op), rather than pre-deleting them? Yes/no?
5. Release wave: do you accept that both portals have had zero broadcast subscribers all
   along (not "4 until the next deploy"), making Memex#152's record the live fix rather than a
   pre-deploy chore? Yes/no?
6. PluginCatalog__RegistryToken (memex authenticates to the plugin registry with a plaintext inline token that SHADOWS the Key Vault one — and the two differ #3201): authorise the sequence — establish the live token,
   vault it on both namespaces, delete both inline entries, rotate — as one change, ahead of any
   other Deployment-spec edit? Yes/no?
7. Systemorph/Memex#175: merge it now that memex cannot self-update: the portal image is ahead of its schema (db_version 54 < 55) and only helm upgrade can run the migration Job #3207 is closed and Config key coverage is green on
   Memex main? Yes/no?
8. Gate gap A: should Chart Drift fail on the reverse direction — a key that is live and
   helm-owned but no longer rendered — so a chart retirement's deletion list is reviewed before
   the deploy? (Systemorph/Memex#152 asked for exactly this; the 13 above are what it would have
   named.) Yes/no?
9. Gate gap B: should the checker's scope widen from memex-portal-deployment to every
   Deployment in the namespace? It would have named portal-next-deployment in July. Yes/no?
10. This issue: re-scope it to "Chart Drift is green" with the per-group owners above, or close it
    in favour of Memex#148 + memex authenticates to the plugin registry with a plaintext inline token that SHADOWS the Key Vault one — and the two differ #3201 + Memex#172 and keep only the gate-gap items here? Yes = re-scope.

(Read-only throughout: the workflow's own log, az aks command invoke reads projected to key names,
lengths and manifest membership, az acr repository show-tags, and two HTTP HEAD-equivalent probes.
No secret value was read, printed or stored.)

[rbuergi]

Closing this — the last thing that lived only here now has code (#3320)

The 09-04 re-measurement put ten yes/no questions to the maintainer. That was too many, and on re-reading the thread it is the wrong shape: nine of the ten belong to other issues that already own them, and answering them here would just move the decision, not make it. So rather than add an eleventh re-measurement, this comment settles what is left in this repo, lands it, and closes.

What this issue asked for, against what is true now

The title is "91 divergences … 8 DIFFERS a deploy would overwrite, and nobody owns the backlog". All three halves are now measured false or resolved:

the claim	measured
a deploy would overwrite / delete these	False. helm upgrade does not delete CLUSTER-ONLY and does not overwrite DIFFERS — measured on helm v3.21.1 (the binary az aks command invoke runs) and v4.2.4, with a positive control. Recorded in Doc/Architecture/ChartDriftSemantics. The one shape where a deploy does delete is a chart retirement of a key helm still owns — which is what #3320 now detects.
8 DIFFERS	7, and none is a deploy hazard: 2 on memex are the probe initialDelaySeconds, ruled inert (a startupProbe suspends liveness and readiness entirely, and memex-cloud runs the identical block with none and is healthy); 5 on memex-cloud are one committed-but-undeployed gated-bake intent, i.e. deploy-drift's class.
nobody owns the backlog	Every group has a named owner and an open PR: Systemorph/Memex#175 (the cluster-only migration, unblocked — #3207 closed and Config key coverage green), #3201 (the registry credential, as a rotation), Systemorph/Memex#172 (portal-next-deployment), Systemorph/Memex#152/#173 (the release-wave record).

The count has never been the finding — 91 → 76 → 81 moved mostly because #3168 re-partitioned the classes and because a chart change made five Authentication__* keys conditional. A count that rises is not automatically drift.

The one thing that had no owner anywhere, and now has code

Chart Drift computes D vs L (chart-as-rendered vs live). hosting-audit computes M vs L (release manifest vs live). Nobody computed D vs M — and that is precisely where the real deletion hazard lives: a key that is live and helm-owned but no longer rendered is a pending deletion the next helm upgrade will perform. Measured by hand on 09-04: 13 of 36 CLUSTER-ONLY findings were owned-but-retired, sorted in among 23 inert ones, with nothing re-deriving the split on the next run. Systemorph/Memex#152 asked for exactly this check.

#3320 makes the gate compute it, with the properties this issue has spent a month arguing for:

• the release manifest is a fail-closed input — an unreadable or empty manifest is a hard failure naming helm get manifest, never a silent "no pending deletions";

• findings distinguish a pending deletion of a non-empty value from a no-op deletion of a zero-length one — which is the whole point, since all 13 found on 09-04 were empty and reporting them on key names alone would have raised a false alarm;

• and it is mutation-proved twice, not merely asserted. Forcing manifest membership to always-false turned the suite red with, verbatim:

  ::error::the finding matching 'CLUSTER-ONLY *ConfigMap LogWatch__DefaultRepository' does not say 'PENDING DELETION' — helm owns this key and the chart no longer renders it, so the next upgrade removes it.

  and disabling the emptiness branch turned it red on exactly the two zero-length cases. Both restored; suite green with 49 ok lines. There is a negative control too — a key that is live and in the manifest and still rendered must not be reported at all — without which the check could pass by flagging everything.

That closes the loop this issue opened: "a detector that is permanently red stops being read." The remaining red is now a worklist with owners, and the class that actually justifies urgency is computed rather than hand-derived once a week.

Answering the parts of the 10 questions that are mine to answer

• Probes (Q1): the chart is authoritative; do not add initialDelaySeconds. That reasoning now sits beside the probes so it stops being re-derived — this is the fifth time.
• Gate gap A (Q8): done, in fix(chart-drift): compute the pending-deletion half of CLUSTER-ONLY #3320.
• Gate gap B (Q9): deliberately not done, and I would not do it as asked. k8s-dashboard-deployment is a free win (the chart renders it unconditionally). But portal-next-deployment is an unmanaged whole object — no helm annotations, kubectl set on 2026-08-07, 0/1 ready on an image tag ACR does not have — and the comparator has no vocabulary for that shape; it classifies fields within named objects. hosting-audit's $unmanagedObjects already has the right primitive. Widening the drift comparator would also rewrite every label its eleven anchored test assertions match, which is a real regression risk for no new coverage. Recommendation: extend hosting-audit, not this comparator — tracked on Systemorph/Memex#172.
• Everything else (Q2–Q7, Q10) is a deploy or a policy call on Systemorph/Memex and belongs on those issues, not here.

Closing

Not because the drift is zero — it is 81, and every entry still wants doing — but because this issue's own premise is resolved and its worklist has moved. Keeping it open re-derives the same measurement every few days; that has now happened five times, and the fifth one is committed to Doc/Architecture/ChartDriftSemantics so the sixth reader inherits it instead.

I will close it once #3320 merges, not before — closing on an open PR would be exactly the "authored, therefore done" error this thread caught on #2194's alert. The daily red is the tracker afterwards; reopen if Chart Drift starts reporting a class nobody owns.

_{Read-only throughout: the workflow's own log, the committed overlays, and az aks command invoke reads projected to key names and lengths. No cluster object was mutated and no secret value was read.}