chore(chart): point spawned ingestor at the 0.8 line (D16 write path) — HOLD until v0.8.0 image

[LukasWodka]

DRAFT — image exists now; held on validation + release go. ghcr.io/tracebloc/ingestor:0.8 is published and signed (v0.8.0 2026-07-30; the 0.8 line is at v0.8.2 since the 2026-08-04 promotion, data-ingestors#433). The remaining hold is functional, not artifact existence — see checklist.

What (reworked 2026-08-04 onto the #494 channelTags structure)

• images.ingestor.channelTags.prod 0.7 → 0.8 — the ds_ per-ingestion write path (data-ingestors#408) ships in the 0.8 ingestor line; a prod edge left on 0.7 spawns a pre-D16 ingestor and silently falls back to legacy shared tables even with perIngestionTables on (found on staging by Divya). dev/stg are untouched — they track the :dev/:stg internal channels (backend#1360).
• images.ingestor.prodDigest → sha256:05e12494… — the v0.8.2 multi-arch index (== the current :0.8 float; == the v0.8.2 GitHub Release digest). Verified against ghcr.io 2026-08-04: OCI image index, linux/amd64 + linux/arm64. This is the actual spawn target on prod edges (prodPin: true).
• _helpers.tpl last-resort literal 0.7 → 0.8 (the --reuse-values fallback for releases predating channelTags) + comment.
• helm-unittest digest/tag assertions updated — 325/325 green locally; helm lint failure is the pre-existing clientId/clientPassword schema minLength (identical on develop).
• Chart.yaml 1.9.12 → 1.9.13 — the chart only publishes on a version change.

Why the write path is still inert after this merges

perIngestionTables stays default false (client#472 knob → PER_INGESTION_TABLES on the spawned ingestor). This PR only makes the prod fleet capable of D16; the knob flip stays a per-fleet operational decision.

Still TODO before un-drafting (checklist)

• :0.8 image published + signed (v0.8.0, 2026-07-30; line now v0.8.2)
• prodDigest on the 0.8-line multi-arch index (v0.8.2, verified 2026-08-04)
• Rebased onto develop post-feat(ingestor): resolve the spawned tag per environment (:dev / :stg) #494 (channelTags) / post-feat(chart): provision per-experiment DB credentials (RFC-0003 D10, backend#1181) #503; chart 1.9.13
• D16 E2E validation on a dev edge using the v0.8.2 release digest as the ingestion image (digest override, per backend#1360: :dev/:stg are different builds than the release artifact): ingest → backend row carries physical_table → training reads ds_<hex> → tb data delete drops table + salt + journal
• Release-owner go: un-draft, review, merge → rides the next chart Release (prod edges start spawning the 0.8 ingestor on auto-upgrade; write path still knob-gated)

Epic: tracebloc/backend#1151 (critical path #3) · RFC-CLI-0003 D16 · ingestor image channel: backend#1360

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes the default ingestor image prod edges spawn after chart auto-upgrade; a mis-pinned digest or tag could break ingestion pulls or leave edges on legacy table behavior until perIngestionTables is enabled, but scope is chart defaults with CI unittest and documented verification.

Overview
Bumps the Helm chart to 1.9.13 and retargets the spawned ingestor on prod edges from the 0.7 release line to 0.8, which carries the D16 ds_<hex> per-ingestion write path (data-ingestors#408).

images.ingestor.channelTags.prod moves from 0.7 to 0.8, and prodDigest is updated to the v0.8.2 multi-arch OCI index (sha256:05e12494…). Prod jobs-manager still spawns ingestion Jobs via INGESTOR_IMAGE_DIGEST when prodPin is true; dev/stg keep floating on :dev / :stg channels.

The tracebloc.ingestorTag helper’s --reuse-values fallback literal changes from 0.7 to 0.8 so older releases without channelTags do not spawn a pre-D16 ingestor. Helm unittest expectations for tag and digest are aligned with the new defaults.

perIngestionTables remains default false in this PR—only the ingestor image capability changes; enabling the write path is still a separate fleet knob flip.

^{Reviewed by Cursor Bugbot for commit 674fbb7. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 37 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#222 — fix(sklearn): unwrap Pipeline to terminal estimator before merge dispatch (Prod: Set kubeconfig context namespace after Helm install for tracebloc  #209) · author: @divyasinghds · no reviewer assigned
• averaging-service#225 — fix(averaging): fail fast + dead-letter on permission-denied weight write (release: promote develop → main (installer hardening — OOM gate, apt/needrestart fixes, drift CI) #221) · author: @divyasinghds · no reviewer assigned
• averaging-service#245 — chore(ci): record gitleaks history baseline (triaged false positives) · author: @LukasWodka · no reviewer assigned
• averaging-service#246 — feat(chart): ServiceAccount + IRSA + blobStore env for the S3 backend (#1162) · author: @saadqbal · reviewer: @aptracebloc
• averaging-service#247 — fix(averaging): classify S3 access-denied weight writes as permanent (fix(mysql): probe over TCP so a moved socket can't kill a healthy DB #225 follow-up) · author: @divyasinghds · no reviewer assigned
• backend#1201 — docs(rfc): Data spaces & federated dataset composition (draft) · author: @LukasWodka · reviewer: @saadqbal
• backend#1217 — docs(rfc): Dataset integrity & score reproducibility (draft) · author: @LukasWodka · reviewer: @saadqbal
• backend#1261 — RFC-BACKEND-0008: board & process reset (decision RFC — for markup) · author: @LukasWodka · reviewer: @saadqbal
• backend#1284 — docs(deploy): backend deploy runbook skeleton (#1263) · author: @LukasWodka · no reviewer assigned
• backend#1293 — fix(heartbeat): tolerate unknown experiment_key in training_info (#1287) · author: @divyasinghds · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[LukasWodka]

Rebuilt against develop — the original change had become wrong, not just conflicting. Still a draft.

Why it needed a rework, not a rebase. #494 (backend#1360) landed the per-environment channel scheme while this sat on HOLD. images.ingestor.tag is now an explicit force-override that is empty by default; the effective tag comes from channelTags[CLIENT_ENV]. So the original tag: "0.7" → "0.8" would have pinned 0.8 on every edge and silently defeated the new :dev / :stg routing.

Also: develop had already published chart 1.9.9 via #494, so the original bump to 1.9.9 would have collided.

What this now does

Change	Why
channelTags.prod 0.7 → 0.8	dev/stg already track :dev/:stg; prod was the only entry still on a pre-D16 line
prodDigest → sha256:d4af3342…c6e614	the 0.8 multi-arch index, via resolve-ingestor-digest.sh --write (never hand-typed) and cross-checked against ghcr's docker-content-digest
_helpers.tpl literal 0.7 → 0.8	only fires for a release predating these keys under --reuse-values — but there it would spawn a pre-D16 ingestor
Chart.yaml 1.9.9 → 1.9.10 (version and appVersion)	a Helm repo publishes only on a version change

The red X is fully explained, and the HOLD is satisfied.

• data-ingestors#419 merged 2026-07-30 10:52Z; v0.8.0 released 11:45Z.
• The ingestor-multiarch job ran at 10:55Z — 50 minutes before the image existed. That's why it reported Platforms: empty.

All four refs the gate now inspects verify multi-arch (linux/amd64 + linux/arm64) against ghcr.io today:

dev     PASS linux/amd64 linux/arm64
stg     PASS linux/amd64 linux/arm64
0.8     PASS linux/amd64 linux/arm64
sha256:d4af33420520d0fb01df3af4cf5259f664c89e95c7dff0681e8aa335a0c6e614  PASS linux/amd64 linux/arm64

Verification: helm unittest 320/320 across 27 suites · helm lint --strict clean on all four platform files (aks/bm/eks/oc) · gen-manifest.sh --check and check-drift.sh clean.

Left as a draft on purpose — I have not flipped it ready. Two things are your call, @LukasWodka:

1. Moving prodDigest reaches every prod edge on the next fleet auto-upgrade (chart default, --reset-then-reuse-values). That's the intended D16 rollout, but it is a prod-facing change and wants your go-ahead rather than mine.
2. Whether prod should move to 0.8 now or ride a later chart release.

[LukasWodka]

⛔ Hold — do not mark ready or merge until Divya's D16 validation passes

This is the prod deployment of the ingestor 0.8 line, not a config tidy-up. The diff moves:

- prodDigest: "sha256:9098b3c9…"   # 0.7 lineage
+ prodDigest: "sha256:d4af3342…"   # v0.8.0, verified against ghcr.io 2026-07-31

prodDigest is a chart default, so it reaches every prod edge on the next fleet auto-upgrade (backend#1245) — no per-edge action required. Divya is currently validating exactly this image (:0.8.0) on a staging edge under D16. Shipping it to prod before that validation completes would invert the entire point of the dev/staging channel work in backend#1360: test first, then promote.

Gate: Divya's D16 sign-off. Then un-draft, review, merge, and let the chart release carry it.

Also worth recording why this PR changed shape: the original one-liner had become wrong, not merely conflicting. #494 repurposed images.ingestor.tag into an empty force-override, so the old tag: "0.8" would have defeated the new :dev/:stg routing for every non-prod edge. The correct change is channelTags.prod → 0.8 plus a refreshed prodDigest, which is what this now does. Chart 1.9.9 → 1.9.10 because #494 already took 1.9.9.

Current state: 17 checks SUCCESS (including the previously-red gate — that was purely timing: it ran at 10:55Z, v0.8.0 published at 11:45Z), helm unittest 320/320, helm lint --strict clean on all four platform files, zero unresolved threads. Ready in every respect except the validation gate.

[LukasWodka]

@divyasinghds — status update, and a correction to something I nearly told you.

This PR is ready except for your D16 sign-off

Everything else has landed. Chart 1.9.9 is on main and published, so the per-environment ingestor channels from backend#1360 are live. This PR is the last step: it points prodDigest at the 0.8 line.

Which image to validate — the digest, not :stg

I was about to suggest you could now just use the :stg channel instead of a digest override. That would have been wrong, so flagging it rather than letting it mislead:

:stg     -> sha256:ff92490567f368f7…   built from the staging branch, 07-31
:0.8.0   -> sha256:d4af33420520d0fb…   the released v0.8.0 tag build   <- what THIS PR pins

Both are ingestor 0.8.0 by version string, but they are different builds. Since #490 pins the :0.8.0 digest, that digest is what every prod edge will run — so the validation that gates this PR should be against sha256:d4af33420520d0fb01df3af4cf5259f664c89e95c7dff0681e8aa335a0c6e614, exactly as before. Validating :stg would sign off on an artifact prod won't run.

Pin it on your staging edge with:

images:
  ingestor:
    digest: "sha256:d4af33420520d0fb01df3af4cf5259f664c89e95c7dff0681e8aa335a0c6e614"

(digest wins over any tag or channel, so it overrides the new routing cleanly.)

What the channel work changes — for next time, not this one

From the next ingestor change onward you won't need a digest override at all: a staging edge on chart ≥ 1.9.9 automatically spawns ingestion Jobs from :stg, which tracks the data-ingestors staging branch. That's the point of backend#1360 — the reason this validation needed a hand-pinned digest was that the ingestor image only existed as a byproduct of a production release. That's fixed; it just doesn't retroactively help this particular sign-off.

When you're done

Comment here or advance the card and I'll un-draft, get it reviewed and merged. No rush from my side — nothing else is blocked on it, and prodDigest reaching every prod edge on the next fleet auto-upgrade is exactly the kind of change worth gating on a real run.

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 674fbb7. Configure here.}

[LukasWodka]

D16 E2E validation runbook (the un-draft gate) — dev or staging edge

Validates the exact artifact prod will pin: ghcr.io/tracebloc/ingestor@sha256:05e124945a2ef61868661a9137abb5e75dac96a090cb41a7b4e8be520c4b5873 (the v0.8.2 release digest). Deliberately not :dev/:stg — those are different builds of the same code (backend#1360), and prod runs prodDigest.

Substrate note: staging is the cleaner substrate — on dev, tb data list may 401 on /dataset/admin/ (backend#1180, account-ops, unrelated to this change). If dev misbehaves that way, switch edge, don't debug here.

Preconditions (all live on dev/staging today)

• Edge chart ≥ 1.9.9 (per-edge images.ingestor.digest pin + channel resolution)
• Engine read-path (resolve_physical_table, tracebloc-engine#543) is on the :dev/:stg engine channels — fine here. Prod enablement separately waits on a tracebloc-engine master promotion (none since 2026-07-27).
• Dataset: tabular or time-series (file-bearing categories are refused under the flag until client-runtime#203 Track B is enabled).

1 · Flip the edge (survives auto-upgrades — these become user-supplied values)

helm -n <ns> get values <release> > values-backup-$(date +%F).yaml
helm -n <ns> upgrade <release> <chart-ref> --reuse-values \
  --set images.ingestor.digest="sha256:05e124945a2ef61868661a9137abb5e75dac96a090cb41a7b4e8be520c4b5873" \
  --set perIngestionTables=true

(<chart-ref> = the published repo at the edge's installed version, e.g. tracebloc/client --version <installed>.) jobs-manager restarts with PER_INGESTION_TABLES=1; the per-edge digest beats every tag/channel for spawned ingestion Jobs.

2 · Ingest

Normal flow: tb data ingest … with a small tabular set.

3 · Verify the write path (on the edge)

# a) the ingestion Job runs the pinned digest and carries the flag
kubectl -n <ns> get jobs --sort-by=.metadata.creationTimestamp \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].image}{"\n"}{end}' | tail -3
kubectl -n <ns> get job <ingest-job> -o yaml | grep -A1 "PER_INGESTION_TABLES"

-- b) one fresh immutable table + bookkeeping (mysql service: mysql-client)
SHOW TABLES LIKE 'ds\_%';                                        -- new ds_<32 hex>
SELECT table_name, registered FROM tracebloc_ingest_runs
  ORDER BY 1 DESC LIMIT 3;                                       -- its row: registered = 1
SELECT table_name FROM tracebloc_ingest_meta;                    -- its salt row exists

Expected: exactly one new ds_<32 hex> table for this run (re-ingesting the same name+schema must create a second table, never append — that's D16/D17; worth one repeat ingest to see it).

4 · Backend row + read path

• tb data list shows the new dataset (label, not the physical name).
• Start a short training on it; the training pod log resolves the physical handle (ds_<hex>) and the run completes → proves the backend data_info.tables → engine resolve_physical_table chain end-to-end.

5 · Delete means gone

tb data delete <dataset>

SHOW TABLES LIKE 'ds\_<hex>%';                                   -- gone
SELECT * FROM tracebloc_ingest_runs WHERE table_name = 'ds_<hex>'; -- 0 rows (journal)
SELECT * FROM tracebloc_ingest_meta WHERE table_name = 'ds_<hex>'; -- 0 rows (salt)

6 · Record → un-draft

Paste the evidence here (job image line, SHOW TABLES before/after, journal/salt rows, training log line), tick the checklist, un-draft this PR.

Not satisfied by this run: backend#1360's close trigger — that wants the next ingestor change validated with no digest override; this run overrides on purpose to hit the release artifact.

Claude prepared this; the flip + run need edge kube access (Lukas).

[LukasWodka]

Runbook precondition correction: the engine-promotion caveat is void — master promoted 2026-08-04 08:08 (train #584) and carries resolve_physical_table (core/utils/database.py:396); :prod engine images rebuilt green the same morning. Prod D16 reads gate on this PR + the knob only.