release-train: staging -> main - #571
Merged
Merged
Conversation
The guard classified only `client/templates/**` and `client/values.yaml`, but release-helm-chart.yaml packages BOTH `./client` and `./ingestor` (lines 131-132) into one shared index.yaml (line 193). So an unbumped `ingestor/**` edit merged green — the exact dark ship the guard exists to stop (PR #472). Raised by Bugbot on the develop->staging promotion PR #519. Measured on develop: 6 of 9 commits touching ingestor chart content never bumped ingestor/Chart.yaml, whose version has been 0.2.0 since 2026-05-20. Because `helm package ./ingestor` passes no --version override, ingestor's published version IS that file, so those edits did not go nowhere — they overwrote an already-published version. ingestor-0.2.0.tgz was replaced 5x between 2026-05-20 and 2026-07-29 and `helm repo index --merge` refreshed the digest in place, so two installs of "0.2.0" months apart are not the same chart. Helm also caches by version, so an existing client may never pick the change up at all. - derive the guarded chart list from the release workflow's `helm package` lines instead of hardcoding it, so a third chart is guarded on day one - require a bump of the chart's OWN Chart.yaml (bumping client no longer satisfies an ingestor change) - add client/values.schema.json (packaged, and Helm validates user values against it at install time), plus charts/** and crds/** pre-emptively - report every unbumped chart in one run rather than one per push - fail closed when the chart list cannot be read, or when a packaged chart has no Chart.yaml Move the logic into scripts/chart-version-guard.sh so it can be tested: the new scripts/tests/chart-version-guard.bats (23 cases, real throwaway git repo, no stubbed git) runs under the required `Unit tests` check, which this workflow is not. 13 of the 23 fail against the previous inline logic; the 10 that pass on both are the client-side semantics, deliberately unchanged. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The caller's header described a "FR on dev" -> "Ready for staging" transition that does not exist. RFC-BACKEND-1405 D6 retired the dev-side review -- "On dev" is set automatically when the train merges to develop -- so /fr-pass applies at exactly one gate: FR on staging -> Ready for prod. The reusable workflow it calls has only ever implemented the staging hop and documents that correctly at the top of the file, so this was the comment drifting away from the code, not a behaviour change. Comment-only; no logic touched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Code review WIP limit of 30 was set for a human-paced queue. The column now sits around 50 with items turning over in days, so the nudge fires on nearly every PR and carries no signal. A limit that is always exceeded is noise, not a limit. Nothing gated on this - it never blocked a merge and no required check is affected. Refs: tracebloc/backend#1405
chore(ci): retire the WIP-limit nudge
chore(release): bump chart to 1.9.10 after the v1.9.9 release
… clamped (bash half of #417) (#445) * fix(installer): memory truth — machine RAM vs Docker's budget as two lines (#417, bash half) Rebuilt on top of #513, which landed the CLAMP half of this work while this PR sat open. #513 already gives every SHOWN figure `_pf_clamp_mem_gb` (physical − PF_OS_RESERVE_GB, floored at PF_MIN_MEM_GB) and hard-fails a sub-floor Docker VM in the post-Docker recheck. This PR's own `_pf_mem_targets` clamp helper is therefore dropped as redundant — it reuses #513's helper instead. What #513 did NOT fix, and this does: the flip-flop. `_pf_total_mem_kb` preferred the runtime view over the host, so the SAME machine reported "16 GB (host)" on a cold run and "6 GB (Docker VM)" on a warm one, purely on whether Docker happened to be running. Two of its tests asserted that behaviour — one was literally named "the Mac trap". - the `_pf_total_mem_kb` memory selector is deleted. Memory has two distinct truths and each caller now names the one it means: `_pf_host_mem_kb` for a hardware fact, `_pf_runtime_mem_kb` for the budget the pods actually get. (CPU keeps its fallback selector — there is no equivalent advice split.) - `_pf_memory` gates on the MACHINE and prints `Memory: N GB (machine)`. The Linux hard-fail gate, the 64 MiB grace and the MemAvailable check are all unchanged. On a machine below the floor the macOS branch no longer offers a Docker resize remedy — no Docker setting fixes too little physical RAM. - Docker's budget becomes its OWN second line via `_pf_runtime_mem_status`, shown only when a runtime is up AND its budget is meaningfully smaller than the machine (the VM case). Native Linux, where the daemon sees all host RAM, no longer repeats the same number twice. - `_pf_hw_summary_line` reports host RAM — it had the same flip-flop in miniature ("7 GB memory" on a 15 GB WSL2 box). - Linux budget hints drop the Docker Desktop dead end (Bugbot #445): a headless box has no Desktop UI, so the remedy names the VM/cgroup limit instead. - `PF_RUNTIME_MEM_WARNED` latches the budget warning so one run never warns twice about the identical condition. It is tested INSIDE the warn branch, never at the top of `_pf_recheck_runtime_mem`, so it can never gate #513's sub-floor hard-fail; a test pins that (latch set + sub-floor VM still exits non-zero). #513's reviewed recheck copy is left exactly as-is. Tests: preflight.bats 89/89 (11 new — the (machine) label, the two-line output, the Linux no-duplicate case, host-unreadable fallback, the clamped/floored advice, both OS hint shapes, the latch, and the latch-can't-gate-the-hard-fail guard; the two Mac-trap selector tests are replaced by a guard that the selector stays gone). Full suite 659/660 — the one failure is `validate_config: valid config passes`, pre-existing on clean develop and macOS-only (the /var symlink; fixed by #443). bash -n + shellcheck --severity=error + check-style clean; manifest regenerated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(installer): one threshold + one copy for the Docker budget; no dead-end advice on a tiny host (Bugbot #445 r2) Two findings from Bugbot on c50df6b, both reproduced and both real. 1) The recheck never actually used the "shared" copy, and graded differently. `_pf_runtime_mem_status` was documented as the single copy for preflight AND the post-Docker recheck, but the recheck still printed its own text — so the COLD install path (Docker starts mid-run, the common case) got no colima guidance on macOS and no hint at all on Linux. Worse, the two compared against different thresholds: the helper against the clamped target, the recheck against the raw PF_WARN_MEM_GB. Measured on an 8 GB host (clamped warn = 6) with a 6 GB budget, one run printed both: ✔ Docker's memory budget: 6 GB ⚠ Docker is running with 6 GB — recommended ≥ 6 GB (6 GB to train) Grading now lives only in the helper, and the recheck calls it. The helper takes MiB so it uses the same PF_VM_MEM_GRACE_MIB tolerance as the recheck — rounding to whole GB first misgraded a VM sized to exactly the documented floor (4900 MiB guest) as sub-floor. A `quiet_ok` flag keeps the recheck silent on a healthy budget, so no run prints the same ✔ twice. 2) A machine too small for the floor was still told to resize Docker. On a 4 GB Mac the budget line advised "colima start --memory 5" — more than the machine has, undercutting the honest "use a larger machine" stop the recheck owns. The helper now detects host − PF_OS_RESERVE_GB < PF_MIN_MEM_GB and points at the machine instead. This mirrors the same fix on the PowerShell side (#444), so both installers now agree on the same hardware. The sub-floor HARD-FAIL is untouched and still unconditional: the latch is tested inside the warn branch only, and a test pins that a set latch plus a sub-floor VM still exits non-zero. Tests: preflight.bats 94/94 (5 new — host-too-small gets no resize, a host that CAN reach the floor still does, the preflight-OK'd budget is never re-warned, the recheck is silent when healthy, and the cold path carries the colima guidance). Full suite 664/665 — the one failure is the pre-existing macOS-only `validate_config` case fixed by #443. shellcheck --severity=error and --warning both clean on preflight.sh; bash -n, check-style and check-drift clean; manifest regenerated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(review): three diverging-copy findings, all closed via one shared predicate Bugbot r3 on #445. All three were the SAME shape as the bug this PR exists to remove — two copies of one judgement disagreeing — surviving on paths the first pass missed. 1. Sub-floor remedy contradicted the hard-fail. On a warm run with a sub-floor budget, _pf_runtime_mem_status hinted "Give Docker <rec>" while _pf_recheck_runtime_mem hard-failed with "raise to <warn>" — the latch suppresses a duplicate warning, deliberately never the hard-fail, so both printed. The remedy now quotes the size that failure quotes; the between-floor-and-warn branch, where no hard-fail follows, still aims at the train figure. 2. Budget GB omitted the VM grace. rt_mib/1024 showed a VM configured at exactly the documented floor as one GB BELOW it — graded correctly by the grace-aware thresholds, displayed as a contradiction. Now (mib + grace)/1024, matching the PowerShell peer. 3. "Enough to run" contradicted "use a larger machine". _pf_memory compared host RAM straight against the Docker floor, ignoring the OS reserve, so a 5-6 GB Mac was graded enough-to-run on one line and told to use a larger machine two lines later. Both now read ONE predicate, _pf_host_too_small_for_floor, which fails safe on unknown input. Native Linux keeps its original wording: the daemon sees host RAM, so the reserve arithmetic does not apply. 696 bats pass (4 new: sub-floor remedy agreement, floor-sized VM display, the machine-line verdict, and the predicate incl. junk input). shellcheck clean; manifest regenerated. * fix(preflight): grace the WARN threshold too, and make the r3 tests enforce Two gaps in 718af66, which otherwise stands as-is — the shared _pf_host_too_small_for_floor predicate is the right shape and is kept. 1. The display became grace-aware but the WARN threshold did not (only the floor one was), so the same self-contradiction reopened one boundary up. Measured on 718af66 with a 32 GB host (warn_eff 8): rt_mib=7680 -> ⚠ budget: 8 GB — recommended ≥ 8 GB rt_mib=8000 -> ⚠ budget: 8 GB — recommended ≥ 8 GB rt_mib=8191 -> ⚠ budget: 8 GB — recommended ≥ 8 GB rt_mib=8192 -> ✔ budget: 8 GB A ~512 MiB band telling the operator to raise a budget to the size it already reports — and Docker Desktop's own defaults land in it. The warn threshold now carries the same grace, so shown == target implies the ✔ branch at BOTH boundaries rather than just the floor. 2. The four r3 tests were only partially enforcing. Under Bats 1.13 a failing bare `[[ ]]` that is not the LAST command in a test body does not fail the test, so `budget: 5 GB`, `!= enough to run`, `Give Docker <warn_eff>` and every line but the last of the predicate test were advisory — they would have passed against broken code. All 16 assertions in those tests now carry `|| return 1`. Verified by mutation, not by inspection: - reverting the warn-threshold grace (i.e. 718af66's shipped state) fails the new boundary test — so this is a real gap, not a hypothetical one; - reverting the display grace fails 2 tests; - neutering _pf_host_too_small_for_floor fails 3, including the r3 predicate test that only became capable of failing once hardened. Baseline and restored are clean in every case. Scope note: only the r3 tests are hardened here. 170 of ~698 tests in this suite share the un-hardened pattern; that sweep needs its own PR because hardening will surface previously-vacuous failures that each need triage (real installer bug vs stale assertion), and burying that in this PR would hide it. Gates: bats scripts/tests/*.bats -> plan 697, ok 697, not ok 0 (complete TAP run); shellcheck --severity=error over the CI file set -> rc=0; check-style clean; check-drift no drift; gen-manifest.sh --check current. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(review): make _pf_memory's thresholds match the grace on its own display Bugbot r5 (High), and correcting my own assumption: the grace on _pf_memory's shown GB came in from develop via the merge, not from my r4 edit. Either way the halves disagreed — the display added PF_VM_MEM_GRACE_MIB while the floor gate kept a 64 MiB tolerance, so a 5 GB VM reporting ~4900 MiB printed "Memory: 5 GB — below the 5 GB the client needs" and, on Linux, hard-failed on it. - floor_mib and warn_mib now use PF_VM_MEM_GRACE_MIB, the same tolerance the display uses and the same one _pf_runtime_mem_status already used for its floor and warn tests. All three now agree on the boundaries. - _pf_host_too_small_for_floor is now fed $(_pf_host_mem_gb) instead of $gb. A shared predicate only prevents divergence if both call sites pass the same input; _pf_memory was passing a grace-adjusted VM-or-host figure while the status path passed raw host GB. MemAvailable is deliberately left ungraced: it is a live measurement, not a configured size, so adjusting it would mask a real shortage. 713 bats pass (2 new: the floor-sized-VM message, and a guard that both call sites feed the predicate the same figure). shellcheck clean; manifest regenerated. * fix(review): render every memory GB through one converter Bugbot r6 found a FOURTH site: _pf_hw_summary_line computed its own memory GB, so the collapsed summary could print a different size from the memory line in the same preflight. Investigating it turned up something worse, and it corrects the record on r4/r5: my r4 fix for _pf_recheck_runtime_mem anchored on a two-line pattern that also existed in _pf_memory, and the replace took the FIRST match — so the fix landed in _pf_memory and the recheck never got it. The grace on _pf_memory's display, which I attributed on the PR to develop via the merge, was actually that misapplied edit. The r5 High finding was a direct consequence. Structural fix rather than a fifth patch: _pf_display_gb_from_mib is now the single definition, used by _pf_memory, _pf_runtime_mem_status, _pf_recheck_runtime_mem and _pf_hw_summary_line. rt_gb goes through it too — it feeds the 'is the VM meaningfully smaller than the machine' comparison, and grading one grace-adjusted side against a raw other side is precisely the mistake these six rounds keep rediscovering. Deliberately still raw: MemAvailable (a live measurement — inflating it would hide a real shortage), disk, and _pf_host_mem_gb (physical RAM needs no compensation, and it is the input _pf_host_too_small_for_floor grades). Every replacement in this commit asserted its anchor matched EXACTLY once and refused otherwise — the guard that would have caught r4's error. 715 bats pass (2 new: a source-level invariant that no site renders its own memory GB, and summary-vs-memory-line agreement). shellcheck clean; manifest regenerated. * fix(review): apply the too-small predicate on every OS in the recheck Bugbot r7 (High). Two divergences in one branch: the reserve arithmetic was inlined instead of calling _pf_host_too_small_for_floor, and the branch was gated OS != Linux while _pf_runtime_mem_status applies it everywhere. On a warm Linux install with a sub-floor cgroup/VM budget, preflight said 'use a larger machine' and this hard-fail then advised raising Docker to a size that machine cannot give. Now calls the shared predicate on every OS, with an OS-appropriate noun so the Mac wording is preserved. Verified both messages agree on a 6 GB Linux host with a 3 GB budget: both say 'use a larger machine'. Same class as r1-r6 on a new axis: not two values disagreeing but two OS GATES disagreeing about when one judgement applies. 717 bats pass (2 new: the Linux path, and an invariant that the reserve arithmetic exists in exactly one place). shellcheck clean; manifest regenerated. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…es progress (#532) Two Bugbot findings on the release-train promotion PR client#515. 1. verify-index leaked through its own guard (fail-open). The prerelease check was `printf '%s\n' "$idx" | grep -qF "${TAG#v}"` under `set -o pipefail`: grep -q closes the pipe on its FIRST match, printf takes SIGPIPE and exits 141, pipefail makes the pipeline 141, and the `if` reads a REAL LEAK as "invariants hold". Reproduced at 380KB of index. Same class as the chart-version guard's own SIGPIPE bug. The logic moves to scripts/index-invariants.sh (the chart-version-guard precedent) so it is testable under the required `Unit tests` check, which this release-only workflow is not. Nothing greps a pipe now: every check greps the fetched FILE and branches on grep's own three exit codes -- 0 found, 1 not found, >=2 could-not-check -- and could-not-check fails closed rather than reading as clean. 2. $script:JobInit did not set $ProgressPreference. A fresh job runspace resets it to 'Continue', so PS 5.1 Invoke-WebRequest inside Invoke-WithHeartbeat throttled unless every caller remembered to silence it. Set it in the init script so every runspace inherits it. Tests: new scripts/tests/index-invariants.bats (17) covers both invariants, every fail-closed path, literal-vs-regex tag matching and two "past the 64KB pipe buffer" cases that fail against the old shape. Three Pester cases pin the JobInit silence (runspace, through Invoke-WithHeartbeat, and a source gate that it lives in JobInit itself). manifest.sha256 regenerated for the install-k8s.ps1 edit. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…533) CLAUDE.md told every author to assign every PR to saadqbal unconditionally. That was true when he was the de-facto code owner for this repo; it is not true now, and it contradicts RFC-BACKEND-0008 D31 (assignee = whoever is doing the work, set by the author) and the org CLAUDE.md. A fixed assignee also re-creates the bystander effect the author-picks model was adopted to remove: if every PR is assigned to the same person, nobody owns any of them. Refs: tracebloc/backend#1405
…rst-launch prompt (#526) * feat(#430): macOS lifecycle — login autostart, no-admin remedy, first-launch prompt macOS had the weakest lifecycle of the three OSes: no autostart at all (a rebooted headless Mac stayed down until someone re-ran manually), the up-front admin gate failed managed/no-admin users with a generic sudo error, and Docker Desktop's privileged-helper dialog was never mentioned. - _install_macos_autostart: writes a per-user LaunchAgent (no admin needed) that starts the runtime at each login — `open -a Docker` on a GUI Mac, `colima start` on a headless one — with RunAtLoad. Combined with the k3d --restart unless-stopped policy, a rebooted Mac (GUI or headless) returns with ZERO human action. Best-effort; sets TB_MACOS_AUTOSTART=1 so the summary can honestly promise it. - _macos_require_admin (+ _macos_user_is_admin): fail FAST on a no-admin Mac with a named, IT-facing remedy (the macOS analog of Linux prepare-host) instead of preflight_sudo's generic "sudo authentication failed" after a wasted prompt. Admins/root pass through. - First-launch: name the privileged-helper prompt ("macOS asks for your admin password once") alongside the license note, so the auth dialog isn't a surprise. - summary _reboot_note: macOS now says "restarts automatically (login item configured)" when autostart is set; the no-autostart output is byte-identical (golden-safe). Tests: new scripts/tests/setup-macos-lifecycle.bats (admin detection, no-admin remedy, LaunchAgent GUI/headless/best-effort-failure, reboot-note both ways). Separate file from setup-macos.bats / setup-macos-arch.bats to avoid a file-add clash across parallel PRs. shellcheck/style/drift clean; summary + copy-catalog golden green; manifest regenerated. Closes #430 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#430): autostart truly best-effort + accurate no-admin remedy (Bugbot) Two Bugbot findings: 1. (Medium) _install_macos_autostart returns 1 on a mkdir/write failure and install_macos called it BARE under set -e — aborting the whole install after Docker + tools were already in, contradicting the best-effort contract. Call it with `|| true`. New test drives install_macos with a failing autostart and asserts it still exits 0. 2. (High) The no-admin remedy was inaccurate: re-running as the same non-admin account just hits _macos_require_admin again, and there is NO macOS prepare-host (run_prepare_host errors on Darwin). Rewrote it to name the remedies that actually unblock the install — grant THIS account admin rights (then re-run as yourself), or install from an account that already has admin. Dropped the misleading "install Docker + re-run" loop and the nonexistent prepare-host reference. Test updated accordingly. shellcheck/style clean; 10/10 setup-macos-lifecycle.bats; manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#430): headless autostart via LaunchDaemon + honor TRACEBLOC_NO_AUTOSTART (Bugbot) Two more Bugbot findings: 1. (High) A per-user LaunchAgent only loads inside a GUI/Aqua login session — which a HEADLESS Mac never has — so the headless branch wrote an agent that never runs at boot, making the "restarts automatically" promise false. Headless now installs a system LaunchDaemon (/Library/LaunchDaemons, root) that runs `colima start` at BOOT as the install user, with HOME + PATH set (a boot daemon has no user env). GUI keeps the LaunchAgent. Factored the shared plist skeleton into _emit_launch_plist. 2. (Medium) _install_macos_autostart ignored TRACEBLOC_NO_AUTOSTART, the opt-out that already gates ensure_cluster_autostart (Linux) and the Windows peer. It now short- circuits on that flag, so macOS no longer configures autostart or promises auto- restart when the operator opted out. Tests: headless test now asserts a LaunchDaemon (UserName + EnvironmentVariables + sudo launchctl, boot not login); new opt-out test; GUI + best-effort tests unchanged. 11/11 setup-macos-lifecycle.bats; shellcheck/style/drift clean; manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#430): per-user autostart log path + accurate reboot note (Bugbot) Two more Bugbot findings on the LaunchDaemon change: 1. (Medium) Both plists logged to a fixed /tmp/tracebloc-autostart.log. With the installer's umask 077 the first account creates it 0600, so a second account's job can't open it (EX_CONFIG → runtime never starts), and /tmp is symlink-plantable on a shared Mac. _emit_launch_plist now takes a per-user log path: the LaunchAgent logs to $HOME/Library/Logs and the LaunchDaemon to the install user's ~/Library/Logs. 2. (Low) _reboot_note said "login item configured" for every macOS autostart, but a headless install uses a system LaunchDaemon (/Library/LaunchDaemons), not a login item — so IT would look in the wrong place. Dropped the mechanism label: "After a reboot, tracebloc restarts automatically." (accurate for both agent and daemon). Tests: assert the plist log path is per-user (Library/Logs), not /tmp; summary + copy-catalog golden green (no-autostart line unchanged). shellcheck clean; manifest regen. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#430): headless daemon creates its log dir + resilient boot start (Bugbot) Two more Bugbot findings on the headless LaunchDaemon: 1. (High) The daemon's StandardOutPath is ~/Library/Logs/... but that dir was never created (the GUI path mkdir's its own). On a fresh headless account without ~/Library/Logs, launchd fails EX_CONFIG before colima runs, yet TB_MACOS_AUTOSTART=1 still promised recovery. Now mkdir -p "${_home}/Library/Logs" before writing the plist. 2. (High) The daemon ran a bare oneshot `colima start` at boot with no retry — the VZ+Rosetta stack commonly leaves stale VM state across a reboot, so the first start fails and the edge never comes back. Replaced with a resilient wrapper: /bin/bash -c 'until colima start; retry up to 3x, colima stop + sleep 15 between attempts' — force-stopping clears the stale state. Loop body has no </>/& so it stays valid inside the plist <string>. Tests: headless test now asserts the log dir is created, the /bin/bash resilient wrapper, and colima stop (retry). 11/11 setup-macos-lifecycle.bats; shellcheck/style clean; manifest regen. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#430): boot retry uses colima stop --force to clear stale VZ state (Bugbot) The headless LaunchDaemon retry ran a bare `colima stop` between failed starts, but the intent is a FORCE stop: without --force, orphaned VZ driver state isn't cleared (and a bare stop can hang), so all three attempts fail and a rebooted headless Mac stays down despite TB_MACOS_AUTOSTART=1. Use `colima stop --force`. Test asserts the flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#430): headless autostart only when colima is the runtime + resolve its real path (Bugbot) Headless autostart always wrote a colima LaunchDaemon and set TB_MACOS_AUTOSTART=1, but install_docker_desktop installs colima only when Docker was DOWN — if Docker was already up by other means colima may be absent, so the daemon was bogus and the auto-restart promise false. The `|| echo /usr/local/bin/colima` fallback also baked a path that's wrong on Apple Silicon (Homebrew there is /opt/homebrew/bin). Now resolve colima via `command -v` (its REAL path on either chip) and, if it isn't installed, skip autostart honestly (best-effort return 1; caller's `|| true`) so the summary won't promise recovery via a runtime that isn't there. New test: headless + colima absent -> skip, no daemon, flag unset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…lling
The template job installed kubeconform with:
curl -sSL .../releases/latest/download/kubeconform-linux-amd64.tar.gz \
| tar xz -C /usr/local/bin
Two independent problems. `releases/latest` is a mutable pointer, so the
binary this job executed changed whenever upstream cut a release — we
were not pinned to anything. And piping curl into tar extracts the bytes
as they arrive, so there was no moment at which a digest could have been
checked even if we had one; the archive was unpacked onto PATH first and
inspected never.
What could previously execute unreviewed: whatever those bytes happened
to be. A replaced release asset, a compromised upstream account, or a
MITM on the download would land an executable in /usr/local/bin and the
next step ran it. This job runs on the matrix for all four platforms, so
it happened four times per CI run.
What now cannot: the download is pinned to an explicit version, written
to a temp file, verified against a digest pinned in the workflow, and
only installed once it matches. Bytes that do not match the digest never
become an executable on PATH.
Follows the pattern already used in tracebloc/.github's actionlint.yml
(version + SHA-256 + `sha256sum -c`).
kubeconform 0.8.0, kubeconform-linux-amd64.tar.gz
sha256 9bc2bffbf71f261128533edaf912153948b7ff238f9a531ae6d34466ec287883
source: the release's own CHECKSUMS asset (not computed from a download)
Version pin is behaviour-preserving: `releases/latest` currently
redirects to v0.8.0.
The digest is asserted to be 64 hex characters before use. `sha256sum -c`
treats a malformed line as "no properly formatted checksum lines found",
and whether that exits non-zero depends on the coreutils build — so an
empty or truncated variable could otherwise verify nothing while the step
still went green. Every failure mode was tested on ubuntu:22.04 (GNU
coreutils 8.32): digest mismatch, empty digest, truncated digest,
uppercase digest, HTTP 404, and a real substitution (pinned v0.8.0 digest
against the v0.7.0 asset). All exit non-zero and leave no binary on PATH.
Refs tracebloc/backend#1426
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…tamination (#529) * fix(#459): isolate e2e path 2 from path 1's --reuse-values replay contamination `helm upgrade --reuse-values` records the previous release's COMPUTED values (chart defaults + overrides) as the new release's user-supplied values. Every later --reset-then-reuse-values (the fleet auto-upgrade) then replays those frozen defaults as if an operator set them, so an edge ever hand-upgraded with --reuse-values silently stops receiving chart-default updates — most importantly images.ingestor.prodDigest. e2e-auto-upgrade.sh modelled this on ONE release: path 1 (--reuse-values) contaminated the recorded values, then path 2 (the fleet auto-upgrade) ran on that contaminated release. Path 2's pin assertion passed only because the baseline pin and the working-tree pin coincided — the ERA NOTE tripwire: the first prodDigest bump would trip it. Resolution (issue decision 1 — isolate path 2 from path 1): between the paths, reset the release's recorded values to just the genuine install-time overrides (helm upgrade --reset-values --set clientId/clientPassword/storageClass), so paths 2-4 assert CLEAN-edge auto-upgrade behavior — the fleet's real contract on an edge no one hand-upgraded. Added an assertion using the issue's own contamination fingerprint (`helm get values` WITHOUT --all must not carry the chart-default prodDigest key), and rewrote the ERA NOTE to document the resolution. A contaminated REAL edge stays a separate fleet-audit concern; its remediation is exactly this reset. shellcheck + check-style clean; bash -n ok. (Runs in the label-gated e2e job — needs a real k3d cluster + the published chart, so it's not exercised by default PR CI.) Closes #459 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#459): reset to the PUBLISHED chart for isolation so path 2 stays a real test (Bugbot) The isolation reset targeted $CHART_DIR (the local working-tree chart), so it pre-applied the local chart's new defaults (working-tree prod pin, egress gateway) BEFORE path 2. Path 2 then became a same-version no-op whose live prodDigest / "new defaults flowed" assertions already held from the isolation step — a --reset-then-reuse-values → --reuse-values regression would slip through (computed values would still carry the local pin). Reset to the PUBLISHED chart ($PREV) instead, so path 2 is a genuine published→local upgrade that MUST pull the new defaults for its assertions to pass. shellcheck + check-style + bash -n clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bugbot, client#534. The verify-index job (added in #532) checked out with bare actions/checkout@v4 and no ref, while all three sibling jobs in this workflow use a SHA-pinned checkout plus ref: github.event.release.tag_name. On a release: published run github.ref intermittently arrives empty (actions/runner#2788), and checkout then falls back to the DEFAULT BRANCH. So the post-publish index backstop would verify the published index using whatever scripts/index-invariants.sh is on develop rather than the one that shipped -- or fail to find it. A backstop reading a different script than the release is worse than no backstop, because it still reports. Now identical to its siblings: same pinned SHA, same ref. Refs: tracebloc/backend#1426
…RFC D3/D4) (#528) * feat(#435): single-source cross-OS installer facts + CI parity gate (RFC D3/D4) The costliest drift class of the installer sweep was FACTS diverging between the three OS implementations — the #410 incident (k3d/helm pins bumped in bash #382 but not PowerShell #410) failed a real customer install. Copy already had the byte-exact catalog; behavior facts get the same treatment here. - scripts/spec/facts.env: the single source of truth for cross-OS facts. Tool version pins (K3D_VERSION / HELM_VERSION / K8S_VERSION) + the READY_TIMEOUT budget. - scripts/check-facts.sh: --write stamps the spec into every consumer (bash common.sh + summary.sh, PowerShell install-k8s.ps1); --check is the CI gate (mirrors gen-manifest's write/check split). Nothing is sourced at runtime — consumers carry literal values, so the single-file verified bootstrap (R8) is untouched. - CI: installer-tests.yaml runs `check-facts.sh --check`, failing the PR if any consumer drifted from the spec — so the #410 incident (a pin in one OS path but not the other) can no longer ship. install-k8s.ps1's lockstep comment updated to point at the spec. Tests: scripts/tests/check-facts.bats — the #410 incident reproduced as a red check (both directions), --write round-trips for versions + the timeout, fail-closed on a missing pattern, bad-mode rejection. Scope note: hosts are already single-sourced + drift-checked (check-drift Checks 1 & 5); memory floors (bash-only today) and the behavior-parity matrix are follow-ups on this same mechanism — a new fact is one row in facts.env + check-facts.sh. Closes #435 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#435): enforce K8S_VERSION in PowerShell too + shellcheck check-facts.sh (Bugbot) Two Bugbot findings: 1. (High) install-k8s.ps1 ALSO pins K8S_VERSION ($K8S_VERSION default, passed to k3d as --image rancher/k3s:$K8S_VERSION), but check-facts treated the pin as bash-only — so bumping the spec + --write updated only common.sh, Windows stayed stale, and --check stayed green: the exact #410 hole for this pin. Added install-k8s.ps1 as a K8S_VERSION consumer (extract + rewrite), updated the facts.env note. 2. (Low) scripts/check-facts.sh was wired for --check but not in the explicit shellcheck --severity=error file list (unlike gen-manifest.sh) — a regression in the facts gate wouldn't fail static CI. Added it to both shellcheck lines. Tests: K8S_VERSION drift in PowerShell -> RED; a K8S bump stamps BOTH consumers. 11/11. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
release-train: develop -> staging
…unner (RFC D5) (#540) * feat(#436): real Windows installer e2e on a self-hosted nested-virt runner (RFC D5) GitHub's windows-latest runners can't nest virtualization, so install-k8s.ps1 had only mocked Pester while the bash installer gets a 9-distro prereq matrix + real-k3d e2e on every push. This adds the missing Windows e2e leg (RFC-CLIENT-0003 D5), on a self-hosted runner that DOES support nested virt. - .github/workflows/windows-e2e.yaml: schedule-only (nightly) + workflow_dispatch, NOT per-PR; runs-on [self-hosted, windows, nested-virt]; concurrency-guarded; 45-min cap; uploads the install log as an artifact on failure; tears the cluster down every run. - scripts/tests/e2e-windows.ps1: the credential-free driver, mirroring e2e-journey.sh — dot-sources install-k8s.ps1 with TB_PESTER=1 (main() does not run), verifies Docker/WSL are up (runner prereqs; it does NOT reinstall Docker Desktop per run), installs the tools, runs the installer's real New-K3dCluster, applies a credential-free stub the CLI discovery keys off (labels + the ingestor SA), asserts the discovery-shaped state, and checks the installer's cluster-create copy landed in the log. Stops before Invoke-ProvisionClient (Steps 5-6 mint a real backend credential — out of scope), exactly as e2e-journey stops before the CLI connects. - docs/WINDOWS-E2E.md: one-time runner setup (labels, WSL2 + Docker Desktop + nested virt, runner account rights) + what green/red means. - installer-tests.yaml: lint e2e-windows.ps1 with PSScriptAnalyzer on every push so a syntax/verb regression fails fast, not at the nightly run. Chose the self-hosted-runner route (Azure isn't available to us). Validated here: YAML parses, PSScriptAnalyzer clean (0 errors; the Write-Host/empty-catch/unicode warnings match install-k8s.ps1's own tolerated set), check-style clean. It CANNOT be validated off a nested-virt Windows host — the first scheduled run on the registered runner shakes it out, which is exactly the issue's acceptance (a broken installer commit turns it red with the install log attached). Closes #436 * fix(#436): admin assert + non-vacuous copy check + bounded kubectl in the e2e driver (Bugbot) Three Bugbot findings on the Windows e2e driver: 1. (High) TB_PESTER=1 skips the installer's self-elevation gate, but Initialize-ToolDir creates %ProgramFiles%\tracebloc\bin and writes the Machine PATH — admin-only. The docs wrongly said a non-admin account suffices. Assert Administrator up front (fail fast with a pointer) and correct docs/WINDOWS-E2E.md: the runner MUST run elevated (the Windows installer is inherently admin). 2. (Medium) The install-log copy assertion was SKIPPED when LOG_FILE was missing, yet the script still printed PASS — a vacuous seal. Now a missing log fails (never report PASS on an unverified check). 3. (Medium) Several kubectl calls (wait/create/apply/get) had no --request-timeout, so a wedged API server would hang to the 45-min job cap. Added --request-timeout=30s to each (the PowerShell analog of the bash journey's watchdog). PSScriptAnalyzer: 0 errors (the remaining warnings match install-k8s.ps1's tolerated set); YAML valid; check-style clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#436): bound docker info + teardown, and assert the FULL discovery selector (Bugbot r2) Two more Bugbot findings on the e2e driver: 1. (High) `docker info` and the finally `k3d cluster delete` had no deadline, so a wedged Docker/WSL or a stuck delete would hang the shared runner to the 45-min job cap — undercutting the --request-timeout hardening already on every kubectl call. Added an Invoke-Bounded helper (Start-Process + the installer's killing Wait-ProcessWithDeadline) and ran both through it (30s / 120s). 2. (Medium) The discovery assertion only filtered app.kubernetes.io/name=client, so a stub missing managed-by=Helm or a non -jobs-manager name still reached PASS. Now it matches the FULL selector DiscoverParentRelease uses: name=client AND managed-by=Helm, AND the Deployment name ends in -jobs-manager. PSScriptAnalyzer 0 errors; warnings unchanged (match install-k8s.ps1's tolerated set). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#436): fresh-cluster pre-clean, log survives timeout-cancel, bounded safety-net (Bugbot r3) Three findings from the persistent-runner reality: 1. (High) A leftover tbe2ewin cluster sends New-K3dCluster down its reuse path (which still logs 'Creating k3d cluster'), so the copy check + PASS could succeed WITHOUT a real create. Pre-clean any stale cluster (bounded) before New-K3dCluster so every run genuinely creates one. 2. (High) The install-log upload was gated on failure() only, but a timeout-minutes hit CANCELS the job (failure() is false), and the always() teardown then wiped the log the acceptance criteria need. Upload on !success() (failed OR cancelled/timed out), before the teardown. 3. (Medium) The workflow safety-net k3d delete was unbounded. Bound it with Start-Process + Wait-Process -Timeout 120 (+ force-kill), matching the driver's Invoke-Bounded. YAML valid; PSScriptAnalyzer 0 errors; check-style clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(#436): put the installer tool dir on GITHUB_PATH so teardown's k3d resolves (Bugbot r4) (High) The always-on safety-net teardown runs bare `k3d` in a fresh step process, but Initialize-ToolDir writes the MACHINE PATH — which a subsequent step doesn't pick up mid-job — so k3d could be unresolved there and, when the driver was killed before its finally, the tbe2ewin cluster would leak on the shared runner. Prepend %ProgramFiles%\tracebloc\bin to GITHUB_PATH in the setup step so every later step resolves k3d (the driver's own calls already resolve via RefreshPath in-process). YAML valid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
docs(ci): /fr-pass advances staging → prod only (backend#1405 D6)
security(ci): verify kubeconform against a pinned digest before installing
…#199-1) (#537) Test-only, no behavior change — the regression net for the egress-lockdown default flip (#199-2). The egress mechanism is fully shipped (client-runtime#102 / #378); #199 is just flipping allowExternalHttps to false by default, which drops the rule-2 external-HTTPS hole and shifts the remaining egress indices. - Pin allowExternalHttps: true in the three cases that assert the rule-2 hole or the MySQL rule at a rule-2-dependent index (external-443, MySQL, OpenShift), so they keep passing once the default flips. - Add two locked-down cases (allowExternalHttps: false): the 0.0.0.0/0 rule is gone (MySQL shifts to egress[1]), the UNCONDITIONAL requests-proxy egress (8888) survives so pods can still POST results/FLOPs, and the egress-proxy allowlist rule (3128) renders when egressProxy.enabled. 13 network-policy tests (was 11), 322 helm suite green. Lands first so #199-2 is a green-to-green flip. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…PASS pre-clean (Bugbot #542) (#544) check-facts.sh: _spec_get and _extract piped `sed -n … | head -1` under `set -o pipefail`. On a second match large enough to fill the pipe buffer, head closes after line 1, sed takes SIGPIPE, and the pipeline exits 141. In _extract the pipeline is the function's terminal command, so that 141 aborts the facts gate before any drift message prints — a crash / fail-open on duplicate input. Both helpers now capture the whole output and take the first line with `${all%%$'\n'*}` (the repo's gpu-nvidia.sh idiom), leaving no pipe to break. Empty (no-match) results still yield empty, so callers are unchanged. e2e-windows.ps1: the pre-clean `k3d cluster delete` ran via `Invoke-Bounded … | Out-Null`, discarding its exit code. On the persistent Windows runner a timed-out (124) or failed delete leaves the tbe2ewin cluster in place; New-K3dCluster then takes its REUSE path, which still logs "Creating k3d cluster", so the copy check + PASS succeed WITHOUT a real create — the exact false-green the pre-clean was added to prevent. The pre-clean now gates on the exit code (k3d delete is idempotent → 0 when absent) and Stop-E2e's the run on non-zero, so a stale cluster can never fall through. Tests: two bats cases in check-facts.bats feed duplicate matches past the pipe buffer and assert --check still passes (the _extract case reproduced exit 141 on the pre-fix code). e2e-windows.ps1 has no Pester suite (it's an integration driver dot-sourcing the real installer against live Docker/k3d), so it relies on the source-level guarantee; PSScriptAnalyzer lints it in CI and still passes. Both findings surfaced by Cursor Bugbot on the release-train PR client#542. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…WO-PVC prod incident) (#549) * fix(chart): jobs-manager must use Recreate, not RollingUpdate (RWO-PVC deadlock) Production incident: a client's hi-jobs-manager sat Pending for 4+ hours, blocking dataset ingestion. Root cause is entirely ours: jobs-manager mounts two ReadWriteOnce PVCs (client-pvc, client-logs-pvc; local-path, WaitForFirstConsumer) but used strategy RollingUpdate with maxSurge:1. maxSurge brings up a SECOND pod before the old one is gone. On a healthy single node this happens to work (RWO permits two pods on the same node), so it ran for days. After a routine WSL2/Docker restart the PVCs must re-provision, and now the rollout presents TWO unscheduled consumers to local-path — WaitForFirstConsumer can't pick one, so the scheduler's VolumeBinding PreBind times out ("context deadline exceeded") and the pod never schedules. The image-refresh CronJob then compounds it: it writes its "last-refreshed digest" annotation only after `kubectl rollout status` SUCCEEDS, which now never happens, so every 15 min it sees recorded != latest and restarts again — a self-perpetuating loop (19 revisions). mysql-client starves on the same wedged provisioner. jobs-manager never goes Ready, so its Service has no endpoints and the ingestor submit job's curl to jobs-manager:8080 fails. Fix: strategy Recreate (drop maxSurge) — exactly one RWO consumer at a time, so provisioning and restarts always converge, even after a node restart. This matches mysql-deployment, which already uses Recreate for the identical RWO-PVC reason. Regression guard: jobs_manager_test.yaml now asserts strategy.type == Recreate and no rollingUpdate block, so this can't silently regress. Chart 1.9.10 -> 1.9.11 (chart content change must bump the version to reach installs). Follow-ups filed for the compounding factors: image-refresh restart loop, K8S version pin reaching clients, and node-restart survival. * fix(chart): image-refresh skips a tick when jobs-manager isn't settled (no restart storm) Second half of the RWO-deadlock incident (#546, folded in to avoid a chart-version collision with the Recreate fix — same incident, same 1.9.11 bump). image-refresh records its digest annotation only after `kubectl rollout status` succeeds. When a rollout can't complete (the RWO deadlock, or any stall), the annotation never lands, so every 15-min tick sees recorded != latest and restarts AGAIN — a sequential restart storm (the incident showed 19 restarts). concurrencyPolicy: Forbid only stops OVERLAPPING jobs, not this. Fix: before touching the deployment, `kubectl rollout status --timeout=10s`; if it isn't settled (rollout in progress, or pod stuck Pending on volume binding), log and exit 0 — a restart can't help an unschedulable pod and only churns ReplicaSets, which on a single-node local-path cluster wedges provisioning further. Retry on the next tick once settled. Regression guard in image_refresh_test.yaml asserts the skip is present in the rendered script. (image_refresh_test's 2 pre-existing failures are local helm v4 schema strictness — identical on clean develop; CI's pinned helm v3.15.4 passes the suite.) Closes #546
…184) (#541) * test(seal-check): run the egress-enforcement probe live in k3d CI (#1184) RFC-0003 D12: the chart's enforcement probes ship as `helm.sh/hook: test` Jobs, but `helm test` ran nowhere in CI — SEAL-CHECK §8.4 recorded the k3s NetworkPolicy substrate as verified (#504) while the full-chart egress probe run stayed "pending". This closes that gap. - scripts/tests/e2e-seal-check.sh: install the local chart on a real k3d cluster with public images + the egress lockdown engaged (allowExternalHttps=false), then `helm test --filter` the egress-enforcement seal-check. Requires BOTH a zero exit AND the probe's `OK egress lockdown verified` marker in the logs — guarding the helm-test-`--filter`-matches-nothing silent-pass trap. - .github/workflows/helm-ci.yaml: a `seal-check-e2e` job mirroring upgrade-e2e (stock ubuntu runner, zero secrets — public curl vs 1.1.1.1). - client/tests/egress_enforcement_check_test.yaml: pin the probe Job's metadata.name so the e2e --filter can never silently drift off it. Local: shellcheck clean · helm template renders the probe Job · helm-unittest 27 suites / 320 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(seal-check): install k3d/helm/kubectl before create_cluster (exit 127) The sourced libs define install_kubectl/install_k3d/install_helm but do not call them; create_cluster + helm need the binaries on PATH first, and a stock runner has none preinstalled. Mirror e2e-auto-upgrade.sh's prerequisite block + the post-create node-ready wait. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(seal-check): drive helm test off exit code, not --logs (Job hook) The egress-enforcement probe is a Job-type test hook with hook-delete-policy hook-succeeded. `helm test --logs` looks up the pod by the Job's bare name, but a Job's pod has a generated suffix ("pods not found"), and Helm deletes the Job on success anyway — so --logs errored even though the probe passed. Drive off the exit code instead (the probe exits 0 only when egress is verified blocked), guard the --filter-silent-pass by asserting the hook is in `helm get hooks` first, and dump the pod log via kubectl only on failure (the Job persists when it fails). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(seal-check): address Bugbot review (autostart, paths, shellcheck) Three Bugbot findings on the new e2e seal-check: - Medium: set TRACEBLOC_NO_AUTOSTART=1 before create_cluster (like the sibling e2e-*.sh) so it never mutates the host's Docker restart policy / runs `systemctl enable docker`. - Medium: add scripts/tests/e2e-seal-check.sh to helm-ci.yaml on.push/ on.pull_request paths so script-only edits re-trigger the k3d job. - Low: enumerate the script in the installer-tests ShellCheck gate (both the error and warning passes), matching the other e2e entrypoints. The High finding ("Job logs never reach marker check") was already resolved in the prior commit — the script drives off `helm test`'s exit code and no longer greps --logs output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(seal-check): avoid SIGPIPE/pipefail false-fail in hook guard (Bugbot) `helm get hooks | grep -q` lets grep close the pipe on first match, SIGPIPE-ing helm mid-write; under set -o pipefail that false-fails the guard even when the hook exists. Capture helm output to a var, then grep a here-string — no pipe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(seal-check): bump job timeout 20m->30m (Bugbot) The script's own bounds (create_cluster up to 15m + helm test 360s + tool install + helm install) can exceed a 20m GHA cap on a slow cluster bring-up, false-failing even while each component is inside its own timeout. Match the sibling k3d job (upgrade-e2e = 30m). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(seal-check): positive control + probe timeout + lib paths + NS (Saqlain) - Positive control (blocking): before trusting a BLOCKED probe, prove the cluster can REACH the host from a non-policied pod (in `default`, ungoverned by the namespace-scoped training-egress policy). Without it, egress failing for an unrelated reason (runner firewall / target outage / rate-limit) would make the probe print OK and pass green while the NetworkPolicy did nothing. A reachable positive + a blocked training pod = the block is attributable to the policy. Positive-control failure now fails the seal-check as inconclusive. - Bump enforcementProbeTimeoutSeconds 60s->240s (blocking): on a cold GHA runner k3s can take >60s to program the pod iptables while the chart installs; the probe is single-shot (backoffLimit 0), so 60s false-fails. 240s is well inside the 360s helm-test budget. - Add scripts/lib/** to helm-ci paths (blocking): the script sources scripts/lib/{common,setup-linux,cluster,preflight}.sh, so a lib-only edit must re-trigger seal-check-e2e + upgrade-e2e (both depend on it). - Derive NS from CLUSTER_NAME so a CLUSTER_NAME override isolates a run under one name instead of desyncing cluster vs release/namespace. shellcheck clean; workflow parses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(seal-check): --tlsv1.2 (house rule) + SA wait + required Lint list (Bugbot) - Add --tlsv1.2 to the positive-control curl — the curl-tls house rule (the required quality/house-rules gate) rejects a curl that could negotiate a downgraded TLS version. - Wait for the default ServiceAccount before the positive-control kubectl run: a fast runner can schedule the pod before the SA exists, aborting under set -e with 'serviceaccount default not found' before the attribution message. - Add e2e-seal-check.sh to the standard-checks.yml Lint shellcheck list (the REQUIRED branch-protection gate) — installer-tests had it, the required Lint did not, so a shellcheck regression in the new script could miss the gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add direct `lengthEqual` assertions on spec.egress for the two locked-down cases so a stray appended rule is caught, not just an index shift. Correction vs the review suggestion: the locked-down case is 4 rules, not 3. `egressProxy.enabled` DEFAULTS TRUE (values.yaml — only `routeWorkloads` defaults false), so the egress-proxy permit rule renders even when the case doesn't set it: DNS + MySQL + requests-proxy + egress-proxy. Both locked-down cases assert count 4. Fast-follow to the #537 review (non-blocking nit, deferred at merge). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bugbot, release-train staging hop #542, High. The always() safety-net teardown ran k3d cluster delete via Start-Process with no -ErrorAction Stop and no null-check on $p, inside an empty catch, and the inline step did not set $ErrorActionPreference. A SPAWN failure left $p null, $p.HasExited threw into the empty catch, and the shared runner kept a leftover tbe2ewin cluster -- the exact state this net exists to clear, and the one that sends the next run's New-K3dCluster down its reuse path for a false PASS with no real create. Now: ErrorActionPreference=Stop, an explicit null-check, a timeout-kill that surfaces rather than hides, and an exit-code check. On any failure it does NOT fail the run (this is always() teardown, the run is over) but emits a ::warning:: naming the stale cluster and the manual fix, so a leftover can never be silent again. Refs: tracebloc/backend#1426
release-train: develop -> staging
…550) Saqlain flagged (PR #541, thread on e2e-seal-check.sh:36) that the bring-up prelude is copy-pasted near-verbatim across scripts/tests/e2e-*.sh, and multiple Bugbot rounds have had to edit every copy in lockstep. Extract the two truly-identical, drift-prone blocks into scripts/tests/lib/e2e-common.sh: - e2e_isolate_env <name> — USER + CLUSTER_NAME default + TRACEBLOC_NO_AUTOSTART - e2e_install_prereqs — has docker + umask + install_{kubectl,k3d,helm} e2e-cluster / e2e-proxy / e2e-journey / e2e-auto-upgrade now source the lib and call these; each keeps its own CLUSTER_NAME default (passed as the arg) and its distinct logic. auto-upgrade keeps its extra `has jq` guard before the call. Deliberately NOT unified (would change behavior): the sub-lib `source` set (proxy/journey source 3 libs, not preflight — a pre-existing inconsistency, flagged not fixed), the cleanup/trap bodies (each reaps its own squid/work dirs), and CHART_DIR (only the chart-installing scripts). e2e-seal-check.sh (open on #541) adopts the lib as a fast-follow once both land — kept non-stacked. Added the lib to both shellcheck gates (installer-tests + the required standard-checks Lint). shellcheck --severity=error/warning clean; bash -n ok. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ollow) (#566) Promised on the #541 review threads, now that #550 (the shared lib) and #541 (the seal-check script) are both on develop: - e2e-seal-check.sh now sources scripts/tests/lib/e2e-common.sh and uses e2e_isolate_env / e2e_install_prereqs, matching the other e2e-*.sh (drops the inlined isolation-env + install block). Keeps its own NS=$CLUSTER_NAME and local fail() for its assertions. - Harden the positive control (Saqlain nit): pin networkPolicy.training. enforcementProbeHost to a single $HOST var the install passes AND the positive control targets, so the probe and the control can never drift onto different hosts (was: HOST hardcoded while the probe used the chart default). shellcheck --severity=error/warning clean; helm template renders the probe Job with HOST=1.1.1.1 pinned. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
) * fix(installer): stop the k3s version pin from silently drifting (#547) Root-caused from the Windows stuck-install incident: a client ran k3s v1.35.5 while the pin was v1.29.4-k3s1. Addresses three of the four compounding gaps the audit found (F3 left as a tracked checklist item): - F1: the header docs advertised `default: latest`, inviting users to set K8S_VERSION=latest, which floats to k3d's bundled default k3s. Fix the docs in both installers, and warn loudly at create time when `latest` is used. - F2: the reuse/adopt path never re-checked the running node's k3s version, so a cluster born unpinned (old installer / latest / manual create) persisted forever across later correctly-pinned re-runs — the single best explanation for the observation. Add _check_existing_cluster_k8s_version (bash) and a parity check in New-K3dCluster (PowerShell): warn + recreate remedy on drift. - F4: check-facts.sh only compared the pinned version STRINGS, not the create wiring, so `--image rancher/k3s:` could be dropped while CI stayed green. Add a structural guard asserting the pin is wired in cluster.sh + install-k8s.ps1. With --image now guaranteed on create, k3d's own version no longer floats k3s, so F3 (winget installs unpinned k3d) is de-risked and tracked in #547. Tests (only added): +7 bats for _check_existing_cluster_k8s_version, +5 Pester source guards, check-facts.bats fixture extended with the wiring line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(installer): regenerate manifest.sha256 for the k3s-pin edits (#547) install-k8s.sh / cluster.sh / install-k8s.ps1 hashes changed; the supply-chain R8 gate (gen-manifest.sh --check) requires the committed manifest to match. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): run k3s-drift check on the healthy fast-path too (Bugbot #565) Bugbot: the drift check only lived on the full reuse path (_handle_existing_cluster / New-K3dCluster), but both installers short-circuit earlier when a re-run classifies as healthy (bash assess_existing_install, PS completed+healthy fast-path). A healthy-but-drifted cluster — the #547 STEADY STATE — would hit "already set up / nothing to do" and never see the warning, exactly the population the check is meant to help. - bash: assess_existing_install's healthy branch now calls _check_existing_cluster_k8s_version before the handoff (guarded by declare -F). - PS: extracted the inline reuse-path check into Test-K3sVersionDrift and call it from BOTH New-K3dCluster and the completed+healthy fast-path in main. Tests: +2 assess.bats (healthy runs it; --force skips it); Pester #547 block updated to assert the shared function + both call sites. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(check-facts): report a missing --image pin as a WIRING gap, not "run --write" (Bugbot #565) Bugbot: the F4 wiring guard incremented the same `drift` counter as version-string mismatches, so a missing create-time --image pin ended with "fact(s) drifted... Run 'check-facts.sh --write'". But --write only restamps version strings and cannot restore create-time wiring — the summary pointed developers at a no-op fix. Track wiring failures in a separate counter and emit a wiring-specific message (this is a WIRING gap; restore the --image rancher/k3s:${K8S_VERSION} flag by hand). +1 check-facts.bats: a missing --image pin fails with the WIRING message and never the --write hint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(install): idempotent hostpath dataset staging for Windows/macOS/Linux (#547) A client re-running the dataset-copy step hit "already exists" from non-idempotent `mkdir` + `Copy-Item -Recurse`. The repo only documented the Linux `kubectl cp` staging path, with no hostpath/Windows guidance. Add an idempotent hostpath staging section: Windows uses `New-Item -Force` + `robocopy /E` (merges into an existing target, safe to re-run); macOS/Linux use `mkdir -p` + `cp -R`. Notes the plain-mkdir "already exists" error is harmless (data already staged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(installer): bound the k3s-drift docker inspect probe (Bugbot #565) Bugbot: _check_existing_cluster_k8s_version (bash) and Test-K3sVersionDrift (PS) ran a bare `docker inspect` with no deadline, and both healthy fast-paths now call them — a wedged Docker engine could hang a headless "already healthy" re-run AFTER success was printed, violating the installer's bounded-probe rule. - bash: wrap the inspect in _bounded (timeout/gtimeout; 124 on timeout → the existing `|| return 0` makes it a silent no-op). - PS: run it via Start-Job + Wait-JobWithProgress -TimeoutSec 15 (mirrors Test-ClusterRunning); on timeout, skip the check with a log line. Tests: cluster.bats setup overrides _bounded so the docker shell-function mock is exercised on Linux CI too (timeout can't exec a function); Pester asserts the bounded Start-Job pattern tied to the "Checking k3s version" probe. Manifest regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(check-facts): wiring-gap hint names the correct pin literal per shell (Bugbot #565) Bugbot: the remediation hint told devs to restore `rancher/k3s:${K8S_VERSION}` in both files, but the PowerShell guard matches the fixed string `rancher/k3s:$K8S_VERSION` (no braces) — following the hint in the PS create path would leave CI red even though --image is correctly wired. Reword the hint to name BOTH shell forms (bash cluster.sh uses ${K8S_VERSION}; PowerShell install-k8s.ps1 uses $K8S_VERSION) and point at the exact literal each ✖ line already prints. +2 assertions in check-facts.bats locking both forms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
release-train: develop -> staging
Contributor
Author
|
bugbot run |
tracebloc-release-train
Bot
added
gate-nudge
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…ign skip Bugbot on the prod promotion (client#571), Medium. The settled-gate ran `kubectl rollout status` and treated ANY non-zero as 'in progress -- skip' (exit 0). But non-zero is also NotFound, RBAC denial, and API errors -- so a misconfigured or unreachable deployment left the CronJob GREEN forever while image refresh never ran. And the call had no --request-timeout (only --timeout, the rollout wait), so a wedged API server could hang the tick with no activeDeadlineSeconds, and concurrencyPolicy: Forbid would then block every later tick. Now: a non-zero rollout status is disambiguated -- if `kubectl get deployment` still succeeds the deployment is genuinely present-but-unsettled (legit skip, exit 0); otherwise it is an error the job SURFACES (exit 1). --request-timeout=15s bounds each API call, and activeDeadlineSeconds=300 caps the whole tick as a backstop. Refs: tracebloc/backend#1426
…ign skip Bugbot on the prod promotion (client#571), Medium. The settled-gate treated ANY non-zero `kubectl rollout status` as 'in progress -- skip' (exit 0). But non-zero is also NotFound, RBAC denial, and API errors, so a misconfigured or unreachable deployment left the CronJob GREEN forever while refresh never ran. The call also had no --request-timeout (only --timeout, the rollout wait), so a wedged API could hang the tick with no activeDeadlineSeconds -- and concurrencyPolicy: Forbid then blocks every later tick. Now a non-zero status is disambiguated: if `kubectl get deployment` still succeeds the deployment is genuinely present-but-unsettled (legit skip, exit 0); otherwise it is surfaced (exit 1). --request-timeout=15s bounds each API call and activeDeadlineSeconds=300 caps the whole tick. (Corrects a prior no-op commit on this branch that committed the file unmodified because a patch anchor mismatched the real indentation.) Refs: tracebloc/backend#1426
…ercut it Bugbot on client#572, High. My 300s deadline was BELOW the default imageRefresh.rolloutTimeout of 10m -- a tick doing rollout restart + rollout status could be DeadlineExceeded mid-wait, so the post-success annotation never lands and later ticks re-restart forever. Raised to 1800s (well above 10m) with a comment tying it to rolloutTimeout; it is only a backstop since every call is --request-timeout-bounded.
fix(chart): image-refresh CronJob must not treat every error as a benign skip (#571)
release-train: develop -> staging
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6766c1a. Configure here.
tracebloc-release-train
Bot
added
gate-nudge
shujaatTracebloc
added a commit
that referenced
this pull request
Aug 13, 2026
…lient#569)
Bugbot, Medium — and, as it notes, the SAME helper-vs-runtime disagreement class
this PR already had to fix for requests-proxy. The rule was written twice and the
two copies drifted:
* `tracebloc.imageRefreshEnabled` treated resource-monitor as done only when
`images.resourceMonitor.digest` was set.
* The CronJob's RESOURCE_MONITOR_PINNED env ALSO treated `resourceMonitor:
false` as done — correctly, since with no DaemonSet a cross-namespace
`set image` would just fail the tick.
So `resourceMonitor: false` plus both class-1 images pinned kept rendering a
CronJob that skipped every image and exited green every 15 minutes, forever.
Before #569 that combination retired the CronJob cleanly. It is also exactly the
green-forever-while-doing-nothing failure mode the script's own #571 comment
warns about.
Both consumers now read one helper, `tracebloc.resourceMonitorRefreshPinned`,
which is the single place the "nothing to do for resource-monitor" rule lives:
an explicit digest pin, or the DaemonSet disabled outright. Nil-safe, and an
absent `resourceMonitor` key reads as enabled to match the
`ne .Values.resourceMonitor false` gate on the DaemonSet itself.
Three tests: the combination now retires both the CronJob and its RBAC, and the
opposite direction is guarded too — disabling resource-monitor must NOT retire
the CronJob while jobs-manager or pods-monitor can still drift.
Verified: helm unittest 420 passed, failures still exactly develop's baseline
(zero new). Rendering confirms image-refresh is gone for the retiring
combination (only auto-upgrade's CronJob remains) and present in both keep
cases. helm lint, check-style clean; all four client/ci value sets render.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
shujaatTracebloc
added a commit
that referenced
this pull request
Aug 13, 2026
…on Always (#569) (#705) * fix(chart): re-image control-plane pods by digest instead of relying on Always (client#569) The always-running control-plane pods rendered `imagePullPolicy: Always`, so a Docker Desktop / WSL2 restart forced a registry round-trip and landed in ImagePullBackOff even with the image already cached in containerd — the edge came back only once docker.io was reachable, up to ~6h behind Docker Hub's anonymous pull-rate limit. `Always` could not simply be flipped: it IS the update mechanism. The image-refresh CronJob's `kubectl rollout restart` only picks up a new build because the pull policy re-resolves the floating tag. Offline-safety and restart-driven updates are mutually exclusive unless the image REFERENCE changes on update. So the reference is now what changes. - All four control-plane call sites render IfNotPresent unconditionally: jobs-manager (api + pods-monitor sidecar), requests-proxy, resource-monitor. - image-refresh swaps `rollout restart` for `kubectl set image repo@digest`. - Two workloads come under refresh for the first time, both quietly broken before: requests-proxy runs the SAME jobs-manager image but was never reconciled (it skewed until an unrelated restart, then jumped to whatever the tag pointed at), and resource-monitor had no deliberate update path at all. - requests-proxy follows the jobs-manager digest in the same tick, no second registry HEAD. images.requestsProxy.digest opts it out. - resource-monitor needs a second Role in the node-agents namespace: get/patch resourceNames-scoped to the one DaemonSet, list/watch namespace-wide and read-only (RBAC ignores resourceNames for collection verbs, and `rollout status` requires them). Not rendered when resourceMonitor: false. - imageRefreshEnabled now requires all three refreshed images pinned before retiring the CronJob; pinning only the two class-1 images used to render it away, leaving the DaemonSet with no update path. - Private mirrors: the script resolves digests from docker.io, so pinning one onto a mirrored reference could pin an image the mirror does not hold. It logs and goes inert. Under `rollout restart` that mismatch was merely useless; with `set image` it has to fail closed. The first-tick "record without acting" contract is kept deliberately: re-imaging on the first tick would rewrite repo:tag to repo@digest for byte-identical content on every fresh install, rolling the Deployment and the DaemonSet on every node for nothing. A fresh edge therefore runs repo:tag until the first real digest change — restart-safe offline, just not yet reproducible. Two bounded limitations are documented in the script header rather than hidden: a chart version bump re-renders repo:tag and this tick will not re-pin (the annotation still matches), and skew predating this change is prevented but not repaired. Both self-heal at the next upstream release and neither can break a running edge. The proper fix for both is reconciling against each workload's live container image instead of a shared annotation — which `set image` makes possible for the first time, and which is a deliberate follow-up. Verified: helm unittest 405 passed, failures exactly develop's pre-existing baseline (5 failed / 5 errored, diffed against a stashed baseline run — zero new). helm lint clean, all four client/ci value sets render, check-style and check-facts pass, gen-manifest --check up to date. Chart 1.9.38 -> 1.9.39, version + appVersion in lockstep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): close two Bugbot findings on the digest-on-update reconcile (client#569) 1. Jobs-manager pin skipped requests-proxy (_helpers.tpl:219). requests-proxy runs the jobs-manager IMAGE, and both the helper and the values docs claimed it follows the jobs-manager pin — but the Deployment read only `images.requestsProxy.digest`. Pinning jobs-manager ALONE was therefore a silent trap: the proxy kept rendering the floating `repo:tag` while jobs-manager ran the pinned digest, AND image-refresh skips pinned images, so nothing ever wrote a `set image` for the proxy either. It froze on the tag indefinitely, running a different build of the same image — precisely the skew #569 exists to close, re-introduced through the pinning path. `images.requestsProxy.digest` now falls back to `images.jobsManager.digest` when empty, so the claim is true by construction. The proxy key remains an explicit per-workload override. The helper comment now records that the test depends on that fallback, so removing it forces requests-proxy back into the "nothing left to do" test. 2. Refresh budget too small for three rollouts (image-refresh-cronjob.yaml). A tick now waits on up to three sequential `rollout status` calls, each up to rolloutTimeout (10m), while activeDeadlineSeconds was a hardcoded 1800 — exactly 3 x 10m, zero headroom. Blowing it is not a benign timeout: the Job is killed mid-wait, so under `set -e` the post-success annotate never runs, the recorded digest never advances, and the shared `refresh-attempt` counter stays incremented. Three such ticks trip the #563 flap lockout for EVERY control-plane image at once, while the CronJob still looks healthy. - activeDeadlineSeconds is now `imageRefresh.activeDeadlineSeconds`, default 3600 (3 x the default rolloutTimeout plus 100% slack), with a schema entry (minimum 60) and the raise-both-together constraint documented on rolloutTimeout. - The resource-monitor DaemonSet gets an explicit `updateStrategy.rollingUpdate.maxUnavailable: 10%`. Kubernetes defaults to maxUnavailable: 1, so a digest change converged in (nodes x pull+start) and blew the 10m wait on any multi-node cluster long before the image was bad. Safe to widen here specifically: resource-monitor is a read-only node metrics reader, so a briefly absent pod degrades scheduling telemetry and nothing else. Kubernetes rounds 10% down and floors it at 1, so small clusters keep today's one-at-a-time behaviour. Verified: helm unittest 411 passed, failures still exactly develop's pre-existing baseline (5 failed / 5 errored, diffed against a stashed baseline run — zero new). helm lint clean, check-style clean. Rendering confirms requests-proxy resolves to the jobs-manager digest when only jobsManager is pinned, activeDeadlineSeconds renders 3600 and honours an override, the DaemonSet carries maxUnavailable 10%, and the schema rejects a sub-60 deadline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): keep Always where the digest reconcile cannot run (client#569) Bugbot, High severity — a regression introduced by this PR's first commit. Making `imagePullPolicy: IfNotPresent` UNCONDITIONAL removed the only update path from every edge where the replacement mechanism cannot run. `Always` on a floating tag is not merely offline-fragility; it IS an update path — restart the pod and the kubelet re-resolves the tag. Two configurations relied on exactly that and were left frozen on their cached image forever, with a green CronJob and no signal: * `global.imageRegistry` (private mirror). The reconcile resolves digests from docker.io, so this PR deliberately makes it inert there rather than pin a digest the mirror may not hold. With IfNotPresent on top, syncing the mirror and restarting kept serving the cached tag. * `imageRefresh.enabled: false`. values.schema.json has always promised these operators the image stays put "until manual restart" — true only with Always. The inert-path log message this PR added made it worse by telling operators to "sync your mirror and restart the workloads", which under IfNotPresent does nothing. That guidance is now correct because the policy is correct. The policy is now resolved per image by one helper, `tracebloc.controlPlanePullPolicy`, so the four call sites cannot disagree: 1. explicit `digest` pin -> IfNotPresent (immutable reference; updates come from changing the pin) 2. reconcile can run here -> IfNotPresent (`set image` changes the REFERENCE, which is what the kubelet pulls) i.e. the CronJob renders AND images come from docker.io 3. neither -> Always (floating tag + restart is the only update path that edge has) The trade is deliberate: offline-restart safety is delivered precisely where the digest reconcile can deliver updates. An edge that opts out of the mechanism keeps pre-#569 semantics rather than silently freezing — a frozen control plane with no signal is worse than a restart that needs the network. Verified by rendering the full matrix: default and all-pinned resolve IfNotPresent across jobs-manager (both containers), requests-proxy and resource-monitor; mirror and refresh-disabled resolve Always across all of them; a pin plus refresh-disabled correctly splits (pinned container IfNotPresent, unpinned sibling Always). helm unittest 417 passed, failures still exactly develop's pre-existing baseline (5 failed / 5 errored — zero new). helm lint, check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(chart): correct statements #569 made false in values.yaml and the schema (client#569) Self-audit follow-through, same class as the Bugbot findings on this PR: behaviour changed and the prose describing it did not. Per CLAUDE.md, a change that makes a statement false fixes that statement in the same PR. Docker Hub rate-limit arithmetic (the operationally significant one). #569 raised the per-tick cost from 2 manifest HEADs to 3 — resource-monitor joined jobs-manager and pods-monitor; requests-proxy adds none because it reuses the jobs-manager digest. values.yaml still claimed "2 images x 4/hr = 8/hr, well under the cap". It is now 12/hr, so 72 per 6h against the anonymous cap of 100, and the headroom left for OTHER workloads sharing the egress IP fell from ~52 to ~28. That is worth more than an arithmetic fix: exhausting this cap behind a shared corporate NAT is one of the failure modes in the incident #569 exists to fix — a rate-limited edge cannot pull for up to ~6h. The comment now states the real numbers, names the shared-IP risk, and gives the concrete remedy (a 3/hr schedule restores ~9/hr, and a digest pin drops that image's HEAD entirely). The default schedule is deliberately UNCHANGED: it is still within the cap, and quietly slowing drift pickup fleet-wide is a call for a human, not a side effect of a doc fix. values.schema.json descriptions, which were describing the pre-#569 mechanism: - imageRefresh: said it "rolls the deployment"; now describes `kubectl set image` across the three workloads, and why that is what permits IfNotPresent. - imageRefresh.enabled: says explicitly that disabling falls back to Always, so the long-standing "until manual restart" promise stays true. - imageRefresh.schedule: three manifests per tick, not two, with the numbers. - imageRefresh.maxRefreshAttempts: re-imaging, not re-restarting; notes the counter is shared across workloads. - images.jobsManager.digest: records that requests-proxy inherits it. - images.requestsProxy.digest: documents the follow-jobs-manager default (it had no description at all), so the pinning trap Bugbot found is discoverable from the schema rather than only from the template comment. - images.podsMonitor / resourceMonitor.digest: descriptions added. Checked and deliberately NOT changed: docs/SECURITY.md 6.1's `rollout restart` after a secret rotation restarts the pod to re-read the Secret, not to pull an image, so it is unaffected by the pull-policy change. Verified: helm unittest 417 passed, failures still exactly develop's baseline (zero new). helm lint and check-style clean; schema is valid JSON. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): one definition for "resource-monitor needs no refresh" (client#569) Bugbot, Medium — and, as it notes, the SAME helper-vs-runtime disagreement class this PR already had to fix for requests-proxy. The rule was written twice and the two copies drifted: * `tracebloc.imageRefreshEnabled` treated resource-monitor as done only when `images.resourceMonitor.digest` was set. * The CronJob's RESOURCE_MONITOR_PINNED env ALSO treated `resourceMonitor: false` as done — correctly, since with no DaemonSet a cross-namespace `set image` would just fail the tick. So `resourceMonitor: false` plus both class-1 images pinned kept rendering a CronJob that skipped every image and exited green every 15 minutes, forever. Before #569 that combination retired the CronJob cleanly. It is also exactly the green-forever-while-doing-nothing failure mode the script's own #571 comment warns about. Both consumers now read one helper, `tracebloc.resourceMonitorRefreshPinned`, which is the single place the "nothing to do for resource-monitor" rule lives: an explicit digest pin, or the DaemonSet disabled outright. Nil-safe, and an absent `resourceMonitor` key reads as enabled to match the `ne .Values.resourceMonitor false` gate on the DaemonSet itself. Three tests: the combination now retires both the CronJob and its RBAC, and the opposite direction is guarded too — disabling resource-monitor must NOT retire the CronJob while jobs-manager or pods-monitor can still drift. Verified: helm unittest 420 passed, failures still exactly develop's baseline (zero new). Rendering confirms image-refresh is gone for the retiring combination (only auto-upgrade's CronJob remains) and present in both keep cases. helm lint, check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): give the node-agents image-refresh RBAC a distinct name (client#569) Bugbot, High. The node-agents Role and RoleBinding added by #569 reused `tracebloc.imageRefreshName` — the same name as the release-namespace pair. `nodeAgents.namespace.name` pointing back at the release namespace is a SUPPORTED layout, not a misconfiguration: node-agents-namespace.yaml documents it and deliberately skips creating the Namespace in that case. In that layout both pairs land in one namespace, so the chart rendered two Roles and two RoleBindings with identical names in identical namespaces. Confirmed by rendering before the fix: Role tracebloc t-image-refresh x2 RoleBinding tracebloc t-image-refresh x2 Helm then either refuses the release or lets the later DaemonSet-only Role overwrite the deployments Role. The second outcome is the dangerous one: it is silent, and it strips image-refresh's patch on jobs-manager and requests-proxy. Because #569 also moves those pods to IfNotPresent, they would be left with no update path at all — the exact failure this PR exists to prevent, reached through a different door. The Role and RoleBinding now use `tracebloc.imageRefreshNodeAgentsName` (`<release>-image-refresh-node-agents`). The RoleBinding SUBJECT deliberately keeps the un-suffixed name: there is only one ServiceAccount and it lives in the release namespace. A distinct name is correct in BOTH layouts — split namespaces get one Role each, and the collapsed layout gets two complementary Roles (deployments, daemonsets) bound to the same SA, which is the intended grant. Verified by rendering both layouts: no duplicate (kind, namespace, name) in either, subjects and roleRefs resolve to the right objects in both. Tests pin the new names, the subject/roleRef split, and add a collapsed-layout regression case. helm unittest 421 passed, failures still exactly develop's baseline (zero new); helm lint and check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(chart): collapse the requests-proxy name to one definition, pin the rest (client#569) Proactive follow-up, not a review finding. Three of the five Bugbot findings on this PR were the same mistake: one side of a two-sided contract moved and the other did not (the requests-proxy digest pin, the resource-monitor pin signal, and the node-agents RBAC name). That is one habit, not three coincidences, so I audited the rest of the diff for the same shape and found two more live instances — both in contracts #569 itself created or started depending on. 1. Workload names. #569 made image-refresh reconcile workloads BY NAME with `kubectl set image`, which gives those names a second consumer. A rename that reached only the workload template would leave the CronJob patching something that does not exist: the tick fails, the digest record freezes, and the shared flap counter eventually locks out refresh for every control-plane image. `<release>-requests-proxy` had exactly two call sites — the Deployment, and the CronJob env I added — so it is now one definition, `tracebloc.requestsProxyName`. resource-monitor already went through `tracebloc.resourceMonitorName`. `<release>-jobs-manager` is deliberately NOT unified here. It has six call sites across five files (NOTES.txt, the PDB, tracebloc.serviceAccountName, ...), most of which this change does not otherwise touch, and the repo convention is that refactors ship separately from behaviour changes. Half-migrating it would BE the bug this commit is about. A contract test pins the two sides in the meantime. 2. Container names. `kubectl set image <workload> <container>=<ref>` fails outright on a wrong container name, so `api`, `pods-monitor-container`, `proxy` and `tracebloc-resource-monitor` are a hard contract between the script and the workload templates, previously asserted from neither side. Tests now pin both sides of everything still spelled out twice: the three reconcile target names and all four container names, asserted from the CronJob AND from jobs_manager / requests_proxy / resource_monitor. Renaming either side alone now fails CI instead of silently breaking refresh on the fleet. Verified: rendered names are byte-identical before and after the unification (t-requests-proxy in both the Deployment and REQUESTS_PROXY_DEPLOYMENT), and the requests-proxy Service selector is untouched — it matches the pod label `app: requests-proxy`, not the Deployment name. helm unittest 426 passed, failures still exactly develop's baseline (zero new); helm lint and check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-mainbranch (a mirror ofstaging), so it never collides with a human PR. Merged only when the fr-gate is green.Note
High Risk
Promotes many customer-facing changes at once—chart deployment strategy, egress lockdown behavior, and installer cluster/version semantics—plus broad CI/release workflow edits where prior bugs were fail-open supply-chain checks.
Overview
Release train promotion (
staging→main) bundling chart 1.9.12, installer/CI hardening, and operational doc updates accumulated on the integration branch.Helm client chart bumps to 1.9.12 and fixes production incidents around RWO PVCs and image-refresh:
jobs-managerswitches fromRollingUpdateto Recreate so a second pod cannot deadlock volume binding; the refresh CronJob skips ticks when the deployment is not settled, distinguishes real API errors from benign in-progress rollouts, and gets a higher activeDeadlineSeconds so legitimate rollouts are not killed mid-flight. Unit tests add regression coverage for locked-down egress (allowExternalHttps: false) and pin the egress-enforcement test Job name for livehelm testfiltering.CI and supply chain move fragile inline bash into tested scripts (
chart-version-guard.shnow guards bothclientand ingestor from the release workflow;index-invariants.shavoids SIGPIPE false greens). Helm CI adds a k3d seal-check e2e for egress enforcement, pins kubeconform by version+digest, and wires new script paths into lint/unit gates. Installer CI gainscheck-facts.sh --checkagainstscripts/spec/facts.env, extends ShellCheck/PSScriptAnalyzer to Windows e2e driver, and adds a nightly self-hosted Windows e2e workflow. Release post-publish index verification checks out the released tag and runs the extracted invariants script on a fetched index file.Installers (bash + PowerShell) single-source cross-OS pins via
facts.env, enforce k3s--imagewiring in CI, warn on k3s version drift when reusing clusters (#547), and tighten preflight memory messaging (host RAM vs Docker VM budget, shared grading). macOS adds a non-admin fail-fast, login/boot autostart for Docker/colima, and an updated reboot note when autostart succeeds. E2e scripts sharee2e-common.sh; auto-upgrade e2e resets recorded values between paths so--reset-then-reuse-valuesis tested on a clean edge (#459).Smaller changes: INSTALL.md idempotent dataset staging on hostPath installs; docs/WINDOWS-E2E.md; CLAUDE.md assignee convention; removal of wip-limit-check workflow; FR-pass caller comment aligned with retired dev FR column.
Reviewed by Cursor Bugbot for commit 6766c1a. Bugbot is set up for automated code reviews on this repo. Configure here.