release-train: develop -> staging - #713
Merged
Merged
Conversation
#701) * sec(#1876): grant jobs-manager list nodes for GPU->CPU pending fallback The check_pending_jobs GPU->CPU fallback (client-runtime#217) calls list_node() to tell an autoscaling-in GPU node from a genuinely absent one. nodes is cluster-scoped, so the grant can only live in the clusterScope: true ClusterRole (like tokenreviews). Without it list_node() 403s and the runtime fail-safe assumes a GPU node is present, silently disabling the fallback — a Pending GPU pod wedges forever while the backend shows RUNNING. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): bump client 1.9.38 → 1.9.39 for the list-nodes RBAC grant The rbac.yaml ClusterRole change is packaged chart content, so it only reaches installs via a new chart version (chart-version-guard). Merge current develop in and bump version + appVersion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ckend#1853) (#697) * fix(ci): watch every mutable label that points at a pinned digest (backend#1853) On 2026-08-12 the ingestor's `channelTags.prod: "0.8"` float moved to v0.8.8, which removed the legacy edgeuser DB_USER fallback. Nothing noticed; a manual sweep found it. Default prod edges were spared only because prodDigest pins v0.8.2 and prodPin defaults to true. DESIGNED TO SAADQBAL'S DIAGNOSIS, not to the symptom: "The disease is a moving label pointing at an immutable trust decision, with no watcher on the label. Anything about the build could have been the thing that changed -- the fallback just got there first." So this script asserts NOTHING about a build's contents. There is no DB_USER check, deliberately: such a check goes green the next time something else moves, which is the failure this watches for rather than a variant of it. It asks one property-agnostic question per pin -- does the float still resolve to the digest we decided to trust? The trusted versions are registered where they already were: the digest:/ prodDigest: fields of client/values.yaml. No second list to keep in sync; adding a pin enrols it automatically. Against the real chart: 3 pins found, squid agrees, the ingestor DRIFTS (the real finding), and mysqlClient is reported UNWATCHABLE -- it carries a pin but declares no repository, so nothing can tell you when its trust decision goes stale. That is a genuine modelling gap, reported rather than skipped. TWO BUGS OF MY OWN, both found by running it, not reading it: 1. images:-scoped discovery watched 1 of the 3 pins. squid's pin lives OUTSIDE images:, and an "empty field means skip" rule dropped mysqlClient without a word. Rewritten pin-driven over the whole file: a pin is watched or REPORTED, never skipped. 2. IFS=$'\t' collapses runs of tabs, because tab is IFS whitespace. A record with an empty repository AND tag slid the pin into the wrong variable, leaving $pin empty, and the row was skipped in silence. This is the exact defect release-train's own parse-repos suite pins by name; 0x1f is not whitespace. Also in this commit, per review: * client/values.yaml no longer states the float's version. It said "at v0.8.4" while the float was at v0.8.8 -- and any version written there is stale on the next release, reading as reassurance ("two patches behind") for a gap that may be far larger. The comment now says the float moves without us and points at this watcher. * docs/SECURITY.md 4.1.1 is reframed as an explicit CEILING with a table: v0.8.0-v0.8.4 safe with the flag off, v0.8.8+ NOT (config.py read at each tag). It previously implied the unsafe release was hypothetical; it exists and is what the float points at. Tests: 16 bats cases, registry stubbed via a documented seam that prints STUBBED on every run so a log cannot pass as a real audit. The repo's bats-hygiene guard caught that 31 of my assertions were advisory -- a bare [ ] on a non-final line cannot fail its test -- so all are now || return 1. Mutation-verified after that fix: comparison always true -> 4 tests fail discovery restricted to the images: block -> test 6 fails a pin with no repository silently skipped -> tests 9 + 11 fail (An earlier sed-based mutation of the third reported 0 failures; the pattern contained backticks and never matched. Inert mutation, not coverage -- re-done with an asserted anchor.) NOT in `make check`: needs network + docker, and is knowingly red today (the drift IS the finding). Runs daily via digest-drift.yml; the bats suite is in `make bats`, which needs neither. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(ci): bound the digest-drift registry lookups + bump chart version (backend#1853) - resolve_index_digest ran docker buildx imagetools inspect / docker manifest inspect with no timeout, so a wedged daemon or stuck registry hung the daily job instead of failing closed as UNRESOLVED. Wrap them in _tmout (timeout/gtimeout, 30s) so a stuck call is non-zero -> UNRESOLVED. - client/values.yaml changed, so the chart-content gate requires a Chart.yaml version bump: 1.9.34 -> 1.9.35. * fix(chart): match appVersion to version (Bugbot, #1853) The version bump left appVersion at 1.9.34; app.kubernetes.io/version follows appVersion, so installed objects would advertise the old chart version. Keep them in lockstep as this chart does: appVersion 1.9.35. * fix(ci): discover single-quoted and off-structure digest pins, fix checkout label check-digest-drift.sh only matched a double-quoted sha256 at exactly four-space indent, so a single-quoted (digest: SQ...SQ) or more-deeply nested pin was dropped in silence -- and the PINS==0 guard cannot catch that while any one conforming pin remains, so a run could print 'no drift' with a pin unwatched. Make discovery quote- and indent-agnostic: a canonical pin is watched as before; a pin off the structure is REPORTED unwatchable, never skipped. Adds bats cases for the single-quoted repro, the good-masks-sneaky case, and the off-structure -> UNWATCHABLE path. Also correct digest-drift.yml's checkout pin comment: 11d5960a is v4.4.0 (repo-standard, on releases/v4), not v5.0.0; drop the stray double space. addresses @saadqbal review, Bugbot, client#697. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…on Always (#569) (#705) * fix(chart): re-image control-plane pods by digest instead of relying on Always (client#569) The always-running control-plane pods rendered `imagePullPolicy: Always`, so a Docker Desktop / WSL2 restart forced a registry round-trip and landed in ImagePullBackOff even with the image already cached in containerd — the edge came back only once docker.io was reachable, up to ~6h behind Docker Hub's anonymous pull-rate limit. `Always` could not simply be flipped: it IS the update mechanism. The image-refresh CronJob's `kubectl rollout restart` only picks up a new build because the pull policy re-resolves the floating tag. Offline-safety and restart-driven updates are mutually exclusive unless the image REFERENCE changes on update. So the reference is now what changes. - All four control-plane call sites render IfNotPresent unconditionally: jobs-manager (api + pods-monitor sidecar), requests-proxy, resource-monitor. - image-refresh swaps `rollout restart` for `kubectl set image repo@digest`. - Two workloads come under refresh for the first time, both quietly broken before: requests-proxy runs the SAME jobs-manager image but was never reconciled (it skewed until an unrelated restart, then jumped to whatever the tag pointed at), and resource-monitor had no deliberate update path at all. - requests-proxy follows the jobs-manager digest in the same tick, no second registry HEAD. images.requestsProxy.digest opts it out. - resource-monitor needs a second Role in the node-agents namespace: get/patch resourceNames-scoped to the one DaemonSet, list/watch namespace-wide and read-only (RBAC ignores resourceNames for collection verbs, and `rollout status` requires them). Not rendered when resourceMonitor: false. - imageRefreshEnabled now requires all three refreshed images pinned before retiring the CronJob; pinning only the two class-1 images used to render it away, leaving the DaemonSet with no update path. - Private mirrors: the script resolves digests from docker.io, so pinning one onto a mirrored reference could pin an image the mirror does not hold. It logs and goes inert. Under `rollout restart` that mismatch was merely useless; with `set image` it has to fail closed. The first-tick "record without acting" contract is kept deliberately: re-imaging on the first tick would rewrite repo:tag to repo@digest for byte-identical content on every fresh install, rolling the Deployment and the DaemonSet on every node for nothing. A fresh edge therefore runs repo:tag until the first real digest change — restart-safe offline, just not yet reproducible. Two bounded limitations are documented in the script header rather than hidden: a chart version bump re-renders repo:tag and this tick will not re-pin (the annotation still matches), and skew predating this change is prevented but not repaired. Both self-heal at the next upstream release and neither can break a running edge. The proper fix for both is reconciling against each workload's live container image instead of a shared annotation — which `set image` makes possible for the first time, and which is a deliberate follow-up. Verified: helm unittest 405 passed, failures exactly develop's pre-existing baseline (5 failed / 5 errored, diffed against a stashed baseline run — zero new). helm lint clean, all four client/ci value sets render, check-style and check-facts pass, gen-manifest --check up to date. Chart 1.9.38 -> 1.9.39, version + appVersion in lockstep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): close two Bugbot findings on the digest-on-update reconcile (client#569) 1. Jobs-manager pin skipped requests-proxy (_helpers.tpl:219). requests-proxy runs the jobs-manager IMAGE, and both the helper and the values docs claimed it follows the jobs-manager pin — but the Deployment read only `images.requestsProxy.digest`. Pinning jobs-manager ALONE was therefore a silent trap: the proxy kept rendering the floating `repo:tag` while jobs-manager ran the pinned digest, AND image-refresh skips pinned images, so nothing ever wrote a `set image` for the proxy either. It froze on the tag indefinitely, running a different build of the same image — precisely the skew #569 exists to close, re-introduced through the pinning path. `images.requestsProxy.digest` now falls back to `images.jobsManager.digest` when empty, so the claim is true by construction. The proxy key remains an explicit per-workload override. The helper comment now records that the test depends on that fallback, so removing it forces requests-proxy back into the "nothing left to do" test. 2. Refresh budget too small for three rollouts (image-refresh-cronjob.yaml). A tick now waits on up to three sequential `rollout status` calls, each up to rolloutTimeout (10m), while activeDeadlineSeconds was a hardcoded 1800 — exactly 3 x 10m, zero headroom. Blowing it is not a benign timeout: the Job is killed mid-wait, so under `set -e` the post-success annotate never runs, the recorded digest never advances, and the shared `refresh-attempt` counter stays incremented. Three such ticks trip the #563 flap lockout for EVERY control-plane image at once, while the CronJob still looks healthy. - activeDeadlineSeconds is now `imageRefresh.activeDeadlineSeconds`, default 3600 (3 x the default rolloutTimeout plus 100% slack), with a schema entry (minimum 60) and the raise-both-together constraint documented on rolloutTimeout. - The resource-monitor DaemonSet gets an explicit `updateStrategy.rollingUpdate.maxUnavailable: 10%`. Kubernetes defaults to maxUnavailable: 1, so a digest change converged in (nodes x pull+start) and blew the 10m wait on any multi-node cluster long before the image was bad. Safe to widen here specifically: resource-monitor is a read-only node metrics reader, so a briefly absent pod degrades scheduling telemetry and nothing else. Kubernetes rounds 10% down and floors it at 1, so small clusters keep today's one-at-a-time behaviour. Verified: helm unittest 411 passed, failures still exactly develop's pre-existing baseline (5 failed / 5 errored, diffed against a stashed baseline run — zero new). helm lint clean, check-style clean. Rendering confirms requests-proxy resolves to the jobs-manager digest when only jobsManager is pinned, activeDeadlineSeconds renders 3600 and honours an override, the DaemonSet carries maxUnavailable 10%, and the schema rejects a sub-60 deadline. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): keep Always where the digest reconcile cannot run (client#569) Bugbot, High severity — a regression introduced by this PR's first commit. Making `imagePullPolicy: IfNotPresent` UNCONDITIONAL removed the only update path from every edge where the replacement mechanism cannot run. `Always` on a floating tag is not merely offline-fragility; it IS an update path — restart the pod and the kubelet re-resolves the tag. Two configurations relied on exactly that and were left frozen on their cached image forever, with a green CronJob and no signal: * `global.imageRegistry` (private mirror). The reconcile resolves digests from docker.io, so this PR deliberately makes it inert there rather than pin a digest the mirror may not hold. With IfNotPresent on top, syncing the mirror and restarting kept serving the cached tag. * `imageRefresh.enabled: false`. values.schema.json has always promised these operators the image stays put "until manual restart" — true only with Always. The inert-path log message this PR added made it worse by telling operators to "sync your mirror and restart the workloads", which under IfNotPresent does nothing. That guidance is now correct because the policy is correct. The policy is now resolved per image by one helper, `tracebloc.controlPlanePullPolicy`, so the four call sites cannot disagree: 1. explicit `digest` pin -> IfNotPresent (immutable reference; updates come from changing the pin) 2. reconcile can run here -> IfNotPresent (`set image` changes the REFERENCE, which is what the kubelet pulls) i.e. the CronJob renders AND images come from docker.io 3. neither -> Always (floating tag + restart is the only update path that edge has) The trade is deliberate: offline-restart safety is delivered precisely where the digest reconcile can deliver updates. An edge that opts out of the mechanism keeps pre-#569 semantics rather than silently freezing — a frozen control plane with no signal is worse than a restart that needs the network. Verified by rendering the full matrix: default and all-pinned resolve IfNotPresent across jobs-manager (both containers), requests-proxy and resource-monitor; mirror and refresh-disabled resolve Always across all of them; a pin plus refresh-disabled correctly splits (pinned container IfNotPresent, unpinned sibling Always). helm unittest 417 passed, failures still exactly develop's pre-existing baseline (5 failed / 5 errored — zero new). helm lint, check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(chart): correct statements #569 made false in values.yaml and the schema (client#569) Self-audit follow-through, same class as the Bugbot findings on this PR: behaviour changed and the prose describing it did not. Per CLAUDE.md, a change that makes a statement false fixes that statement in the same PR. Docker Hub rate-limit arithmetic (the operationally significant one). #569 raised the per-tick cost from 2 manifest HEADs to 3 — resource-monitor joined jobs-manager and pods-monitor; requests-proxy adds none because it reuses the jobs-manager digest. values.yaml still claimed "2 images x 4/hr = 8/hr, well under the cap". It is now 12/hr, so 72 per 6h against the anonymous cap of 100, and the headroom left for OTHER workloads sharing the egress IP fell from ~52 to ~28. That is worth more than an arithmetic fix: exhausting this cap behind a shared corporate NAT is one of the failure modes in the incident #569 exists to fix — a rate-limited edge cannot pull for up to ~6h. The comment now states the real numbers, names the shared-IP risk, and gives the concrete remedy (a 3/hr schedule restores ~9/hr, and a digest pin drops that image's HEAD entirely). The default schedule is deliberately UNCHANGED: it is still within the cap, and quietly slowing drift pickup fleet-wide is a call for a human, not a side effect of a doc fix. values.schema.json descriptions, which were describing the pre-#569 mechanism: - imageRefresh: said it "rolls the deployment"; now describes `kubectl set image` across the three workloads, and why that is what permits IfNotPresent. - imageRefresh.enabled: says explicitly that disabling falls back to Always, so the long-standing "until manual restart" promise stays true. - imageRefresh.schedule: three manifests per tick, not two, with the numbers. - imageRefresh.maxRefreshAttempts: re-imaging, not re-restarting; notes the counter is shared across workloads. - images.jobsManager.digest: records that requests-proxy inherits it. - images.requestsProxy.digest: documents the follow-jobs-manager default (it had no description at all), so the pinning trap Bugbot found is discoverable from the schema rather than only from the template comment. - images.podsMonitor / resourceMonitor.digest: descriptions added. Checked and deliberately NOT changed: docs/SECURITY.md 6.1's `rollout restart` after a secret rotation restarts the pod to re-read the Secret, not to pull an image, so it is unaffected by the pull-policy change. Verified: helm unittest 417 passed, failures still exactly develop's baseline (zero new). helm lint and check-style clean; schema is valid JSON. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): one definition for "resource-monitor needs no refresh" (client#569) Bugbot, Medium — and, as it notes, the SAME helper-vs-runtime disagreement class this PR already had to fix for requests-proxy. The rule was written twice and the two copies drifted: * `tracebloc.imageRefreshEnabled` treated resource-monitor as done only when `images.resourceMonitor.digest` was set. * The CronJob's RESOURCE_MONITOR_PINNED env ALSO treated `resourceMonitor: false` as done — correctly, since with no DaemonSet a cross-namespace `set image` would just fail the tick. So `resourceMonitor: false` plus both class-1 images pinned kept rendering a CronJob that skipped every image and exited green every 15 minutes, forever. Before #569 that combination retired the CronJob cleanly. It is also exactly the green-forever-while-doing-nothing failure mode the script's own #571 comment warns about. Both consumers now read one helper, `tracebloc.resourceMonitorRefreshPinned`, which is the single place the "nothing to do for resource-monitor" rule lives: an explicit digest pin, or the DaemonSet disabled outright. Nil-safe, and an absent `resourceMonitor` key reads as enabled to match the `ne .Values.resourceMonitor false` gate on the DaemonSet itself. Three tests: the combination now retires both the CronJob and its RBAC, and the opposite direction is guarded too — disabling resource-monitor must NOT retire the CronJob while jobs-manager or pods-monitor can still drift. Verified: helm unittest 420 passed, failures still exactly develop's baseline (zero new). Rendering confirms image-refresh is gone for the retiring combination (only auto-upgrade's CronJob remains) and present in both keep cases. helm lint, check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(chart): give the node-agents image-refresh RBAC a distinct name (client#569) Bugbot, High. The node-agents Role and RoleBinding added by #569 reused `tracebloc.imageRefreshName` — the same name as the release-namespace pair. `nodeAgents.namespace.name` pointing back at the release namespace is a SUPPORTED layout, not a misconfiguration: node-agents-namespace.yaml documents it and deliberately skips creating the Namespace in that case. In that layout both pairs land in one namespace, so the chart rendered two Roles and two RoleBindings with identical names in identical namespaces. Confirmed by rendering before the fix: Role tracebloc t-image-refresh x2 RoleBinding tracebloc t-image-refresh x2 Helm then either refuses the release or lets the later DaemonSet-only Role overwrite the deployments Role. The second outcome is the dangerous one: it is silent, and it strips image-refresh's patch on jobs-manager and requests-proxy. Because #569 also moves those pods to IfNotPresent, they would be left with no update path at all — the exact failure this PR exists to prevent, reached through a different door. The Role and RoleBinding now use `tracebloc.imageRefreshNodeAgentsName` (`<release>-image-refresh-node-agents`). The RoleBinding SUBJECT deliberately keeps the un-suffixed name: there is only one ServiceAccount and it lives in the release namespace. A distinct name is correct in BOTH layouts — split namespaces get one Role each, and the collapsed layout gets two complementary Roles (deployments, daemonsets) bound to the same SA, which is the intended grant. Verified by rendering both layouts: no duplicate (kind, namespace, name) in either, subjects and roleRefs resolve to the right objects in both. Tests pin the new names, the subject/roleRef split, and add a collapsed-layout regression case. helm unittest 421 passed, failures still exactly develop's baseline (zero new); helm lint and check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(chart): collapse the requests-proxy name to one definition, pin the rest (client#569) Proactive follow-up, not a review finding. Three of the five Bugbot findings on this PR were the same mistake: one side of a two-sided contract moved and the other did not (the requests-proxy digest pin, the resource-monitor pin signal, and the node-agents RBAC name). That is one habit, not three coincidences, so I audited the rest of the diff for the same shape and found two more live instances — both in contracts #569 itself created or started depending on. 1. Workload names. #569 made image-refresh reconcile workloads BY NAME with `kubectl set image`, which gives those names a second consumer. A rename that reached only the workload template would leave the CronJob patching something that does not exist: the tick fails, the digest record freezes, and the shared flap counter eventually locks out refresh for every control-plane image. `<release>-requests-proxy` had exactly two call sites — the Deployment, and the CronJob env I added — so it is now one definition, `tracebloc.requestsProxyName`. resource-monitor already went through `tracebloc.resourceMonitorName`. `<release>-jobs-manager` is deliberately NOT unified here. It has six call sites across five files (NOTES.txt, the PDB, tracebloc.serviceAccountName, ...), most of which this change does not otherwise touch, and the repo convention is that refactors ship separately from behaviour changes. Half-migrating it would BE the bug this commit is about. A contract test pins the two sides in the meantime. 2. Container names. `kubectl set image <workload> <container>=<ref>` fails outright on a wrong container name, so `api`, `pods-monitor-container`, `proxy` and `tracebloc-resource-monitor` are a hard contract between the script and the workload templates, previously asserted from neither side. Tests now pin both sides of everything still spelled out twice: the three reconcile target names and all four container names, asserted from the CronJob AND from jobs_manager / requests_proxy / resource_monitor. Renaming either side alone now fails CI instead of silently breaking refresh on the fleet. Verified: rendered names are byte-identical before and after the unification (t-requests-proxy in both the Deployment and REQUESTS_PROXY_DEPLOYMENT), and the requests-proxy Service selector is untouched — it matches the pod label `app: requests-proxy`, not the Deployment name. helm unittest 426 passed, failures still exactly develop's baseline (zero new); helm lint and check-style clean; all four client/ci value sets render. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…al one (#702) * fix(installer): last failing command wins, so the report names the real one The ERR recorder shipped in #683 kept the FIRST failure. That is wrong, and a field report showed why: the run reported Stopped at .../lib/common.sh:527 (exit 1). command: sudo -n true for a failure two steps later. common.sh:527 is _real_sudo, reached from step a's _probe_privilege, whose `sudo -n true` returns non-zero to mean "a password is needed" — the installer then PRINTS that as a normal row in the host check. The trap fires for every failing command, benign ones included, so first-wins latched onto a routine probe inside a step that SUCCEEDED and refused every later record. The fatal command was never captured. A confidently wrong location is worse than the blank screen #683 replaced: it sends the reader to a line that is working as designed. Last-wins is precise. errexit stops the script AT the fatal command, and the trap fires once per failing command with no per-frame re-firing as the error unwinds — verified on bash 3.2 (macOS) and 5.x. Also: - Re-entrancy guard. `set -E` makes the recorder inherit its own trap, and the new `log` call is exactly the kind of command that fails inside it (its `[[ -n "${LOG_FILE:-}" ]] && …` form returns non-zero with no log open). Without the guard that recurses forever. - install_cleanup disarms the ERR trap before reading the record. Its own lines fail routinely — a `kill` on a dead pid, a false `[[ … ]]` — and under last-wins each would overwrite the fatal command with a cleanup detail. - The full ERR trail now goes to the log. The benign entries are not noise: reading them in order is what identified this bug. Five bats tests, mutation-real against the first-wins guard, including the field shape end to end — a probe that fails inside an `if`, a step that then succeeds, a fatal command afterwards. 982 bats green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(common): make the _record_err re-entrancy test actually exercise the guard (client#702) The recursion test drove the ERR trap with `command false || true`, which does not fire it — a command in a || list is excluded from ERR (bash manual). On bash 5.3 that form fires the trap zero times, so _record_err never ran and SURVIVED printed with or without the guard. bash 3.2 does fire it, which is why it looked green on macOS while being vacuous on Linux CI. `unset LOG_FILE` was the other half: log() is `[[ -n $LOG_FILE ]] && echo …`, so with no log open the write never attempts and nothing inside the recorder fails. The failure has to come from the redirection, so point LOG_FILE at a path whose parent does not exist (fails for root too, unlike chmod 000). Fixing only that is not enough. bash re-enters an ERR trap at most once, so deleting _TB_IN_RECORD_ERR does not hang anything and the survival test passes either way. Add a test for what the guard actually protects: a re-entrant call must not overwrite TB_ERR_* with the recorder's own log failure, which would turn 'died at helm upgrade' into 'died writing its log'. Verified by mutation — with the guard removed, the new test goes red and the survival test stays green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Shujaat Hasan <shujaat@tracebloc.io>
Contributor
Author
|
bugbot run |
… an old CLI (#708) Once a machine is "already set up", no installer on any platform ever updated the tracebloc CLI again. The cluster keeps upgrading itself hourly via the auto-upgrade CronJob; the host binary does not, and nothing noticed. A field machine was found on CLI v0.5.1 from 6 July against a current 0.10.6 — five minors behind — on a box whose chart had meanwhile auto-upgraded 1.8.5 -> 1.9.34. The user re-ran the newest installer and it exited before reaching the CLI step. macOS and Linux (one file, no platform branch, so identical): install-k8s.sh:198 assess -> healthy -> hands off -> exit 0 install-k8s.sh:241 install_tracebloc_cli, the ONLY call site, never reached _assess_cli_present checked presence, not version. Note the asymmetry: a MISSING CLI was correctly degraded/cli-missing and got installed; only a STALE one slipped through, the single state nothing checked. Windows had the same hole plus a worse one. Test-ToolsPresent covers docker/kubectl/k3d/helm — the CLI is not in it at all — and `completed` is set purely from ClientState -eq "connected", which says nothing about the CLI, while Install-TraceblocCli is deliberately non-fatal. So a machine whose CLI install FAILED was still marked complete and never retried: permanently CLI-less, not merely stale. The CLI's own nudge cannot rescue either case. It landed in v0.10.0, is nudge-only, needs an interactive TTY, and is skipped under CI / without a config dir — and by definition cannot reach anyone below v0.10.0, which is exactly the population at risk. Both fast paths now check the version, with 0.10.0 as the floor: the release from which the CLI can keep itself current. It is a FLOOR, not a "must be latest", so it costs no network call per run and never needs raising. Below it -> degraded/cli-outdated (bash) / fall through to Install-TraceblocCli (Windows), which upgrades it once. Both fail OPEN on an unreadable version — that is not evidence of staleness, and reinstalling the CLI on every run would be worse than the staleness. _version_lt is self-contained: no `sort -V` (BSD sort predates it) and no jq, and it orders 0.9.9 below 0.10.0 rather than lexically. Either fix alone leaves a platform broken, so they land together. 988 bats + 670 Pester green; the fast-path guards on both sides are mutation-real. Closes #707 Refs tracebloc/backend#1920 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
tracebloc-release-train
Bot
added
gate-nudge
…175 (client#703) (#704) * fix(macos): give install_macos the Tier 0 path Linux has had since #1175 A Mac where a container runtime is already installed AND running is classified Tier 0 — "zero root, no privileged steps" — and step b then ran the admin gate and primed sudo anyway, demanding an administrator password to install a runtime that was already installed and answering. A field report died exactly there: Host check Container runtime Docker 29.7.2 — docker info OK ✓ → Install tier Tier 0 (zero root) — a container is already runnable. ... step b: install_macos starting (OS=Darwin ARCH=arm64 tier=0) step b: admin check passed <- nothing after this install_linux has short-circuited Tier 0 since RFC 0001 #1175; install_macos was a flat sequence with no tier branch at all, and install_macos_cli_tools hardcoded /usr/local/bin + sudo under the comment "macOS has no Tier/rootless model" — so even skipping the admin gate would still have prompted for a password to write the tools. Tier 0 now skips the admin gate, sudo priming, Homebrew and the Docker Desktop install, and lands the pinned tools in ~/.local/bin with no sudo (the same target _set_tools_target picks on Linux), then persists it on PATH via _persist_tools_on_path — which self-gates on that directory and is already macOS-aware. It still verifies amd64 emulation and still sets up login autostart, both of which are genuinely needed and neither of which wants admin. Skipping the admin gate is deliberate, not incidental: Tier 0 is precisely the case RFC 0001 opened up — a user with NO administrator rights on a machine where someone else already provisioned the runtime. That user could not install at all before this. Every other tier is byte-identical, including an UNSET INSTALL_TIER (a stale bootstrap that never fetched probe.sh), which keeps the privileged path. Seven bats tests; the four that matter are mutation-real — removing either the install_macos branch or the cli-tools target branch fails them. 985 bats green. Closes #703 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(macos): Tier 0 autostart makes no sudo call (Bugbot, client#704) The Tier 0 path prints "no administrator rights needed", then called _install_macos_autostart, whose headless branch writes a system LaunchDaemon via sudo mkdir / sudo tee / sudo launchctl. On a headless Mac (SSH with a TTY) that prompts for a password — the exact step-b failure Tier 0 exists to remove — and a Ctrl-C at the prompt leaves the install half-done before cluster create. _install_macos_autostart now takes an optional "no-sudo" argument. In that mode the GUI LaunchAgent path is unchanged (it never needed sudo and is the correct macOS analogue of a user-level autostart), but the headless LaunchDaemon path — the only branch that needs root — is skipped with instructions on how to enable reboot autostart later, instead of calling sudo. The Tier 0 caller passes "no-sudo"; the privileged path is byte-identical and still installs the daemon. This mirrors Linux, whose Tier 0 already stays out of privileged autostart. Four new mutation-real bats tests (24 green): headless no-sudo makes zero sudo calls and writes no daemon; GUI no-sudo still installs the user LaunchAgent; the Tier 0 caller passes no-sudo; the privileged caller does not. shellcheck clean at error severity; scripts/manifest.sha256 regenerated (R8). Bugbot, client#704. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(macos): headless Tier 0 names colima in the reboot footer, not Docker Desktop (Bugbot, client#704) Headless Tier 0 skips the boot LaunchDaemon rather than prompt for the password Tier 0 exists to avoid, so TB_MACOS_AUTOSTART stays unset. But _reboot_note's not-configured branch is the macOS/Windows GUI fallback: it printed 'open Docker Desktop to bring tracebloc back'. On a headless Mac that names a runtime which is not what runs here, an action there is no desktop to perform, and it directly contradicts the 'colima start' hint the skip printed moments earlier. That footer is the LAST line of a successful install, so it is the advice the operator actually leaves with. Set TB_MACOS_HEADLESS_NO_AUTOSTART at the skip site and give _reboot_note a branch for it. A configured autostart still wins, so a stale marker cannot downgrade a real promise. Tests cover both summary branches, that the skip path actually sets the marker (otherwise the new branch is unreachable and the tests are theatre), and that a GUI session does not. The existing golden 'open Docker Desktop' assertion is unchanged. manifest.sha256 regenerated; make check green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(macos): resolve the headless recovery command, don't assume colima (Bugbot + @saadqbal, client#704) My previous commit fixed the footer naming Docker Desktop on a headless box, but replaced it with an unconditional 'colima start' — the same defect in the other direction, and inconsistent with the privileged path ten lines below, which already resolves `command -v colima` and softens to a generic message when it is absent (#430 Bugbot). Three cases now: * boot daemon ALREADY installed — Tier 0 means someone else provisioned the box, so a prior admin install may have left it. Skipping the write is still right (we hold no sudo), but claiming manual recovery would be false. Set TB_MACOS_AUTOSTART and return 0. * colima resolvable — name 'colima start', as before. * colima absent — say 'start your Docker runtime manually' and name nothing. The summary reads the command the skip site resolved against this host rather than hardcoding one, so the two can't disagree. Tests for all three, plus the generic summary branch. The no-colima test overrides `command -v` for colima alone rather than emptying PATH, which would also remove date(1) via log and test the harness instead of the branch. 31/31 lifecycle, 28/28 summary, hygiene 18/18, make check green, manifest regenerated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Shujaat Hasan <shujaat@tracebloc.io>
…lient pin watchable (#714) * fix(chart): un-vacuum the order-of-ops assertion, and make the mysqlClient pin watchable Two Bugbot Mediums on the staging promotion PR #713. 1. ORDER-OF-OPS ASSERTION WAS VACUOUS. It required `kubectl rollout restart` before `kubectl annotate` -- but #569 REMOVED that command, and a sibling notMatchRegex forbids it on a command line. So the only text it could still match was the header comments explaining the old mechanism, and it passed regardless of what order the real commands ran in. Measured on the rendered script rather than argued: with the post-re-image annotates hoisted above the first `kubectl set image` -- the actual regress, header comments left in place -- the OLD assertion still passes (True) and the NEW one fails (False). On the real script the new one passes. So a regress that recorded the digest before re-imaging would have frozen every workload on its old image with a green suite. Now anchored to COMMAND lines and to the mechanism that exists: `set image` -> `rollout status` -> `annotate`. (Two earlier attempts at that mutation were themselves broken -- one hoisted the annotates above the comments too, one popped by stale indices after inserting. Both "proved" the wrong thing. The mutation above is verified to have actually moved the lines before its result is quoted.) 2. mysqlClient PIN WAS NEVER WATCHED. `images.mysqlClient` carried a real digest but no `repository` and an empty `tag`, because both live as template defaults. check-digest-drift.sh pairs a pin with the repo/float in its own block, found neither, and classified it UNWATCHABLE -- which exits non-zero, so the DAILY DRIFT WATCH STAYS RED FOREVER while the pin it exists to compare is never checked. A guard that cannot pass teaches everyone to ignore it. Stated in values and consumed by the template, so the value is real config rather than decoration. Both are what the templates already defaulted to. Verified: * render byte-identical across aks/bm/eks values (only POD_TOKEN_SIGNING_SECRET differs, which helm regenerates every run); mysql image line unchanged * helm lint clean; helm unittest 36/36 including the edited suite * drift watch: BEFORE "UNWATCHABLE: images.mysqlClient is pinned to sha256:f546…" AFTER "ok tracebloc/mysql-client:prod sha256:f546e47fb339…" * test(image-refresh): pin the order guard to the digest annotate, not any annotate The order-of-ops assertion only required SOME `kubectl annotate deployment` after set-image/rollout-status. The flap-counter-reset annotate (`${ATTEMPT_KEY}- ${FLAP_KEY}-`) already sits there by design, so moving ONLY the digest-recording annotate (`$annotate_args`) above the re-image kept the suite green while the digest was recorded BEFORE the image was applied — the freeze path this guard exists to catch. Require the `$annotate_args` continuation line so the ordering is tied to the digest write itself, distinct from the flap reset. Verified by mutation: the loose pattern stayed green with the digest annotate moved above set-image; the tightened one fails. Bugbot, client#714. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(chart): bump to 1.9.40 for the chart-content changes in this PR The version-bump-gate (scripts/chart-version-guard.sh) requires a Chart.yaml version bump when client/templates|values change, since the Helm repo only publishes a NEW version — an unbumped edit ships dark or overwrites a published one. develop is at 1.9.39; bump to 1.9.40. backend#1468 unrelated — this is client#714. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…end#1729 sweep 5) (#706) * ci(env): one CLIENT_ENV vocabulary, four languages, now checked (backend#1729 sweep 5) Sweep 5 of the inert-verification epic. The epic states the class as: "A verification written in the same vocabulary as the thing it verifies cannot detect a vocabulary error." and its evidence was 366 chart tests covering dev/stg/prod/unset/unknown while NOT ONE set `staging` -- the alias the chart's own docs recommend. That measured gap is now closed (staging has 9 cases, development 2, production 4). What is NOT closed is the structure underneath it: the same three alias->canonical mappings are declared FOUR times, in four languages, and nothing compares them. 1 client/templates/_helpers.tpl $aliases := dict ... Go template 2 client/values.schema.json the CLIENT_ENV enum JSON Schema 3 scripts/lib/common.sh tb_client_env() bash case 4 scripts/install-k8s.ps1 Get-TraceblocClientEnv PowerShell switch Two of the four have already drifted, separately, and been repaired separately: backend#1723 fixed the chart, backend#1745 fixed the bash installer -- whose own comment records the cost, "a raw `staging` fell through to the prod branch, so verify_credentials() checked staging credentials against the production backend and reported them invalid". So adding a seventh spelling to the template leaves both installers silently not reducing it, which is #1745 reintroduced in a repo that has already paid for it once. THE GUARD DERIVES, IT DOES NOT RESTATE. It parses all four declarations and compares them to each other; it holds no copy of the vocabulary, because a fifth hand-written list is the defect rather than the fix (the lesson of backend#1780 and backend#1828, where hand-copied declarations each claimed the others kept them honest and nothing crossed the boundary). It also asserts every accepted spelling is exercised by at least one helm-unittest case -- the specific thing #1729 measured. ARMED WHILE GREEN, deliberately: all four agree today and all six spellings are tested, so this imports no backlog. Arming a red check trains people to skip the tier -- the same reasoning as .github#235 and the opposite of what happened when a fleet-wide copies: bump reddened the org audit for 2h20m on 2026-08-12. Mutations, all five behaving correctly: template gains a 4th alias, installers unchanged 3 findings bash reducer drops staging (the #1745 shape) 1 finding PowerShell maps staging -> dev 1 finding schema accepts a spelling no reducer maps 2 findings a parser goes stale (dict renamed) EXIT 2, fail-closed That last one matters most: zero parsed pairs compares equal to zero parsed pairs, so a stale parser would report agreement between four declarations it never read. It exits 2 with a diagnostic instead. Wired into SHELLCHECK_FILES and `make check` beside its sibling, and into helm-ci with path filters covering all four declarations -- including scripts/lib/common.sh and scripts/install-k8s.ps1, so a change to either installer runs it. make check green; chart-env-vocabulary 28/28; gen-manifest --check clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(env-check): don't misreport a missing python3 as a closed-vocabulary gap env-vocabulary-agreement.sh reads the CLIENT_ENV enum out of values.schema.json with python3, but `make setup` never installed (or even checked for) python3, so on the pre-push `make check` path a missing interpreter failed closed with the false diagnosis that the schema has no CLIENT_ENV enum and the vocabulary is no longer closed. The Go/bash/PowerShell reducers are jq-free by rule, so python3 is the JSON parser here and was the one unguarded dependency. Preflight `command -v python3` before the enum is read, and at the call site branch on the helper's exit status: 3 (its sys.exit for a genuinely-absent enum) still reports the real closed-vocabulary finding, while any other non-zero (a python3 that fails to run, malformed JSON) reports a distinct tooling/parse error. Add python3 to `make setup`'s prereq check so the gap is caught up front. Bugbot, client#706. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * ci(env): shellcheck env-vocabulary-agreement in the static job The Makefile's SHELLCHECK_FILES already lists env-vocabulary-agreement.sh, but installer-tests.yaml's static job shellchecked only through chart-env-vocabulary.sh, so CI never linted the new guard. Append it to both the error-gate and warning-advisory invocations so the workflow list matches the Makefile again (same files, same order). Bugbot, client#706. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 3e97d10. Configure here.
tracebloc-release-train
Bot
added
gate-nudge
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated promotion by the release train (RFC-0008 D14). Head is the train-managed
release-train/to-stagingbranch (a mirror ofdevelop), so it never collides with a human PR. Merged only when the fr-gate is green.Note
High Risk
Changes how control-plane images update and when pods pull from the registry, adds cross-namespace RBAC for image-refresh, and documents live ingestor float/pin drift—the scheduled drift job may fail until pins are reconciled.
Overview
This release train bundles several operational and chart changes, led by #569 rework of control-plane image lifecycle.
Image-refresh now reconciles floating tags with
kubectl set image repo@digest(replacing rollout restart) so jobs-manager, pods-monitor, requests-proxy (same image as jobs-manager), and resource-monitor can useIfNotPresentwhere digest reconcile or explicit pins provide an update path, while mirrors andimageRefresh.enabled: falsekeepAlwaysso edges are not left frozen. New helpers unify pull policy, resource-monitor “pinned” semantics, and workload names; node-agents RBAC and DaemonSetmaxUnavailable: 10%support cross-namespace refresh without blowing the Job deadline (defaultactiveDeadlineSeconds: 3600).Supply-chain watching:
scripts/check-digest-drift.shplus a dailydigest-driftworkflow compare every chart pin to its mutable float (ingestor0.8vsprodDigest, etc.); CI/Makefile wire vocabulary agreement and the new bats suite.docs/SECURITY.mdand values comments document that the prod float now resolves past the safe ingestor ceiling (v0.8.8).Installers: macOS Tier 0 skips admin/Docker when a runtime already works (CLI to
~/.local/bin); bash/Windows assess paths treat CLI < 0.10.0 as degraded; ERR recording is last-wins with re-entrancy guards. ClusterRole gainslist nodesfor jobs-manager GPU→CPU pending fallback (backend#1876). Chart version 1.9.38 → 1.9.40.Reviewed by Cursor Bugbot for commit 3e97d10. Bugbot is set up for automated code reviews on this repo. Configure here.