fix(installer): fail fast on k3d spawn failure; bound the create wait - #439
Conversation
…#412, #426 Windows half) Two fixes to New-K3dCluster's create path, both observed live: 1. Start-Process failure ('%1 is not a valid Win32 application' from a broken k3d.exe) is non-terminating by default, leaving $k3dProc null — and 'while (-not $null.HasExited)' is always true, so the installer spun 'Creating compute environment...' forever over a dead install. The spawn now runs under -ErrorAction Stop in a try/catch that cleans up the temp logs + proxy config and fails with the real exception, the log path, and a remedy. 2. k3d cluster create --wait has no timeout of its own, so a stalled image pull spun the spinner indefinitely. The wait is now bounded (15 min default, TB_CREATE_TIMEOUT_MIN override): on expiry the process is killed, the last stderr lines and the install-log path are printed, and the install fails loudly. Extracted as Wait-ProcessWithDeadline so the deadline/kill path is unit-tested. Closes #412. Windows half of #426 (the bash half — k3d create + helm timeouts in cluster.sh / install-client-helm.sh — follows separately). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
bugbot run |
# Conflicts: # scripts/manifest.sha256
… out (Bugbot #439) Killing k3d mid --wait skips its own rollback, so the timeout path left a half-created cluster behind — and the next run's reuse path would see serversRunning > 0 and print 'Compute environment already running' over a broken environment. The timeout path now deletes the partial cluster (bounded at 2 min via Wait-ProcessWithDeadline) before failing, and tells the operator the exact manual command if the delete itself fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
bugbot run |
# Conflicts: # scripts/manifest.sha256
|
bugbot run |
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit a8a38c4. Configure here.
… (Bugbot #442 r2) 1. spin's deadline path captured no child PIDs, so once the TERM'd wrapper died, its children reparented to init and the pkill -KILL -P sweep found nothing — a TERM-immune k3d could survive the backstop. Child PIDs are now captured BEFORE any signal and the KILL sweep addresses them directly; test proves a trap-''-TERM child dies. 2. When the create backstop fires, the k3d log is often empty (hung daemon) — the operator saw a bare failure with no timeout hint. The 124 path now names the timeout, points at TB_CREATE_TIMEOUT_MIN, and deletes the partially created cluster (bounded) so a re-run cannot adopt it via the 'already exists' branch — parity with the Windows fix on #439. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
shujaatTracebloc
left a comment
There was a problem hiding this comment.
Reviewed the code and ran the full Pester suite locally at HEAD (a8a38c41) with the exact CI invocation (TB_PESTER=1, whole scripts/tests dir) on pwsh 7.6.4 / Pester 6.0.1: 173 passed, 0 failed, 8 skipped (the 8 skips are Windows-only CIM/drive probes, unrelated to this change). All 5 new tests in the Bounded cluster-create wait (#412 / #426) block pass, and the manifest integrity gate re-confirms the regenerated manifest.sha256.
The change is sound:
- #412 fixed — spawn wrapped in
try/catch+-ErrorAction Stop, fails fast instead of spinning over a null $k3dProc. - #426 (Windows half) fixed — create wait bounded at 15 min via
Wait-ProcessWithDeadline(TB_CREATE_TIMEOUT_MINoverride). - Bugbot 'Timeout orphans cluster on re-run' addressed — bounded
k3d cluster deleteon timeout, with a clear operator-warning fallback; thread resolved and latest Bugbot run clean. - Cleanup of temp logs + the proxy-cred dir is consistent across all three exit branches.
Non-blocking nits (fine to defer): the timeout-path k3d cluster delete spawn doesn't redirect stdout/stderr, so its output can interleave with the spinner (cosmetic); TB_CREATE_TIMEOUT_MIN=0 is accepted → immediate timeout (no floor).
Merge note: land #438 → #439 in order (#438 should be retargeted to develop now that #437 is merged).
LGTM 🚀
|
Ran the full Pester suite locally at HEAD ( All 5 new tests in the Bounded cluster-create wait (#412 / #426) block pass, and the manifest integrity gate re-confirms the regenerated Note: the installer test suite doesn't run in CI here because this PR targets the |
…#439) * fix(installer): fail fast on k3d spawn failure; bound the create wait (#412, #426 Windows half) Two fixes to New-K3dCluster's create path, both observed live: 1. Start-Process failure ('%1 is not a valid Win32 application' from a broken k3d.exe) is non-terminating by default, leaving $k3dProc null — and 'while (-not $null.HasExited)' is always true, so the installer spun 'Creating compute environment...' forever over a dead install. The spawn now runs under -ErrorAction Stop in a try/catch that cleans up the temp logs + proxy config and fails with the real exception, the log path, and a remedy. 2. k3d cluster create --wait has no timeout of its own, so a stalled image pull spun the spinner indefinitely. The wait is now bounded (15 min default, TB_CREATE_TIMEOUT_MIN override): on expiry the process is killed, the last stderr lines and the install-log path are printed, and the install fails loudly. Extracted as Wait-ProcessWithDeadline so the deadline/kill path is unit-tested. Closes #412. Windows half of #426 (the bash half — k3d create + helm timeouts in cluster.sh / install-client-helm.sh — follows separately). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): remove the partial cluster when the create wait times out (Bugbot #439) Killing k3d mid --wait skips its own rollback, so the timeout path left a half-created cluster behind — and the next run's reuse path would see serversRunning > 0 and print 'Compute environment already running' over a broken environment. The timeout path now deletes the partial cluster (bounded at 2 min via Wait-ProcessWithDeadline) before failing, and tells the operator the exact manual command if the delete itself fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…447) * fix(installer): fail fast on k3d spawn failure; bound the create wait (#439) * fix(installer): fail fast on k3d spawn failure; bound the create wait (#412, #426 Windows half) Two fixes to New-K3dCluster's create path, both observed live: 1. Start-Process failure ('%1 is not a valid Win32 application' from a broken k3d.exe) is non-terminating by default, leaving $k3dProc null — and 'while (-not $null.HasExited)' is always true, so the installer spun 'Creating compute environment...' forever over a dead install. The spawn now runs under -ErrorAction Stop in a try/catch that cleans up the temp logs + proxy config and fails with the real exception, the log path, and a remedy. 2. k3d cluster create --wait has no timeout of its own, so a stalled image pull spun the spinner indefinitely. The wait is now bounded (15 min default, TB_CREATE_TIMEOUT_MIN override): on expiry the process is killed, the last stderr lines and the install-log path are printed, and the install fails loudly. Extracted as Wait-ProcessWithDeadline so the deadline/kill path is unit-tested. Closes #412. Windows half of #426 (the bash half — k3d create + helm timeouts in cluster.sh / install-client-helm.sh — follows separately). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): remove the partial cluster when the create wait times out (Bugbot #439) Killing k3d mid --wait skips its own rollback, so the timeout path left a half-created cluster behind — and the next run's reuse path would see serversRunning > 0 and print 'Compute environment already running' over a broken environment. The timeout path now deletes the partial cluster (bounded at 2 min via Wait-ProcessWithDeadline) before failing, and tells the operator the exact manual command if the delete itself fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * fix(installer): Docker engine wait — 10 minutes, elapsed progress, named failure state (#440) * fix(installer): Docker engine wait — 10 minutes, elapsed progress, named failure state (#413) A first-ever Docker Desktop start on AV-heavy corporate machines routinely needs 5-10 minutes (WSL bootstrap, image unpack). The old 3-minute cap turned a normal cold start into a failed install plus a manual re-paste of the one-liner — observed as a recurring wait-then- re-run loop on hospital installs. - Wait bound: 3 min -> 10 min default, TB_DOCKER_WAIT_MIN override. - After the first minute the spinner shows elapsed minutes and the expected worst case, so the wait doesn't read as a hang. - On expiry the failure names the observed state: Docker Desktop process gone (start it / fix its error window) vs. running with the engine still down (tray-icon guidance) - instead of one generic line. Closes #413 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): exited-Docker path drops the slow-start advice (Bugbot #440) When Docker Desktop's process has exited, 'a first start can be slow' and the TB_DOCKER_WAIT_MIN hint contradict the diagnosis and steer operators toward raising the wait instead of restarting/fixing the crash. The slow-start reassurance + override hint now print only on the engine-still-starting path; the exited path fails with its own start-and-fix remedy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…angs (#426, bash half) k3d cluster create --wait had no deadline of its own and its spinner none either: a stalled image pull (rate-limited registry, TLS-intercepting proxy) span the create forever. Both helm invocations (install + reconcile) ran under the deadline-less spin_cmd and could hang the same way against a wedged kube-apiserver. - create: --wait now always pairs with --timeout (default 15m, TB_CREATE_TIMEOUT_MIN override — the same env knob the Windows installer adopted); k3d aborts with a real error that the existing failure path dumps. spin() gained an optional deadline, used as a +5min backstop in case k3d itself wedges past its own timeout. - helm install + reconcile: new spin_cmd_bounded (spin_cmd + hard deadline, rc 124 with an explicit timeout note + log tail), default 10m via TB_HELM_TIMEOUT_MIN. Completes #426 (the Windows half shipped with the #439 stack). Closes #426 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (Bugbot #442 r2) 1. spin's deadline path captured no child PIDs, so once the TERM'd wrapper died, its children reparented to init and the pkill -KILL -P sweep found nothing — a TERM-immune k3d could survive the backstop. Child PIDs are now captured BEFORE any signal and the KILL sweep addresses them directly; test proves a trap-''-TERM child dies. 2. When the create backstop fires, the k3d log is often empty (hung daemon) — the operator saw a bare failure with no timeout hint. The 124 path now names the timeout, points at TB_CREATE_TIMEOUT_MIN, and deletes the partially created cluster (bounded) so a re-run cannot adopt it via the 'already exists' branch — parity with the Windows fix on #439. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…angs (bash half of #426) (#442) * fix(installer): bound k3d create and the helm calls — no indefinite hangs (#426, bash half) k3d cluster create --wait had no deadline of its own and its spinner none either: a stalled image pull (rate-limited registry, TLS-intercepting proxy) span the create forever. Both helm invocations (install + reconcile) ran under the deadline-less spin_cmd and could hang the same way against a wedged kube-apiserver. - create: --wait now always pairs with --timeout (default 15m, TB_CREATE_TIMEOUT_MIN override — the same env knob the Windows installer adopted); k3d aborts with a real error that the existing failure path dumps. spin() gained an optional deadline, used as a +5min backstop in case k3d itself wedges past its own timeout. - helm install + reconcile: new spin_cmd_bounded (spin_cmd + hard deadline, rc 124 with an explicit timeout note + log tail), default 10m via TB_HELM_TIMEOUT_MIN. Completes #426 (the Windows half shipped with the #439 stack). Closes #426 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): spin's deadline kills the wrapper's children too (Bugbot #442) The create backstop passes the `( k3d … )` wrapper-subshell PID into spin, and the deadline path signalled only that PID — orphaning k3d, which kept creating the cluster after the installer had already exited as failed, racing any retry. The deadline path now TERMs the children while the parent is still alive (afterwards they reparent to init and pkill -P can't see them), then the wrapper, with a KILL sweep after the grace period. Harmless when bash exec-optimizes the wrapper away. Test proves the child dies with the wrapper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): reparent-safe KILL sweep + explicit create-timeout UX (Bugbot #442 r2) 1. spin's deadline path captured no child PIDs, so once the TERM'd wrapper died, its children reparented to init and the pkill -KILL -P sweep found nothing — a TERM-immune k3d could survive the backstop. Child PIDs are now captured BEFORE any signal and the KILL sweep addresses them directly; test proves a trap-''-TERM child dies. 2. When the create backstop fires, the k3d log is often empty (hung daemon) — the operator saw a bare failure with no timeout hint. The 124 path now names the timeout, points at TB_CREATE_TIMEOUT_MIN, and deletes the partially created cluster (bounded) so a re-run cannot adopt it via the 'already exists' branch — parity with the Windows fix on #439. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): spin's deadline path is set -e-proof (Bugbot #442 r3) Bare pkill (returns 1 with no children), kill on a reaped pid, and wait after a kill (reports the signal) could each abort the deadline path before 'return 124' under the installer's set -e — callers would see 143/1: no timeout copy, no partial-cluster cleanup, no bounded note. Every signal step is now failure-proofed, and a bash -c 'set -euo pipefail' end-to-end test pins the 124 contract for the childless (exec-optimized) case that reproduces the abort. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): helm timeout names the pending-release unwedge (Bugbot #442 r4) A SIGKILLed helm (spin_cmd_bounded deadline) can leave the release wedged as pending-install/pending-upgrade — the next run then fails with Helm's 'another operation is in progress' and no guidance. Both call sites now capture the rc (if-! discarded it) and, on 124, print the exact unwedge commands (uninstall for a half-installed release, rollback for an upgrade) before the error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(installer): fix over-broad grep assertions on the helm-bound tests Count invocation/hint lines, not comment mentions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): rollback hint names the release, not the namespace (Bugbot #442 r5) The adopt path tracks release ($_rel) and namespace ($_ns) separately; the reconcile-timeout unwedge hint said 'helm rollback $_ns', which is a non-working command whenever they differ. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(installer): base-10-normalize the timeout knobs — octal trap defused (Bugbot #442 r6) TB_CREATE_TIMEOUT_MIN / TB_HELM_TIMEOUT_MIN sanitizers accepted 08/09, which bash arithmetic reads as invalid octal — aborting $(( … )) under set -e mid-create (k3d already backgrounded → partial cluster left behind), and 010 silently became 8. One shared, unit-tested helper (tb_minutes_or) now normalizes via 10# at all three call sites. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Summary
Fixes the infinite "Creating compute environment..." spinner observed live on a Windows test machine (2026-07-27), and bounds the create wait on top of it.
Bug 1 — null-process spin (#412): when
Start-Processfork3d cluster createfails (e.g. a brokenk3d.exe:%1 is not a valid Win32 application), the error is non-terminating,$k3dProcstays$null, andwhile (-not $null.HasExited)is always true → live spinner over a dead install, forever.Bug 2 — unbounded wait (#426, Windows half):
k3d cluster create --waithas no--timeout, so a stalled image pull (rate-limited registry, TLS-intercepting proxy) also spins forever — even with a healthy process object.Change
-ErrorAction Stop+ try/catch: temp logs and the proxy-config dir (contains proxy credentials) are cleaned up, then a real error with the exception text, install-log path, and remedy.Wait-ProcessWithDeadline(spinner + deadline + best-effort kill, unit-testable): create wait bounded at 15 min (TB_CREATE_TIMEOUT_MINoverride). On expiry: last 5 lines of k3d stderr, the install-log path, cleanup, loud failure.HasExitedloop stays gone.manifest.sha256regenerated.The bash half of #426 (same missing bounds in
cluster.sh+install-client-helm.sh) comes as its own PR — different test framework, same pattern.Stacked on #438 → #437 — merge in that order; this branch contains both.
Type
Bug fix
Test plan
Full Pester suite locally on pwsh 7.5 (CI invocation): 169 passed, 0 failed, 8 skipped (4 new).
gen-manifest.sh --checkclean.🤖 Generated with Claude Code
Note
Low Risk
Changes are confined to the Windows install script’s k3d create/wait path with new unit tests; no auth, data, or production runtime behavior is affected.
Overview
Fixes two Windows installer hangs during k3d cluster create: a failed
Start-Processleft$k3dProcnull and an infinite spinner (#412), andk3d cluster create --waithad no upper bound on stalled image pulls (#426).Spawn path:
Start-Processnow uses-ErrorAction Stopwith try/catch so a brokenk3d.exeexits immediately after cleaning temp logs and proxy config, with a clear error instead of spinning forever.Bounded wait: Cluster creation uses new
Wait-ProcessWithDeadline(spinner, deadline, best-effort kill). Default timeout is 15 minutes, overridable viaTB_CREATE_TIMEOUT_MIN. On timeout the installer surfaces the last lines of k3d stderr/stdout, cleans up, runs a 2-minute-boundedk3d cluster deleteto avoid leaving a partial cluster that the next run would reuse (#439), then fails with remediation hints.Pester covers deadline/kill behavior and script-level regression gates;
manifest.sha256is updated forinstall-k8s.ps1.Reviewed by Cursor Bugbot for commit a8a38c4. Bugbot is set up for automated code reviews on this repo. Configure here.