Skip to content

fix(installer): fail fast on k3d spawn failure; bound the create wait - #439

Merged
LukasWodka merged 4 commits into
fix/410-pin-k3d-helmfrom
fix/412-create-deadline
Jul 27, 2026
Merged

fix(installer): fail fast on k3d spawn failure; bound the create wait#439
LukasWodka merged 4 commits into
fix/410-pin-k3d-helmfrom
fix/412-create-deadline

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the infinite "Creating compute environment..." spinner observed live on a Windows test machine (2026-07-27), and bounds the create wait on top of it.

Bug 1 — null-process spin (#412): when Start-Process for k3d cluster create fails (e.g. a broken k3d.exe: %1 is not a valid Win32 application), the error is non-terminating, $k3dProc stays $null, and while (-not $null.HasExited) is always true → live spinner over a dead install, forever.

Bug 2 — unbounded wait (#426, Windows half): k3d cluster create --wait has no --timeout, so a stalled image pull (rate-limited registry, TLS-intercepting proxy) also spins forever — even with a healthy process object.

Change

  • Spawn under -ErrorAction Stop + try/catch: temp logs and the proxy-config dir (contains proxy credentials) are cleaned up, then a real error with the exception text, install-log path, and remedy.
  • New Wait-ProcessWithDeadline (spinner + deadline + best-effort kill, unit-testable): create wait bounded at 15 min (TB_CREATE_TIMEOUT_MIN override). On expiry: last 5 lines of k3d stderr, the install-log path, cleanup, loud failure.
  • Pester: deadline/kill behavior on fake processes, spawn-fails-fast gate, and a regression gate asserting the unbounded HasExited loop stays gone.
  • manifest.sha256 regenerated.

The bash half of #426 (same missing bounds in cluster.sh + install-client-helm.sh) comes as its own PR — different test framework, same pattern.

Stacked on #438#437 — merge in that order; this branch contains both.

Type

Bug fix

Test plan

Full Pester suite locally on pwsh 7.5 (CI invocation): 169 passed, 0 failed, 8 skipped (4 new). gen-manifest.sh --check clean.

🤖 Generated with Claude Code


Note

Low Risk
Changes are confined to the Windows install script’s k3d create/wait path with new unit tests; no auth, data, or production runtime behavior is affected.

Overview
Fixes two Windows installer hangs during k3d cluster create: a failed Start-Process left $k3dProc null and an infinite spinner (#412), and k3d cluster create --wait had no upper bound on stalled image pulls (#426).

Spawn path: Start-Process now uses -ErrorAction Stop with try/catch so a broken k3d.exe exits immediately after cleaning temp logs and proxy config, with a clear error instead of spinning forever.

Bounded wait: Cluster creation uses new Wait-ProcessWithDeadline (spinner, deadline, best-effort kill). Default timeout is 15 minutes, overridable via TB_CREATE_TIMEOUT_MIN. On timeout the installer surfaces the last lines of k3d stderr/stdout, cleans up, runs a 2-minute-bounded k3d cluster delete to avoid leaving a partial cluster that the next run would reuse (#439), then fails with remediation hints.

Pester covers deadline/kill behavior and script-level regression gates; manifest.sha256 is updated for install-k8s.ps1.

Reviewed by Cursor Bugbot for commit a8a38c4. Bugbot is set up for automated code reviews on this repo. Configure here.

…#412, #426 Windows half)

Two fixes to New-K3dCluster's create path, both observed live:

1. Start-Process failure ('%1 is not a valid Win32 application' from a
   broken k3d.exe) is non-terminating by default, leaving $k3dProc null —
   and 'while (-not $null.HasExited)' is always true, so the installer
   spun 'Creating compute environment...' forever over a dead install.
   The spawn now runs under -ErrorAction Stop in a try/catch that cleans
   up the temp logs + proxy config and fails with the real exception,
   the log path, and a remedy.

2. k3d cluster create --wait has no timeout of its own, so a stalled
   image pull spun the spinner indefinitely. The wait is now bounded
   (15 min default, TB_CREATE_TIMEOUT_MIN override): on expiry the
   process is killed, the last stderr lines and the install-log path are
   printed, and the install fails loudly. Extracted as
   Wait-ProcessWithDeadline so the deadline/kill path is unit-tested.

Closes #412. Windows half of #426 (the bash half — k3d create + helm
timeouts in cluster.sh / install-client-helm.sh — follows separately).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka
LukasWodka requested a review from saadqbal as a code owner July 27, 2026 13:53
@LukasWodka LukasWodka self-assigned this Jul 27, 2026
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

Comment thread scripts/install-k8s.ps1
… out (Bugbot #439)

Killing k3d mid --wait skips its own rollback, so the timeout path left
a half-created cluster behind — and the next run's reuse path would see
serversRunning > 0 and print 'Compute environment already running' over
a broken environment. The timeout path now deletes the partial cluster
(bounded at 2 min via Wait-ProcessWithDeadline) before failing, and
tells the operator the exact manual command if the delete itself fails.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a8a38c4. Configure here.

LukasWodka added a commit that referenced this pull request Jul 27, 2026
… (Bugbot #442 r2)

1. spin's deadline path captured no child PIDs, so once the TERM'd
   wrapper died, its children reparented to init and the pkill -KILL -P
   sweep found nothing — a TERM-immune k3d could survive the backstop.
   Child PIDs are now captured BEFORE any signal and the KILL sweep
   addresses them directly; test proves a trap-''-TERM child dies.

2. When the create backstop fires, the k3d log is often empty (hung
   daemon) — the operator saw a bare failure with no timeout hint. The
   124 path now names the timeout, points at TB_CREATE_TIMEOUT_MIN, and
   deletes the partially created cluster (bounded) so a re-run cannot
   adopt it via the 'already exists' branch — parity with the Windows
   fix on #439.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@shujaatTracebloc shujaatTracebloc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the code and ran the full Pester suite locally at HEAD (a8a38c41) with the exact CI invocation (TB_PESTER=1, whole scripts/tests dir) on pwsh 7.6.4 / Pester 6.0.1: 173 passed, 0 failed, 8 skipped (the 8 skips are Windows-only CIM/drive probes, unrelated to this change). All 5 new tests in the Bounded cluster-create wait (#412 / #426) block pass, and the manifest integrity gate re-confirms the regenerated manifest.sha256.

The change is sound:

  • #412 fixed — spawn wrapped in try/catch + -ErrorAction Stop, fails fast instead of spinning over a null $k3dProc.
  • #426 (Windows half) fixed — create wait bounded at 15 min via Wait-ProcessWithDeadline (TB_CREATE_TIMEOUT_MIN override).
  • Bugbot 'Timeout orphans cluster on re-run' addressed — bounded k3d cluster delete on timeout, with a clear operator-warning fallback; thread resolved and latest Bugbot run clean.
  • Cleanup of temp logs + the proxy-cred dir is consistent across all three exit branches.

Non-blocking nits (fine to defer): the timeout-path k3d cluster delete spawn doesn't redirect stdout/stderr, so its output can interleave with the spinner (cosmetic); TB_CREATE_TIMEOUT_MIN=0 is accepted → immediate timeout (no floor).

Merge note: land #438#439 in order (#438 should be retargeted to develop now that #437 is merged).

LGTM 🚀

@LukasWodka
LukasWodka merged commit 3b6d9e3 into fix/410-pin-k3d-helm Jul 27, 2026
4 checks passed
@shujaatTracebloc

Copy link
Copy Markdown
Contributor

Ran the full Pester suite locally at HEAD (a8a38c41) with the exact CI invocation (TB_PESTER=1, whole scripts/tests dir) on pwsh 7.6.4 / Pester 6.0.1:

Tests Passed: 173, Failed: 0, Skipped: 8, Inconclusive: 0

All 5 new tests in the Bounded cluster-create wait (#412 / #426) block pass, and the manifest integrity gate re-confirms the regenerated manifest.sha256. The 8 skips are Windows-only CIM/drive probes (Get-PfCpu/Get-PfMemGb/Get-PfVirtualization/Get-PfFsType), skipped on macOS and unrelated to this change — CI's windows-latest leg exercises them.

Note: the installer test suite doesn't run in CI here because this PR targets the fix/410-pin-k3d-helm feature branch (installer-tests.yaml only triggers on PRs to main/develop/openshift), so this is the local stand-in for that coverage. ✅

LukasWodka added a commit that referenced this pull request Jul 27, 2026
…#439)

* fix(installer): fail fast on k3d spawn failure; bound the create wait (#412, #426 Windows half)

Two fixes to New-K3dCluster's create path, both observed live:

1. Start-Process failure ('%1 is not a valid Win32 application' from a
   broken k3d.exe) is non-terminating by default, leaving $k3dProc null —
   and 'while (-not $null.HasExited)' is always true, so the installer
   spun 'Creating compute environment...' forever over a dead install.
   The spawn now runs under -ErrorAction Stop in a try/catch that cleans
   up the temp logs + proxy config and fails with the real exception,
   the log path, and a remedy.

2. k3d cluster create --wait has no timeout of its own, so a stalled
   image pull spun the spinner indefinitely. The wait is now bounded
   (15 min default, TB_CREATE_TIMEOUT_MIN override): on expiry the
   process is killed, the last stderr lines and the install-log path are
   printed, and the install fails loudly. Extracted as
   Wait-ProcessWithDeadline so the deadline/kill path is unit-tested.

Closes #412. Windows half of #426 (the bash half — k3d create + helm
timeouts in cluster.sh / install-client-helm.sh — follows separately).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): remove the partial cluster when the create wait times out (Bugbot #439)

Killing k3d mid --wait skips its own rollback, so the timeout path left
a half-created cluster behind — and the next run's reuse path would see
serversRunning > 0 and print 'Compute environment already running' over
a broken environment. The timeout path now deletes the partial cluster
(bounded at 2 min via Wait-ProcessWithDeadline) before failing, and
tells the operator the exact manual command if the delete itself fails.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Jul 27, 2026
…447)

* fix(installer): fail fast on k3d spawn failure; bound the create wait (#439)

* fix(installer): fail fast on k3d spawn failure; bound the create wait (#412, #426 Windows half)

Two fixes to New-K3dCluster's create path, both observed live:

1. Start-Process failure ('%1 is not a valid Win32 application' from a
   broken k3d.exe) is non-terminating by default, leaving $k3dProc null —
   and 'while (-not $null.HasExited)' is always true, so the installer
   spun 'Creating compute environment...' forever over a dead install.
   The spawn now runs under -ErrorAction Stop in a try/catch that cleans
   up the temp logs + proxy config and fails with the real exception,
   the log path, and a remedy.

2. k3d cluster create --wait has no timeout of its own, so a stalled
   image pull spun the spinner indefinitely. The wait is now bounded
   (15 min default, TB_CREATE_TIMEOUT_MIN override): on expiry the
   process is killed, the last stderr lines and the install-log path are
   printed, and the install fails loudly. Extracted as
   Wait-ProcessWithDeadline so the deadline/kill path is unit-tested.

Closes #412. Windows half of #426 (the bash half — k3d create + helm
timeouts in cluster.sh / install-client-helm.sh — follows separately).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): remove the partial cluster when the create wait times out (Bugbot #439)

Killing k3d mid --wait skips its own rollback, so the timeout path left
a half-created cluster behind — and the next run's reuse path would see
serversRunning > 0 and print 'Compute environment already running' over
a broken environment. The timeout path now deletes the partial cluster
(bounded at 2 min via Wait-ProcessWithDeadline) before failing, and
tells the operator the exact manual command if the delete itself fails.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): Docker engine wait — 10 minutes, elapsed progress, named failure state (#440)

* fix(installer): Docker engine wait — 10 minutes, elapsed progress, named failure state (#413)

A first-ever Docker Desktop start on AV-heavy corporate machines
routinely needs 5-10 minutes (WSL bootstrap, image unpack). The old
3-minute cap turned a normal cold start into a failed install plus a
manual re-paste of the one-liner — observed as a recurring wait-then-
re-run loop on hospital installs.

- Wait bound: 3 min -> 10 min default, TB_DOCKER_WAIT_MIN override.
- After the first minute the spinner shows elapsed minutes and the
  expected worst case, so the wait doesn't read as a hang.
- On expiry the failure names the observed state: Docker Desktop
  process gone (start it / fix its error window) vs. running with the
  engine still down (tray-icon guidance) - instead of one generic line.

Closes #413

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): exited-Docker path drops the slow-start advice (Bugbot #440)

When Docker Desktop's process has exited, 'a first start can be slow'
and the TB_DOCKER_WAIT_MIN hint contradict the diagnosis and steer
operators toward raising the wait instead of restarting/fixing the
crash. The slow-start reassurance + override hint now print only on
the engine-still-starting path; the exited path fails with its own
start-and-fix remedy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Jul 27, 2026
…angs (#426, bash half)

k3d cluster create --wait had no deadline of its own and its spinner none
either: a stalled image pull (rate-limited registry, TLS-intercepting
proxy) span the create forever. Both helm invocations (install +
reconcile) ran under the deadline-less spin_cmd and could hang the same
way against a wedged kube-apiserver.

- create: --wait now always pairs with --timeout (default 15m,
  TB_CREATE_TIMEOUT_MIN override — the same env knob the Windows
  installer adopted); k3d aborts with a real error that the existing
  failure path dumps. spin() gained an optional deadline, used as a
  +5min backstop in case k3d itself wedges past its own timeout.
- helm install + reconcile: new spin_cmd_bounded (spin_cmd + hard
  deadline, rc 124 with an explicit timeout note + log tail), default
  10m via TB_HELM_TIMEOUT_MIN.

Completes #426 (the Windows half shipped with the #439 stack).

Closes #426

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Jul 27, 2026
… (Bugbot #442 r2)

1. spin's deadline path captured no child PIDs, so once the TERM'd
   wrapper died, its children reparented to init and the pkill -KILL -P
   sweep found nothing — a TERM-immune k3d could survive the backstop.
   Child PIDs are now captured BEFORE any signal and the KILL sweep
   addresses them directly; test proves a trap-''-TERM child dies.

2. When the create backstop fires, the k3d log is often empty (hung
   daemon) — the operator saw a bare failure with no timeout hint. The
   124 path now names the timeout, points at TB_CREATE_TIMEOUT_MIN, and
   deletes the partially created cluster (bounded) so a re-run cannot
   adopt it via the 'already exists' branch — parity with the Windows
   fix on #439.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LukasWodka added a commit that referenced this pull request Jul 28, 2026
…angs (bash half of #426) (#442)

* fix(installer): bound k3d create and the helm calls — no indefinite hangs (#426, bash half)

k3d cluster create --wait had no deadline of its own and its spinner none
either: a stalled image pull (rate-limited registry, TLS-intercepting
proxy) span the create forever. Both helm invocations (install +
reconcile) ran under the deadline-less spin_cmd and could hang the same
way against a wedged kube-apiserver.

- create: --wait now always pairs with --timeout (default 15m,
  TB_CREATE_TIMEOUT_MIN override — the same env knob the Windows
  installer adopted); k3d aborts with a real error that the existing
  failure path dumps. spin() gained an optional deadline, used as a
  +5min backstop in case k3d itself wedges past its own timeout.
- helm install + reconcile: new spin_cmd_bounded (spin_cmd + hard
  deadline, rc 124 with an explicit timeout note + log tail), default
  10m via TB_HELM_TIMEOUT_MIN.

Completes #426 (the Windows half shipped with the #439 stack).

Closes #426

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): spin's deadline kills the wrapper's children too (Bugbot #442)

The create backstop passes the `( k3d … )` wrapper-subshell PID into
spin, and the deadline path signalled only that PID — orphaning k3d,
which kept creating the cluster after the installer had already exited
as failed, racing any retry. The deadline path now TERMs the children
while the parent is still alive (afterwards they reparent to init and
pkill -P can't see them), then the wrapper, with a KILL sweep after the
grace period. Harmless when bash exec-optimizes the wrapper away. Test
proves the child dies with the wrapper.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): reparent-safe KILL sweep + explicit create-timeout UX (Bugbot #442 r2)

1. spin's deadline path captured no child PIDs, so once the TERM'd
   wrapper died, its children reparented to init and the pkill -KILL -P
   sweep found nothing — a TERM-immune k3d could survive the backstop.
   Child PIDs are now captured BEFORE any signal and the KILL sweep
   addresses them directly; test proves a trap-''-TERM child dies.

2. When the create backstop fires, the k3d log is often empty (hung
   daemon) — the operator saw a bare failure with no timeout hint. The
   124 path now names the timeout, points at TB_CREATE_TIMEOUT_MIN, and
   deletes the partially created cluster (bounded) so a re-run cannot
   adopt it via the 'already exists' branch — parity with the Windows
   fix on #439.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): spin's deadline path is set -e-proof (Bugbot #442 r3)

Bare pkill (returns 1 with no children), kill on a reaped pid, and wait
after a kill (reports the signal) could each abort the deadline path
before 'return 124' under the installer's set -e — callers would see
143/1: no timeout copy, no partial-cluster cleanup, no bounded note.
Every signal step is now failure-proofed, and a bash -c 'set -euo
pipefail' end-to-end test pins the 124 contract for the childless
(exec-optimized) case that reproduces the abort.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): helm timeout names the pending-release unwedge (Bugbot #442 r4)

A SIGKILLed helm (spin_cmd_bounded deadline) can leave the release
wedged as pending-install/pending-upgrade — the next run then fails
with Helm's 'another operation is in progress' and no guidance. Both
call sites now capture the rc (if-! discarded it) and, on 124, print
the exact unwedge commands (uninstall for a half-installed release,
rollback for an upgrade) before the error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(installer): fix over-broad grep assertions on the helm-bound tests

Count invocation/hint lines, not comment mentions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): rollback hint names the release, not the namespace (Bugbot #442 r5)

The adopt path tracks release ($_rel) and namespace ($_ns) separately;
the reconcile-timeout unwedge hint said 'helm rollback $_ns', which is
a non-working command whenever they differ.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(installer): base-10-normalize the timeout knobs — octal trap defused (Bugbot #442 r6)

TB_CREATE_TIMEOUT_MIN / TB_HELM_TIMEOUT_MIN sanitizers accepted 08/09,
which bash arithmetic reads as invalid octal — aborting $(( … )) under
set -e mid-create (k3d already backgrounded → partial cluster left
behind), and 010 silently became 8. One shared, unit-tested helper
(tb_minutes_or) now normalizes via 10# at all three call sites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants