Skip to content

fix(#425): stuck image pulls — classify and tell the truth - #505

Merged
shujaatTracebloc merged 4 commits into
developfrom
fix/425-honest-pull-status
Jul 31, 2026
Merged

fix(#425): stuck image pulls — classify and tell the truth#505
shujaatTracebloc merged 4 commits into
developfrom
fix/425-honest-pull-status

Conversation

@shujaatTracebloc

@shujaatTracebloc shujaatTracebloc commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

#425 — stuck image pulls: classify and tell the truth

helm upgrade --install runs with no --wait, so helm "succeeds" before any image is pulled. The pull-progress helper then waited TB_PULL_TIMEOUT and, on expiry, always printed "Services are still downloading — they'll finish starting in the background." — even when the pulls had permanently failed (x509, blocked registry, auth). A hard failure was reported in success-flavored copy; the user walked away believing the install would complete.

Bash — scripts/lib/install-client-helm.sh

  • _pull_failure_detail — on a visible ImagePullBackOff/ErrImagePull/InvalidImageName, prints the failing pod line + the matching failed to pull … event (bounded, non-fatal, self-contained so no cross-lib sourcing).
  • _progress_end_message (pure) — maps (pulled, total, max_pulled, has_fail)done | failed | downloading | stalled. A permanent failure never maps to downloading, so it can't be sold as background progress.
  • _download_services_progress now tracks max_pulled and, on timeout, classifies: a failure warns loudly with the event text; the "downloading in the background" copy prints only when pulls demonstrably progressed; otherwise a neutral "not pulling yet — see the diagnosis".

Windows — scripts/install-k8s.ps1

Wait-ForClientReadyGet-NotReadyState already classified pull failures into honest red-X summary branches (image_pull/image_pull_ca/crash); the "downloading" copy only ever shows for a genuine starting state. This PR also makes the failure copy carry the event text: Get-NotReadyState captures the pull event (or the failing pod line) into $script:NotReadyDetail, and the failure branches print it via the new Write-NotReadyDetail — matching the acceptance on both platforms.

Acceptance

  • A permanently failing pull is never described as background progress.
  • The failure copy contains the pod event text (both bash and Windows).

Tests

bats: _progress_end_message (all four outcomes; failure wins over partial progress), _pull_failure_detail (failing → detail + rc 0; healthy → nothing + rc 1), and a guard that the end copy routes through the selector. Pester: Get-NotReadyState detail capture (x509 / non-x509 / no-event fallback) + Write-NotReadyDetail (labelled block; no-op when empty). Full suites: bats install-client-helm green (one unrelated pre-existing _extract_yaml_value failure exists on develop, macOS-bash-specific — out of scope), Pester 333 passed / 0 failed; shellcheck + check-style + check-drift clean; scripts/manifest.sha256 regenerated (R8).

Closes #425


Note

Low Risk
Installer UX and messaging only; bounded kubectl reads with no change to cluster install or auth behavior.

Overview
Fixes #425: permanent image pull failures were often reported as "still downloading in the background" even when pulls had already failed (x509, registry, auth).

On bash, _download_services_progress now uses _pull_failure_detail and the pure _progress_end_message selector (done / failed / downloading / stalled) so timeouts warn with pod/event lines instead of implying background progress when nothing will recover.

On Windows, Get-NotReadyState fills $script:NotReadyDetail from scoped pull events or pod status (including crash loops), and failure summaries call Write-NotReadyDetail under "What the cluster reported:".

Adds bats and Pester coverage; scripts/manifest.sha256 updated.

Reviewed by Cursor Bugbot for commit 7e7b072. Bugbot is set up for automated code reviews on this repo. Configure here.

…s as "downloading"

helm runs without --wait, so the pull-progress helper waited TB_PULL_TIMEOUT and, on
expiry, always printed "Services are still downloading — they'll finish starting in
the background." — even when the pulls had PERMANENTLY failed (x509 / blocked
registry / auth). A hard failure was reported in success-flavored copy.

Bash (scripts/lib/install-client-helm.sh):
- New _pull_failure_detail: on a visible ImagePullBackOff/ErrImagePull, prints the
  failing pod line + the matching pull event (self-contained; no cross-lib source).
- New pure _progress_end_message: maps (pulled,total,max_pulled,has_fail) -> one of
  done|failed|downloading|stalled. A permanent failure NEVER maps to "downloading",
  so it can't be sold as background progress.
- _download_services_progress tracks max_pulled and, on timeout, classifies: a
  failure warns loudly with the event text; "downloading in the background" prints
  ONLY when pulls demonstrably progressed; otherwise a neutral "not pulling yet".

Windows (scripts/install-k8s.ps1) — the Wait-ForClientReady path already classified
pull failures (image_pull/image_pull_ca/crash) into honest red-X summary branches;
"downloading" only shows for a genuine starting state. Now it also carries the event
text: Get-NotReadyState captures the pull event (or the failing pod line) into
$script:NotReadyDetail, and the failure summary branches print it via the new
Write-NotReadyDetail helper — matching the bash acceptance on both platforms.

Tests: bats for _progress_end_message (all four outcomes; failure wins over partial
progress) + _pull_failure_detail (failing prints detail/0, healthy prints nothing/1)
+ a source guard that the end copy routes through the selector; Pester for
Get-NotReadyState detail capture (x509 / non-x509 / no-event fallback) and
Write-NotReadyDetail (prints under a label; no-op when empty).

Closes #425
@shujaatTracebloc shujaatTracebloc self-assigned this Jul 31, 2026
@shujaatTracebloc
shujaatTracebloc marked this pull request as ready for review July 31, 2026 06:52

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 97db9e1. Configure here.

Comment thread scripts/lib/install-client-helm.sh Outdated
shujaatTracebloc and others added 2 commits July 31, 2026 09:02
…l-status

# Conflicts:
#	scripts/manifest.sha256
…gbot)

Bugbot: the event grep matched bare x509/TLS strings alongside failed to pull/
ErrImagePull, so with tail -n 3 an UNRELATED x509 event elsewhere in the namespace
could displace the real pull-failure line and show the wrong reason. Scope to
failed to pull|ErrImagePull only — matching summary.sh::_diagnose_not_ready and the
PowerShell path. A genuine x509 pull failure is on a "failed to pull ..." line, so
its detail is still captured; an unrelated x509 event is not.

Test: unrelated x509 events don't displace the real pull reason (403 survives; x509
scoped out). Also merges develop (manifest regenerated).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
saadqbal
saadqbal previously approved these changes Jul 31, 2026

@saadqbal saadqbal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fix — the classifier is clean and the pure _progress_end_message is exactly the right seam to unit-test. Approving. Two small non-blockers inline: the failed copy can now over-claim in the opposite direction on a transient backoff, and bad prints unbounded despite the "Bounded" comment.

Comment thread scripts/lib/install-client-helm.sh Outdated
Comment thread scripts/lib/install-client-helm.sh
…ad review)

- ImagePullBackOff/ErrImagePull can also be a transient blip or a registry 429 that
  kubelet keeps retrying, so wait_for_client_ready may still reach "connected". The
  absolute "this won't finish on its own" could contradict a later ✔ Connected —
  soften to "look stuck pulling — this usually needs action, not just more time".
- _pull_failure_detail printed $bad unbounded though the header says "Bounded" and
  the PowerShell path caps at 3; add `| head -n 3` so a many-failing-pods cluster
  doesn't print a wall of lines (matches Select-Object -First 3).

Test copy assertion updated to the new wording.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@shujaatTracebloc
shujaatTracebloc merged commit 5a8b3ff into develop Jul 31, 2026
84 of 86 checks passed
@shujaatTracebloc
shujaatTracebloc deleted the fix/425-honest-pull-status branch July 31, 2026 08:33
@LukasWodka

Copy link
Copy Markdown
Contributor

/fr-pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants