fix(#425): stuck image pulls — classify and tell the truth - #505
Merged
Conversation
…s as "downloading" helm runs without --wait, so the pull-progress helper waited TB_PULL_TIMEOUT and, on expiry, always printed "Services are still downloading — they'll finish starting in the background." — even when the pulls had PERMANENTLY failed (x509 / blocked registry / auth). A hard failure was reported in success-flavored copy. Bash (scripts/lib/install-client-helm.sh): - New _pull_failure_detail: on a visible ImagePullBackOff/ErrImagePull, prints the failing pod line + the matching pull event (self-contained; no cross-lib source). - New pure _progress_end_message: maps (pulled,total,max_pulled,has_fail) -> one of done|failed|downloading|stalled. A permanent failure NEVER maps to "downloading", so it can't be sold as background progress. - _download_services_progress tracks max_pulled and, on timeout, classifies: a failure warns loudly with the event text; "downloading in the background" prints ONLY when pulls demonstrably progressed; otherwise a neutral "not pulling yet". Windows (scripts/install-k8s.ps1) — the Wait-ForClientReady path already classified pull failures (image_pull/image_pull_ca/crash) into honest red-X summary branches; "downloading" only shows for a genuine starting state. Now it also carries the event text: Get-NotReadyState captures the pull event (or the failing pod line) into $script:NotReadyDetail, and the failure summary branches print it via the new Write-NotReadyDetail helper — matching the bash acceptance on both platforms. Tests: bats for _progress_end_message (all four outcomes; failure wins over partial progress) + _pull_failure_detail (failing prints detail/0, healthy prints nothing/1) + a source guard that the end copy routes through the selector; Pester for Get-NotReadyState detail capture (x509 / non-x509 / no-event fallback) and Write-NotReadyDetail (prints under a label; no-op when empty). Closes #425
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 97db9e1. Configure here.
…l-status # Conflicts: # scripts/manifest.sha256
…gbot) Bugbot: the event grep matched bare x509/TLS strings alongside failed to pull/ ErrImagePull, so with tail -n 3 an UNRELATED x509 event elsewhere in the namespace could displace the real pull-failure line and show the wrong reason. Scope to failed to pull|ErrImagePull only — matching summary.sh::_diagnose_not_ready and the PowerShell path. A genuine x509 pull failure is on a "failed to pull ..." line, so its detail is still captured; an unrelated x509 event is not. Test: unrelated x509 events don't displace the real pull reason (403 survives; x509 scoped out). Also merges develop (manifest regenerated). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
saadqbal
previously approved these changes
Jul 31, 2026
saadqbal
left a comment
Contributor
There was a problem hiding this comment.
Nice fix — the classifier is clean and the pure _progress_end_message is exactly the right seam to unit-test. Approving. Two small non-blockers inline: the failed copy can now over-claim in the opposite direction on a transient backoff, and bad prints unbounded despite the "Bounded" comment.
…ad review) - ImagePullBackOff/ErrImagePull can also be a transient blip or a registry 429 that kubelet keeps retrying, so wait_for_client_ready may still reach "connected". The absolute "this won't finish on its own" could contradict a later ✔ Connected — soften to "look stuck pulling — this usually needs action, not just more time". - _pull_failure_detail printed $bad unbounded though the header says "Bounded" and the PowerShell path caps at 3; add `| head -n 3` so a many-failing-pods cluster doesn't print a wall of lines (matches Select-Object -First 3). Test copy assertion updated to the new wording. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
aptracebloc
approved these changes
Jul 31, 2026
Contributor
|
/fr-pass |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

#425 — stuck image pulls: classify and tell the truth
helm upgrade --installruns with no--wait, so helm "succeeds" before any image is pulled. The pull-progress helper then waitedTB_PULL_TIMEOUTand, on expiry, always printed "Services are still downloading — they'll finish starting in the background." — even when the pulls had permanently failed (x509, blocked registry, auth). A hard failure was reported in success-flavored copy; the user walked away believing the install would complete.Bash —
scripts/lib/install-client-helm.sh_pull_failure_detail— on a visibleImagePullBackOff/ErrImagePull/InvalidImageName, prints the failing pod line + the matchingfailed to pull …event (bounded, non-fatal, self-contained so no cross-lib sourcing)._progress_end_message(pure) — maps(pulled, total, max_pulled, has_fail)→done | failed | downloading | stalled. A permanent failure never maps todownloading, so it can't be sold as background progress._download_services_progressnow tracksmax_pulledand, on timeout, classifies: a failure warns loudly with the event text; the "downloading in the background" copy prints only when pulls demonstrably progressed; otherwise a neutral "not pulling yet — see the diagnosis".Windows —
scripts/install-k8s.ps1Wait-ForClientReady→Get-NotReadyStatealready classified pull failures into honest red-X summary branches (image_pull/image_pull_ca/crash); the "downloading" copy only ever shows for a genuinestartingstate. This PR also makes the failure copy carry the event text:Get-NotReadyStatecaptures the pull event (or the failing pod line) into$script:NotReadyDetail, and the failure branches print it via the newWrite-NotReadyDetail— matching the acceptance on both platforms.Acceptance
Tests
bats:
_progress_end_message(all four outcomes; failure wins over partial progress),_pull_failure_detail(failing → detail + rc 0; healthy → nothing + rc 1), and a guard that the end copy routes through the selector. Pester:Get-NotReadyStatedetail capture (x509 / non-x509 / no-event fallback) +Write-NotReadyDetail(labelled block; no-op when empty). Full suites: bats install-client-helm green (one unrelated pre-existing_extract_yaml_valuefailure exists ondevelop, macOS-bash-specific — out of scope), Pester 333 passed / 0 failed; shellcheck + check-style + check-drift clean;scripts/manifest.sha256regenerated (R8).Closes #425
Note
Low Risk
Installer UX and messaging only; bounded kubectl reads with no change to cluster install or auth behavior.
Overview
Fixes #425: permanent image pull failures were often reported as "still downloading in the background" even when pulls had already failed (x509, registry, auth).
On bash,
_download_services_progressnow uses_pull_failure_detailand the pure_progress_end_messageselector (done/failed/downloading/stalled) so timeouts warn with pod/event lines instead of implying background progress when nothing will recover.On Windows,
Get-NotReadyStatefills$script:NotReadyDetailfrom scoped pull events or pod status (including crash loops), and failure summaries callWrite-NotReadyDetailunder "What the cluster reported:".Adds bats and Pester coverage;
scripts/manifest.sha256updated.Reviewed by Cursor Bugbot for commit 7e7b072. Bugbot is set up for automated code reviews on this repo. Configure here.