Skip to content

Retry prek hook installation in CI when toolchain downloads fail - #72382

Draft
rjgoyln wants to merge 1 commit into
apache:mainfrom
rjgoyln:fix/ci-retry-prek-install-hooks
Draft

Retry prek hook installation in CI when toolchain downloads fail#72382
rjgoyln wants to merge 1 commit into
apache:mainfrom
rjgoyln:fix/ci-retry-prek-install-hooks

Conversation

@rjgoyln

@rjgoyln rjgoyln commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

prek install-hooks downloads hook toolchains without retrying. A transient 5xx therefore fails the job before any work starts.

The prek cache step already retries its downloads, leaving hook installation as the last unguarded external dependency. Upgrading prek does not fix this: the download path is unchanged between 0.4.14 and 0.5.1 and still uses go.dev.

#72489 addresses how often this path is reached: a cache key mismatch currently leaves four job groups reinstalling every hook on every run. It does not remove the need for a retry, since hook installation still has to happen whenever a .pre-commit-config.yaml or uv version change invalidates the key.

Tests

Reproduced the same go.dev HTTP 500 failure with prek 0.4.14 and verified:

  • Before: fails with the expected error.
  • After: two 500s are retried with 20s/40s backoff; the third attempt succeeds and installs Go 1.25.0.
  • Persistent failure: still fails after 4 attempts.
  • Interrupted download: leaves nothing under tools/, so retries safely re-download the toolchain.

The retry budget covers roughly two minutes, which is sufficient for the observed outage.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

prek provisions hook toolchains by downloading them and does not retry, so a
transient 5xx from an upstream download host fails the job before it does any
work. Only the jobs that reach this step before the prek cache tarball is
written are exposed, which is why it lands on unrelated PRs at random.

The cache restore step in this action is pinned to a revision that retries its
own download for the same class of failure; hook installation was the last
external dependency here without a guard.
@rjgoyln rjgoyln changed the title CI: Retry prek hook installation when a toolchain download fails Retry prek hook installation when a toolchain download fails Sep 1, 2026
@rjgoyln
rjgoyln force-pushed the fix/ci-retry-prek-install-hooks branch from fcb5f88 to 0575b26 Compare September 1, 2026 17:11
@rjgoyln
rjgoyln marked this pull request as ready for review September 1, 2026 17:49
@rjgoyln rjgoyln changed the title Retry prek hook installation when a toolchain download fails Retry prek hook installation in CI when toolchain downloads fail Sep 1, 2026
@rjgoyln
rjgoyln marked this pull request as draft September 3, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant