ci: bound the build jobs and the package fetches they hang on (#351) - #352
Merged
Conversation
The build (PG 15, x86_64) job hung indefinitely on its package install step on two consecutive PRs, #347 for 40 minutes and #350 for 18, against a normal runtime of about 70 seconds. Every other build passed both times, including PG 15 on aarch64 -- same major, same compiler, different runner -- and the step that wedged is package installation, before any project code compiles. Three things were missing, and all three are needed. The build and build-beta jobs had no timeout-minutes at all; only suites did. So a stalled mirror produced a check that stayed pending forever rather than failing. That is worse than a failure: the PR shows "1 pending" and the reviewer either waits or merges on an incomplete gate. It came up twice in a row here, and a gate that is routinely bypassed stops being a gate. curl was invoked as "curl -fsSL" with no connect or total timeout, so it waits forever on a stalled connection rather than failing and retrying. It now bounds both, and retries with backoff. apt-get had no retry configuration and no outer bound. It now sets Acquire::Retries and runs under timeout, so a wedged mirror costs a bounded amount of the job budget instead of all of it. The same unbounded curl and apt appear in nightly.yml, whose jobs do have timeouts, so a hang there fails after up to two hours rather than never. Hardened identically, since it is the same defect with a slower fuse. docs.yml had no job timeouts either; added. Every job across the three workflows is now bounded, and every network fetch is retried and time-limited. Worth recording for whoever hits this next: when the hung run is cancelled, GitHub may retry the job as a new attempt, and the PR check then shows green from that retry while the cancelled attempt sits at conclusion=cancelled with Build: skipped. On #350 the cancel produced a green-looking check over a job that compiled nothing. Check the retried job's steps, not the check mark. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
Collaborator
|
Reviewed — sound and complete, mergeable.
One optional follow-up, not a blocker: caching the PG apt install (your option 3) would remove the mirror as a failure surface entirely, but the timeout+retry already turns "hang forever" into "fail fast and retry," which is the part that was hurting. No objections. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #351.
What was wrong
build (PG 15, x86_64)hung indefinitely onInstall PostgreSQL 15 and codec headerson two consecutive PRs -- #347 for 40 minutes, #350 for 18 -- against a normal runtime of ~70 seconds. Both times every other build passed, including PG 15 on aarch64: same major, same compiler, different runner. The step that wedges is package installation, before any project code compiles, so it cannot be a property of the change under test.Three separate gaps, all of which had to be closed:
buildandbuild-betahad notimeout-minutes(onlysuitesdid)curl -fsSLwith no connect or total timeoutapt-getwith no retries and no outer boundThe missing job timeout is the one that matters most. A check stuck at pending is worse than a red one: the PR shows "1 pending" and a reviewer either waits or merges on an incomplete gate. That temptation arose on two consecutive PRs here, and a gate that is routinely bypassed stops being a gate.
What changed
timeout-minutesonbuild(15) andbuild-beta(20). Normal runtime is ~70s, so these bound a hang without being tight enough to fail a slow-but-working runner.curlnow sets--connect-timeout 15 --max-time 120and retries with--retry 5 --retry-delay 5 --retry-all-errors.apt-getruns with-o Acquire::Retries=5under an outertimeout.curl/aptinnightly.ymlhardened identically -- same defect, slower fuse, since its jobs at least have timeouts and so fail after up to two hours rather than never.docs.ymlhad no job timeouts either; added.Every job across the three workflows is now bounded, and every network fetch is retried and time-limited:
A trap worth knowing about
When the hung run is cancelled, GitHub may retry the job as a new attempt, and the PR check then shows green from the retry while the cancelled attempt sits at
conclusion=cancelledwithBuild: skipped. On #350 that produced a green-lookingbuild (PG 15, x86_64)check over a job that compiled nothing -- I caught it only by inspecting the job's steps rather than the check mark.This PR does not fix that (it is GitHub's behaviour), but with the timeout in place the cancel-and-retry dance should stop being necessary.
Verification
All three workflow files parse (
yaml.safe_load). The real proof is this PR's own CI: it exercises the changedbuildandbuild-betajobs on all majors and both architectures.🤖 Generated with Claude Code
https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8