Skip to content

ci: bound the build jobs and the package fetches they hang on (#351) - #352

Merged
jdatcmd merged 1 commit into
mainfrom
fix/351-ci-install-hang
Aug 3, 2026
Merged

ci: bound the build jobs and the package fetches they hang on (#351)#352
jdatcmd merged 1 commit into
mainfrom
fix/351-ci-install-hang

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Fixes #351.

What was wrong

build (PG 15, x86_64) hung indefinitely on Install PostgreSQL 15 and codec headers on two consecutive PRs -- #347 for 40 minutes, #350 for 18 -- against a normal runtime of ~70 seconds. Both times every other build passed, including PG 15 on aarch64: same major, same compiler, different runner. The step that wedges is package installation, before any project code compiles, so it cannot be a property of the change under test.

Three separate gaps, all of which had to be closed:

gap effect
build and build-beta had no timeout-minutes (only suites did) a stall produced a check pending forever rather than failing
curl -fsSL with no connect or total timeout waits indefinitely on a stalled connection instead of failing and retrying
apt-get with no retries and no outer bound a wedged mirror consumes the entire job

The missing job timeout is the one that matters most. A check stuck at pending is worse than a red one: the PR shows "1 pending" and a reviewer either waits or merges on an incomplete gate. That temptation arose on two consecutive PRs here, and a gate that is routinely bypassed stops being a gate.

What changed

  • timeout-minutes on build (15) and build-beta (20). Normal runtime is ~70s, so these bound a hang without being tight enough to fail a slow-but-working runner.
  • curl now sets --connect-timeout 15 --max-time 120 and retries with --retry 5 --retry-delay 5 --retry-all-errors.
  • apt-get runs with -o Acquire::Retries=5 under an outer timeout.
  • The same unbounded curl/apt in nightly.yml hardened identically -- same defect, slower fuse, since its jobs at least have timeouts and so fail after up to two hours rather than never.
  • docs.yml had no job timeouts either; added.

Every job across the three workflows is now bounded, and every network fetch is retried and time-limited:

ci.yml       build       timeout=15
ci.yml       build-beta  timeout=20
ci.yml       suites      timeout=60
docs.yml     build       timeout=20
docs.yml     deploy      timeout=20
nightly.yml  suites      timeout=60
nightly.yml  sanitizer   timeout=120
nightly.yml  coverage    timeout=90

A trap worth knowing about

When the hung run is cancelled, GitHub may retry the job as a new attempt, and the PR check then shows green from the retry while the cancelled attempt sits at conclusion=cancelled with Build: skipped. On #350 that produced a green-looking build (PG 15, x86_64) check over a job that compiled nothing -- I caught it only by inspecting the job's steps rather than the check mark.

This PR does not fix that (it is GitHub's behaviour), but with the timeout in place the cancel-and-retry dance should stop being necessary.

Verification

All three workflow files parse (yaml.safe_load). The real proof is this PR's own CI: it exercises the changed build and build-beta jobs on all majors and both architectures.

🤖 Generated with Claude Code

https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8

The build (PG 15, x86_64) job hung indefinitely on its package install step
on two consecutive PRs, #347 for 40 minutes and #350 for 18, against a normal
runtime of about 70 seconds. Every other build passed both times, including
PG 15 on aarch64 -- same major, same compiler, different runner -- and the
step that wedged is package installation, before any project code compiles.

Three things were missing, and all three are needed.

The build and build-beta jobs had no timeout-minutes at all; only suites did.
So a stalled mirror produced a check that stayed pending forever rather than
failing. That is worse than a failure: the PR shows "1 pending" and the
reviewer either waits or merges on an incomplete gate. It came up twice in a
row here, and a gate that is routinely bypassed stops being a gate.

curl was invoked as "curl -fsSL" with no connect or total timeout, so it
waits forever on a stalled connection rather than failing and retrying. It
now bounds both, and retries with backoff.

apt-get had no retry configuration and no outer bound. It now sets
Acquire::Retries and runs under timeout, so a wedged mirror costs a bounded
amount of the job budget instead of all of it.

The same unbounded curl and apt appear in nightly.yml, whose jobs do have
timeouts, so a hang there fails after up to two hours rather than never.
Hardened identically, since it is the same defect with a slower fuse.
docs.yml had no job timeouts either; added.

Every job across the three workflows is now bounded, and every network fetch
is retried and time-limited.

Worth recording for whoever hits this next: when the hung run is cancelled,
GitHub may retry the job as a new attempt, and the PR check then shows green
from that retry while the cancelled attempt sits at conclusion=cancelled with
Build: skipped. On #350 the cancel produced a green-looking check over a job
that compiled nothing. Check the retried job's steps, not the check mark.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
@ChronicallyJD

Copy link
Copy Markdown
Collaborator

Reviewed — sound and complete, mergeable.

  • The timeout-minutes is the load-bearing change: it converts the indefinite pending into a red failure. A red gate blocks; a pending gate is what tempted the incomplete-gate merges you flagged. Correct root fix.
  • I checked for a partial-coverage gap and there isn't one: the bounded-curl + timeout/Acquire::Retries apt hardening is applied to all three install-doing jobsbuild (15m), build-beta (20m), and suites (60m) — not just the one that hung. So no job is left with an unbounded fetch.
  • Retries (--retry 5 --retry-all-errors, Acquire::Retries=5) heal a transient mirror blip so the new bound doesn't spuriously red a recoverable stall, while the outer timeout 300/600 keeps a genuinely wedged mirror from eating the whole job.
  • Bounds are generous vs the ~70s normal runtime (15m build), so a slow-but-working runner won't trip them.

One optional follow-up, not a blocker: caching the PG apt install (your option 3) would remove the mirror as a failure surface entirely, but the timeout+retry already turns "hang forever" into "fail fast and retry," which is the part that was hurting. No objections.

@jdatcmd
jdatcmd merged commit bc08f36 into main Aug 3, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: build (PG 15, x86_64) hangs indefinitely on the package install step, blocking PRs

2 participants