ci: bound apt-get so a dead mirror cannot wedge a job - #3924
Open
hubcio wants to merge 2 commits into
Open
Conversation
Runners intermittently lose egress to azure.archive.ubuntu.com. apt falls back through the mirror list, then stalls with no output at all, so the job sits dead until it burns the full 60 minute budget. Three consecutive doctest runs on PR #3921 died this way, each on a different runner, while sibling jobs on the same run finished apt in nine seconds. apt's own Acquire::*::Timeout does not bound this - the observed stalls ran half an hour past the 120s default - so scripts/ci/apt-install.sh caps both the index refresh and the install with timeout(1). A wedged runner now fails loudly in minutes instead of dying quietly at the job limit. Installation still runs after a failed update, so a merely stale index set is not fatal. Every call site routes through it rather than repeating the loop inline.
mmodzelewski
approved these changes
Aug 19, 2026
spetz
approved these changes
Aug 19, 2026
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3924 +/- ##
============================================
- Coverage 83.82% 83.69% -0.13%
Complexity 1358 1358
============================================
Files 1212 1212
Lines 166148 166191 +43
Branches 133622 133790 +168
============================================
- Hits 139272 139095 -177
- Misses 23253 23318 +65
- Partials 3623 3778 +155
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runners intermittently lose egress to azure.archive.ubuntu.com.
apt falls back through the mirror list, then stalls with no
output at all, so the job sits dead until it burns the full 60
minute budget. Three consecutive doctest runs on PR #3921 died
this way, each on a different runner, while sibling jobs on the
same run finished apt in nine seconds.
apt's own Acquire::*::Timeout does not bound this - the observed
stalls ran half an hour past the 120s default - so
scripts/ci/apt-install.sh caps both the index refresh and the
install with timeout(1). A wedged runner now fails loudly in
minutes instead of dying quietly at the job limit. Installation
still runs after a failed update, so a merely stale index set is
not fatal. Every call site routes through it rather than
repeating the loop inline.