ci: keep the runner image's git off HTTP/2 so dependency clones stop failing - #401
Merged
Conversation
…failing Six self-hosted jobs on run 33627249007 (clang-release, clang-tsan, both all-optional-features legs, both Application ladder legs) died in Configure with 17 occurrences of fatal: could not read Username for 'https://github.com': No such device or address fatal: expected flush after ref listing Had to git clone more than once: 3 times. CMake Error ... Failed to clone repository: 'https://github.com/stephenberry/glaze.git' That is CMakeLists.txt:151's FetchContent fallback for glaze, the only anonymous clone the build makes. It reads as a credentials or rate-limit problem and is neither. Reproduced from a running runner container: GitHub answers the info/refs GET with 200 and then answers the git-upload-pack POST on the same reused HTTP/2 connection with a spurious 401 and `www-authenticate: Basic realm="GitHub"`, which git turns into the username prompt above. Measured 7 failures in 10 ls-remotes; with -c http.version=HTTP/1.1 or -c protocol.version=0, 10 in 10 succeed. It is the client stack, not the network, the IP, or GitHub throttling: the same clone from the Docker host -- same public address -- is 10 in 10, and the anonymous API rate limit reads 60/60 remaining. It is not git's version either: git 2.55 from ppa:git-core/ppa inside the image fails at the same rate as the distro's 2.43, because both link Ubuntu 24.04's libcurl 8.5.0 / nghttp2 1.59 (the host's is 8.21). 24.04 has no newer libcurl to install, so the fix is to keep git off HTTP/2. Only authenticated requests escape the 401 -- git retries those with credentials -- which is why actions/checkout has always worked here and only the dependency clone breaks, and why the GitHub-hosted path never showed this. Applied system-wide rather than per-repo because the bug is in the HTTP stack, not in anything specific to that remote. Containers built from an older image keep failing until recreated; the README records the in-place `docker exec -u root ... git config --system http.version HTTP/1.1` for that, which was run against the five online runners (each verified 10 in 10 afterwards) and turned the rerun of those six jobs past Configure. The README's host inventory goes with it. It listed three hosts, two of which no longer exist, and it will drift again -- Settings -> Actions -> Runners and `gh api repos/LASTRADA-Software/morph/actions/runners` are the live answer. The sizing and fastcached recipes stay, reworded as per-platform guidance rather than claims about particular machines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EZCZ8SoqHKrcmXxwb7q22D
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
Six self-hosted jobs on run 33627249007 (
clang-release,clang-tsan, bothall optional featureslegs, bothApplication ladderlegs) failed in Configure, 17 occurrences of:That is
CMakeLists.txt:151's FetchContent fallback for glaze — the only anonymous clone in the build.Root cause
It looks like a credentials or rate-limit problem and is neither. Reproduced live from a running runner container: GitHub answers the
info/refsGET with 200, then answers thegit-upload-packPOST on the same reused HTTP/2 connection with a spurious401+www-authenticate: Basic realm="GitHub", which git turns into the username prompt.ls-remotes succeed-c http.version=HTTP/1.1-c protocol.version=0ppa:git-core/ppaSo it is not the network, not the IP, not GitHub throttling (anonymous API rate limit read 60/60 remaining), and not git's version — it is Ubuntu 24.04's libcurl 8.5.0 / nghttp2 1.59, which 24.04 has no newer replacement for. Only authenticated requests escape the 401, because git retries them with credentials — which is why
actions/checkoutalways worked here and only the dependency clone broke, and why GitHub-hosted runs never showed it.The change
Dockerfile:RUN git config --system http.version HTTP/1.1, system-wide because the bug is in the HTTP stack rather than in anything specific to that remote.README.md: new Dependency clones and HTTP/2 section with the symptom, the evidence, and the in-place fix for containers built from an older image (docker exec -u root <name> git config --system http.version HTTP/1.1).README.md: host inventory removed — it listed three hosts, two of which no longer exist, and would drift again. Settings → Actions → Runners andgh api repos/LASTRADA-Software/morph/actions/runnersare the live answer. Sizing and fastcached recipes stay, reworded as per-platform guidance rather than claims about particular machines.Verification
The in-place config was applied to all five online runners, each then verified at 10 of 10 clones (from 3 of 10). Rerunning the six failed jobs on #395 carried them past Configure, where they had previously died in 27–44 s.
🤖 Generated with Claude Code
https://claude.ai/code/session_01EZCZ8SoqHKrcmXxwb7q22D