From 890559c6b342e7291893762c56fb255586323afe Mon Sep 17 00:00:00 2001 From: Yaraslau Tamashevich Date: Wed, 2 Sep 2026 16:56:46 +0200 Subject: [PATCH] ci: keep the runner image's git off HTTP/2 so dependency clones stop failing Six self-hosted jobs on run 33627249007 (clang-release, clang-tsan, both all-optional-features legs, both Application ladder legs) died in Configure with 17 occurrences of fatal: could not read Username for 'https://github.com': No such device or address fatal: expected flush after ref listing Had to git clone more than once: 3 times. CMake Error ... Failed to clone repository: 'https://github.com/stephenberry/glaze.git' That is CMakeLists.txt:151's FetchContent fallback for glaze, the only anonymous clone the build makes. It reads as a credentials or rate-limit problem and is neither. Reproduced from a running runner container: GitHub answers the info/refs GET with 200 and then answers the git-upload-pack POST on the same reused HTTP/2 connection with a spurious 401 and `www-authenticate: Basic realm="GitHub"`, which git turns into the username prompt above. Measured 7 failures in 10 ls-remotes; with -c http.version=HTTP/1.1 or -c protocol.version=0, 10 in 10 succeed. It is the client stack, not the network, the IP, or GitHub throttling: the same clone from the Docker host -- same public address -- is 10 in 10, and the anonymous API rate limit reads 60/60 remaining. It is not git's version either: git 2.55 from ppa:git-core/ppa inside the image fails at the same rate as the distro's 2.43, because both link Ubuntu 24.04's libcurl 8.5.0 / nghttp2 1.59 (the host's is 8.21). 24.04 has no newer libcurl to install, so the fix is to keep git off HTTP/2. Only authenticated requests escape the 401 -- git retries those with credentials -- which is why actions/checkout has always worked here and only the dependency clone breaks, and why the GitHub-hosted path never showed this. Applied system-wide rather than per-repo because the bug is in the HTTP stack, not in anything specific to that remote. Containers built from an older image keep failing until recreated; the README records the in-place `docker exec -u root ... git config --system http.version HTTP/1.1` for that, which was run against the five online runners (each verified 10 in 10 afterwards) and turned the rerun of those six jobs past Configure. The README's host inventory goes with it. It listed three hosts, two of which no longer exist, and it will drift again -- Settings -> Actions -> Runners and `gh api repos/LASTRADA-Software/morph/actions/runners` are the live answer. The sizing and fastcached recipes stay, reworded as per-platform guidance rather than claims about particular machines. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EZCZ8SoqHKrcmXxwb7q22D --- .github/self-hosted-runner/Dockerfile | 20 +++++ .github/self-hosted-runner/README.md | 106 ++++++++++++++++---------- 2 files changed, 85 insertions(+), 41 deletions(-) diff --git a/.github/self-hosted-runner/Dockerfile b/.github/self-hosted-runner/Dockerfile index 97723d63..0a186bdd 100644 --- a/.github/self-hosted-runner/Dockerfile +++ b/.github/self-hosted-runner/Dockerfile @@ -41,6 +41,26 @@ RUN apt-get update -q && apt-get install -y --no-install-recommends \ build-essential \ && rm -rf /var/lib/apt/lists/* +# Anonymous HTTPS clones from this image are unreliable over HTTP/2, and +# CMakeLists.txt's FetchContent fallback for glaze does exactly one of +# those per configure: GitHub answers the info/refs GET with 200 and then +# the git-upload-pack POST on the same reused HTTP/2 connection with a +# spurious 401 (`www-authenticate: Basic realm="GitHub"`), which git turns +# into `could not read Username for 'https://github.com'` and CMake into +# "Failed to clone repository". Measured here at ~7 failures in 10 +# ls-remotes; with -c http.version=HTTP/1.1 (or protocol.version=0) it is +# 10 in 10. It is the client stack, not the network or the IP -- the same +# clone from the Docker host, same public address, is 10 in 10 -- and not +# git's version either: git 2.55 from ppa:git-core/ppa in this image fails +# at the same rate as the distro's 2.43, because both link Ubuntu 24.04's +# libcurl 8.5.0/nghttp2 1.59. There is no newer libcurl to install on +# 24.04, so the fix is to keep git off HTTP/2 entirely. Only +# authenticated requests escape the 401 (git retries with credentials), +# which is why actions/checkout has always worked here and only the +# dependency clone fails. System-wide rather than per-repo: the bug is in +# the HTTP stack, so it applies to every clone any job makes. +RUN git config --system http.version HTTP/1.1 + # Ubuntu 24.04's apt package is CMake 3.28.3. GitHub-hosted ubuntu-24.04 # ships a newer CMake from Kitware's own release archive rather than the # distro package, and that gap is not cosmetic: linux-all-features' diff --git a/.github/self-hosted-runner/README.md b/.github/self-hosted-runner/README.md index 81788b87..c5e3029f 100644 --- a/.github/self-hosted-runner/README.md +++ b/.github/self-hosted-runner/README.md @@ -5,30 +5,14 @@ Docker containers. Used by `ci.yml`'s `linux-compilers`, `linux-sanitizers`, and `linux-all-features` jobs whenever a runner is online and idle (see **ci.yml integration** below). -Currently running across three hosts: - -- the maintainer's Windows machine (Docker Desktop, Linux containers): 4 - containers, each capped at 6 CPUs (`docker run --cpus=6`) on a - 32-logical-processor box — 24 cores committed, 8 left as headroom for the - host OS and Docker Desktop itself. -- a Hetzner Cloud VM (8 CPU / 15 GiB): 3 workers at 2 CPU/4 GiB each plus - its own `fastcached` daemon (1 CPU/2 GiB/10 GiB-disk), all sized by - `bootstrap-cloud-node.sh` from the box's actual resources — see - **Bootstrapping a cloud node with its own fastcached** below. -- the maintainer's Linux workstation (12 logical processors / 61 GiB, plain - Docker Engine): 5 containers pinned to 2 processors each (`--cpuset-cpus`, - not `--cpus` — see below) with a 6 GiB cap, plus a released `fastcached` - running as a host service on the Docker bridge address — 10 of 12 - processors committed to runners, the rest to the host and the cache. - Started by hand rather than by `bootstrap-cloud-node.sh`, which is written - for a dedicated VM it may install Docker on and give the whole box to; see - **Running more than one runner** below for the exact commands, and **A - workstation that already runs its own fastcached** for why the runners get - their own daemon rather than the one already serving the developer's local - builds. - -All three hosts' runners register to the same repo and are indistinguishable -to `ci.yml` — a job lands on whichever is online and idle. +Any number of hosts can register runners to this repo; they are +indistinguishable to `ci.yml` — a job lands on whichever is online and +idle. Which machines are currently registered is not recorded here (it +changes): read it off **Settings → Actions → Runners**, or +`gh api repos/LASTRADA-Software/morph/actions/runners`. Registrations for +hosts that no longer exist stay listed there as `offline` and are harmless +— the probe counts only online, non-busy ones — but are worth deleting so +the list reflects what actually runs. Multiple runners exist so a multi-leg matrix (`linux-compilers` has 4, `linux-sanitizers` has 3) actually runs its legs in parallel instead of @@ -263,8 +247,9 @@ directly. Each container is one runner process. To add capacity (another container here, or a registration on a second machine), repeat the Quick start with a distinct `RUNNER_NAME` per container/host — no coordination between them -is needed, they all just poll the same repo's job queue. This is exactly -how the current 4 containers are set up: +is needed, they all just poll the same repo's job queue. On a host where +`host.docker.internal` resolves by itself (Docker Desktop), that is just +the Quick start in a loop: ```bash for i in 1 2 3 4; do @@ -285,8 +270,8 @@ the image needs to know about — size it to (host logical processors) ÷ itself, and adjust down if the containers are still oversubscribing the box under load. -On a **plain Linux Docker Engine** host two more flags are needed, and the -Linux workstation's 5 containers are started exactly like this: +On a **plain Linux Docker Engine** host two more flags are needed. A +12-processor / 61 GiB box running five containers, for example: ```bash for i in 1 2 3 4 5; do @@ -332,10 +317,11 @@ done The registration token is single-use, which is why it is minted inside the loop rather than once before it. -The Windows host's four containers use `--cpus=6` and carry the same latent -hazard — 32 logical processors reported to every build inside a 6-CPU -container. It has not bitten there because those containers are not memory -capped, so the oversubscription costs wall-clock rather than killed compiles. +Any host that starts its containers with `--cpus=N` instead carries the +same latent hazard — the box's full logical-processor count is reported to +every build inside an N-CPU container. It only bites where a memory cap is +also set: without one the oversubscription costs wall-clock rather than +killed compiles. ## ci.yml integration @@ -369,6 +355,45 @@ needed for that case. the remaining Linux jobs (valgrind, Qt, ladder tests, clang-tidy) are intentionally left on `ubuntu-24.04` for now. +## Dependency clones and HTTP/2 + +`CMakeLists.txt` falls back to `FetchContent` for glaze when no installed +copy is found, so every Linux configure step does one anonymous +`git clone https://github.com/stephenberry/glaze.git` — the only +unauthenticated clone in the build. Inside this image that clone fails +most of the time: GitHub answers the `info/refs` GET with 200 and then +the `git-upload-pack` POST on the same reused HTTP/2 connection with a +spurious `401` and `www-authenticate: Basic realm="GitHub"`, which +surfaces as + +``` +fatal: could not read Username for 'https://github.com': No such device or address +fatal: expected flush after ref listing +Had to git clone more than once: 3 times. +CMake Error ... Failed to clone repository: 'https://github.com/stephenberry/glaze.git' +``` + +and fails Configure. It looks like a credentials or rate-limit problem and +is neither: it is Ubuntu 24.04's libcurl 8.5.0 / nghttp2 1.59 speaking +HTTP/2. Measured from a running runner container, ~7 of 10 `ls-remote`s +fail; with `-c http.version=HTTP/1.1` or `-c protocol.version=0`, 10 of +10 succeed. The same clone from the Docker host (same public address, +libcurl 8.21) is 10 of 10, and upgrading git inside the container to 2.55 +from `ppa:git-core/ppa` changes nothing — the libcurl underneath is the +same, and 24.04 has no newer one to install. Only *authenticated* +requests escape the 401, because git retries them with credentials, which +is why `actions/checkout` has always worked here and only the dependency +clone breaks. + +The Dockerfile therefore pins `git config --system http.version HTTP/1.1`. +Containers built from an older image keep failing until they are +recreated; to fix a running one in place, without disturbing the job it +may be executing: + +```bash +docker exec -u root morph-runner-1 git config --system http.version HTTP/1.1 +``` + ## Compiler cache: fastcache-cc `linux-compilers`, `linux-sanitizers`, and `linux-all-features` each set @@ -386,14 +411,13 @@ daemon reachable at that address from inside its containers**. There is no single shared cache across hosts; each host caches its own compiles. How `host.docker.internal` resolves differs by platform: -- **Docker Desktop (the Windows machine)**: resolves automatically to +- **Docker Desktop (Windows/macOS hosts)**: resolves automatically to whatever the host's `127.0.0.1` means — i.e. that machine's own - `fastcached` service (see `D:\caching` on that machine; **not** part of - this repository). That service must be **running** and its + `fastcached` service (kept outside this repository). That service must be **running** and its `fastcached.yaml` must **bind `0.0.0.0`**, not the default `127.0.0.1`, or a container cannot reach it at all (`127.0.0.1` inside a container means the container itself). -- **Plain Linux Docker Engine (the Hetzner box, or any cloud VM)**: does +- **Plain Linux Docker Engine (a workstation or a cloud VM)**: does **not** provide `host.docker.internal` automatically the way Docker Desktop does. `bootstrap-cloud-node.sh` adds it explicitly via `--add-host=host.docker.internal:host-gateway` on each worker @@ -410,7 +434,7 @@ How `host.docker.internal` resolves differs by platform: A developer machine's `fastcached` listens on loopback, which inside a container means the container itself. Two ways to give the runners a cache -on such a host, and this repository's Linux workstation uses the second: +on such a host; the second is what these runners are set up for: 1. **Add a listener on the Docker bridge gateway** to the existing daemon — `listeners:` in `fastcached.yaml` supersedes `bind`/`port`, so both @@ -430,8 +454,8 @@ on such a host, and this repository's Linux workstation uses the second: 2. **Run a second daemon, as a host process**, bound to the bridge address only and left on a released version, with the personal one untouched on - loopback. This is what the Linux workstation does, as a `systemd --user` - unit (`fastcached-ci.service`) running the 0.1.1 release binary: + loopback — e.g. as a `systemd --user` unit (`fastcached-ci.service`) + running the 0.1.1 release binary: ```yaml # ~/.local/opt/fastcached-0.1.1/fastcached-ci.yaml @@ -468,8 +492,8 @@ on such a host, and this repository's Linux workstation uses the second: `FASTCACHE_AUTO_INSTALL=ON` fetches the newest **released** `fastcache-cc` from the `fastcached` project's GitHub Releases. A daemon built from that project's `master` can be far ahead of its last release and **refuses that -client**, which is why the workstation runs a release build rather than the -one already on the box. +client**, which is why the daemon serving the runners has to be a release +build, not whatever from-source one is already on the box. Measured here: `fastcached 0.1.1-599-gea414a2` (a from-source master build) against the auto-installed `fastcache-cc` 0.1.1, configuring from inside a