CI: every self-hosted job can fail in Configure — FetchContent's anonymous git clone is answered with a credential challenge

[Yaraslaut]

Summary

Since #263 merged (d45dfeb1), CI on master and on every PR is red. The
failing jobs are exclusively the self-hosted ones, and they fail in the
Configure step, before any compilation: FetchContent's git clone of a
dependency is answered with a credential challenge, and git — having no tty —
aborts.

[1/9] Performing download step (git clone) for 'glaze-populate'
Cloning into 'glaze-src'...
fatal: could not read Username for 'https://github.com': No such device or address
fatal: expected flush after ref listing
Cloning into 'glaze-src'...
fatal: could not read Username for 'https://github.com': No such device or address
fatal: expected flush after ref listing
Cloning into 'glaze-src'...
fatal: could not read Username for 'https://github.com': No such device or address
fatal: expected flush after ref listing
Had to git clone more than once: 3 times.
CMake Error at glaze-subbuild/glaze-populate-prefix/tmp/glaze-populate-gitclone.cmake:50 (message):
  Failed to clone repository: 'https://github.com/stephenberry/glaze.git'
CMake Error at /usr/local/share/cmake-3.31/Modules/FetchContent.cmake:1918 (message):
  Build step for glaze failed: 1

stephenberry/glaze is public, so an anonymous clone needs no credentials. A
credential prompt means the server answered the ref advertisement with an
auth challenge rather than the refs.

Verification status

Measured on runs 33617999890 (master, d45dfeb1), 33626533084 and
33627249007 (PRs #394 and #395), all on 2026-09-02.

Confirmed:

• Self-hosted only. Every failing job in every one of the three runs carries
  runs-on = self-hosted,Linux,X64,morph-docker. No GitHub-hosted job failed in
  any of them.
• Not caused by the PRs. master at d45dfeb1 fails on its own, with no PR
  involved. PR ci: flip != back to == so those assertions are mutated at all #394 changes only scripts/scenario/*.py and cannot affect a
  CMake configure.
• Not a general network fault. In the same job, apt-get update over HTTPS
  to apt.llvm.org, archive.ubuntu.com and ppa.launchpadcontent.net all
  succeed, seconds before the clone fails. DNS, TLS and egress are fine.
• Not a static misconfiguration of the image. .github/self-hosted-runner/Dockerfile
  and entrypoint.sh set no proxy, no insteadOf, no credential helper and no
  GIT_CONFIG. The same job name and preset (Linux / clang-tsan,
  clang-tsan) succeeded on morph-docker-5 and failed on morph-docker-4
  inside run 33617999890. A bad ~/.gitconfig would fail every time.
• The failing dependency varies. glaze in one run, reflection-cpp in
  another — so it is not one repository or one URL.

Load correlation, measured — self-hosted jobs per run, and how many failed:

run	when	self-hosted jobs	failed
33617999890 (master, alone)	10:09Z	11	2
33626533084 (PR #394)	11:48Z	11	3
33627249007 (PR #395, overlapping the above)	11:56Z	11	6

The two PR runs overlap, putting ~22 self-hosted jobs across 5 containers into
the same ten-minute window. Failure timestamps cluster tightly inside it —
11:57:11Z (×2), 11:59:08Z, 11:59:28Z, 11:59:31Z, 12:01:02Z. The
failure rate rises monotonically with how many jobs are configuring at once.

Inferred, not proven: that the challenge is GitHub throttling
unauthenticated git traffic from a single source IP. The five runners are five
containers on one host (.github/self-hosted-runner/README.md), so they share
one egress IP, and each configure clones several dependencies plus their
transitive ones. That fits every observation above — intermittency, load
correlation, which-dependency-varies, and an otherwise healthy network — but
the HTTP status code is not in the logs, so the mechanism is inferred from
behaviour rather than read off the wire.

Not verified:

• The actual HTTP response. git does not log it without GIT_CURL_VERBOSE=1 /
  GIT_TRACE=1, neither of which is set.
• Whether GitHub-hosted runners survive because they are not throttled, or
  because they never share an IP for long. Not tested.
• I initially took "no git clone lines in a successful job's log" as evidence
  that successful jobs never clone. That was wrong — FetchContent only
  echoes its subbuild output when the subbuild fails, so a successful clone is
  silent. Recording it because it is an easy trap for the next reader.

What would change the verdict

• Confirmed if a failing configure is re-run with GIT_CURL_VERBOSE=1 and
  shows a 401/429 from github.com, or if authenticating the clones (below)
  makes the failures stop.
• Refuted if the failures continue at the same rate with authenticated
  clones, or if a run with exactly one self-hosted job in flight fails at the
  same rate as a saturated one.

Suggested direction, not yet implemented

Authenticate the FetchContent clones on the self-hosted path, so they are
charged against a token's quota rather than the shared IP's anonymous one:

- name: Authenticate git for FetchContent
  run: |
    git config --global \
      url."https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/".insteadOf \
      "https://github.com/"

That is the standard remedy and needs no change to CMakeLists.txt. A vendored
or host-side dependency mirror would also remove the exposure and would cut
configure time, but is a larger change.

Filed rather than fixed: this is CI infrastructure that #263 has just landed,
and whoever owns that setup should choose the remedy.

[Yaraslaut]

Evidence that this is fixed, from CI runs either side of #401

Recording this rather than closing, since PR #401 declared no Closes #397 and
the omission may have been deliberate.

Before — run 33617999890
on d45dfeb1, 2026-09-02T10:20Z. Two self-hosted jobs failed, both in checkout,
with exactly the symptom this issue describes:

fatal: could not read Username for 'https://github.com': No such device or address
fatal: expected flush after ref listing
  Failed to clone repository:

• Application ladder / ASan+UBSan — failure
• Linux / all optional features (clang) — failure

After — PR #401 ("keep the runner image's git off HTTP/2 so dependency clones
stop failing") merged at 14:58Z. Run
33645959731
on 130eb316, created 15:02Z:

• Probe self-hosted runner — success
• Linux / all optional features (clang) — success
• Application ladder / ASan+UBSan — cancelled (concurrency group, superseded by
  the next merge), so this job has no post-fix green yet.

Not verified: Application ladder / ASan+UBSan on a post-#401 commit. It is
in_progress right now on e675253a
(33650531644).
That run is the remaining evidence; if it goes green, every job named in this
issue has cloned successfully after the fix.

Worth being explicit about one coincidence that misled me briefly:
Linux / all optional features (clang) is also the job named in morph#391's
segfault report. The two are unrelated — #391 is a SegFault in ctest, this is a
checkout failure before any test runs.

[Yaraslaut]

Closing on the evidence recorded above: PR #401 fixed this, it simply carried no Closes reference.

Before, run 33617999890 on d45dfeb1 at 10:20Z — Application ladder / ASan+UBSan and Linux / all optional features (clang) both failed in checkout with fatal: could not read Username for 'https://github.com' / Failed to clone repository.

After — #401 merged 14:58Z; run 33645959731 on 130eb316 at 15:02Z had Probe self-hosted runner success and Linux / all optional features (clang) success. Self-hosted jobs have cloned successfully on every run since.

One thing deliberately not claimed: Application ladder / ASan+UBSan was cancelled by the concurrency group on that particular run, so it has no post-fix green of its own on that commit. It has run green on later commits, and no job has failed with this symptom since 14:58Z.

Unrelated to morph#391 despite sharing a job name — that one is a SegFault inside ctest, this was a checkout failure before any test ran.