Skip to content

Authenticate CI git fetches of GitHub repos - #610

Merged
samuelburnham merged 1 commit into
mainfrom
sb/authenticate-ci-git-fetches
Sep 2, 2026
Merged

Authenticate CI git fetches of GitHub repos#610
samuelburnham merged 1 commit into
mainfrom
sb/authenticate-ci-git-fetches

Conversation

@samuelburnham

Copy link
Copy Markdown
Member

GitHub throttles anonymous git HTTPS traffic per source IP, and Warp runners share egress IPs across tenants. Starting 2026-09-02 ~12:30 UTC that pool was throttled, so unauthenticated clones of public repos began failing with 401s ("could not read Username for 'https://github.com'"): Nix's eval-time builtins.fetchGit of Lake manifest dependencies broke Nix CI, and Lake's own dependency clone broke cuda-compile, across unrelated PRs.

Add an authenticate-github-fetches composite action that rewrites github.com URLs via url.insteadOf to carry the job's ephemeral GITHUB_TOKEN, moving every git-CLI fetch (Lake dependencies, eval-time fetchGit, script clones) off the shared anonymous IP pool and onto per-token limits. Wire it into:

  • setup-rust-toolchain, alongside CARGO_NET_GIT_FETCH_WITH_CLI=true so cargo's git dependencies fetch through the git CLI and pick up the rewrite (cargo's built-in fetcher ignores insteadOf);
  • both nix.yml jobs, which don't use that wrapper (install-nix-action's github_access_token covers only github: flake inputs, not git+https fetchGit);
  • the bench compile jobs in bench-main and bench-pr, whose lake build Compile<env> clones mathlib and friends without the wrapper.

In bench-pr this puts the token in ~/.gitconfig where checked-out PR code can read it, which persist-credentials: false previously kept off disk. That is acceptable: the workflow's permissions block pins the token to read-only on a public repo (the dispatch job's actions:write never runs PR code), so the exposure matches an ordinary pull_request run. Toolchain-only lean-action jobs (build: false) clone nothing and are left alone, as is CodeQL, which manages its own fetches.

GitHub throttles anonymous git HTTPS traffic per source IP, and Warp
runners share egress IPs across tenants. Starting 2026-09-02 ~12:30 UTC
that pool was throttled, so unauthenticated clones of public repos began
failing with 401s ("could not read Username for 'https://github.com'"):
Nix's eval-time builtins.fetchGit of Lake manifest dependencies broke
Nix CI, and Lake's own dependency clone broke cuda-compile, across
unrelated PRs.

Add an authenticate-github-fetches composite action that rewrites
github.com URLs via `url.insteadOf` to carry the job's ephemeral
GITHUB_TOKEN, moving every git-CLI fetch (Lake dependencies, eval-time
fetchGit, script clones) off the shared anonymous IP pool and onto
per-token limits. Wire it into:

- setup-rust-toolchain, alongside CARGO_NET_GIT_FETCH_WITH_CLI=true so
  cargo's git dependencies fetch through the git CLI and pick up the
  rewrite (cargo's built-in fetcher ignores insteadOf);
- both nix.yml jobs, which don't use that wrapper
  (install-nix-action's github_access_token covers only `github:`
  flake inputs, not git+https fetchGit);
- the bench compile jobs in bench-main and bench-pr, whose
  `lake build Compile<env>` clones mathlib and friends without the
  wrapper.

In bench-pr this puts the token in ~/.gitconfig where checked-out PR
code can read it, which persist-credentials: false previously kept off
disk. That is acceptable: the workflow's permissions block pins the
token to read-only on a public repo (the dispatch job's actions:write
never runs PR code), so the exposure matches an ordinary pull_request
run. Toolchain-only lean-action jobs (build: false) clone nothing and
are left alone, as is CodeQL, which manages its own fetches.
@samuelburnham
samuelburnham marked this pull request as ready for review September 2, 2026 19:10
@samuelburnham
samuelburnham added this pull request to the merge queue Sep 2, 2026
Merged via the queue into main with commit 74e7e25 Sep 2, 2026
14 checks passed
@samuelburnham
samuelburnham deleted the sb/authenticate-ci-git-fetches branch September 2, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants