Intermittent multi-minute connection timeouts to r2u.stat.illinois.edu from GitHub Actions amd64 runners (arm64 unaffected)

[jmgirard]

Summary

First, thanks for r2u — it's the backbone of a couple of Docker images I maintain, and binary installs make them far smaller and faster to build.

I'm hitting an intermittent connectivity issue when building r2u-based images in GitHub Actions, and the failure is oddly architecture-specific, so I wanted to report it in case it points at something on the mirror side (or a mirror I should be using instead).

Setup

• Multi-arch image built on top of rocker/r2u in GitHub Actions, one native runner per architecture:
  • amd64 on ubuntu-latest (GitHub's Azure-hosted runners)
  • arm64 on ubuntu-24.04-arm (GitHub's native ARM runners)
• During the build I run install.packages(...) for a batch of CRAN packages, which bspm resolves to apt fetches from r2u.stat.illinois.edu.

Symptom

The amd64 legs intermittently fail during that install with connection timeouts:

Unable to connect to r2u.stat.illinois.edu:http:
Could not connect to r2u.stat.illinois.edu:443 (192.17.190.167), connection timed out

When it happens, it's sustained, not a momentary blip — I retried the install 5 times over ~10 minutes from the same runner and every attempt timed out.

The part that seems diagnostic

Across four CI runs:

• arm64 (native ARM runners): 4/4 legs succeeded. Never once failed.
• amd64 (Azure runners): roughly half of the legs failed with the timeout above.
• Crucially, in the same run, a failing amd64 leg and a succeeding arm64 leg (and sometimes a succeeding amd64 leg) were running concurrently — so it isn't a global mirror outage. It looks tied to the specific runner's egress IP.
• r2u.stat.illinois.edu is IPv4-only (no AAAA record), so this isn't an IPv6 fallback problem and Acquire::ForceIPv4 doesn't apply. That pattern reads to me like per-source-IP reachability — e.g. rate-limiting / connection throttling of certain source ranges, or a eering issue between the Azure ranges and Internet2 — but I can't tell from the outside.

Questions

1. Is there any server-side connection throttling / rate-limiting / fail2ban-style blocking that could trip on bursty CI IPs?
2. Is there a CDN-backed or secondary mirror you'd recommend for CI use?
3. Is Azure ↔ Internet2 flakiness to the Illinois host something you've seen before?

What I've done meanwhile (so this isn't urgent)

I'm not blocked — I fail fast and re-run failed legs on fresh runners (new egress IP), which almost always connects, and the native arm64 builds are rock-solid. So please treat this as informational; I mostly want to know whether it's something on the mirror side, a mirror I should switch to, or just network peering I should keep working around.

Happy to provide full CI logs or run IDs if useful.

[eddelbuettel]

Thanks for the long write-up ... and I am aware, and see it each time I build packages. This has been ongoing for many months now. The cynical side in me sees this as planned brownouts by GH. I think I am also seeing amd64 timeout, but I did not keep statistics so your observation that is happens more at amd64 may or may not be correct. Either way both are the same network hops away from the main server.

And no, I have no general fix at this time. It seems to be mostly inside the client networking side at GH. I never see anything server-side (where I have access to logs).

[eddelbuettel]

You keen observation that this only appears to matter for amd64 is still in the back of my mind. Maybe I should shift the 'binary-all' packages (not containing compiled code) from amd64 (default) to arm64 (where I so far only build what need compilation).

If you have any other hints I'd be all ears :)

[jmgirard]

I wrote a small probe that repeatedly tries to reach r2u.stat.illinois.edu from fresh amd64 and arm64 runners — ~30 runs each, across ~24 amd64 IP ranges and ~11 hours (including the time of day my one real failure hit). Every run reached the mirror fine, both architectures.

So two honest caveats to my original post:

• It's intermittent and currently much rarer than the ~50% I saw — I couldn't trigger a single failure.
• I can't independently confirm the amd64-vs-arm64 difference — nothing failed either way, so the direction still rests only on my earlier logs.

One observation from re-reading that earlier failure log (same data, not new): the runner couldn't reach the mirror for the whole job (~9 min, retries included), so retrying inside the build didn't help — re-running on a fresh runner cleared it.

@eddelbuettel — consistent with your "may or may not be correct": I couldn't make the difference (or any failures) show up on demand.

[eddelbuettel]

Excellent idea trying to force it. Surprised it didn't tickle it.

I more or less can guarantee failure.

For r2u, when I build 'early morning my time' (here in Chicagoland) I generally have 60 to 80 packages. I tend to have at least two fail, often more. If we wanted to be clever we could look at the GitHub Actions log of the repo r2u-builder (under my handle). You will see that most runs require more than one take (and often finished on the 2nd run 're-run failures'. Not sure we need to go the extra mile of extracting logs but it should suit as a quick eyeball check for you.

Similar, other CI runs relying on r2u to supply binaries are hampered the same way. Today it was the Rcpp repo building the Docker containers I create / use for CI / provide there. Same thing: access to r2u times out, the r2u server never has any marks in its logs, this still seems 100% local to GHA.

[jmgirard]

Thanks, r2u-builder was exactly the right place to look.

Your impression checks out: of 1,203 build.yaml runs, only 38% finished on the first attempt.

And the amd64/arm64 difference is real in your own logs. The failures live in attempt 1 (the API's default job list only shows the latest attempt, so re-run-fixed ones look invisible). Taking only cases where the same package was built in the same run on both arches — 994 pairs:

• both succeeded: 913
• both failed: 38 (genuinely broken packages)
• only amd64 failed: 42 — only arm64 failed: 1

Same picture on noble and resolute separately, and the failures are unambiguously this issue (Could not connect to r2u.stat.illinois.edu:443 (192.17.190.167), connection timed out).

One hypothesis for why: the CPU itself shouldn't matter. So "amd64" is probably standing in for something else, and the obvious candidate is the source IP range: across ~30 probe runs per arch, I found the amd64 and arm64 runner pools drawing from almost entirely disjoint Azure blocks (22 /16s seen only on amd64, 13 only on arm64, 2 shared). That would make this a network-path effect rather than an architecture one.

Which suggests one cheap change: add a curl -s https://api.ipify.org step to the builder jobs. Your builds already fail often enough that within days you'd know exactly which prefixes are affected. Combined with "connection timeouts but nothing in the server logs," that gives campus networking something concrete to check at the border.

[eddelbuettel]

I just added an entry to the r2u FAQ referencing this as it is likely that it bites other users.

Also, GitHub is having a bad day too and Actions are generally down so it may take a moment until this is updated on the public page. Until then there is a screenshot from a local rendering here.

PS Now updated given that the Action outage is over.

[eddelbuettel]

"Anecdata" from a small run but

i) arm really does not seem to fail and
ii) maybe amd64 fails more on 24.04 (== higher usage?) than 26.04?