GitHub Community
Subject: ghcr.io blob downloads reset mid-transfer over IPv6, IPv4 unaffected #203736
Replies: 4 comments 1 reply
|
💬 Your Product Feedback Has Been Submitted 🎉 Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users. Here's what you can expect moving forward ⏩
Where to look to see what's shipping 👀
What you can do in the meantime 💻
As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities. Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐ |
|
Possibly related, with packet-level evidence: #203736 Same region (FRA edges), but the failure mode there is specific to IPv6: If anyone here is on a dual-stack line, forcing IPv4 for |
|
this is a very solid report... the matching hop limit is a useful signal that the reset was not generated anywhere close to the client. that said, it still does not tell us whether it came from the cache process itself or from an L4 device somewhere else inside the same POP. at this point, the ball is mostly in GitHub/Fastly's court. the most useful next step would be to correlate the supplied request IDs with the corresponding TCP 4-tuples and inspect the IPv6 frontend state, particularly:
two additional control tests could help narrow this down:
if transfers stay clean at concurrency 1 but start falling over as concurrency increases, that would be a pretty strong signal of edge state handling rather than PMTU, routing, or a local firewall issue. forcing IPv4 is a perfectly reasonable workaround for now, but the evidence here is already strong enough to justify a proper investigation on the provider side :) |
|
Thanks — both of your points were worth chasing, and the second one changed the picture. First, a correction to my own report. You are right to ask about ordering, and my claim does not What does survive is on the second connection in the capture. After the RST arrives, the edge arrival 0.947 RST Those cannot have overtaken the RST, and a stack that had reset the connection would not retransmit Second, your HTTP/1.1 idea. I ran it interleaved so both protocols see the same conditions - HTTP/2 160 ok / 16 failed of 176 9.1 % But the raw counts understate it, because the two failure modes are not the same: HTTP/2 16x curl 56, http 200, aborted mid-body after 11.6 - 23.6 MB Every HTTP/2 failure is the mid-transfer abort. The two HTTP/1.1 failures never got a response line Your third suggestion also produced something. Pinning every request to a single edge address made HTTP/2, spread over all four addresses 45 ok / 3 failed So concurrency against one edge raises the rate rather than distributing it away, which points at One more detail that may be a hint: the abort positions cluster. Of the 16 HTTP/2 failures, eight Revised summary: the defect is in the HTTP/2 path of the edge, it needs concurrency against the same One more control, since the question of local interference kept coming up. I repeated the HTTP/2 run HTTP/2, all local inspection disabled 54 ok / 10 failed of 64 15.6 % The rate did not drop. In the same capture the resets arrive with hop limit 56 and 57, matching the The abort positions in that run are worth repeating, because they are not spread out: 11.4 11.5 11.7 12.0 12.1 MB Ten of ten fall into those two narrow bands, and the same two bands showed up in the earlier runs. Practical note for anyone finding this: forcing HTTP/1.1 is a workaround for direct downloads, but |
|
nice follow-up. the retransmits after the RST, with the HTTP/2 vs HTTP/1.1 result is probably the biggest clue here. same for the pinned-edge test - that makes it look a lot more like per-edge state than some random network issue. flow control is definitely a reasonable guess, though i would not call it proven just from the clustered offsets yet. really nice debugging work :)) |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
🏷️ Discussion Type
Bug
Body
Pulling image layers from ghcr.io fails intermittently over IPv6. The edge resets the
connection in the middle of the body. Over IPv4 the same download, from the same host,
at the same time, with the same concurrency, never fails.
Since the original report the picture has narrowed considerably. Current state:
The follow-up runs and all request IDs are in the "Update" section further down.
Measurement, 2026-08-02 10:20-10:22 UTC
16 concurrent requests, 6 rounds per address family, same 24.5 MB object, both families
over the same interface and the same uplink. Client is a Vodafone DE connection in
Germany, so the requests land in FRA.
Both address families terminate in the same destination AS. 2606:50c0:8001::/48 and
185.199.108.0/22 are both announced by AS54113 (Fastly), so this is not a case of the two
protocols reaching different providers or different infrastructure. Same company, same
edge software, same origin - only the address family differs, and only one of them fails.
Visible path towards the failing edge (the remaining hops do not answer):
Payload arrives with hop limit 56 and 57, so roughly seven to eight hops between your edge
and me.
The eight failures, with your request IDs:
8 different FRA edges are involved, so this does not look like a single bad
machine. The shield node was cache-iad-kcgs7200053-IAD for every request in the run,
successful ones included.
What the packet capture shows
Client-side capture of one failure (local port 36762, epoch timestamps):
The transfer is healthy up to the reset: every segment acknowledged immediately, receive
window around 1 MB, no retransmissions, no duplicate ACKs, no zero-window. The RST is
sequence-correct. Six milliseconds later the same edge continues sending payload on the
connection it just reset. The second failure in the capture shows two RSTs 24 ms apart,
also followed by more payload.
On the ordering of those two packets: a bare RST carries no TCP timestamp, so arrival order
alone does not prove the edge sent the payload after the reset. What does settle it is the
second connection in the same capture. After the RST arrives, the edge retransmits the same
segment repeatedly, with its own TSval advancing each time:
Those cannot have overtaken the RST, and a stack that had truly reset the connection would
not retransmit on it at all.
What I ruled out here
The RSTs arrive with IPv6 hop limit 56 and 57, the same values as your payload packets
(133766 packets at 56, 353346 at 57). A reset injected inside my network would arrive
with 63 or 64, so it is not a middlebox on my side.
My firewall logged nothing for this host or for 2606:50c0::/32 in the relevant seconds.
All of its inspection features were enabled during the test, and the same host does
appear in that log at other times, so it is not silently filtering this traffic.
No ICMPv6 and no retransmissions before the resets, so it is not path loss or PMTU.
No 429 and no 403. The response is a normal 200 and the abort happens deep in the body,
so it does not look like ordinary rate limiting either.
Why this matters in practice
Talos Linux pulls its bootstrap images from ghcr.io. containerd fetches three layers in
parallel per node, and a three-node control plane boots at once, so about nine concurrent
large transfers hit your edge. Every abort makes containerd retry the whole layer. Worst
case here was a node stuck for 21 minutes on ghcr.io/siderolabs/kubelet:v1.35.6 before a
retry got through. The same install over IPv4 completes in about two minutes.
Forcing HTTP/1.1 avoids the problem for direct downloads, but it is not a usable workaround
for image pulls: containerd negotiates HTTP/2 via ALPN and offers no switch to turn that off.
Update 2026-08-02, 13:08-13:29 UTC: HTTP/2 vs HTTP/1.1, and request IDs for all of it
Collected here so the identifiers are in one place. Positions below are in MiB (2^20); the
figures in the section above use the same base.
Summary of the follow-up runs
The two failure modes are not comparable. All 16 HTTP/2 failures are the mid-body abort
described above: http 200, curl exit 56, connection dropped deep into the body. The two
HTTP/1.1 failures never received a response line (http 000, zero bytes), which is a
connection-level problem and not the same defect. On that measure the mid-body abort is
16 of 176 over HTTP/2 and 0 of 176 over HTTP/1.1.
Runs were interleaved - alternating rounds of 16 concurrent requests - so both protocols
saw the same conditions, same object, same interface, IPv6 throughout.
Pinning every request to one edge address made HTTP/2 worse rather than better:
Request IDs, all 16 HTTP/2 failures
15 distinct FRA edges across these 16 failures, behind two shields
(cache-iad-kiad7000171-IAD and cache-iad-kiad7000049-IAD).
One clarification about "pinned": forcing the destination address does not pin the edge. The
48 pinned HTTP/2 requests all went to 2606:50c0:8000::154 and were still answered by 47
distinct FRA edges - essentially the same spread as the 48 unpinned ones, which saw 44. So
the higher failure rate under pinning is not the effect of repeatedly hitting one machine.
Whatever a single destination address concentrates, it sits upstream of edge selection.
Control run with local inspection disabled, 13:29 UTC
Repeated over HTTP/2 with region blocking, intrusion prevention and content filtering
switched off on my gateway:
The rate did not drop. In the capture taken during this run the resets again arrive with
IPv6 hop limit 56 and 57, matching the payload exactly (88462 packets at 56, 171557 at 57),
so they do not originate near me.
I do not have x-fastly-request-id for these ten - that run logged only the local port and
x-served-by, and I did not notice the omission until now. The edges and timestamps:
I can repeat that run with request IDs recorded if it is useful.
Where the transfers stop
All 26 HTTP/2 aborts from the runs above, sorted, in MiB of a 24.5 MiB object:
Thirteen in each band, and nothing at all between 12.3 and 21.0 MiB - across three separate
runs, two shields and more than twenty distinct edges. I am deliberately not claiming this
proves a flow-control window; it is an observation about where the transfers stop, and the
mechanism is yours to determine. It is simply hard to reconcile with random loss.
What would help
Can you check what makes the FRA edges reset these connections over IPv6 on the HTTP/2 path?
The request IDs above should let you locate the failing transfers. I have the full
per-request logs for every run and the packet capture from the control run, and can supply
either on request, along with a larger sample if that helps.
All reactions