Severe transfer degradation and connection resets when downloading individual files from private repository via REST API

[mjmokhtar]

🏷️ Discussion Type

Question

Body

Hello GitHub Support,

I am experiencing a reproducible issue when downloading individual files from a private repository through the GitHub REST API.

Repository: mjmokhtar/docs-for-learning

The Contents API successfully returns metadata, authentication works correctly, and downloading the entire private repository archive works normally. However, requesting individual binary files using the raw media type becomes extremely slow and eventually terminates or is reset.

Example request:

GET /repos/mjmokhtar/docs-for-learning/contents/career/roadmap.png

with:

Accept: application/vnd.github.raw

One reproduction resulted in:

HTTP: 200

Expected file size: ~372 KB

Received: 162426 bytes

Average speed: 533 bytes/sec

Duration: 304 seconds

curl: (56) Recv failure: Connection reset by peer

A 2.38 MB PDF from the same repository also repeatedly fails. Node.js/Undici reports:

UND_ERR_SOCKET: other side closed

In contrast, downloading the entire private repository archive through:

GET /repos/mjmokhtar/docs-for-learning/tarball/main

successfully transferred approximately 1.69 GB at ~3.2 MB/s.

A public repository archive downloaded through api.github.com also transferred approximately 12 MB in 4.75 seconds (~2.5 MB/s).

Therefore, general connectivity from this server to GitHub appears healthy. The degradation seems specific to individual authenticated raw/blob file delivery.

I can reproduce the problem directly with curl, without my application, reverse proxy, or Cloudflare involved.

Could you please investigate whether there is an issue with the backend/CDN path serving authenticated raw/blob objects for private repositories, or advise what additional diagnostics would help identify the problem?

Thank you.

[kittimzhe]

Your connectivity is clearly fine (tarball at 3.2 MB/s, public archive at 2.5 MB/s), so this is specific to the authenticated raw/blob single-object path — not your server.

Two things are going on, and both are best solved by switching retrieval path:

1. The Contents API isn't built for binary/file streaming. Accept: application/vnd.github.raw on GET /repos/{owner}/{repo}/contents/{path} is meant for source text and tops out around 1 MB. Your 2.38 MB PDF is firmly over that; the ~372 KB PNG is under it, but you're on the same non-optimized path, which matches the slow-then-reset symptom (curl 56 / UND_ERR_SOCKET).

2. Use the Git Blobs API for individual files (the supported path, up to 100 MB):

TOKEN=...
sha=$(curl -sH "Authorization: token $TOKEN" -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/mjmokhtar/docs-for-learning/contents/career/roadmap.png" | jq -r .sha)

curl -sH "Authorization: token $TOKEN" -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/mjmokhtar/docs-for-learning/git/blobs/$sha" \
  | jq -r .content | tr -d '\n' | base64 -d > roadmap.png

The blob endpoint returns base64 (.content + .encoding: "base64"); decode it. It goes through a different backend path than the raw media type and transfers reliably.

3. For raw bytes directly, point at the raw host instead of api.github.com:

curl -sH "Authorization: token $TOKEN" \
  "https://raw.githubusercontent.com/mjmokhtar/docs-for-learning/main/career/roadmap.png" -o roadmap.png

Worth adding --retry 3 / -C - so a transient mid-stream reset auto-resumes rather than killing the transfer.

Since tarball is orders of magnitude faster than single raw objects, this also smells like a cold-object/CDN stall on the authenticated raw path — worth a staff look — but the Blobs API above unblocks you right now.

[mjmokhtar]

Thanks — this is helpful and matches what I've been seeing.

One important detail: I already tested the Git Blobs API as well, and unfortunately it shows the same behavior.

For a 2,381,428-byte PDF, I first retrieved its SHA successfully:

696ee2289d4f886f5564d0d47844e65e9922803b

Then I tested:

GET /repos/mjmokhtar/docs-for-learning/git/blobs/

with Accept: application/vnd.github+json.

The request returned HTTP 200, but the transfer stalled and terminated after receiving only ~767 KB (~23% of the JSON/base64 response) after ~69 seconds with:

curl: (18) Transferred a partial file

I also tested the same Git Blobs endpoint with:

Accept: application/vnd.github.raw

That transferred only ~801 KB (~33%) before failing after ~52 seconds with:

curl: (92) HTTP/2 stream 1 was not closed cleanly: CANCEL (err 8)

I also reproduced the issue with a much smaller file (roadmap.png, ~372 KB) through the Contents raw endpoint. It received only 162,426 bytes in 304 seconds (~533 B/s) before:

curl: (56) Recv failure: Connection reset by peer

Meanwhile, a ~1.69 GB private tarball of the same repository downloads successfully at ~3.2 MB/s.

So unfortunately, switching from Contents API to Git Blobs API does not appear to bypass the problematic path in my case.

I agree that this increasingly looks like something specific to the single-object delivery path. If you know another endpoint/path worth testing, I'm happy to run it.