GitHub Releases: Status 503 Egress is over the account limit.

[AustinTSchaffer]

Hello! I tried installing sharp for a new project this morning and ran into an issue with this repository.

What I did:

npm install sharp

What I got:

> sharp@0.29.3 install {my new project directory}
> (node install/libvips && node install/dll-copy && prebuild-install) || (node install/can-compile && node-gyp rebuild && node install/dll-copy)

sharp: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.11.3/libvips-8.11.3-darwin-x64.tar.br
sharp: Installation error: Status 503 Egress is over the account limit.
sharp: Please see https://sharp.pixelplumbing.com/install for required dependencies

I reran the install command with the AWS Lambda instructions, which then worked, though it looks like that command also downloaded resources from this repository. For future reference, should I just keep retrying the install command until GitHub sends the binaries?

[lovell]

Egress is over the account limit

This message is from (the Microsoft-owned) Azure and says (the Microsoft-owned) GitHub has reached the egress limit on their blob storage account.

The Azure docs mention an egress limit of 120Gbps, which I guess might apply to the whole of GitHub Releases.

https://docs.microsoft.com/en-us/azure/storage/common/scalability-targets-standard-account

@MylesBorins Is this a known limitation of GitHub Releases?

[lovell]

The only other example I can find of this specific error message in relation to GitHub Releases is from 3 days ago, which suggests this might be a new problem.

codecov/uploader#532 (comment)

[MylesBorins]

@lovell would you be able to open a ticket with GitHub support so we can track this internally. I'll escalate to the appropriate folks

[lovell]

@MylesBorins Thank you for the quick response, support ticket 1419586 created for this.

[lovell]

I got a reply from GitHub support, which quotes the same URL I provided:

Thanks for reaching out to GitHub Support!

The 503: Egress is over the account limit. error is pretty uncommon and usually transient.

You can find more information related to this error here:

https://docs.microsoft.com/en-us/azure/storage/common/scalability-targets-standard-account

If the error persists though, please let us know and we can ask our engineers to investigate further.

This appears to confirm that GitHub Releases:

• serves assets directly from Azure blob storage
• has a peak traffic level greater than the 120Gbps egress limit

Many others are reporting the same problem at community/community#8535

Anyone else experiencing this should contact GitHub support to provide more evidence via https://support.github.com

[ascorbic]

I wonder if it would make sense to load the binary via e.g. Cloudflare instead of direct from GitHub

[lovell]

Follow up response from GitHub support:

We have indeed had multiple reports of this bug over the last couple of hours and our engineers are working on a fix. It's very useful for us to have more data points, so thank you for submitting this report! I'll pass it on to the team.

[alexandrosgkesos]

Got the same while downloading kubectl

Error: Failed to install provider
Error while installing gavinbunney/kubectl v1.13.1: unsuccessful request to
https://github.com/gavinbunney/terraform-provider-kubectl/releases/download/v1.13.1/terraform-provider-kubectl_1.13.1_linux_amd64.zip:
503 Egress is over the account limit.

[lovell]

Further response from GitHub support:

Just wanted to update you on this: the team has identified and resolved the underlying issue, so you should no longer see any of these 503 errors.

[lovell]

It looks like release assets are being served via S3 again, specifically the us-east-1 region, so my best guess would be that this was a cascading failure situation related to yesterday's AWS outage.

[AustinTSchaffer]

@lovell

Thank you for keeping this thread updated on what was going on with GitHub, Azure, and AWS, despite it being well beyond the scope of this project. Best wishes and happy holidays.

[jgoux]

It doesn't seem fully resolved, we just hit the error in our CI right now. 😅

sharp: Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.11.3/libvips-8.11.3-linux-x64.tar.br
sharp: Installation error: Status 503 Egress is over the account limit.

[lovell]

If you are experiencing this problem, please contact GitHub support via https://support.github.com