Can't upgrade/'Bandwidth exceeded'

[rwoodpecker]

Visting https://yt-dl.org/downloads/latest/youtube-dl shows 'Bandwidth exceeded'. Unable to upgrade or download youtube-dl.

[yan12125]

@phihag Are these servers still managed by you?

[phihag]

Yes they are. Unfortunately, it seems we're a victim of our popularity here; the hoster just contacted me because we're using way too much bandwidth. I'll look into it.

[Hrxn]

Maybe we should do some fundraising (i.e. donations)?

I'd be willing to contribute..

Edit:

Also, in addition to hosting binaries on GitHub, maybe split the load to other sites like this: https://bintray.com/

Short-term methods for traffic alleviation ..

[phihag]

While we certainly do appreciate donations, we'll need another hoster. We're in the process of moving right now.

For now, updating should work again, certainly did for me.

[yan12125]

@Hrxn Does bintray.com impose bandwidth limitations?

[Hrxn]

Well, not that I'm aware of. I just checked their site, and they have different account types and service levels. But they offer a Community account type, similar to GitHub, i.e. for open source projects / public projects.

Free for OSS distribution
Downloads: Included
Data Storage: Included
CDN: Yes

The limitations for that account type are: No private repositories, no commercial-content repositories, there's a limit to the amount of REST API calls, you can't use your own domain name and advanced rights/access management is not available.

Not something really needed, so looks good to me.
The ToS don't mention anything either.

And they offer nice automation features, as well as a good integration with GitHub, which is pretty useful, I think.

W.R.T. Donations:

You accept Bitcoins, much appreciated, was planning to get some anyway. I haven't touched this stuff in the last couple years, can you maybe point me to a good exchange or something?

[yan12125]

According to statistics provided by @phihag, neither Github nor Bintray is suitable. Maybe reduce the traffic in an update is the way. There are several approaches:

1. For Windows users, do not update the whole youtube-dl.exe but download a new zipball and use it.
2. Do not update the whole zip file but just a zip with updated .py files. This requires hosting servers with compression utilities.
3. Distribute BZ2 tarballs instead of zipballs and load them from a bootstrap script. BZ2 can support all versions from Python 2.6 to 3.5.

[phihag]

Are there any other hosters that can take 10TB/month? Squarspace famously advertises with unlimited traffic.

[kidol]

Don't know what requirements you have, but for a test you could check out scaleway.com (https://www.scaleway.com/pricing/). Will cost you 3,50€ (incl. VAT) a month with 200mbit/s, 2GB RAM & 50GB SSD.

Re unlimited bandwidth: https://community.scaleway.com/t/need-advice-for-distributing-game/800

Partial question from user:

So let's say we need at least 15TB of bandwidth a month...

Admin later answers:

There is no maximum time for the 200mbit/s, you can use it as long as you want.
If you go above 200mbit/s for a long period of time, you will need to take the premium bandwith.

[Hrxn]

According to statistics provided by phihag, neither Github nor Bintray is suitable.

Why is that? Did you find something about a Bintray limitation (traffic volume, concurrent bandwith) somewhere hidden on their site? Or is it API access or something like this?

[yan12125]

Bintray's pricing plans cost too much and our account can't afford that.
For Github: I've mailed them and they said Github doesn't like 10TB/month.

[yan12125]

Wait. Does bintray offer free plans for open source projects, no matter how much traffic?

[Hrxn]

Yes, free plans for open source. I thought I said that. 😉

I searched their site and ToS, and it didn't said anything about traffic limitations.
So, yes. But no limits usually means "within reasonable limits", but I think 10 TB / month is pretty reasonable.

But binary distribution is the purpose of this site, as opposed to GitHub, who focus explicitly on source code. I am not surprised that they wouldn't be too happy here, considering that they didn't allow binary distribution at all first.

[yan12125]

Bintray's response:

-250MB limit for single file uploads
-10GB of storage quota per account
-2TB of monthly downloads

Bintray is out, too. 😞

[Hrxn]

Oh, wow. 2TB / month. Fine if you basically have no users, because that's like almost nothing. I don't know what they expect.

If they want to to succeed commercially with their site, they have to do something for their popularity and visibility.

I don't think that they're well-known yet. Or is it just me, and everyone else has already heard about these guys? 😉

[rg3]

At this point, I would ask prgmr.com explaining the full situation, of course. Their plans have prioritized traffic but no specific data limit. Also, is all the traffic legit? Doing some simple math results in millions of downloads a month. Is that real traffic or are there any specific abusers?

[phihag]

@rg3 I'm a little bit surprised, too. I haven't done detailed analysis, but we get about 300k requests on youtube-dl.exe, and 200k on youtube-dl per release.

But the statistics by hosts don't show any surprising patterns. 600GB is caused by hosts with failed reverse DNS lookup. But no 2 in the list by traffic is already only 35GB, from a dynamic IP in Taiwan. 125.17.12.0 is hitting us 300M times/month, but only for /ip, so that traffic is miniscule.

The only thing that does look strange is user agents (number of requests):
980K BubbleUPnP UPnP/1.1
220K youtube-dl masquerading as Firefox 20 or 44
230K Python-urllib/2.7 (I guess very old youtube-dl?)
200K InnoTools_Downloader (whatever that is)
70K NSISDL
50K wget
300K various real-looking webbrowsers in the top 15

Maybe it would be a good idea to encode youtube-dl's platform, version number and file path in update requests. In addition, BubbleUPnP seems to stand out, but my guess is that they are responsible for the massive number of requests to /ip.

[rg3]

I have contacted prgmr.com and they don't think their plans would work. However, they were kind enough to point me to CloudFlare. It seems they have free plans that could work for our case.

/cc @FiloSottile who works there BTW.

PS: I forgot the link: https://www.cloudflare.com/plans/

[FiloSottile]

We can definitely help. It should work out of the box: throw a Free plan in front of the domain, and add a Page Rule to set the Cache Level to Cache Everything on the downloads, and one to limit the cache lifetime of the updates JSON. We'll probably eat most of the bandwidth and make the downloads much faster :)

Let me know if you have any issues.

[FiloSottile]

Or even better, Cache Everything on all downloads, and call the CF API to purge the cache of /latest/ and of the json as part of the release process.

[WassimAttar]

youtube-dl.exe : 2.11 MB
youtube-dl : 1.2 MB
According to @phihag for each release 300k downloads for youtube-dl.exe and 200k downloads for youtube-dl.
So, each release uses 300kX2.11 = 633MB + 200kX1.2 = 240MB => 873MB of bandwidth.
There is an average of 25 releases / month, so 873MBx25 = 21GB / month of bandwidth.
@phihag says that he is using 10TB / month of bandwidth.
It's 480 times more than is needed.
Where all the 9.9TB bandwidth is going ?

[WassimAttar]

In the other hand, let's assume that 10TB are used monthly to download binaries.
It means that each month 6.3 millions copies are downloaded.
Isn't a lot ?

In any case, something is wrong.

[phihag]

@WassimAttar Please ensure that your calculations and your technical understanding of the matters at hand are sound. In May, we had ~10 releases.

(300K * 2MB + 200K * 1MB) * 10 = (600GB + 200 GB) * 10 = 8TB

[WassimAttar]

Oups, i made 100 factor error.
But according to @dstftw "95% only update when experience some breakage."
It means that 6M downloads are from new users.
6M new users / month ?
For a command line program ?
Are you sure about that ?
It's more than any trendy app.
At the end of year you'll have more than 50M users and in 12 months more than twitter users.
Something still incorrect.

[dstftw]

It's been a month since we temporarily (?) moved to GitHub for hosting releases and during this time we've consumed expected 8 TiB of bandwidth:

total downloads traffic: 8.19TiB (8999693569268 bytes)

@rg3 have you received any concrete complaints from GitHub staff yet?

[rg3]

None so far. Cloudflare is still an option, though.

On Sun, Jul 3, 2016, at 17:27, Sergey M. wrote:

It's been a month since we temporarily (?) moved to GitHub and during
this time we consumed expected 8
TiB
of bandwidth.

total downloads traffic: 8.19TiB (8999693569268 bytes)

@rg3 have you received any concrete complaints from GitHub staff yet?

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#9690 (comment)

[yan12125]

As reported by @N1CEyk at #10123 (comment), Github.com is blocked in China. Although I'm not sure whether Cloudflare would be also blocked or not.

[N1CEyk]

I have tried Cloudflare just now.it is OK and needn't proxy at present.