We need to move out of GH Downloads

[FiloSottile]

From a mail I sent to @phihag

https://github.com/blog/1302-goodbye-uploads

Put short: GH is discontinuing the Downloads section and now web uploads are disabled.
Less-than-24h-after-I-started-using-it.

What does this mean?
Well, this depends on wether API uploads are still active (I have to check, I'll have time in the evening).

If they are, then we simply have 90 days to find a new downloads location, upload a new version that updates from the new location, and wait for it to propagate.
No big troubles, only annoying, as I have to redo half of the work.

If they are not, then we are screwed. Because we have ~700 people with an un-updateable version.
I have some contacts at GH and their support is usually really emphatic and skilled, so we will probably get them to push one last version for us. Then, everything will be as above.

I'm really sorry that this happened, but I couldn't foresee it. Sorry.

My suggestion on the new place:

• web server (offered by @gcmalloc) + CloudFlare
• gh-pages (for the index) + S3 + CloudFlare (my favorite, as it leaves control to repo pushers, whoever they might become)

[phihag]

The latter one looks fine. We'll support gh-pages for the foreseeable future anyways. I'm not sure why we need S3 and CloudFlare.

Alternatively, we could also use youtube-dl.org, but gh-pages is more convenient for the time.

[FiloSottile]

Well, the best would be to host a index on gh-pages with pointers to outside URLs, so that repository weight doesn't get more out of control that it is already (my fault, I have to admit). gh-pages files get always cloned along with master...

[phihag]

Oh, that didn't translate well to text, and my lack of sleep didn't help: I'm not sure why we need S3 and CloudFlare. One of them should be sufficient, shouldn't it?

[FiloSottile]

Oh, sorry. Well, money ;) you pay for bandwidth on S3.

[rg3]

On Wed, Dec 12, 2012, at 16:15, Filippo Valsorda wrote:

Oh, sorry. Well, money ;) you pay for bandwidth on S3.

This is no doubt a step back for Github. SourceForge has been serving
file releases for decades with no issues.

[phihag]

Assigning to me, I'm in the process of setting up the needed storage space, access, and download mechanism. @FiloSottile Please keep everything in place for now. We'll redirect to the new URLs soon.

[FiloSottile]

Ok, I was about to start, too, but I'll leave it to you.
So have you checked if API uploads work?

[phihag]

I'll simply set up a dedicated server + domain, and give SFTP access to the devs (i.e. you, and @rg3 if he wants). By using an indirection over gh-pages, the whole mechanism should work in the future as well if we should decide to switch to another file hoster.

[FiloSottile]

Great. And I can confirm that we are still able to upload via the API.
This is great, because we can simply push a new version with the new updating scheme, and wait for people to update.

[phihag]

Server is set up, domain ordered. Steps to do:

• One-step Windows building
• make build which builds the whole shebang(incl. Windows) and creates build/$VERSION/{youtube-dl,youtube-dl.exe,youtube-dl-src-$VERSION.tar.gz}
• Make devscripts/release.sh do some smoke tests on all of the above files (with strings) to make sure all bear the right version number
• Make devscripts/release.sh use the current version instead of specifying a new one.
• Make devscripts/release.sh upload the newest files to $SFTP_URL, with he same directory structure as in build/
• Make devscripts/release.sh write into the gh-pages branch and:
  • Update the current hashsums
  • Update all links to $URL_BASE/$VERSION/{youtube-dl,youtube-dl.exe,youtube-dl-src-$VERSION.tar.gz}
  • Create a file in gh-pages: update/$VERSION and write $URL_BASE/$VERSION/youtube-dl in there
  • Create/update a file in gh-pages: update/LATEST and write $URL_BASE/$VERSION/youtube-dl in there
  • Commit and push that
• Ensure that devscripts/release.sh updates LATEST_VERSION
• Move devscripts/release.sh into Makefile, so that make release works, and implies build
• Update youtube-dl(.exe) to:
  1. Fetch LATEST_VERSION
  2. If that is newer, fetch gh-pages:update/$VERSION
  3. And download that URL
• Update the placeholder versions in master and downloads to always fetch the URL in gh-pages:/update/LATEST

[gcmalloc]

bitbucket could also be an alternative, as they provide everything github already have, including the download section with the corresponding api.
Edit: bitbucket doesn't support travis yet, according to travis-ci/travis-ci#667.

[phihag]

@gcmalloc We'd still leave all the issues + git on github, so missing travis wouldn't hinder us (besides, we could simply multipush). But I'm not sure that bitbucket has a download API; at least, I can't find any mention on http://restbrowser.bitbucket.org/ . Can you post me to a link which documents the API?

[gcmalloc]

Seems like there is a location for files, but no api. According to this thread, they are working on it :
https://bitbucket.org/site/master/issue/3313/file-uploads-for-add-download

[FiloSottile]

My idea was that of putting a update_info.json on gh-pages like below, so that we depend only on that branch, and not on master. LGTY?

{
   latest: "2012.12.12",
   versions: [
      ...
   ]
}

It is also easier to build "old versions" tables this way.

[phihag]

That's fine too, but we'd have to fetch that every time. While the file is likely to stay small, a directory where each version corresponds to a subdirect/file should work just as well, shouldn't it? And it would have the advantage of being compatible with uscan et all, which do not support JSON.

[FiloSottile]

Sure, but it would be slightly less flexible hosting-wide (we need control over directories on the server and can't easily put files in separate places). We can keep a uscan-compatible layout also with the JSON.

Anyway I am ok with anything that does not depend on master.

My idea of solution ATM would be: LATEST_VERSION on gh-pages, small to fetch; update-index.json on gh-pages; uscan compatible layout on the server.

[phihag]

I don't really care where LATEST_VERSION is, but moving everything built to gh-pages sounds like a great idea. I'll be offline and occupied until Tuesday, so feel free to reassign this issue and work on it.