Output crate pages so that the latest version is easily scrapable

[brson]

Debian uses automation to scrape web pages for updates to libraries, and for some reason these tools don't seem to easily understand crates.io. Find out what kind of structure they want and implement it.

[sanxiyn]

Re: "for some reason". crates.io is a JavaScript webapp. Static HTML would be more amenable to scraping.

[anguslees]

Right, crates.io is a javascript app that hits the crates.io API (JSON over HTTP). The good stuff is actually buried in a URL like https://crates.io/api/v1/crates/$crate/versions

From our (Debian's) point of view, we need an HTML page somewhere that our tools can scrape looking for <a href=...> links to discover/download new versions. This limitation is completely a consequence of the Debian tools and isn't something the Rust community needs to solve, except that I suspect the solution might be reuseable for other distros/tools too.

Option 1:
We add something to the crates.io website to return a suitable HTML page somewhere (perhaps as a new API endpoint that returns HTML rather than JSON?)

Option 2:
We do something similar within Debian infrastructure by adding a crates.io-specific piece of logic to http://anonscm.debian.org/viewvc/qa/trunk/cgi-bin/fakeupstream.cgi?view=markup - this would be a proxy that does the JSON API query and reformats it as an HTML webpage of download links.

I don't want this to be a big thing, and I'm ok with either approach.

[alexcrichton]

I suspect that if y'all have a tool already to turn JSON into HTML that'll be the easiest way forward, it unfortunately isn't trivial to add HTML endpoints for us :(

[alexcrichton]

Now that being said we could also just add an endpoint that returns a string corresponding to the latest version

[anguslees]

Bug (with patch) adding crates.io support to fakeupstream.cgi: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=808790

[anguslees]

My fakeupstream.cgi patch is now live, eg: https://qa.debian.org/cgi-bin/fakeupstream.cgi?upstream=crates.io/libc

I think we can close this issue now (or reduce its urgency).

[alexcrichton]

Thanks @anguslees!