Switch to Accept: application/json when downloading crates

[alexcrichton]

Currently we fetch a url on crates.io which redirects us to S3, but crates.io has a feature where if you specify Accept: application/json we'll get a JSON blob with the url instead. We should switch Cargo to this interface so we can learn auxiliary information in the future if necessary, and we still know how to get the final crate!

[ehuss]

What extra information might it learn or display?

[alexcrichton]

We discussed this with some crates.io folks briefly today and @sgrif mentioned that one possible use is to send auxiliary information about things like yank reasons, although this'd largely be future proofing moreso than anything else I think. @sgrif or other crates.io folks may have more to fill in though!

[sgrif]

Can confirm this is mostly future proofing. The way we'd be most likely to use this in the near future is to provide information about insecure dependencies

[epage]

Would there be any compatibility issues with alternative registries?

[Eh2406]

For backwards compatibility cargo would have to be robust to receiving any of:

• the actual crate
• the json data with a url
• a redirect to either of the previous

As long as cargo accepts whatever response it gets for the request, and we document the JSON structures it should work. Although whether it's still the best way to get this data into cargo 🤷