Properly index available rust versions

[foresterre]

We do something pretty dumb right now: simply take the minor versions and decrease it. But we do have all versions available, even ordered already, so instead we should be slightly smarter and first build and index of available versions, and then for each latests available minor version run the check.

[foresterre]

Options

• Use https://raw.githubusercontent.com/rust-lang/rust/master/RELEASES.md
  • Only stable versions
  • Should still check channel manifest for latest version, since the upcoming release may be included in RELEASES.md
• Parse from (historical) channel manifests
  • Historical manifests indexed at: https://static.rust-lang.org/manifests.txt
  • Requires downloading, parsing and assembling a lot of smaller files to a create an index; depending on the connection speed, limiting, etc., the speed of this process may vary
• Download S3 index
  • Manual: aws --no-sign-request s3 ls static-rust-lang-org/dist/ > dists.txt
  • Slow download, heavily rate limited, approx 45s currently
  • One big index

[foresterre]

Progress:

• Ability to index stable, beta, nightly versions using second option (Parse from (historical) channel manifests)
• Proof of concept repo: https://github.com/foresterre/manifesta
• Downloading all indexes is severely rate limited (30-120sec for stable channel manifests), and the release manifests are sufficiently large that parsing them, even though we only need a small part takes a few seconds

[foresterre]

closed by #47