Mine Rust crates

[pombredanne]

We need to collect all the crates and index them.
There is an API on crates.io and also a git repo with an index:

• https://github.com/rust-lang/crates.io-index

[alok1304]

@pombredanne, suppose we collected and indexing all the crates from Crates.io. Once we have all crates indexed, could you clarify the next steps? Should we focus on setting up querying functionality, integrating the index into PurlDB, or enriching the data in any particular way? Any guidance on priorities or additional requirements would be greatly appreciated!

[pombredanne]

@alok1304 re:

suppose we collected and indexing all the crates from Crates.io. Once we have all crates indexed, could you clarify the next steps? Should we focus on setting up querying functionality, integrating the index into PurlDB, or enriching the data in any particular way? Any guidance on priorities or additional requirements would be greatly appreciated!

Getting the index is straightforward, this is just a git repo. With this the steps would be:

For batch operations:

1. Parse the index and populate the PurlDB with basic cargo packages
2. For one package from 1., further collect detailed metadata from the API
3. For one package from 2., further integrate the download and scan of the crate code

For the on-demand API endpoint:

1. For one package PURL as an input, do the 2. and 3. from above, on demand

This would establish the data in PurlDB to collect, index, match and have a comprehensive references for all the crates.

[pombredanne]

In terms of priority, the on-demand operations are likely higher priority to get some results fast to than the full batch operations that come afterwards. Unless Batch 1. is really easy, which it may be.

[pombredanne]

@ziadhany since you are likely to look into this, some extra pointers:

• https://crates.io/data-access

Crate index

The crates.io sparse index is available at index.crates.io, which adheres to the Cargo index format. The sparse index provides an extremely efficient way of accessing metadata on a single or small number of crates.

Each index file provides newline delimited JSON metadata on all published versions of the crate, organised into index files. For example, information on the base64 crate can be found at https://index.crates.io/ba/se/base64.

No rate limits are required to use data from the sparse crate index.
Legacy Git crate index

Older versions of Cargo use the crate index provided in the rust-lang/crates.io-index repository on GitHub. This remains available for use, and may be a more efficient way of accessing crate metadata for projects that require most or all crates to be included than the sparse index.

As the Git index is hosted on GitHub, GitHub's Acceptable Use Policies apply.
RSS feeds

crates.io provides a couple of RSS feeds that contain information on new crates and crate updates:

https://static.crates.io/rss/crates.xml:
The latest new crates registered on crates.io (the past 60 minutes, but at least 50 new crates).
https://static.crates.io/rss/updates.xml:
The latest version updates on crates.io (the past 60 minutes, but at least 100 versions).
e.g. https://static.crates.io/rss/crates/serde.xml:
The latest version updates of the serde crate (the past 24 hours, but at least 10 versions).

Database dumps

crates.io database dumps contain all information available through the crates.io API in a single download. They are updated every 24 hours.

The latest dump is available at the address https://static.crates.io/db-dump.tar.gz. Information on using the dump is contained in the tarball. You can find the changelog for database dumps in GitHub issue #3617.

[pombredanne]

The first step to get all cargo PURLs is tracked in:

• purl-next: Collect PURLs for Rust ecosystem #666