ClassyFire Streaming Downloader

Rust crawler for building a local copy of ClassyFire labels for PubChem compounds.

It streams the PubChem CID-InChI-Key table, queries ClassyFire through GET /entities/{InChIKey}.json, writes successful responses into compressed shards, and publishes weekly partial releases.

Warning

ClassyFire is usable for slow, long-run recovery, not for fast bulk export. This project keeps the request rate conservative and treats local archives as the durable output. Download the Zenodo releases directly for any practical use of the data.

Quick Start, then wait 20+ years

1. Download the PubChem file and decompress it to plain TSV:

curl -L -o CID-InChI-Key.gz \
  https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Extras/CID-InChI-Key.gz
gzip -cd CID-InChI-Key.gz > CID-InChI-Key.tsv

2. Create the environment file:

cp .env.example .env

Required variables:

• ZENODO_TOKEN
• CLASSYFIRE_ZENODO_DEPOSIT_ID

3. Run the crawler and wait 20+ years:

cargo run --release -- run \
  --input ./CID-InChI-Key.tsv \
  --output-dir ./classyfire-run

Progress Report

Render a single SVG snapshot from the current local crawl state:

cargo run --release --bin classyfire-progress-report -- \
  --output-dir ./classyfire-run \
  --output ./classyfire-progress.svg

Render from the latest Zenodo release instead:

cargo run --release --bin classyfire-progress-report -- \
  --zenodo-doi \
  --output ./classyfire-progress.svg

If the latest release does not yet include progress-summary.json, pass --pubchem-total-rows so the tool can fall back to the published dataset and manifest.

License

MIT