Skip to content

v0.2.0

Latest

Choose a tag to compare

@skearnes skearnes released this 31 Aug 00:41
· 14 commits to main since this release
0a67aeb

data/ is Parquet-only: the 550 legacy .pb.gz files are removed, leaving the same 53 .parquet datasets that v0.1.1 already carried.

This is the compatibility break — v0.0.0 through v0.1.1 all resolve .pb.gz paths, and this release does not.

Breaking changes

data/ no longer contains .pb.gz files, and message_helpers.load_message has no Parquet path. Read datasets with ord_schema.datasets.load_dataset(path), which dispatches on suffix and returns a streaming DatasetView; pass as_dataset=True for a materialized Dataset.

from ord_schema.datasets import load_dataset

dataset = load_dataset("data/4d/ord_dataset-4d431564f3ef4e9c91d8da5836f4eae6.parquet")

The legacy .pb.gz files are still retrievable

They are removed from data/ at this tag, not deleted from storage. A checkout of v0.1.1, or of any earlier commit, still resolves every one of them:

git checkout v0.1.1
git lfs pull --include="data/**/*.pb.gz"

Both LFS stores retain the objects. The Hugging Face mirror serves all 603 objects referenced by v0.1.1 even though its HEAD no longer references the 550 .pb.gz, and GitHub's LFS store serves them as well:

git -c lfs.url="https://github.com/open-reaction-database/ord-data.git/info/lfs" \
    lfs pull --include="data/**/*.pb.gz"

Retired dataset IDs

499 dataset IDs no longer address a file, because two groups of shards are published as single consolidated datasets:

Dataset ID Sources
uspto-grants ord_dataset-1158e351757f315b93cbcbe7bc55f38e 489 monthly uspto-grants-YYYY_MM datasets
Training data from https://doi.org/10.1039/C8SC04228D ord_dataset-e7830cd6b11158b43994ccfb5ee9acb3 10 (N/10) shards

retired_datasets.csv maps each retired ID to the dataset that replaced it. Every other dataset kept its ID; only the suffix changed. Per-reaction patent provenance for the USPTO data is on Reaction.provenance.patent, so the monthly bucket a reaction came from stays recoverable.

No reactions were lost in the removal: the two consolidated datasets match their shard totals exactly, and every dataset deleted here has a Parquet counterpart with identical reactions.

Reaction data lives in Git LFS, so the auto-generated source archives below contain pointer files, not reactions. Clone the repository, or run python scripts/download_from_huggingface.py; LFS reads are served by the Hugging Face mirror.

What's Changed

  • Remove legacy pb.gz datasets in favor of their Parquet replacements by @skearnes in #284
  • Assert validation coverage and finish the Parquet-only migration by @skearnes in #256

Full Changelog: v0.1.1...v0.2.0

🤖 Generated with Claude Code