feat(put): Hugging Face Hub storage backend ([71]) - #246
Merged
Conversation
roar put hf://<owner>/<repo>[/path] (and hf://datasets/<owner>/<repo>) uploads artifacts to a HF repo and records the public resolve URL as the artifact's source_url/downloadLocation — the output-side of the lineage story: a reproduced model gets a canonical, stranger-fetchable home, so the whole DAG is verifiable end to end (infra-spec §3). - storage/hf.py: HFBackend (create_repo idempotent + upload_file via huggingface_hub, lazy-imported like the gcs backend; model default, datasets/spaces prefixes; token from HF_TOKEN/~/.hf_token). Mirrors the hf:// download URL convention. - publish.py: wire 'hf' into the put backend factory. - put_preparation.py: record 'hf' as a composite source_type -> downloadLocation. - put.py: hf:// destinations in --help. - pyproject: huggingface_hub as a dev/optional dep (lazy at runtime, like boto3/gcs). - tests: 9 unit tests (mocked HfApi) — model/dataset/space parsing, resolve URLs, upload args, exists. ruff + mypy clean. Output-side sibling of [17] (roar get --cache). Should ride the 0.4.1 release. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
roar put hf://<owner>/<repo>[/path](andhf://datasets/<owner>/<repo>) uploads artifacts to a HF repo and records the publicresolveURL as the artifact'ssource_url/downloadLocation.Why ([71], infra-spec §3)
The output-side of the lineage story — a reproduced model gets a canonical, stranger-fetchable home on HF (~14 TB free public), so the whole DAG is verifiable end to end (inputs from public sources, outputs pullable + hash-checkable). Closes "sourced-but-not-obtainable" for our own outputs. Sibling of [17] (
roar get --cache).How
storage/hf.py:HFBackend—create_repo(idempotent) +upload_fileviahuggingface_hub, lazy-imported exactly like the GCS backend (only needed when you actuallyput hf://). Model repo by default;datasets//spaces/prefixes supported; token fromHF_TOKEN/~/.hf_token. URL convention matches the hf:// download side.hfinto the put backend factory; recorded as a compositesource_type→downloadLocation; added toput --help;huggingface_hubas a dev/optional dep.Security note (campaign)
The deliberate publish uses a scoped, publish-time write token, distinct from the operator run's read-only token (risks C4/C5). Not an ambient credential.
Tests
9 unit tests (mocked
HfApi) — parsing (model/dataset/space + subpaths), resolve URLs, upload args, exists-on-error. ruff + mypy + format clean.Should ride the 0.4.1 release.
🤖 Generated with Claude Code