fix: reuse converted release datasets

[SilvioGiancola]

Summary

• reuse an existing split JSON before downloading Parquet/WebDataset shards from Hugging Face
• expose whether a download was skipped and cover the cache-hit behavior with a regression test
• align classification release fixtures with per-split media roots and excluded dataset labels
• normalize detected CUDA versions before selecting PyTorch wheels
• document the reused JSON behavior

Validation

• cache-hit smoke test passed
• Python compilation passed for the modified transfer module and tests
• git diff --check passed
• full pytest suite not run because pytest is unavailable in the current shell

API impact

Backward-compatible: Parquet download results now include download_skipped. Existing converted JSON files are reused.

[Copilot]

Pull request overview

This PR updates the Hugging Face transfer utilities to reuse already-converted per-split JSON outputs (avoiding redundant Parquet/WebDataset downloads), exposes whether a download was skipped in the returned result, and aligns release test fixtures and documentation with the per-split output layout. It also normalizes detected CUDA versions into PyTorch wheel tags when selecting the PyTorch index URL.

Changes:

• Reuse an existing <split>.json for Parquet/WebDataset transfers and expose a download_skipped flag (plus regression coverage).
• Adjust classification release fixtures to (a) exclude dataset-hardcoded labels and (b) resolve source_path per split for sharded datasets.
• Normalize CUDA version strings (e.g., "13.0" → "cu130") before selecting PyTorch wheel indexes, and document the JSON reuse behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
opensportslib/tools/hf_transfer.py	Adds a cache-hit fast path for Parquet split downloads and includes download_skipped in results.
tests/test_hf_transfer_tools.py	Adds regression coverage for cache-hit behavior and asserts download_skipped on Parquet paths.
tests/release/test_classification_release.py	Aligns release fixtures with dataset label exclusions and per-split media roots for sharded datasets.
opensportslib/setup/setup.py	Normalizes detected CUDA version strings into PyTorch wheel tags before selecting the torch index URL.
README.md	Documents that existing per-split JSON outputs are reused for Parquet/WebDataset downloads.
tests/release/README.md	Documents the per-split JSON reuse behavior in release test tooling.

Suppressed comments (1)

opensportslib/setup/setup.py:77

• When CUDA_VERSION is not detected (CPU-only machine), install_torch() enters the for cuda in CUDA_SUPPORT: loop, installs plain CPU wheels, but then returns the first CUDA tag in the list (currently cu126). That return value is misleading and can cause callers/logs to report the wrong install target.

    # get_cuda_version() returns the dotted version reported by nvidia-smi
    # (e.g. "13.0"), not a pip wheel tag (e.g. "cu130") -- convert before
    # comparing, the same way install_pyg() below already does.
    detected_tag = f"cu{CUDA_VERSION.replace('.', '')}" if CUDA_VERSION else None
    if detected_tag in CUDA_SUPPORT:
        cuda = detected_tag
        subprocess.check_call([
            python, "-m", "pip", "install",
            "torch", "torchvision", "torchaudio",
            "--index-url",
            f"https://download.pytorch.org/whl/{cuda}"
        ])
        print(f"\nSuccess with {cuda}")
        return cuda
    for cuda in CUDA_SUPPORT:

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.