Data Fetch

Data Fetch

Reference databases and read datasets are hosted on Zenodo and pulled with a single script, scripts/build-db/fetch_bakeoff_data.sh. The script downloads the bundle parts, verifies their sha256 checksums, reassembles and decompresses them, and extracts everything into the data/... layout the pipeline expects — including the post-extract NCBI taxonomy step.

Prerequisite: the data-fetch system tools (curl, jq, zstd ≥ 1.4, tar, sha256sum). See Installation.

Quick start

From the repository root, fetch everything:

scripts/build-db/fetch_bakeoff_data.sh --root .

Files land under <root>/data/.... When it finishes, confirm the layout:

ls data/

How bundles work

Each bundle is a tar.zst archive split into numbered parts (<bundle>.tar.zst.part-NNN) plus checksum sidecars. The script discovers the available bundles directly from the configured Zenodo records, so the authoritative list always comes from --list rather than this page:

scripts/build-db/fetch_bakeoff_data.sh --list

Bundles are split by reference-DB / platform / dataset so you can download only what a given figure needs (for example bakeoff_refdb_default_sylph, bakeoff_mock_ont, bakeoff_mock_reference).

Selective fetch

Pass a space-separated subset to --bundles:

# Just the ONT mock reads + the reference needed to profile them
scripts/build-db/fetch_bakeoff_data.sh --root . \
  --bundles "bakeoff_refdb_default_sylph bakeoff_mock_ont bakeoff_mock_reference"

Run --list first to see the exact names for the published records.

Options

Flag	Effect
--root DIR	Extraction root; files land at <root>/data/... (default: $PWD)
--bundles "a b c"	Subset to fetch (default: all bundles in the records)
--list	Print available bundles and exit
--staging DIR	Where parts download to (default: <root>/_staging)
--keep-staging	Keep downloaded parts after a successful extract
--skip-verify	Skip sha256 verification (faster, not recommended)

Downloads resume on interruption (curl -C -) and retry on transient failures, so re-running the same command continues where it left off. Already-extracted parts are reported as cached.

Verification

By default every part is checked against <bundle>.parts.sha256 before extraction. If all parts match, the reassembled stream is correct by construction. To re-verify a finished fetch, re-run with --keep-staging and inspect the staging directory, or compare your extracted data/ tree against the bundle's <bundle>.manifest.txt.

Manual extraction fallback

If the script can't run, the parts can be reassembled by hand:

cat <bundle>.tar.zst.part-* | zstd -dc --long=27 | tar -xf - -C <root>

After extracting the reference-DB bundle, expand the taxonomy snapshot the tools expect:

mkdir -p data/ref_db/refseq03032025/taxdump
tar -C data/ref_db/refseq03032025/taxdump \
    -xzf data/ref_db/refseq03032025/taxdump0303.tar.gz

(The script does this automatically.)

Troubleshooting

• ERROR: Zenodo record '…' is a placeholder — the record IDs at the top of the script haven't been set for your checkout. Edit ZENODO_RECORDS once the records are published.
• Checksum mismatch — delete the offending part in the staging directory and re-run; the download resumes.
• missing dependency: … — install the named system tool (see Installation).

Next steps

• Database Build — optionally, you can build the unified DB from scratch instead of fetching
• Reproduction — regenerate the manuscript figures and tables