Data Fetch

Zenodo hosts the reference databases that the pipeline reads - the unified RefSeq-228 family plus the Ganon2 default index. Mock-community reads remain on SRA; simulated reads and their truth tables ship in a separate Zenodo record cited from Datasets. The other per-tool default databases (Centrifuge, Centrifuger, Kraken2, Sourmash, Sylph) are not bundled - install each from the tool's own published index (default DBs (not bundled) below).

All Zenodo records below are grouped in the Treangen Lab Bakeoff community, which gives a single page to browse, cite, or monitor for new revisions.

Prerequisite: curl, zstd ≥ 1.4, tar. See Installation.

What's on Zenodo

Each bundle below lives in its own Zenodo record. Both metadata bundles ship in a single combined record (20481656), and the Kraken2 unified index is split across two records that are both required.

Bundle	Contents	Approx. size (compressed)	Zenodo record
`bakeoff_refdb_shared_metadata`	taxdump + ete3 sqlite + lineage / accession-to-taxid tables + audit lists. Required alongside any unified bundle.	~5 GB	`20481656`
`bakeoff_refdb_unified_kraken2`	Kraken2 unified index. Split across two Zenodo records; both required and concatenated by `fetch_bakeoff_data.sh`.	~121 GB	`20484934` + `20496297`
`bakeoff_refdb_unified_centrifuge`	Centrifuge unified index	~58 GB	`20479369`
`bakeoff_refdb_unified_centrifuger`	Centrifuger unified index	~57 GB	`20479514`
`bakeoff_refdb_unified_ganon2`	Ganon2 unified index	~32 GB	`20481847`
`bakeoff_refdb_unified_sourmash`	Sourmash unified sketches	~2 GB	`20481715`
`bakeoff_refdb_unified_sylph`	Sylph unified sketches	~5 GB	`20481766`
`bakeoff_refdb_default_ganon2`	Ganon2 default index (the only default DB redistributed; sized + licence allow it)	~95 GB	`20482294`
`bakeoff_refdb_default_metadata`	Taxonomy snapshots + sylph-tax metadata that the default-DB pipeline runs need but upstream default indexes don't include. Required alongside any `--db default` workflow that touches Centrifuge, Sourmash, or Sylph. Ships in the same Zenodo record as `shared_metadata`; downloading record `20481656` covers both.	~500 MB	`20481656`

Each large bundle is split into 45 GB parts named <bundle>.tar.zst.part01, .part02, ... and shipped with a <bundle>.manifest.txt listing the archive contents.

Pick a download set

You almost never need all of it. Pair the right bundles for what you want to do:

Goal	Bundles to download
Reproduce unified-DB results for one tool T	`shared_metadata` + `unified_<T>`
Reproduce default-DB Ganon2 results	`default_ganon2`
Reproduce default-DB results for any other tool	self-install the tool's index (Default DBs (not bundled)) + `default_metadata` for the matching taxonomy snapshot
Rebuild a unified index from scratch	`shared_metadata` - then follow Database Build

shared_metadata is mandatory for every unified bundle; the unified indexes reference its taxdump and ete3 snapshot.

Reassemble and extract a bundle

The Zenodo files for one bundle (e.g. bakeoff_refdb_unified_kraken2) look like:

bakeoff_refdb_unified_kraken2.tar.zst.part01
bakeoff_refdb_unified_kraken2.tar.zst.part02
bakeoff_refdb_unified_kraken2.tar.zst.part03
bakeoff_refdb_unified_kraken2.manifest.txt        # inventory; optional

Concatenate the parts and pipe straight through zstd | tar into your bakeoff project root - no intermediate .tar.zst is materialised, so disk usage during extraction is just the unpacked DB:

PROJECT_ROOT=/path/to/bakeoff       # the dir containing data/ , scripts/ , ...
cd "$PROJECT_ROOT"

cat bakeoff_refdb_unified_kraken2.tar.zst.part?? \
    | zstd -dc --long=27 \
    | tar -xf -

The leading cat *.part?? glob expands in lexicographic order, which matches the numeric partNN naming. Run from inside the project root so the relative archive paths (data/ref_db/...) land at the right place.

The archive paths are stored relative (data/ref_db/...), so a single pass restores the canonical layout the pipeline expects - no mv or symlinks needed.

For the smaller, un-split bundles (sourmash, sylph, shared_metadata), the .partNN glob still works because cat <onefile>.tar.zst just emits that one file:

cat bakeoff_refdb_unified_sourmash.tar.zst \
    | zstd -dc --long=27 \
    | tar -xf -

Integrity of each downloaded part is already covered by the MD5 Zenodo computes for every file on the record page - compare it against md5sum <part> if you want a per-part check before extraction. The pipe-through form above is enough for everyday use; nothing further is needed once tar -xf returns 0.

One post-extract step: expand the taxdump

shared_metadata ships the NCBI taxdump as a compressed tarball; the tools want it expanded into a taxdump/ subdir next to it:

mkdir -p data/ref_db/refseq03032025/taxdump
tar -C data/ref_db/refseq03032025/taxdump \
    -xzf data/ref_db/refseq03032025/taxdump0303.tar.gz

That's the only manual post-fetch step. Everything else is already in place.

Resulting file layout

After extracting all unified bundles plus shared_metadata, default_ganon2, and default_metadata, your tree under <PROJECT_ROOT>/data/ref_db/ looks like:

data/ref_db/
├── database_build.md                                # ← shared_metadata
├── RefSeq-release228.catalog.gz                     # ← shared_metadata
│
├── refseq03032025/                                  # the unified DB family
│   ├── assembly_summary_refseq_augmented.txt        # ← shared_metadata
│   ├── ete3_taxa/                                   # ← shared_metadata
│   ├── file.list                                    # ← shared_metadata
│   ├── nucl_gb.accession2taxid.gz                   # ← shared_metadata
│   ├── nucl_wgs.accession2taxid.gz                  # ← shared_metadata
│   ├── refseq03032025_dict.txt                      # ← shared_metadata
│   ├── refseq_030325_lineage.tsv                    # ← shared_metadata
│   ├── refseq_accessions_taxonomy.csv               # ← shared_metadata
│   ├── seqid2taxid.map                              # ← shared_metadata
│   ├── taxdump0303.tar.gz                           # ← shared_metadata (expand to taxdump/)
│   ├── taxdump/                                     # ← created by the post-fetch tar -xzf above
│   │
│   ├── k2_abfv_030325/                              # ← unified_kraken2
│   ├── centrifuge_abv_030325/                       # ← unified_centrifuge
│   ├── centrifuger_abv_030325/                      # ← unified_centrifuger
│   ├── ganon2_abvf_030325/                          # ← unified_ganon2
│   ├── sourmash_abvf_030325/                        # ← unified_sourmash
│   └── sylph_abf_030325/                            # ← unified_sylph
│
└── default_db/                                      # per-tool default DBs
    ├── ganon2_default/                              # ← default_ganon2 (only one bundled)
    │
    ├── cf_default/
    │   ├── ete3_taxa/                               # ← default_metadata (taxa122016 snapshot)
    │   └── ...index files...                            # ← self-install (Centrifuge)
    ├── sm_default/
    │   ├── ete3_taxa/                               # ← default_metadata (taxa032022 snapshot)
    │   └── ...zip files...                              # ← self-install (Sourmash)
    ├── sylph_default/
    │   ├── ete3_taxa/                               # ← default_metadata (taxaGTDB-r220 snapshot)
    │   ├── taxa042024.sqlite{,.traverse.pkl}        # ← default_metadata
    │   └── ...syldb files...                            # ← self-install (Sylph)
    ├── sylph_tax/                                   # ← default_metadata (sylph-tax lineage metadata)
    │
    ├── k2_default/                                  # ← self-install (Kraken2 default)
    └── cfer_default/                                # ← self-install (Centrifuger default)

Path-by-path mapping from bundles is also encoded in each bundle's <bundle>.manifest.txt - useful for a quick spot-check that extraction landed everything where it should.

Default DBs (not bundled)

Five of the six tools' default DBs aren't redistributed (size + licence concerns) - install each from the tool's own published index. Create the directory shells first; that way an unfinished partial install is obvious, and the pipeline's --db default runs report cleanly "missing index" rather than a confusing parser error:

cd "$PROJECT_ROOT"

mkdir -p data/ref_db/default_db/{k2_default,cf_default,cfer_default,sm_default,sylph_default,sylph_tax}

Then populate each from its upstream source. The version numbers in Tools and Databases are what the manuscript uses; match those if you want bit-for-bit reproducibility.

Tool / dir	Source	Notes
`k2_default/` (Kraken2)	https://benlangmead.github.io/aws-indexes/k2 - pick the `k2_standard_*` tarball	Untar in place; the `*.k2d` files go directly under `k2_default/`.
`cf_default/` (Centrifuge)	https://benlangmead.github.io/aws-indexes/centrifuge	Place the `.cf` index files directly under `cf_default/`.
`cfer_default/` (Centrifuger)	https://github.com/mourisl/centrifuger releases - the prebuilt index download	Same layout: index files directly under `cfer_default/`.
`sm_default/` (Sourmash)	https://sourmash.readthedocs.io/en/latest/databases.html - pick the matching zip	Drop the `.zip` files into `sm_default/`.
`sylph_default/` (Sylph)	https://github.com/bluenote-1577/sylph/wiki/Pre%E2%80%90built-databases - sketch + metadata	Sketches (`.syldb`) into `sylph_default/`.
`sylph_tax/`	https://github.com/bluenote-1577/sylph-tax taxonomy bundles	This is the `sylph-tax` lineage-resolution data, not the sketch itself.

After populating, each tool's ete3 snapshot for taxonomy resolution must match the index - see Tools and Databases for the snapshot-to-default-DB mapping. For the default DBs, those snapshots ship in bakeoff_refdb_default_metadata, not shared_metadata (which only carries the unified-DB snapshot, refseq03032025/ete3_taxa/taxa032025.sqlite). Extracting default_metadata drops the right snapshot into each <tool>_default/ete3_taxa/ automatically, and also populates sylph_default/taxa042024.sqlite and the sylph_tax/ metadata directory. Run it once alongside any of the self-installed default indexes:

cat bakeoff_refdb_default_metadata.tar.zst \
    | zstd -dc --long=27 \
    | tar -xf -

Verification

Two sanity checks once a download finishes:

# 1. inventory the extracted files against the bundle's manifest
diff <(cd $PROJECT_ROOT && tar -tvf <unused>.tar.zst | sort) <(sort bakeoff_refdb_<bundle>.manifest.txt)
# (only meaningful if you kept the joined .tar.zst; otherwise rely on a quick file count)

# 2. point-check one expected file per bundle
ls $PROJECT_ROOT/data/ref_db/refseq03032025/k2_abfv_030325/hash.k2d            # kraken2
ls $PROJECT_ROOT/data/ref_db/refseq03032025/centrifuge_abv_030325/*.cf         # centrifuge
ls $PROJECT_ROOT/data/ref_db/refseq03032025/sylph_abf_030325/*.syldb           # sylph
ls $PROJECT_ROOT/data/ref_db/refseq03032025/ete3_taxa/taxa032025.sqlite        # ete3 snapshot

If a file is missing from default_db/<tool>_default/, that's almost always an incomplete download from the upstream source rather than a bundling problem on our side - re-fetch from the URL above.

Troubleshooting

zstd: error 70 : Frame requires too much memory for decoding - pass --long=27 (as shown in every recipe above); the bundles use a 27-bit window.
tar: data/ref_db: Cannot mkdir: Permission denied - you're not in the project root, or you don't own <PROJECT_ROOT>/data/. cd "$PROJECT_ROOT" before the cat | zstd | tar line, and check the directory's permissions.
Index works for one tool but not another - the unified indexes reference the taxdump and ete3 snapshot from shared_metadata; the most common cause is forgetting that bundle.
Downloads stall - Zenodo serves through CDN, but for the 121 GB Kraken2 bundle parts you may want curl -C - (continue) so an interrupted part resumes cleanly.

Next steps

Database Build - optionally rebuild a unified index from scratch instead of downloading it
Reproduction - regenerate the manuscript figures and tables once the data is in place

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Data Fetch

Data Fetch

What's on Zenodo

Pick a download set

Reassemble and extract a bundle

One post-extract step: expand the taxdump

Resulting file layout

Default DBs (not bundled)

Verification

Troubleshooting

Next steps

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Getting Started

Reproducing the Paper

Reference

Contributing

Clone this wiki locally