Database Build

Note

Rebuilding indices for each tool requires a series of steps. Please ensure you follow the instructions exactly to avoid any discrepancies.

Database Build

This page rebuilds the unified RefSeq-228 reference from scratch. Most users should not need it — the prebuilt database ships via Data Fetch. Rebuild only when you need to change the RefSeq snapshot, extend the reference, or audit the build end-to-end.

Critical constraint: if you rebuild the unified reference, rebuild every tool index from the same taxdump snapshot. Bumping a single tool's index against a different taxonomy reintroduces exactly the cross-tool heterogeneity the unified design exists to eliminate.

What the unified DB is

A single reference shared by all six profilers: ~69k RefSeq genomes spanning archaea, bacteria, fungi, and viruses, with the NCBI taxonomy pinned to 2025-02-28. The end product is one directory, data/ref_db/refseq03032025/, holding the taxdump snapshot, the ete3 sqlite (taxa032025.sqlite), and one index per tool.

Build helpers live in [scripts/build-db/](<REPO_URL>/tree/main/scripts/build-db).

Step 1 — Download genomes + taxdump

Pull the RefSeq genomes and the NCBI taxdump corresponding to the manuscript's pin date (2025-02-28). The selected assemblies are inventoried in data/ref_db/refseq03032025/file.list and assembly_summary_refseq_augmented.txt; pair those against an NCBI taxdump snapshot of the same date.

Step 2 — Build the seqid→taxid map

python scripts/build-db/database_dic.py

Writes seqid2taxid.map for the unified library (accession → taxid for the selected RefSeq assemblies).

Step 3 — Build per-tool lineage tables

The unified library feeds several tool-specific lineage formats:

# Top-level lineage table
python scripts/build-db/build_unified_lineage_table.py \
    --file-list refseq03032025/file.list \
    --assembly-summary refseq03032025/assembly_summary_refseq.txt \
    --out refseq03032025/lineage.tsv

Script	Produces
build_unified_lineage_table.py	top-level lineage table
build_ganon_lineage_table.py	Ganon2-format lineage table
build_sourmash_lineage_table.py	Sourmash-format lineage CSV (ident → taxid + NCBI lineage from nodes.dmp/names.dmp)
build_sylph-tax_metadata_table.py	sylph-tax metadata for the custom Sylph DB

Step 4 — Build per-tool indexes

Each tool's index-construction command is tool-native (e.g. kraken2-build, centrifuge-build, sylph sketch, sourmash sketch, ganon build) and documented in data/ref_db/README. Build all of them against the Step 1 taxdump.

ete3 taxonomy snapshots

Each (tool, database) pair resolves names/taxids against a specific ete3 sqlite snapshot, so taxid lookups match the taxonomy the index was built with. The unified DB uses taxa032025.sqlite; the default DBs carry their own snapshots:

Snapshot	Used by
taxa032025.sqlite	unified RefSeq-228 (all tools, unified DB)
taxa122016.sqlite	Centrifuge / Centrifuger default
taxa032022.sqlite	Sourmash default
taxa042024.sqlite	Sylph default
taxaGTDB-r220.sqlite	Sylph default (GTDB lineage)

These are referenced by analysis_prep.py (ETE3_SUBPATHS) and resolved relative to --db-dir. See Tools and Databases for which snapshot pairs with which default DB.

Default databases

The per-tool default databases are each tool's own preferred catalogue (e.g. kraken2-build --download-library); they are vendor-provided and not rebuilt here. Provenance is in manuscript Table 2 and summarized in Tools and Databases.