Databases

Databases

Any Mash sketch database works with mashID. Databases are stored outside the package and outside the repository.

Pre-built databases

mashID_download_db                    # list databases and whether they are installed
mashID_download_db mycobacteriaceae   # download one (MD5-verified)
mashID_download_db --all --dir /db    # download everything somewhere else

Name	Content	Size
mycobacteriaceae (default)	Mycobacteriaceae (NCBI taxon 1762), all NCBI genomes as of 2025-02-20, dereplicated per species at 99.9% identity. k=21, s=1000. Figshare	27 MB
listeria	Listeria spp., all NCBI genomes as of 2025-02-18, dereplicated. Figshare	46 MB
progenomes3	proGenomes v3 representative genomes, all bacteria and archaea. Figshare	331 MB
refseq_bacteria	RefSeq bacteria as of 2023-01-19, dereplicated at 99% identity. Figshare	653 MB

Databases live in $MASHID_DB_DIR if set, else $XDG_DATA_HOME/mashID/db (~/.local/share/mashID/db). Pass a name (-d listeria) or a path (-d /db/x.msh) to mashID. A metadata sidecar (see below) is generated right after download.

Building your own

make_mashID_db -i /path/to/genomes -o /path/to/db -p my_database -s 10000 -k 21 -t 16
# -> /path/to/db/my_database.msh and /path/to/db/my_database.metadata.tsv

Option	Description
-i PATH	Directory searched recursively for .fna/.fa/.fasta[.gz], or a text file listing one genome path per line.
-o DIR, -p NAME	Output directory and database name (.msh is appended).
-s 10000	Sketch size. 10000 is recommended for species-level identification.
-k 21	K-mer size (1–32).
--min-length 100000	Exclude references shorter than this; use 0 for plasmid or viral collections.
--metadata FILE	TSV/CSV with Accession, Organism and optionally TaxID columns.
--assembly-report FILE.jsonl	assembly_data_report.jsonl from an NCBI datasets download genome archive.
--annotate DB.msh	Only (re)write the metadata sidecar of an existing database.

scripts/build_mycobacteriaceae_db.sh in the repository shows a full workflow: download all NCBI genomes of a taxon with datasets, dereplicate them per species with Assembly-dereplicator, and sketch the database.

The metadata sidecar

<database>.metadata.tsv sits next to the .msh file and holds, for every reference: Accession, Organism, TaxID, Length, Hashes, Source_File, Description.

It is the source of Identification and TaxID in the outputs, and of the reference lengths used by --min-ref-length. Organism names come from, in order of preference:

1. --metadata FILE: your own table.
2. --assembly-report: NCBI's report, which carries organism names and TaxIDs for every accession.
3. The first fasta header of each genome, parsed for genus, species and infraspecific ranks (NZ_CP060409.1 Mycolicibacterium fortuitum strain W4 chromosome → Mycolicibacterium fortuitum).

Accessions are recognised in file names (GCF_000195955.2_ASM19595v2_genomic.fna.gz → GCF_000195955.2). If a database has no sidecar, mashID builds one on first use from mash info, and writes it next to the database when the directory is writable. To add or refresh a sidecar, including for downloaded databases:

make_mashID_db --annotate /path/to/db.msh --assembly-report assembly_data_report.jsonl

Choosing a sketch size

The pre-built Mycobacteriaceae database uses s=1000, which resolves species but yields coarse identities (steps of 0.001). For a custom database aimed at close relatives, s=10000 is worth the tenfold larger file: identities become finer and the shared-hash counts more informative.