Skip to content

Configs & data

Jump to bottom
dsmutin edited this page Aug 29, 2026 · 4 revisions

Data

The key concept behind SAMOVAR is the use of a variety of programs, databases, and raw genomes. It’s easy to get lost in this diversity, which is why SAMOVAR includes a configuration system that automatically specifies and stores data about runs.

The main types of large-scale genomic data you’ll inevitably encounter during your runs:

  1. Your metagenomes (real or simulated)

  2. Metagenome databases (e.g., for Kraken2, Kaiju, MetaPhlAn 4, etc.)

  3. Raw genomic data required for regenerating metagenomes

  4. Regenerated metagenomes

While (1) and (4) will be unique to a specific run, the data for (2) and (3) is stored in configuration files so that it can be easily retrieved in future runs, eliminating the need to specify hard-coded paths or download them again each time.

Configs

SAMOVAR has various configuration files.

  • The base configuration defines the relationship with the compilers and programs it will need to work with in the future. By default, it is generated in ~/.config/samovar/config.json

  • During the samovar generate stage, a configuration for the pipeline is generated, as well as configurations for running individual annotators.

The main configuration file specifies the paths to programs and data. This information is then reused during the samovar prepare process, so you don't have to set up the correct environment or upload new data every time.

Main config

This file stores information about the program’s installation:

  • its root directory
  • paths to compilers and workflows
  • paths to databases and genomes
  • program launch indices

Important: The tool does not currently version its environment, so if you overwrite or delete anything, you may encounter difficulties reproducing the results. This is especially important for raw genome data. To avoid having to download them constantly, by default, at the end of the samovar exec pipeline, when the samovar generate --reindex_genomes flag is used, they are overwritten to the directory specified during installation via install.sh (by default, path/to/samovar/genomes).

How to obtain / refresh it

./install.sh

Install discovers compilers and tools on PATH, migrates any previous flat config, and rewrites the nested schema (no duplicated python_path / tools.python / iss_path triples). Optional extras:

./install.sh OPAL      # scoring
./install.sh MultiQC
./install.sh CAMISIM
./install.sh NanoSim   # sidecar conda env
./install.sh ART
./install.sh R-package

You can also edit the JSON by hand. Unknown extra keys are ignored; keep the nested shape below.

Schema

{
  "version": "0.10.19",
  "root": "/path/to/samovar",
  "compilers": {
    "bash": "/bin/bash",
    "python": "/path/to/python",
    "python_libs": [],
    "R": "/path/to/R",
    "R_libs": [],
    "cpp": "/usr/bin/g++",
    "cpp_libs": []
  },
  "API": {
    "ncbi_email": "you@example.com"
  },
  "genomes": {
    "test": ["/path/to/samovar/data/test_genomes"],
    "taxdump": "path/to/NCBI/taxdump",
    "raw": {"default": "/scratch/samovar/genomes"},
    "processed": {"default": "/scratch/samovar/genomes"},
    "data": {
       "10847": ["10847", "GCF_000819615.1", "samovar_database", "GCF_000819615.1.fa.gz"]
       # format
       "taxID": ["species_taxid", "NCBI ID", "database ID", "file name"]
    }
  },
  "databases": {
    "kraken2": {
      "standard_8GB:2025oct": {
        "path": "/path/to/kraken2_db",
        "flags": "--memory-mapping",
        "lazy-download": "#!/bin/bash\ncurl -L … && tar -xzf …",
        "url": "https://genome-idx.s3.amazonaws.com/kraken/k2_standard_08gb_20251015.tar.gz",
        "type": "database",
        "version": "2025oct"
      }
    }
  },
  "workflows": {
    "snakemake": ["annotators", "annotation2iss", "read_processing", "database_prep", "iss_test"],
    "nextflow": ["camisim"],
    "conda": ["nanosim"]
  },
  "tools": {
    "kraken2:2.0.7": {
      "exec": {"env": "", "parser": "bash", "path": "/path/to/kraken2"},
      "type": "annotator",
      "lazy-install": "conda install -y bioconda::kraken2",
      "flags": "",
      "flags-translate": {"--threads": "--threads", "--cores": "--threads"}
    },
    "nanosim": {
      "exec": {"env": "conda", "parser": "nanosim", "path": "/path/to/envs/nanosim"},
      "type": "metagenome_generator"
    }
  }
}

compilers

Interpreters and compilers used to run SamovaR. *_libs lists are extra prefixes (empty = system/default). Do not repeat these paths under tools except as the runnable binaries (python, R, g++).

API

External service credentials. ncbi_email is the Entrez contact. Environment NCBI_EMAIL / ENTREZ_EMAIL / SAMOVAR_EMAIL still wins at runtime and is not overwritten by env if already set.

genomes

  • test: truncated ISS/CI stubs shipped with the package. Never used as an NCBI library.
  • raw / processed: maps folder_id → directory. NCBI downloads belong here (or under $out_dir/.cache at exec time). Do not put multi-GB caches under $HOME.
  • data: optional catalog taxID → [accession, folder_id, file_name].

databases

Annotator indexes, nested like tools: databases.<annotator>.<name:version> objects.

Field Meaning
path Directory, file (.qza, .fa.gz), or centrifuge prefix (stem when stem.1.cf exists)
flags Native CLI tokens for that annotator. At prepare they are merged into AnnotatorConfig.extra (appended to the classifier command). MetaPhlAn also gets --bowtie2db <path> if it is missing.
lazy-download Rebuild recipe (curl/tar) used by samovar export / unfold install.sh
url Official archive; fills lazy-download when the recipe is empty
version Stored in the key as name:version (several versions of the same name can coexist)

Register with samovar tools import --type database (see Custom tools import). prepare accepts the indexed name instead of a filesystem path:

samovar prepare --kraken2-test "kraken2 standard_8GB"
samovar prepare --kaiju-test "kaiju fungi"
samovar prepare --metaphlan-test "metaphlan jan25"

Legacy [name, path, flags] triples still parse. NCBI nodes.dmp / names.dmp stay in genomes.taxdump (env SAMOVAR_TAXDUMP); if that is empty, SamovaR also looks at databases.taxdump.ncbi.

workflows

Named workflow engines available on this install (snakemake Snakefiles, nextflow pipelines, conda sidecar env names).

tools

Every runnable tool, including annotators and generators. Preferred on-disk form is an object keyed name:version (exec, type, flags, lazy-install, flags-translate). Legacy list rows (["env", "workflow", "path", "group", "flags", "inputs"]) still parse.

If you want to add any tool as the permanent possibility (mainly may be usefull for the custom annotators) you may edit this file with such info, or use samovar tools import.

Clone this wiki locally