Configs & data

Data

The key concept behind SAMOVAR is the use of a variety of programs, databases, and raw genomes. It’s easy to get lost in this diversity, which is why SAMOVAR includes a configuration system that automatically specifies and stores data about runs.

The main types of large-scale genomic data you’ll inevitably encounter during your runs:

1. Your metagenomes (real or simulated)

2. Metagenome databases (e.g., for Kraken2, Kaiju, MetaPhlAn 4, etc.)

3. Raw genomic data required for regenerating metagenomes

4. Regenerated metagenomes

While (1) and (4) will be unique to a specific run, the data for (2) and (3) is stored in configuration files so that it can be easily retrieved in future runs, eliminating the need to specify hard-coded paths or download them again each time.

Configs

SAMOVAR has various configuration files.

• The base configuration defines the relationship with the compilers and programs it will need to work with in the future. By default, it is generated in ~/.config/samovar/config.json

• During the samovar generate stage, a configuration for the pipeline is generated, as well as configurations for running individual annotators.

The main configuration file specifies the paths to programs and data. This information is then reused during the samovar prepare process, so you don't have to set up the correct environment or upload new data every time.

Main config

This file stores information about the program’s installation:

• its root directory
• paths to compilers and workflows
• paths to databases and genomes
• program launch indices

Important: The tool does not currently version its environment, so if you overwrite or delete anything, you may encounter difficulties reproducing the results. This is especially important for raw genome data. To avoid having to download them constantly, by default, at the end of the samovar exec pipeline, when the samovar generate --reindex_genomes flag is used, they are overwritten to the directory specified during installation via install.sh (by default, path/to/samovar/genomes).

How to obtain / refresh it

./install.sh

Install discovers compilers and tools on PATH, migrates any previous flat config, and rewrites the nested schema (no duplicated python_path / tools.python / iss_path triples). Optional extras:

./install.sh OPAL      # scoring
./install.sh MultiQC
./install.sh CAMISIM
./install.sh NanoSim   # sidecar conda env
./install.sh ART
./install.sh R-package

You can also edit the JSON by hand. Unknown extra keys are ignored; keep the nested shape below.

Schema

{
  "version": "0.10.19",
  "root": "/path/to/samovar",
  "compilers": {
    "bash": "/bin/bash",
    "python": "/path/to/python",
    "python_libs": [],
    "R": "/path/to/R",
    "R_libs": [],
    "cpp": "/usr/bin/g++",
    "cpp_libs": []
  },
  "API": {
    "ncbi_email": "you@example.com"
  },
  "genomes": {
    "test": ["/path/to/samovar/data/test_genomes"],
    "taxdump": "path/to/NCBI/taxdump",
    "raw": {"default": "/scratch/samovar/genomes"},
    "processed": {"default": "/scratch/samovar/genomes"},
    "data": {
       "10847": ["10847", "GCF_000819615.1", "samovar_database", "GCF_000819615.1.fa.gz"]
       # format
       "taxID": ["species_taxid", "NCBI ID", "database ID", "file name"]
    }
  },
  "databases": {
    "kraken2": ["standard_8GB", "/path/to/kraken2_db", ""]
  },
  "workflows": {
    "snakemake": ["annotators", "annotation2iss", "read_processing", "database_prep", "iss_test"],
    "nextflow": ["camisim"],
    "conda": ["nanosim"]
  },
  "tools": {
    "kraken2": ["", "bash", "/path/to/kraken2", "annotator"],
    "nanosim": ["conda", "nanosim", "/path/to/envs/nanosim", "metagenome_generator"]
  }
}

compilers

Interpreters and compilers used to run SamovaR. *_libs lists are extra prefixes (empty = system/default). Do not repeat these paths under tools except as the runnable binaries (python, R, g++).

API

External service credentials. ncbi_email is the Entrez contact. Environment NCBI_EMAIL / ENTREZ_EMAIL / SAMOVAR_EMAIL still wins at runtime and is not overwritten by env if already set.

genomes

• test: truncated ISS/CI stubs shipped with the package. Never used as an NCBI library.
• raw / processed: maps folder_id → directory. NCBI downloads belong here (or under $out_dir/.cache at exec time). Do not put multi-GB caches under $HOME.
• data: optional catalog taxID → [accession, folder_id, file_name].

databases

Per-tool index records: [database_name, database_path, database_flags]. Flags are extra CLI tokens that index needs (empty if none). A tool may have several rows (list of triples).

workflows

Named workflow engines available on this install (snakemake Snakefiles, nextflow pipelines, conda sidecar env names).

tools

Every runnable tool, including annotators and generators:

"name": ["env", "workflow_name", "path", "tool_group"]

If you want to add any tool as the permanent possibility (mainly may be usefull for the custom annotators) you may edit this file with such info