Skip to content

Custom tools import

dsmutin edited this page Aug 30, 2026 · 8 revisions

samovar tools import

Register an executable in the install config (tools.<name>) so prepare / exec / resolve_executable find it without a full ./install.sh. Annotator indexes use the same CLI with --type database (writes databases.*, not tools.*).

samovar tools import \
  --name kaiju \
  --env "" \
  --exec kaiju \
  --exec-path /usr/bin/kaiju \
  --type annotator
Flag Alias Meaning
--name -n Config key. Also the prepare token (--kaiju-test "kaiju /db").
--env "" (host) or conda (path is a sidecar prefix). Default "".
--exec Binary name (default: --name). Used as prefix/bin/<exec> when --exec-path is a conda env directory.
--exec-path File, conda prefix, or a name on PATH. If omitted, PATH is searched for --exec then --name.
--type -t Stage / tool_group, or database / db to write databases.*.
--tool Annotator that uses this DB (--type database). Also accepted as -n kraken2:dbname.
--flags Native CLI flags. For a tool: stored on tools.*. For a database: merged into that annotator's extra at prepare.
--lazy-install Rebuild recipe for a binary (@file or - for stdin).
--lazy-download Rebuild recipe for a database (@file / --lazy-download-file).
--url Official archive URL for a database (feeds lazy-download).
--tool-version Version stored in the key (name:version).
--inputs --glob Input glob under the run output dir (6th tools.* slot). Scoring/viz default: *annotations.

--type values: annotator (a), reads_generator (reads), metagenome_generator (meta), qc (QC, trim), table_reads_generator (table), scoring (score, viz), reprofiler (ml), annotation_converter (converter), workflow, runtime, compiler, database (db).


Databases (--type database)

Writes databases.<annotator>.<name:version> (same object shape as tools: path, flags, lazy-download, url, version). Does not go under tools.*.

samovar tools import --type database \
  -n standard_8GB --tool kraken2 \
  --exec-path /path/to/kraken2_db \
  --tool-version 2025oct \
  --flags "--memory-mapping" \
  --url https://genome-idx.s3.amazonaws.com/kraken/k2_standard_08gb_20251015.tar.gz

-n kraken2:standard_8GB is the same as -n standard_8GB --tool kraken2. --exec-path may be a directory, a file (.qza, .fa.gz), or a centrifuge prefix (p_compressed+h+v when .1.cf exists). --lazy-download @fetch.sh stores a custom rebuild script; if omitted, known official URLs get a curl/tar recipe.

Then prepare uses the indexed name, not the filesystem path:

samovar prepare --output_dir RUN --kraken2-test "kraken2 standard_8GB"

flags on the DB record are native tokens for that classifier and are appended to its command. CLI leftovers after the name (--kraken2-test "kraken2 standard_8GB --threads 8") still apply. samovar build --type kaiju --index NAME --flags "…" also registers this object.

samovar export (full) packs the tree plus db-meta/.../lazy-download.sh; unfold install.sh re-imports with --type database.


Tools (--type annotator and other groups)

Stored row (legacy list form still parses; on disk the preferred form is a name:version object):

"name": ["env", "workflow", "path", "tool_group"]
"name": ["env", "workflow", "path", "tool_group", "flags"]            # --flags set
"name": ["env", "workflow", "path", "tool_group", "flags", "inputs"]  # scoring, or --inputs set

workflow is conda when --env conda, otherwise bash. path is the resolved file, or the conda prefix. The 5th slot is extra CLI flags from samovar tools import --flags "…". Empty flags are omitted unless a 6th inputs slot is present (scoring/viz).

Those flags are merged at prepare / run with samovar prepare --flags TOOL "…". TOOL can be the imported --name, or the group (annotator / a, table_reads_generator / table). Import flags first, then flags from the prepare token (--myclf-test "myclf /db leftover…"), then --flags.

samovar tools import -n myclf --exec-path /scratch/myclf.py --type annotator \
  --flags "--confidence 0.1"
samovar tools import -n myboil --exec-path /scratch/myboil.py --type table \
  --flags "--log-mu 1 --log-sigma 2"

samovar prepare --output_dir RUN \
  --myclf-test "myclf /path/to/db --threads 8" \
  --flags myclf "--keep-tmp" \
  --table_reads_generator myboil \
  --flags table_reads_generator "--distribution replicates"

--flags annotator "…" applies to every annotator in that prepare. --flags myclf "…" applies only to the run whose type / run_name / command basename is myclf. Same for --flags table_reads_generator vs --flags myboil / --flags camisim-table, --flags reads_generator vs --flags iss / --flags myreads, --flags metagenome_generator vs --flags constant_iss / --flags camisim, --flags qc / --flags trim vs --flags gc_filter, --flags scoring / --flags viz vs --flags counts, and --flags ml / --flags reprofiler vs --flags linear / --flags random_forest.

# Native classifier already on PATH
samovar tools import -n kaiju --exec kaiju --type annotator

# Conda sidecar (prefix, not the file)
samovar tools import -n nanosim --env conda --exec simulator.py \
  --exec-path "$SAMOVAR_ROOT/.cache/samovar/envs/nanosim" --type meta

# Your own annotator script
samovar tools import -n myclf --exec-path /scratch/myclf.py --type annotator
chmod +x /scratch/myclf.py

Then:

samovar prepare --output_dir RUN --myclf-test "myclf /path/to/db"

./install.sh rediscovers PATH tools and keeps existing tools.* values.


How a tool must look (by --type)

annotator

Input (SamovaR always passes these flags):

Flag Value
-i R1 FASTQ (.fastq or .fastq.gz)
-I R2 FASTQ (may be empty; still passed)
-d Database path from prepare (--name-test "name DB")
-o Output file path (create/overwrite)
-t Thread count
extra Tokens from import --flags, leftover after db in --name-test "name DB …", and prepare `--flags annotator

Do not require -p unless you are a custom.sh router tool (centrifuge, metauto, assembly_hybrid). Imported annotators are invoked directly. Extra flags are appended at the end of that command (after -t).

Output file (-o): tab-separated, no header, two columns:

<read_id>	<ncbi_taxid>
  • One row per classified read (skip unclassified / taxid 0 if you want).
  • read_id is the FASTQ header token without @, without /1 /2.
  • Empty -o is allowed (zero reads).
  • Snakemake names the file {sample}_{run}.custom_{name}.out. Downstream parsers only care about the two columns.

Processing: classify reads → write that TSV. Do not write logs into -o. Put temp files next to -o or under $TMPDIR and delete them.

Exit 0 on success even if every read is unclassified (empty file). Non-zero abort the annotators Snakemake rule.

Native names (kraken2, kaiju, metaphlan, …) keep their own CLIs; import only records the binary path. Do not import those under a new --name if you still pass --kaiju-test "kaiju …" — the type is taken from the command basename.

reads_generator

Same import pattern as annotators and table regenerators: register a binary, then pass its name to generate and prepare. Extra CLI flags from samovar tools import --flags (5th tools.* slot) are merged with samovar generate --flags / samovar prepare --flags. A reads tool only turns an abundance table (or equal mix from --genome_dir) into FASTQ. Combined community+reads tools use --type meta / --metagenome_generator below.

samovar tools import -n myreads --exec-path /scratch/myreads.py --type reads \
  --flags "--profile hiseq"
chmod +x /scratch/myreads.py

samovar generate --output_dir RUN --genome_dir genomes --host_genome host.fna \
  --reads_generator myreads \
  --flags myreads "--keep-tmp"

samovar prepare --output_dir RUN --reads_generator myreads \
  --flags reads_generator "--threads 4" \
  --kraken2-test "kraken2 /db"

--type reads stores group reads_generator. NanoSim may be --type meta (metagenome_generator); both groups are accepted. --flags reads_generator "…" applies to every reads tool in that run; --flags iss / --flags art / --flags myreads apply only to that name.

Built-ins (no import required):

Name Generate Prepare (from abundance tables)
iss InSilicoSeq; FASTQ {sample}_full_R1/R2.fastq Same mixer; {sample}_{annotator}_R*
art / illumina CAMISIM ART; also {sample}_illumina_R* then {sample}_full_R* ISS mix + read_type:illumina in headers and {sample}_{annotator}_illumina_R* copies
wgsim CAMISIM wgsim Same, read_type:wgsim
nanosim / ont CAMISIM NanoSim3 Same, read_type:ont
hybrid CAMISIM one run per tech; headers read_type:<tech> ISS mix (no fake per-tech split)
camisim CAMISIM community design (--camisim-mode table by default)

--simulator camisim is still valid; --reads_generator art is the same as --simulator camisim --camisim-mode illumina. Import flags on iss / art / wgsim / nanosim are appended to ISS or Nextflow as extra tokens.

Input (custom tools always get these; omit -i when generate uses dirs only):

Flag Value
-i Abundance CSV (taxid, N_<sample>…). Optional if --genome-dir is set.
-o Output directory (create FASTQ here)
-m Samples metadata CSV (omitted when unset)
--genome-dir Community genomes (generate-style)
--host-genome / --host-fraction Host spike-in; fraction or RANDOM
--n-samples / --total-reads / --seed / --model Same as samovar generate
--stage generate or regenerate
--annotator Annotator token on prepare ({sample}_{annotator}_R*)
--read-type Extra filename id (repeatable: illumina, sequence_type, …)
--gzip-reads Write .fastq.gz
extra Import --flags then prepare/generate --flags reads_generator|NAME

Generate without a table uses --genome_dir, --host_genome, --host_fraction, --n_samples, --total_reads like stock ISS. Generate -i abundance.csv feeds that table instead of a uniform community (ISS and custom tools).

Output FASTQ (.fastq or .fastq.gz):

  • Headers must keep a recoverable true taxid: taxid:<digits> (ISS combined genomes already do this; CAMISIM harvest adds \|taxid:). If you have a true label under another token, keep taxid: as well so extract_true_taxid works.
  • Hybrid / multi-tech: add read_type:<token> on the same header. Extra ids may appear in the filename as well as the header: generate {sample}_{sequence_type}_R1.fastq plus the canonical {sample}_full_R1.fastq; prepare {sample}_{annotator}_{sequence_type}_R1.fastq plus {sample}_{annotator}_R1.fastq. Annotators and scoring use the canonical (_full / _{annotator}) stem; tech copies are not classified twice.
  • Paired outputs: {prefix}_R1.fastq and {prefix}_R2.fastq (R2 may be empty for ONT).

Preferred contract — Python module (--exec-path ends in .py):

from typing import Dict, List, Optional
import pandas as pd

def generate(spec: dict, metadata: Optional[pd.DataFrame], config: dict) -> List[str]:
    # spec has abundance_table, genome_dir, host_genome, host_fraction,
    # n_samples, total_reads, output_dir, stage, annotator, extra_ids, gzip_reads
    # config["extra_argv"] is import + CLI flags
    _ = metadata
    out = []
    # write FASTQ under spec["output_dir"] with taxid: in headers
    return out

Alternatively class ReadsGenerator with generate(self, spec, metadata, config) or run(...). Use if __name__ == "__main__" so import does not run the CLI.

If the name is not in the install config (tools.NAME with group reads_generator or metagenome_generator), generate and prepare error. Import first.

ISS/ART/wgsim/NanoSim native CLIs stay unchanged when you only import them for PATH / conda prefix. Their SamovaR wrappers still tag taxid: / read_type: and extra filename ids as above.

metagenome_generator

Combined community design + read simulation in one tool (CAMISIM is the built-in example). Import with --type meta (not --type reads). Used in both samovar generate and samovar prepare.

samovar tools import -n constant_iss \
  --exec-path examples/metagenome_generator/constant_iss.py --type meta \
  --flags "--model hiseq"

samovar generate --output_dir RUN --genome_dir genomes --host_genome host.fna \
  --metagenome_generator constant_iss \
  --flags constant_iss "--n-reads 400"

samovar prepare --output_dir RUN --metagenome_generator constant_iss \
  --flags metagenome_generator "--model hiseq" \
  --test-genomes \
  --kraken2-test "kraken2 /db"

--flags metagenome_generator "…" applies to the combined tool in that run; --flags constant_iss / --flags camisim apply only to that name. Import --flags (5th slot) are merged first.

Built-ins (no import required): camisim (community design + ISS/ART depending on --camisim-mode), hybrid, nanosim. Same as --simulator camisim / --reads_generator hybrid.

Custom contract is the same FASTQ layout as reads_generator (taxid: / read_type:, {sample}_full_R* on generate, {sample}_{annotator}_R* on prepare), but the tool owns abundances: generate may ignore a table and draw a constant/equal community from --genome-dir + host fractions; prepare may flatten observed counts to a constant profile then simulate. Python generate(spec, metadata, config) -> list[str] or the same CLI as reads (-i/-o/--genome-dir/--host-fraction/--stage/--annotator + extra flags).

If NAME is not in the install config with group metagenome_generator, generate and prepare error.

Shipped example: examples/metagenome_generator/constant_iss.py (equal taxid counts + ISS).

qc

FASTQ metagenome → trimmed metagenome. Default is absent: setup_reads / ISS output is treated as already trimmed (the qc_initial / qc_generated checkpoints are identity copies into initial_trimmed/ and regenerated_trimmed/). Annotators always read the trimmed dirs.

Import with --type QC (aliases qc, trim, quality). Default is identity. Native short-read trimmers: fastp, Trimmomatic, Cutadapt (Illumina / BGI / MGI postfix is filled automatically). A GC-content filter is the Python contract example (tests/tools/gc_filter.py).

# fastp / Trimmomatic / Cutadapt: lazy-install if the binary is not on PATH yet
samovar tools import -n fastp --type QC \
  --exec-path fastp \
  --lazy-install 'conda install -y bioconda::fastp' \
  --pytest

samovar tools import -n trimmomatic --type QC \
  --exec-path trimmomatic \
  --lazy-install 'conda install -y bioconda::trimmomatic' \
  --pytest

samovar tools import -n cutadapt --type QC \
  --exec-path cutadapt \
  --lazy-install 'conda install -y bioconda::cutadapt' \
  --pytest

samovar prepare --output_dir RUN --qc fastp \
  --flags qc "--thread 8" \
  --kraken2-test "kraken2 /db"

# hybrid: short-read QC on Illumina/BGI postfixes, identity on ONT
samovar prepare --output_dir RUN --qc cutadapt \
  --qc-postfix ont:identity \
  --kraken2-test "kraken2 /db"

samovar tools import -n gc_filter \
  --exec-path tests/tools/gc_filter.py --type QC \
  --flags "--min-gc 0.3 --max-gc 0.7"

--qc fastp (aliases --qc illumina / --qc bgi once fastp is imported), --qc trimmomatic, or --qc cutadapt sets both stages and fills qc_postfix for illumina / bgi / mgi. --qc-initial / --qc-generated override one side (none / identity / omit = identity). --qc-postfix ont:identity keeps long reads untrimmed. --flags qc "…" applies to every QC tool; --flags fastp "…" / --flags trimmomatic "…" / --flags cutadapt "…" only to that name.

Native CLIs (plus import/prepare extra flags; --threads is translated):

  • fastp: fastp -i R1 -I R2 -o dest_R1 -O dest_R2 --thread N
  • Trimmomatic: trimmomatic PE -threads N R1 R2 dest_R1 unpaired1 dest_R2 unpaired2 MINLEN:1 (default step if none given)
  • Cutadapt: cutadapt --cores N -a … -A … -o dest_R1 -p dest_R2 R1 R2 (dummy poly-G adapter if none given)

Checkpoints: qc_initial (after setup_reads), qc_generated (after sort_reads). YAML keys: qc, qc_initial, qc_generated, qc_postfix: {illumina: fastp, bgi: fastp}.

Preferred contract — Python module (--exec-path ends in .py):

from typing import Dict, List, Optional

def trim(r1: str, r2: Optional[str], dest_r1: str, dest_r2: Optional[str], config: dict) -> List[str]:
    # config may include min_gc / max_gc and extra_argv from --flags qc
    # write dest_r1 / dest_r2 (empty files are allowed); keep mates in lockstep
    return [dest_r1] + ([dest_r2] if dest_r2 else [])

Alternatively class QC with trim / run. Use if __name__ == "__main__" so import does not run a CLI.

CLI fallback if there is no trim():

Flag Value
-i R1 FASTQ
-I R2 FASTQ (optional)
-o Output directory (same filenames as the inputs)
extra Import --flags then prepare --flags qc|NAME (--min-gc / --max-gc for the GC example)

fastp / Trimmomatic / Cutadapt are native binaries, not the generic -i/-I/-o dir CLI. samovar tools import --pytest runs tests/test_tool_contracts.py::test_qc_contract against a Python trim() or one of those binaries.

table_reads_generator

Same idea as a custom annotator: you implement a named tool, import it, then pass that name as the table-regeneration mode.

Built-in modes stay in Python (direct, bootstrap, vae, glm, camisim-table) or optional R (samovar). They all take an Annotation object (the concatenated taxID_* / sample table) and emit per-annotator abundance CSVs (taxid, N_<sample>…). Optional samples metadata is a second table (sample plus any covariates). Built-ins ignore it (null is fine). Your method may use it.

samovar tools import -n myboil --exec-path /scratch/myboil.py --type table
chmod +x /scratch/myboil.py

samovar prepare --output_dir RUN --table_reads_generator myboil \
  --kraken2-test "kraken2 /db"

--regeneration_mode myboil is the same flag. YAML:

regeneration_mode: myboil   # or table_reads_generator: myboil
samples_metadata: /path/to/samples.csv   # optional

If myboil is not in the install config (tools.myboil with group table_reads_generator), prepare and regenerate error. Import first.

Preferred contract — Python module (--exec-path ends in .py):

from typing import Dict, Optional
import pandas as pd
from samovar.parse_annotators import Annotation
from samovar.regenerate import regenerate_preserve

def regenerate(
    annotation: Annotation,
    metadata: Optional[pd.DataFrame],
    config: dict,
) -> Dict[str, pd.DataFrame]:
    # metadata is None unless samples_metadata was set
    _ = metadata
    return regenerate_preserve(
        annotation.DataFrame,
        n_reads=config.get("N_reads"),
    )

config includes N, N_reads, seed, annotation_dir, output_dir, extra_flags / extra_argv (imported --flags plus prepare --flags table_reads_generator "…"), and the rest of the regenerate YAML. Return {annotator: DataFrame} with columns taxid and N_<sample>.

Alternatively define class TableRegenerator with run(self, annotation, metadata, config) (same types).

CLI fallback if there is no regenerate() / TableRegenerator.run:

Flag Value
-i Annotation directory (*.annotation.csv)
-o Output directory (write {annotator}.csv)
-m Samples metadata CSV (omitted when unset)
--seed / --N / --N_reads Same as pipeline config
extra Tokens from import --flags and prepare --flags table_reads_generator "…"

Exit 0 after writing abundance CSVs. Use if __name__ == "__main__" so importing the module does not run the CLI.

See also samovar_regenerate.md.

scoring

OPAL and MultiQC stay native: import only records their path. Custom scoring and visualization wrappers use the same --type scoring (score, viz). They run after each viz checkpoint (viz_initial, viz_regenerated, viz_reprofiled) on files already written under the run output directory.

The 6th tools.* slot is the input glob (relative to $out_dir). Built-in viz/OPAL-style tools consume annotation directories samovar_out/*annotations (initial_annotations, regenerated_annotations, reprofiled_annotations). Future methods can point at a combined table, plot dirs, or anything else; store that glob here.

samovar tools import -n counts \
  --exec-path examples/scoring/count_annotations.py --type scoring \
  --inputs '*annotations' \
  --flags "--min-files 0"

samovar tools import -n table_score \
  --exec-path /scratch/table_score.py --type viz \
  --inputs '*annotations/combined_annotation_table.csv'

samovar prepare --output_dir RUN \
  --kraken2-test "kraken2 /db" \
  --flags scoring "--tag extra" \
  --flags counts "--min-files 1"

--scoring NAME (repeatable) runs only those imported tools; omit it to run every custom scoring tool. --scoring none disables them.

--inputs value Matches under $out_dir
*annotations (default) initial_annotations, regenerated_annotations, reprofiled_annotations
annotation_table / combined_annotation_table.csv *annotations/combined_annotation_table.csv
*_plots / plots *_annotations_plots
comma-separated union of each pattern

Empty 5th slot is kept when the 6th is set: ["", "bash", "/path/script.py", "scoring", "", "*annotations"]. Dict form: inputs / glob.

Preferred contract — Python module (--exec-path ends in .py):

from pathlib import Path
from typing import List, Optional

def score(inputs: List[Path], output_dir: Path, config: dict) -> None:
    # inputs: resolved paths for this tool's glob
    # output_dir: $out_dir/{name}_scores
    # config["extra_argv"] is import + prepare --flags scoring|NAME
    # config["stage"] is viz_initial / viz_regenerated / viz_reprofiled
    output_dir.mkdir(parents=True, exist_ok=True)
    (output_dir / "ok.txt").write_text("\n".join(str(p) for p in inputs))

Alternatively visualize(...) or class Scorer with score / run. Use if __name__ == "__main__" so import does not run a CLI.

CLI fallback if there is no score():

Flag Value
-i Each matched path (repeatable)
-o Output directory ($out_dir/{name}_scores)
extra Import --flags then prepare --flags scoring|NAME

Native opal / opal.py / multiqc are not re-invoked as custom wrappers (the pipeline already runs compare_annotations, OPAL-adjacent viz, and MultiQC).

Shipped example: examples/scoring/count_annotations.py.

reprofiler

Supervised ML on annotation tables with known ground truth (regenerated metagenomes). Built-ins need no import: ensemble (default; RandomForest + AdaBoost, keep the better test score), random_forest / rf, adaboost / ada. Custom methods: --type ml (aliases reprofiler, reprofiling).

The model trains on the regenerated annotation table (true taxid from FASTQ taxid: labels) and may also read ground-truth abundance tables ($out/regenerated/.regenerated_abundance/, the counts used to simulate those reads). It then profiles the initial annotation tables into $out/reprofiled_annotations/ (taxid_SAMOVAR) and writes trained_model.joblib.

samovar tools import -n linear \
  --exec-path examples/reprofiling/linear_classifier.py --type ml \
  --flags "--max-iter 500"

samovar prepare --output_dir RUN --reprofiler linear \
  --flags ml "--C 1" \
  --kraken2-test "kraken2 /db"

--reprofiler / --ml selects the method. --flags ml "…" applies to whichever reprofiler is selected; --flags linear / --flags random_forest apply only to that name.

Preferred contract — Python module:

from typing import Dict
import pandas as pd
from samovar.reprofilers import ReprofileResult, profile_tables_with_model

def reprofile(
    regenerated: pd.DataFrame,
    ground_truth: Dict[str, pd.DataFrame],
    initial: Dict[str, pd.DataFrame],
    config: dict,
) -> ReprofileResult:
    # regenerated: combined regenerated annotation table (column ``true``)
    # ground_truth: {name: abundance CSV} used to generate those metagenomes
    # initial: {sample: annotation table} to profile
    # config["extra_argv"] is import + prepare --flags ml|NAME
    # return tables with taxid_SAMOVAR / taxid_SAMOVAR_confidence, plus model
    ...

CLI fallback: --regenerated, --ground-truth DIR, --initial DIR, -o DIR, plus extra flags. Write {sample}_reprofiled.csv and trained_model.joblib.

See samovar_reprofiling.md.

workflow / runtime / compiler

Snakemake, Nextflow, python, g++: path only.


Built-in annotation tools

centrifuge, metauto, assembly_hybrid, and the dummy constant-taxID classifier still go through src/annotators/custom.sh (-p <name>). Importing centrifuge only updates the binary location used inside that router, not the SamovaR -i/-I/-d/-o contract.

Clone this wiki locally