-
Notifications
You must be signed in to change notification settings - Fork 1
Custom tools import
Register an executable in the install config (tools.<name>) so prepare / exec / resolve_executable find it without a full ./install.sh. Annotator indexes use the same CLI with --type database (writes databases.*, not tools.*).
samovar tools import \
--name kaiju \
--env "" \
--exec kaiju \
--exec-path /usr/bin/kaiju \
--type annotator| Flag | Alias | Meaning |
|---|---|---|
--name |
-n |
Config key. Also the prepare token (--kaiju-test "kaiju /db"). |
--env |
"" (host) or conda (path is a sidecar prefix). Default "". |
|
--exec |
Binary name (default: --name). Used as prefix/bin/<exec> when --exec-path is a conda env directory. |
|
--exec-path |
File, conda prefix, or a name on PATH. If omitted, PATH is searched for --exec then --name. |
|
--type |
-t |
Stage / tool_group, or database / db to write databases.*. |
--tool |
Annotator that uses this DB (--type database). Also accepted as -n kraken2:dbname. |
|
--flags |
Native CLI flags. For a tool: stored on tools.*. For a database: merged into that annotator's extra at prepare. |
|
--lazy-install |
Rebuild recipe for a binary (@file or - for stdin). |
|
--lazy-download |
Rebuild recipe for a database (@file / --lazy-download-file). |
|
--url |
Official archive URL for a database (feeds lazy-download). |
|
--tool-version |
Version stored in the key (name:version). |
|
--inputs |
--glob |
Input glob under the run output dir (6th tools.* slot). Scoring/viz default: *annotations. |
--type values: annotator (a), reads_generator (reads), metagenome_generator (meta), qc (QC, trim), table_reads_generator (table), scoring (score, viz), reprofiler (ml), annotation_converter (converter), workflow, runtime, compiler, database (db).
Writes databases.<annotator>.<name:version> (same object shape as tools: path, flags, lazy-download, url, version). Does not go under tools.*.
samovar tools import --type database \
-n standard_8GB --tool kraken2 \
--exec-path /path/to/kraken2_db \
--tool-version 2025oct \
--flags "--memory-mapping" \
--url https://genome-idx.s3.amazonaws.com/kraken/k2_standard_08gb_20251015.tar.gz-n kraken2:standard_8GB is the same as -n standard_8GB --tool kraken2. --exec-path may be a directory, a file (.qza, .fa.gz), or a centrifuge prefix (p_compressed+h+v when .1.cf exists). --lazy-download @fetch.sh stores a custom rebuild script; if omitted, known official URLs get a curl/tar recipe.
Then prepare uses the indexed name, not the filesystem path:
samovar prepare --output_dir RUN --kraken2-test "kraken2 standard_8GB"flags on the DB record are native tokens for that classifier and are appended to its command. CLI leftovers after the name (--kraken2-test "kraken2 standard_8GB --threads 8") still apply. samovar build --type kaiju --index NAME --flags "…" also registers this object.
samovar export (full) packs the tree plus db-meta/.../lazy-download.sh; unfold install.sh re-imports with --type database.
Stored row (legacy list form still parses; on disk the preferred form is a name:version object):
"name": ["env", "workflow", "path", "tool_group"]
"name": ["env", "workflow", "path", "tool_group", "flags"] # --flags set
"name": ["env", "workflow", "path", "tool_group", "flags", "inputs"] # scoring, or --inputs set
workflow is conda when --env conda, otherwise bash. path is the resolved file, or the conda prefix. The 5th slot is extra CLI flags from samovar tools import --flags "…". Empty flags are omitted unless a 6th inputs slot is present (scoring/viz).
Those flags are merged at prepare / run with samovar prepare --flags TOOL "…". TOOL can be the imported --name, or the group (annotator / a, table_reads_generator / table). Import flags first, then flags from the prepare token (--myclf-test "myclf /db leftover…"), then --flags.
samovar tools import -n myclf --exec-path /scratch/myclf.py --type annotator \
--flags "--confidence 0.1"
samovar tools import -n myboil --exec-path /scratch/myboil.py --type table \
--flags "--log-mu 1 --log-sigma 2"
samovar prepare --output_dir RUN \
--myclf-test "myclf /path/to/db --threads 8" \
--flags myclf "--keep-tmp" \
--table_reads_generator myboil \
--flags table_reads_generator "--distribution replicates"--flags annotator "…" applies to every annotator in that prepare. --flags myclf "…" applies only to the run whose type / run_name / command basename is myclf. Same for --flags table_reads_generator vs --flags myboil / --flags camisim-table, --flags reads_generator vs --flags iss / --flags myreads, --flags metagenome_generator vs --flags constant_iss / --flags camisim, --flags qc / --flags trim vs --flags gc_filter, --flags scoring / --flags viz vs --flags counts, and --flags ml / --flags reprofiler vs --flags linear / --flags random_forest.
# Native classifier already on PATH
samovar tools import -n kaiju --exec kaiju --type annotator
# Conda sidecar (prefix, not the file)
samovar tools import -n nanosim --env conda --exec simulator.py \
--exec-path "$SAMOVAR_ROOT/.cache/samovar/envs/nanosim" --type meta
# Your own annotator script
samovar tools import -n myclf --exec-path /scratch/myclf.py --type annotator
chmod +x /scratch/myclf.pyThen:
samovar prepare --output_dir RUN --myclf-test "myclf /path/to/db"./install.sh rediscovers PATH tools and keeps existing tools.* values.
Input (SamovaR always passes these flags):
| Flag | Value |
|---|---|
-i |
R1 FASTQ (.fastq or .fastq.gz) |
-I |
R2 FASTQ (may be empty; still passed) |
-d |
Database path from prepare (--name-test "name DB") |
-o |
Output file path (create/overwrite) |
-t |
Thread count |
| extra | Tokens from import --flags, leftover after db in --name-test "name DB …", and prepare `--flags annotator |
Do not require -p unless you are a custom.sh router tool (centrifuge, metauto, assembly_hybrid). Imported annotators are invoked directly. Extra flags are appended at the end of that command (after -t).
Output file (-o): tab-separated, no header, two columns:
<read_id> <ncbi_taxid>
- One row per classified read (skip unclassified / taxid
0if you want). -
read_idis the FASTQ header token without@, without/1/2. - Empty
-ois allowed (zero reads). - Snakemake names the file
{sample}_{run}.custom_{name}.out. Downstream parsers only care about the two columns.
Processing: classify reads → write that TSV. Do not write logs into -o. Put temp files next to -o or under $TMPDIR and delete them.
Exit 0 on success even if every read is unclassified (empty file). Non-zero abort the annotators Snakemake rule.
Native names (kraken2, kaiju, metaphlan, …) keep their own CLIs; import only records the binary path. Do not import those under a new --name if you still pass --kaiju-test "kaiju …" — the type is taken from the command basename.
Same import pattern as annotators and table regenerators: register a binary, then pass its name to generate and prepare. Extra CLI flags from samovar tools import --flags (5th tools.* slot) are merged with samovar generate --flags / samovar prepare --flags. A reads tool only turns an abundance table (or equal mix from --genome_dir) into FASTQ. Combined community+reads tools use --type meta / --metagenome_generator below.
samovar tools import -n myreads --exec-path /scratch/myreads.py --type reads \
--flags "--profile hiseq"
chmod +x /scratch/myreads.py
samovar generate --output_dir RUN --genome_dir genomes --host_genome host.fna \
--reads_generator myreads \
--flags myreads "--keep-tmp"
samovar prepare --output_dir RUN --reads_generator myreads \
--flags reads_generator "--threads 4" \
--kraken2-test "kraken2 /db"--type reads stores group reads_generator. NanoSim may be --type meta (metagenome_generator); both groups are accepted. --flags reads_generator "…" applies to every reads tool in that run; --flags iss / --flags art / --flags myreads apply only to that name.
Built-ins (no import required):
| Name | Generate | Prepare (from abundance tables) |
|---|---|---|
iss |
InSilicoSeq; FASTQ {sample}_full_R1/R2.fastq
|
Same mixer; {sample}_{annotator}_R*
|
art / illumina
|
CAMISIM ART; also {sample}_illumina_R* then {sample}_full_R*
|
ISS mix + read_type:illumina in headers and {sample}_{annotator}_illumina_R* copies |
wgsim |
CAMISIM wgsim | Same, read_type:wgsim
|
nanosim / ont
|
CAMISIM NanoSim3 | Same, read_type:ont
|
hybrid |
CAMISIM one run per tech; headers read_type:<tech>
|
ISS mix (no fake per-tech split) |
camisim |
CAMISIM community design (--camisim-mode table by default) |
— |
--simulator camisim is still valid; --reads_generator art is the same as --simulator camisim --camisim-mode illumina. Import flags on iss / art / wgsim / nanosim are appended to ISS or Nextflow as extra tokens.
Input (custom tools always get these; omit -i when generate uses dirs only):
| Flag | Value |
|---|---|
-i |
Abundance CSV (taxid, N_<sample>…). Optional if --genome-dir is set. |
-o |
Output directory (create FASTQ here) |
-m |
Samples metadata CSV (omitted when unset) |
--genome-dir |
Community genomes (generate-style) |
--host-genome / --host-fraction
|
Host spike-in; fraction or RANDOM
|
--n-samples / --total-reads / --seed / --model
|
Same as samovar generate
|
--stage |
generate or regenerate
|
--annotator |
Annotator token on prepare ({sample}_{annotator}_R*) |
--read-type |
Extra filename id (repeatable: illumina, sequence_type, …) |
--gzip-reads |
Write .fastq.gz
|
| extra | Import --flags then prepare/generate --flags reads_generator|NAME
|
Generate without a table uses --genome_dir, --host_genome, --host_fraction, --n_samples, --total_reads like stock ISS. Generate -i abundance.csv feeds that table instead of a uniform community (ISS and custom tools).
Output FASTQ (.fastq or .fastq.gz):
- Headers must keep a recoverable true taxid:
taxid:<digits>(ISS combined genomes already do this; CAMISIM harvest adds\|taxid:). If you have a true label under another token, keeptaxid:as well soextract_true_taxidworks. - Hybrid / multi-tech: add
read_type:<token>on the same header. Extra ids may appear in the filename as well as the header: generate{sample}_{sequence_type}_R1.fastqplus the canonical{sample}_full_R1.fastq; prepare{sample}_{annotator}_{sequence_type}_R1.fastqplus{sample}_{annotator}_R1.fastq. Annotators and scoring use the canonical (_full/_{annotator}) stem; tech copies are not classified twice. - Paired outputs:
{prefix}_R1.fastqand{prefix}_R2.fastq(R2 may be empty for ONT).
Preferred contract — Python module (--exec-path ends in .py):
from typing import Dict, List, Optional
import pandas as pd
def generate(spec: dict, metadata: Optional[pd.DataFrame], config: dict) -> List[str]:
# spec has abundance_table, genome_dir, host_genome, host_fraction,
# n_samples, total_reads, output_dir, stage, annotator, extra_ids, gzip_reads
# config["extra_argv"] is import + CLI flags
_ = metadata
out = []
# write FASTQ under spec["output_dir"] with taxid: in headers
return outAlternatively class ReadsGenerator with generate(self, spec, metadata, config) or run(...). Use if __name__ == "__main__" so import does not run the CLI.
If the name is not in the install config (tools.NAME with group reads_generator or metagenome_generator), generate and prepare error. Import first.
ISS/ART/wgsim/NanoSim native CLIs stay unchanged when you only import them for PATH / conda prefix. Their SamovaR wrappers still tag taxid: / read_type: and extra filename ids as above.
Combined community design + read simulation in one tool (CAMISIM is the built-in example). Import with --type meta (not --type reads). Used in both samovar generate and samovar prepare.
samovar tools import -n constant_iss \
--exec-path examples/metagenome_generator/constant_iss.py --type meta \
--flags "--model hiseq"
samovar generate --output_dir RUN --genome_dir genomes --host_genome host.fna \
--metagenome_generator constant_iss \
--flags constant_iss "--n-reads 400"
samovar prepare --output_dir RUN --metagenome_generator constant_iss \
--flags metagenome_generator "--model hiseq" \
--test-genomes \
--kraken2-test "kraken2 /db"--flags metagenome_generator "…" applies to the combined tool in that run; --flags constant_iss / --flags camisim apply only to that name. Import --flags (5th slot) are merged first.
Built-ins (no import required): camisim (community design + ISS/ART depending on --camisim-mode), hybrid, nanosim. Same as --simulator camisim / --reads_generator hybrid.
Custom contract is the same FASTQ layout as reads_generator (taxid: / read_type:, {sample}_full_R* on generate, {sample}_{annotator}_R* on prepare), but the tool owns abundances: generate may ignore a table and draw a constant/equal community from --genome-dir + host fractions; prepare may flatten observed counts to a constant profile then simulate. Python generate(spec, metadata, config) -> list[str] or the same CLI as reads (-i/-o/--genome-dir/--host-fraction/--stage/--annotator + extra flags).
If NAME is not in the install config with group metagenome_generator, generate and prepare error.
Shipped example: examples/metagenome_generator/constant_iss.py (equal taxid counts + ISS).
FASTQ metagenome → trimmed metagenome. Default is absent: setup_reads / ISS output is treated as already trimmed (the qc_initial / qc_generated checkpoints are identity copies into initial_trimmed/ and regenerated_trimmed/). Annotators always read the trimmed dirs.
Import with --type QC (aliases qc, trim, quality). Default is identity. Native short-read trimmers: fastp, Trimmomatic, Cutadapt (Illumina / BGI / MGI postfix). ONT: Chopper, NanoFilt (ont / nanosim3 postfix). A GC-content filter is the Python contract example (tests/tools/gc_filter.py).
# fastp / Trimmomatic / Cutadapt: lazy-install if the binary is not on PATH yet
samovar tools import -n fastp --type QC \
--exec-path fastp \
--lazy-install 'conda install -y bioconda::fastp' \
--pytest
samovar tools import -n trimmomatic --type QC \
--exec-path trimmomatic \
--lazy-install 'conda install -y bioconda::trimmomatic' \
--pytest
samovar tools import -n cutadapt --type QC \
--exec-path cutadapt \
--lazy-install 'conda install -y bioconda::cutadapt' \
--pytest
samovar tools import -n chopper --type QC \
--exec-path chopper \
--lazy-install 'conda install -y bioconda::chopper' \
--pytest
samovar tools import -n nanofilt --type QC \
--exec-path NanoFilt \
--lazy-install 'conda install -y bioconda::nanofilt' \
--pytest
samovar prepare --output_dir RUN --qc fastp \
--flags qc "--thread 8" \
--kraken2-test "kraken2 /db"
# hybrid: short-read QC on Illumina, Chopper on ONT
samovar prepare --output_dir RUN --qc fastp \
--qc-postfix ont:chopper \
--kraken2-test "kraken2 /db"
samovar tools import -n gc_filter \
--exec-path tests/tools/gc_filter.py --type QC \
--flags "--min-gc 0.3 --max-gc 0.7"--qc fastp (aliases --qc illumina / --qc bgi once fastp is imported), --qc trimmomatic, or --qc cutadapt fills qc_postfix for illumina / bgi / mgi. --qc chopper (aliases --qc ont / --qc nanopore) or --qc nanofilt fills ont / nanosim3. --qc-initial / --qc-generated override one side (none / identity / omit = identity). --flags qc "…" applies to every QC tool; --flags chopper "…" / --flags nanofilt "…" only to that name.
Native CLIs (plus import/prepare extra flags; --threads is translated):
-
fastp:
fastp -i R1 -I R2 -o dest_R1 -O dest_R2 --thread N -
Trimmomatic:
trimmomatic PE -threads N R1 R2 dest_R1 unpaired1 dest_R2 unpaired2 MINLEN:1(default step if none given) -
Cutadapt:
cutadapt --cores N -a … -A … -o dest_R1 -p dest_R2 R1 R2(dummy poly-G adapter if none given) -
Chopper / NanoFilt: stdin FASTQ → stdout (SE; R1 and R2 filtered independently). Defaults
-q 0 -l 1so the contract keeps short reads. Chopper gets--threads N.
Checkpoints: qc_initial (after setup_reads), qc_generated (after sort_reads). YAML keys: qc, qc_initial, qc_generated, qc_postfix: {illumina: fastp, ont: chopper}.
Preferred contract — Python module (--exec-path ends in .py):
from typing import Dict, List, Optional
def trim(r1: str, r2: Optional[str], dest_r1: str, dest_r2: Optional[str], config: dict) -> List[str]:
# config may include min_gc / max_gc and extra_argv from --flags qc
# write dest_r1 / dest_r2 (empty files are allowed); keep mates in lockstep
return [dest_r1] + ([dest_r2] if dest_r2 else [])Alternatively class QC with trim / run. Use if __name__ == "__main__" so import does not run a CLI.
CLI fallback if there is no trim():
| Flag | Value |
|---|---|
-i |
R1 FASTQ |
-I |
R2 FASTQ (optional) |
-o |
Output directory (same filenames as the inputs) |
| extra | Import --flags then prepare --flags qc|NAME (--min-gc / --max-gc for the GC example) |
fastp / Trimmomatic / Cutadapt / Chopper / NanoFilt are native binaries, not the generic -i/-I/-o dir CLI. Import --flags (and --flags-translate, --lazy-install) are appended after SamovaR fills inputs/outputs and threads. Sidecar reports: *.fastp.json, *.cutadapt.json, *.trimlog. samovar tools import --pytest runs tests/test_tool_contracts.py::test_qc_contract. CLI adapters: tests/tools/fastp, cutadapt, trimmomatic, chopper, nanofilt.
Same idea as a custom annotator: you implement a named tool, import it, then pass that name as the table-regeneration mode.
Built-in modes stay in Python (direct, bootstrap, vae, glm, camisim-table) or optional R (samovar). They all take an Annotation object (the concatenated taxID_* / sample table) and emit per-annotator abundance CSVs (taxid, N_<sample>…). Optional samples metadata is a second table (sample plus any covariates). Built-ins ignore it (null is fine). Your method may use it.
samovar tools import -n myboil --exec-path /scratch/myboil.py --type table
chmod +x /scratch/myboil.py
samovar prepare --output_dir RUN --table_reads_generator myboil \
--kraken2-test "kraken2 /db"--regeneration_mode myboil is the same flag. YAML:
regeneration_mode: myboil # or table_reads_generator: myboil
samples_metadata: /path/to/samples.csv # optionalIf myboil is not in the install config (tools.myboil with group table_reads_generator), prepare and regenerate error. Import first.
Preferred contract — Python module (--exec-path ends in .py):
from typing import Dict, Optional
import pandas as pd
from samovar.parse_annotators import Annotation
from samovar.regenerate import regenerate_preserve
def regenerate(
annotation: Annotation,
metadata: Optional[pd.DataFrame],
config: dict,
) -> Dict[str, pd.DataFrame]:
# metadata is None unless samples_metadata was set
_ = metadata
return regenerate_preserve(
annotation.DataFrame,
n_reads=config.get("N_reads"),
)config includes N, N_reads, seed, annotation_dir, output_dir, extra_flags / extra_argv (imported --flags plus prepare --flags table_reads_generator "…"), and the rest of the regenerate YAML. Return {annotator: DataFrame} with columns taxid and N_<sample>.
Alternatively define class TableRegenerator with run(self, annotation, metadata, config) (same types).
CLI fallback if there is no regenerate() / TableRegenerator.run:
| Flag | Value |
|---|---|
-i |
Annotation directory (*.annotation.csv) |
-o |
Output directory (write {annotator}.csv) |
-m |
Samples metadata CSV (omitted when unset) |
--seed / --N / --N_reads
|
Same as pipeline config |
| extra | Tokens from import --flags and prepare --flags table_reads_generator "…"
|
Exit 0 after writing abundance CSVs. Use if __name__ == "__main__" so importing the module does not run the CLI.
See also samovar_regenerate.md.
OPAL and MultiQC stay native: import only records their path. Custom scoring and visualization wrappers use the same --type scoring (score, viz). They run after each viz checkpoint (viz_initial, viz_regenerated, viz_reprofiled) on files already written under the run output directory.
The 6th tools.* slot is the input glob (relative to $out_dir). Built-in viz/OPAL-style tools consume annotation directories samovar_out/*annotations (initial_annotations, regenerated_annotations, reprofiled_annotations). Future methods can point at a combined table, plot dirs, or anything else; store that glob here.
samovar tools import -n counts \
--exec-path examples/scoring/count_annotations.py --type scoring \
--inputs '*annotations' \
--flags "--min-files 0"
samovar tools import -n table_score \
--exec-path /scratch/table_score.py --type viz \
--inputs '*annotations/combined_annotation_table.csv'
samovar prepare --output_dir RUN \
--kraken2-test "kraken2 /db" \
--flags scoring "--tag extra" \
--flags counts "--min-files 1"--scoring NAME (repeatable) runs only those imported tools; omit it to run every custom scoring tool. --scoring none disables them.
--inputs value |
Matches under $out_dir
|
|---|---|
*annotations (default) |
initial_annotations, regenerated_annotations, reprofiled_annotations
|
annotation_table / combined_annotation_table.csv
|
*annotations/combined_annotation_table.csv |
*_plots / plots
|
*_annotations_plots |
| comma-separated | union of each pattern |
Empty 5th slot is kept when the 6th is set: ["", "bash", "/path/script.py", "scoring", "", "*annotations"]. Dict form: inputs / glob.
Preferred contract — Python module (--exec-path ends in .py):
from pathlib import Path
from typing import List, Optional
def score(inputs: List[Path], output_dir: Path, config: dict) -> None:
# inputs: resolved paths for this tool's glob
# output_dir: $out_dir/{name}_scores
# config["extra_argv"] is import + prepare --flags scoring|NAME
# config["stage"] is viz_initial / viz_regenerated / viz_reprofiled
output_dir.mkdir(parents=True, exist_ok=True)
(output_dir / "ok.txt").write_text("\n".join(str(p) for p in inputs))Alternatively visualize(...) or class Scorer with score / run. Use if __name__ == "__main__" so import does not run a CLI.
CLI fallback if there is no score():
| Flag | Value |
|---|---|
-i |
Each matched path (repeatable) |
-o |
Output directory ($out_dir/{name}_scores) |
| extra | Import --flags then prepare --flags scoring|NAME
|
Native opal / opal.py / multiqc are not re-invoked as custom wrappers (the pipeline already runs compare_annotations, OPAL-adjacent viz, and MultiQC).
Shipped example: examples/scoring/count_annotations.py.
Supervised ML on annotation tables with known ground truth (regenerated metagenomes). Built-ins need no import: ensemble (default; RandomForest + AdaBoost, keep the better test score), random_forest / rf, adaboost / ada. Custom methods: --type ml (aliases reprofiler, reprofiling).
The model trains on the regenerated annotation table (true taxid from FASTQ taxid: labels) and may also read ground-truth abundance tables ($out/regenerated/.regenerated_abundance/, the counts used to simulate those reads). It then profiles the initial annotation tables into $out/reprofiled_annotations/ (taxid_SAMOVAR) and writes trained_model.joblib.
samovar tools import -n linear \
--exec-path examples/reprofiling/linear_classifier.py --type ml \
--flags "--max-iter 500"
samovar prepare --output_dir RUN --reprofiler linear \
--flags ml "--C 1" \
--kraken2-test "kraken2 /db"--reprofiler / --ml selects the method. --flags ml "…" applies to whichever reprofiler is selected; --flags linear / --flags random_forest apply only to that name.
Preferred contract — Python module:
from typing import Dict
import pandas as pd
from samovar.reprofilers import ReprofileResult, profile_tables_with_model
def reprofile(
regenerated: pd.DataFrame,
ground_truth: Dict[str, pd.DataFrame],
initial: Dict[str, pd.DataFrame],
config: dict,
) -> ReprofileResult:
# regenerated: combined regenerated annotation table (column ``true``)
# ground_truth: {name: abundance CSV} used to generate those metagenomes
# initial: {sample: annotation table} to profile
# config["extra_argv"] is import + prepare --flags ml|NAME
# return tables with taxid_SAMOVAR / taxid_SAMOVAR_confidence, plus model
...CLI fallback: --regenerated, --ground-truth DIR, --initial DIR, -o DIR, plus extra flags. Write {sample}_reprofiled.csv and trained_model.joblib.
Snakemake, Nextflow, python, g++: path only.
centrifuge, metauto, assembly_hybrid, and the dummy constant-taxID classifier still go through src/annotators/custom.sh (-p <name>). Importing centrifuge only updates the binary location used inside that router, not the SamovaR -i/-I/-d/-o contract.