Snakemake and Scripts Reference

Snakemake & Scripts Reference

What is this?

Where every rule and script lives, what each does, what it calls, and the conventions for writing new ones.

Why does it matter?

Changing one rule can change all 27 builds. Knowing how the files connect tells you what a change will touch.

What do I do?

Look things up here; follow the conventions when you add or change code. How to run the pipeline is on Step 2.

flowchart LR
    SF["Snakefile<br/>wildcards, min_lengths,<br/>rule all"] --> I["ingest.smk"]
    SF --> C["colors.smk"]
    SF --> S["segments.smk"]
    SF --> G["genomes.smk"]
    SF --> R["reports.smk"]
    SF --> SN["snapshot.smk"]
    I -->|"data/{subtype}/{segment}"| S
    I -->|"data/{subtype}/genome"| G
    S -->|"HA Nextclade calls"| G
    C -->|"colors.tsv"| S
    C -->|"colors.tsv"| G
    S --> R
    G --> R
    R --> SN

Loading

The Snakefile includes six rule files in this order; arrows show which outputs feed which file. colors.smk must come before segments.smk and genomes.smk, reports.smk after ingest.smk, and snapshot.smk after reports.smk.

The rule files

All rules live in workflow/snakemake_rules/.

File	What it does	Calls	Main outputs
Snakefile	Wildcard constraints (subtype: h1n1/h3n2/vic, segment: pb2...ns), segment minimum lengths, rule all (27 builds + report)	-	-
ingest.smk	Types JHH segments, builds fludb, writes per-build inputs	scripts/flusort/flusort.py; fludb/scripts/fludb_inititate.py, upload_jhh.py, upload_vaccine.py; scripts/fludb_download_builds.py (→ fludb/scripts/download.py, download_concatenated.py; reads config/subtype_assignment.yaml)	source/flusort_*, fludb.db, data/{subtype}/{segment}/, data/{subtype}/genome/
colors.smk	Syncs subclade colors from nextstrain/seasonal-flu before any export	scripts/sync_subclade_colors.py	config/{subtype}/colors.tsv, config/subclade_colors_source.tsv
segments.smk	One build per subtype × segment	nextclade; scripts/assign_clades.py; scripts/merge_quality_metrics.py; augur	results/{subtype}/{segment}/, auspice/{subtype}/{segment}.json, ..._tip-frequencies.json
genomes.smk	Whole-genome build per subtype	csvtk; augur	results/{subtype}/genome/, auspice/{subtype}/genome.json
reports.smk	Surveillance report; run start/end handlers (provenance, data scan, run report)	scripts/report.py; scripts/run_report.py; scripts/record_provenance.py; scripts/data_guard.sh	reports/, .run/run_logs/, .run/reports/
snapshot.smk	Archive and clean the workspace	scripts/snapshot_clean.sh	snapshots/{timestamp}.tar.gz

Rules in order

ingest.smk: flusort → fludb_inititate → upload_genomes, upload_vaccines → download_segments → download_genomes (+ ingest_complete).

segments.smk (per subtype × segment): fetch_hana_datasets (once per run; downloads the git-ignored HA/NA datasets and logs their tags) → nextclade → assign_clades → merge_quality_metrics → augur_filter → align → raw_tree → refine → annotate_traits, infer_ancestral → translate → export; frequency.

genomes.smk (per subtype): genome_clades → genome_filter → genome_align → genome_tree → genome_refine → genome_traits, genome_ancestral → genome_translate → genome_export; genome_frequencies.

colors.smk: sync_subclade_colors (once per run, before the first export; skip with --config sync_subclade_colors=false).

export and genome_export wait for sync_subclade_colors, then color subclades from config/{subtype}/colors.tsv (Subclade colors).

reports.smk: surveillance_report; handlers onstart, onsuccess, onerror.

snapshot.smk: snapshot_clean (on demand), snapshot_clean_after_build (with --config snapshot_clean=true).

Rule graph

Because wildcards are constrained by subtype and segment, this graph runs for H1N1, H3N2 and B/Victoria for all 8 segments, plus the 3 genome builds.

Regenerate after changing rules (graphviz: conda install -c conda-forge graphviz if missing):

snakemake --rulegraph | dot -Tpng > rulegraph.png

Writing rules

• One .smk file per major feature. A new feature (e.g. t-SNE embeddings, reassortment analysis) gets its own file and one include: line in the Snakefile; keep the include order.
• Keep .smk files declarative: inputs, outputs, params and a short shell:. Small helpers (like clock_rate()) and one-line lambdas are fine; more than a few lines of Python belongs in scripts/. script: is fine for short pandas steps (see scripts/assign_clades.py).
• Every rule has message:, input:, output:, log: (.run/logs/{rule}/{subtype}_{segment}.log), and a shell: starting with exec > {log} 2>&1, Python run as python -u. Add benchmark: (.run/benchmarks/...) for anything slower than a few seconds and threads: for multithreaded tools.
• Outputs go to results/ (intermediate) or auspice/ (final). Rules never write into source/, config/ or tutorial/ (the flusort_* outputs in source/ are a historical exception).
• Use the input variables from ingest.smk (jhh_sequences, ...), not hard-coded source/ paths, so the tutorial config keeps working.
• Use the existing wildcards {subtype} and {segment} (plus genome); new targets in rule all are Tier 2.
• Keep filter values in sync with config/report.yaml (Step 3).
• Name rules in snake_case after their output.
• Test: snakemake -n, snakemake --lint (no new warnings), a tutorial run, regenerate rulegraph.png, update the wiki (page map).

Writing scripts

• All functional code lives in scripts. Anything that calculates or modifies metrics or data for the builds is a script in scripts/ (fludb's own in fludb/scripts/), called by a rule. Never in a notebook, never as a long block inside a .smk file.
• Header docstring: what it does, which rule calls it, inputs, outputs, and, for adapted code, the original source and licence.
• Arguments, not hard-coded paths: argparse for command-line scripts, the snakemake object for script: scripts.
• Small, named, commented functions; comment the why. No unused options, no dead code, no _v2 copies.
• Report what happened (rows read, written, dropped and why) into the rule's log. Fail loudly on bad input; never write partial output.
• No new dependencies without a Tier 3 environment change.
• Naming: snake_case.py, verb first (assign_clades.py).
• Retiring: move to scripts/depreciated/ in its own commit; remove the calling rule in the same PR.

Script inventory

Script	Called by	Purpose
scripts/prepare_source.py	you (Tutorial 2)	Build JHH_sequences.fasta + JHH_metadata.txt from the run FASTAs in source_prep/
scripts/flusort/flusort.py	rule flusort	BLAST typing/subtyping and completeness of JHH segments
fludb/scripts/fludb_inititate.py	rule fludb_inititate	Create the empty fludb.db
fludb/scripts/upload_jhh.py	rule upload_genomes	Load JHH sequences + metadata (database_origin = mostafa_lab)
fludb/scripts/upload_vaccine.py	rule upload_vaccines	Load vaccine.fasta (database_origin = gisaid)
scripts/fludb_download_builds.py	rules download_segments (--segments), download_genomes (--genome)	Writes data/{subtype}/ from fludb for one subtype: fludb/scripts/download.py per segment, or download_concatenated.py for the genome. Which fludb subtypes go into each build (incl. assumed H1xx/xxN1/H3xx/xxN2) is set in config/subtype_assignment.yaml. Replaces fludb_download_seasonal_build.py and fludb_download_seasonal_build_genomes.py
scripts/assign_clades.py	rule assign_clades	Copies Nextclade HA clade/subclade onto each segment's metadata
scripts/merge_quality_metrics.py	rule merge_quality_metrics	Adds Nextclade coverage and QC status to metadata
scripts/report.py	rule surveillance_report; marimo run	Surveillance report (HTML, PowerPoint, tables)
scripts/run_report.py	run end handlers	Per-run timing/memory report in .run/reports/
scripts/record_provenance.py	run start/end handlers; rule fetch_hana_datasets	Writes input-file checksums and record counts (start) and Nextclade dataset tags (end) to the provenance file
scripts/snapshot_clean.sh	rules snapshot_clean*	Archive and clean the workspace
scripts/data_guard.sh	git hooks; run start	GISAID/JHH data protection (maintainer only)
scripts/build_env.sh	you	Build/check the environment, enable hooks
scripts/generate_tutorial_dataset.py	maintainer	Rebuild the tutorial dataset from fludb
scripts/sync_subclade_colors.py	rule sync_subclade_colors; maintainer, by hand	Copy the official Nextstrain subclade colors into config/{subtype}/colors.tsv; --check reports drift (Subclade colors)
scripts/nextstrain_upload.py	a person, by hand only	Upload all builds to the private group, or to the public group with --public; checks files, logs in, asks for typed confirmation (Step 5)
scripts/shannon.py	you	Per-position Shannon entropy of an alignment
scripts/join-genbank.py, join-segments.py	you	Concatenate GenBank/FASTA segments (from nextstrain/avian-flu, unmodified)
scripts/pyflute.py, scripts/flusort/flusort_split.py	you	Legacy segment sorting/filtering helpers
fludb/scripts/upload_gisaid.py	nothing	Bulk GISAID upload; not part of the pipeline
scripts/depreciated/*	nothing	Retired (incl. jhh_unique_strains.py); kept for reference. The folder is git-ignored: scripts retired there in October 2026 (the three old upload scripts, download_datasets.py) stay on the maintainer's computer only

fludb's query options (filters, headers, segments) are documented in fludb/README.md; flusort's in scripts/flusort/README.md; the custom Nextclade datasets in nextclade/README.md.

Troubleshooting / FAQ

• Snakemake wants to rerun everything after a small edit. You changed an input, a rule's code or params used everywhere (e.g. Snakefile). The dry run (snakemake -n) prints the reason for each job.
• AmbiguousRuleException after adding a rule: two rules can make the same file; tighten the output path or wildcard constraints.
• My script: can't find snakemake. That object exists only when run by Snakemake; test with a small rule, not python script.py.

Next step → Glossary