Pipeline Overview

Pipeline Overview

What is this?

A map of every stage, from raw sequencing files to a published tree.

Why does it matter?

When something looks wrong in a tree, knowing which stage produced it tells you where to look.

What do I do?

Read the diagram, then follow the links in order.

flowchart TD
    subgraph S1["Step 1: Source data preparation (by a person)"]
        R["Run FASTAs + run_dates.csv<br/>source_prep/"] -->|prepare_source.py| J["JHH_sequences.fasta<br/>JHH_metadata.txt"]
        G["GISAID vaccine strains"] -->|reformat headers| V["vaccine.fasta"]
        J --> SRC["source/"]
        V --> SRC
    end
    subgraph S2["Step 2: Snakemake run (one command)"]
        SRC --> I["Ingest: flusort typing,<br/>fludb database, per-build inputs<br/>data/"]
        I --> NC["Nextclade: clades + QC"]
        NC --> F["Step 3: Filtering & QC<br/>augur filter"]
        F --> T["Augur: align, tree,<br/>refine (+ clock filter),<br/>traits, ancestral, translate"]
        T --> X["Export<br/>auspice/*.json"]
    end
    X --> REP["Step 4: Reports & outputs<br/>reports/, .run/"]
    X --> PUB["Step 5: Review in Auspice,<br/>upload by hand"]

Loading

Steps 1 and 5 are done by people; Steps 2 to 4 happen automatically inside one Snakemake run. The same chain runs for each of the 27 builds.

The stages in one sentence each

Stage	What happens	Details
Source data preparation	Per-run sequencing files and run dates are combined into two clean JHH files; GISAID vaccine strains are downloaded and reformatted; all three go into source/.	Step 1
Ingest	flusort types each segment (A/B, H and N subtype) by BLAST, everything is loaded into a small database (fludb), and one input set per build is written to data/.	Step 2, Snakemake reference
Clades & QC	Nextclade assigns HA clades/subclades and scores each sequence's quality.	Step 3
Filtering	augur filter drops low-coverage, low-quality, too-short and excluded sequences; augur refine drops molecular-clock outliers.	Step 3
Tree building	Augur aligns, builds and dates the tree, adds traits, ancestral mutations and amino-acid changes.	Snakemake reference
Export	One Auspice JSON (+ tip frequencies) per build in auspice/.	Step 4
Reports	A surveillance report (HTML + PowerPoint) and a run report (timing, provenance).	Step 4
Review & publish	A person inspects every tree and uploads by hand: private group first, public only with approval.	Step 5

The 27 builds

	PB2	PB1	PA	HA	NP	NA	MP	NS	Whole genome
H1N1 (h1n1)	✓	✓	✓	✓	✓	✓	✓	✓	✓
H3N2 (h3n2)	✓	✓	✓	✓	✓	✓	✓	✓	✓
B/Victoria (vic)	✓	✓	✓	✓	✓	✓	✓	✓	✓

Each build lands at auspice/{subtype}/{segment}.json, for example auspice/h3n2/ha.json; whole genomes at auspice/{subtype}/genome.json.

Where files live while it runs

Folder	Holds	In git?
source/	The 3 input files	Never
data/, fludb.db	Per-build inputs and the database	Never
results/	Nextclade output, filter logs, alignments, trees	Never
auspice/	Final trees	Never (uploaded by hand)
reports/, .run/	Reports, logs, provenance	Never
snapshots/	Dated archives of whole runs	Never

The full folder map, with the reason each is ignored, is on Data Protection.

Troubleshooting / FAQ

• A sample is missing from a tree. Walk the filtering stages in order: Step 3.
• Why Nextclade instead of augur clades? See Step 3.
• Why is the genome build smaller than the segment builds? Genome builds need all 8 segments; segment builds keep incomplete genomes.

Next step → Step 1: Source Data Preparation