Step 1 Source Data Preparation

Step 1: Source Data Preparation

What is this?

Building the source/ folder: the three files the pipeline reads.

Why does it matter?

Everything downstream trusts these files. A wrong date or a mislabelled control here ends up in every tree and in the report.

What do I do?

Get access, then build the two JHH files with one script and add the GISAID vaccine file. The hands-on walk-throughs are Tutorial 2 (new sequencing run) and Tutorial 3 (vaccine strains). This page is the reference: formats, rules and what every message means.

Caution

Private data. Run FASTAs, run_dates.csv, everything in source_prep/output/ and everything in source/ are unpublished JHH patient-sample data or licensed GISAID data. They stay on your computer and on the lab SharePoint. Never upload, email or share them, and never use git add -f. Rules: Data Protection.

flowchart LR
    A["Mostafa Lab shared folder<br/>one folder per run"] -->|copy consensus FASTA| B["Lab SharePoint (master)<br/>runs/ + run_dates.csv"]
    B -->|copy down| C["source_prep/<br/>runs/ + run_dates.csv"]
    C -->|python scripts/prepare_source.py<br/>checks + cleans| D{"RESULT"}
    D -->|STOPPED| E["Fix what it lists"] --> C
    D -->|READY| F["source_prep/output/<br/>JHH_sequences.fasta<br/>JHH_metadata.txt"]
    F -->|copy| S["source/"]
    G["GISAID workset"] -->|download + reformat headers| V["vaccine.fasta"] --> S
    S --> P["Ready for Step 2"]

Loading

Raw runs travel from the sequencing team's folder to the lab's SharePoint, then to your computer, where one script checks and combines them.

Getting access

• Lab data (FluCrew): email Dr. Andy Pekosz (apekosz1@jh.edu) to be added to the FluCrew Team. Its SharePoint folder 00_nextstrain_data holds the lab master copy of runs/ and run_dates.csv (the link works only once you are a member).
• Raw sequencing output (one folder per run, e.g. IV26Run8): Mostafa Lab shared folder Influenza-Surveillance (ask Dr. Mostafa for access).
• Contact Dr. Heba Mostafa and Dr. Andy Pekosz for access to the JHH sequencing data.
• GISAID requires your own individual account; credentials are never shared. See Tutorial 3.

Permission notes. The lab SharePoint folder must be owned by the PI or the lab, not a student's personal OneDrive (deleted when they leave). Only people with their own GISAID account may receive vaccine.fasta or anything made from it (Data Protection).

The three input files

Create the working folders once: mkdir -p source data results

File in source/	What it is	How to get it
JHH_sequences.fasta	Every JHH segment sequence, all sequencing runs	Built by scripts/prepare_source.py (Tutorial 2). Never edit by hand
JHH_metadata.txt	One row per JHH specimen	Built together with the FASTA. Never edit by hand or open and save in Excel
vaccine.fasta	Vaccine strains from GISAID	Downloaded and reformatted by hand (Tutorial 3)

nextstrain/
├── source/
│   ├── JHH_metadata.txt
│   ├── JHH_sequences.fasta
│   └── vaccine.fasta
├── data/       # empty until the pipeline runs
└── results/    # empty until the pipeline runs

The pipeline also writes flusort_JHH_sequences.fasta and flusort_JHH_metadata.tsv into source/ during ingest (removed by snapshot_clean).

The source_prep/ folder

nextstrain/
├── source_prep/                 ← the only place you work during preparation
│   ├── README.md                ← pointer to this page
│   ├── .gitignore               ← keeps the data below out of git (do not edit)
│   ├── run_dates.csv            ← the ONLY file anyone types into (copied from SharePoint)
│   ├── runs/                    ← one FASTA per sequencing run (copied from SharePoint)
│   │   ├── ConsensusIV26Run6.fasta
│   │   └── ConsensusIV26Run7.fasta
│   └── output/                  ← written by the script; you copy 2 files to source/
│       ├── JHH_sequences.fasta
│       ├── JHH_metadata.txt
│       └── prep_report.txt      ← everything the script checked and found
├── scripts/prepare_source.py    ← the script (one command, no options needed)
└── source/                      ← the pipeline reads its inputs from here

The lab SharePoint folder is the master copy of runs/ and run_dates.csv. Your source_prep/ is a working copy: change things on SharePoint first, then copy them down. If you change your local run_dates.csv, make the same change on SharePoint, or the next person will not have it.

Format rules

Run names

IV26Run8 = IV + 2-digit year (26 = 2026) + Run + run number in that year. A re-run may have one extra letter: IV25Run11b.

Run FASTA files (source_prep/runs/)

Rule	Details
File name	Use the Mostafa Lab convention ConsensusIV26Run8.fasta. Other names are accepted if they contain the run name (how the run name is read). Extensions .fasta, .fa, .fas, .fna (any case).
One per run	Two files that resolve to the same run name stop the script. Copy the file itself, not the whole run folder.
Location	Directly in runs/, not inside a subfolder.
Content	Plain-text FASTA: each record is a header line starting with >, then the sequence on one or more lines. Never open and save it in Word, Excel or TextEdit.
Header	> + specimen ID + _ + segment number, e.g. >JH23840_4 (segment 4 of specimen JH23840). Anything after a space or | is removed.
Specimen ID	Patient specimens: JH + 2-digit year + number (JH25583, JH241041). Letters, digits, . and - only. No underscores inside the ID: later steps split the header at _.
Segment number	1 to 8 (table below).
Sequence	A C G T and ambiguity codes R Y S W K M B D H V N. Lower case is converted; gaps - are removed.
Controls	Named after the run, e.g. IV25Run6NTC_3. Any non-JH ID containing NTC, NEG, BLANK, WATER, H2O, CTRL, CONTROL, PTC or POS is treated as a control.

How the run name is read

Naming convention. The Consensus + run name + .fasta naming scheme (ConsensusIV#Run#.fasta, e.g. ConsensusIV26Run8.fasta) was defined by the Mostafa Lab and is the name to use. The script does not require it: any file name that contains the run name is accepted, with a [NOTE] saying how it was read.

How the script finds the run in a file name:

1. It looks for IV + 2-digit year + Run + number (1-3 digits) anywhere in the name, in upper or lower case. Nothing may separate the parts (IV26_Run8 is not read).
2. One letter straight after the number marks a re-run (IV25Run11b), but only if no other letter follows it, so IV26Run7final is run IV26Run7.
3. The name is standardized: leading zeros are removed and the letter is made lower case (iv26run008B → IV26Run8b). This standard name is looked up in run_dates.csv and written to the sequencing_run column.
4. If a name contains two run names, the first one is used.
5. The run comes only from the file name. The headers inside are not checked against it, so run 9's sequences saved as ConsensusIV26Run8.fasta are labelled run 8. The only hints are the specimen-year WARNING and the "changes run" NOTE in the comparison with source/.

File name	Read as
ConsensusIV26Run8.fasta (convention)	IV26Run8
IV26Run8.fa, iv26run8.FASTA, JHH_IV26Run8_consensus.fna, IV26Run008.fasta	IV26Run8 (NOTE)
ConsensusIV25Run11B.fasta	IV25Run11b
IV26Run7final.fasta, IV26Run7_final.fasta	IV26Run7
IV26Run8 (1).fasta next to ConsensusIV26Run8.fasta	Both IV26Run8: STOPPED, two files for one run
IV26Run8v2.fasta	IV26Run8v (a letter followed by a digit counts as the re-run letter); STOPPED, no date. Use IV26Run8_v2
IV26_Run8, IV-26-Run8, IV26 Run 8, IV2026Run8, Run8_IV26	STOPPED: no run name found
ConsensusIV26Run8.fasta.txt, ConsensusIV26Run8.fasta.gz	Not read (only a WARNING): the whole run is left out. Rename or unzip to .fasta

Segment numbering. Numbers follow the GenBank references used for genome assembly upstream, so segments 1 and 2 swap names between influenza A and B. flusort detects the type later, so this does not matter during preparation.

Segment number	Influenza A (H1N1, H3N2)	Influenza B (B/Victoria)
1	PB2	PB1
2	PB1	PB2
3	PA	PA
4	HA	HA
5	NP	NP
6	NA	NA
7	MP	MP
8	NS	NS

So JH23840_4 is HA for both types, while JH23840_1 is PB2 for influenza A but PB1 for influenza B.

run_dates.csv

One row per sequencing run:

sequencing_run,date,notes
IV25Run11b,2025-03-07,
IV26Run8,2026-10-06,3 samples re-extracted

Rule	Details
First line	Exactly sequencing_run,date,notes
sequencing_run	Run name as in the FASTA file name, e.g. IV26Run8
date	The date the sequencing team gives for that run, YYYY-MM-DD. Excel's 10/6/2026 (month/day/year) is also read and converted. Not in the future, not before 2015. One date per run.
notes	Optional free text, not used by the pipeline
Same run twice	Different dates: STOPPED. Same date: WARNING; delete one row
Order	Any; new runs usually go at the bottom
Saving from Excel	File > Save As > CSV UTF-8 (Comma delimited) (.csv); choose Keep Current Format if asked. Never save as .xlsx.
Runs without a file yet	Allowed (listed as a NOTE). A row with an empty date is ignored until that run's FASTA is in runs/.

Write it as above. The script is more forgiving, so a slightly different file still works: column names run/run_name and run_date, any capitalization, tab or semicolon separators, Windows Excel CSV encoding ([FIXED]), and most date formats (10/6/2026, 6-Oct-2026, Oct 6, 2026, Excel day numbers such as 46301, a time after the date). Every date is written out as YYYY-MM-DD.

What the script writes (source_prep/output/)

JHH_sequences.fasta: one record per segment, header >ID_segment, 60 bases per line, Unix line endings, ordered by run, specimen, segment. Controls are not included. Each specimen's segments all come from one run.

JHH_metadata.txt: tab-separated, one row per specimen:

Column	Value	Used later by
sequence_ID	Specimen ID from the FASTA headers, e.g. JH25583	flusort and fludb, to join metadata to sequences
sample_ID	Same as sequence_ID	fludb download: becomes the strain name in every build, so it must be unique
sequencing_run	Run the sequences came from, e.g. IV26Run8	fludb; reports
date	Run date from run_dates.csv, YYYY-MM-DD (fludb also accepts XX for an unknown month or day)	Nextstrain dates and frequencies
passage_history	vtm (viral transport medium) for every specimen	fludb

Every specimen in the metadata has at least one sequence, and every sequence has exactly one metadata row.

prep_report.txt: everything printed on screen with full lists, plus the date, who ran it and the MD5 checksum of every run file used. Keep it with the run if anyone asks what went into a build. It contains specimen IDs but no sequences; share it inside JHU only.

vaccine.fasta headers

The GISAID download must be reformatted so every header has these 7 fields, in this order, separated by | (spaces around | are fine):

>Isolate name-Passage details/history | Isolate ID | Collection date | Passage details/history | Segment number | Type | Lineage

>A/Wisconsin/67/2022-E2/E3 | EPI_ISL_XXXXXXXX | 2022-03-15 | E2/E3 | 4 | A | H1N1

• Passage history appears twice: appended to the isolate name in field 1 (A/Wisconsin/67/2022-E2/E3) and on its own in field 4 (E2/E3). The vaccine ingest script parses vaccine records differently from clinical ones and needs both.
• Type is e.g. A or B; lineage e.g. H1N1, H3N2 or Victoria.
• Wrong or missing fields mean a strain fails to load into fludb, gets wrong metadata, or silently drops out of the builds.
• Parsed by fludb/scripts/upload_vaccine.py, called from workflow/snakemake_rules/ingest.smk.

What the script decides for you

• Controls (NTC, water...) are left out of both files. If a control produced a segment of 200 bases or more, it warns POSSIBLE CONTAMINATION. Controls given a JH number look like specimens; they must be listed in config/exclude.tsv. The script notes which exclude.tsv IDs it saw.
• A specimen sequenced in more than one run keeps one run: the one with the most segments; if equal, the most recent. Listed as a NOTE.
• The same segment twice in one run file: identical copies are merged; different versions keep the longer one (fewest N) with a WARNING.
• Short or low-coverage segments are kept. The pipeline filters them later (Step 3). The quality summary only reports them: a table per run (specimens, complete 8/8, partial), segments shorter than the pipeline's min_lengths, segments more than 10% N, and the total of ambiguous bases. It is skipped until every run has a date.
• Run dates out of order within a year are reported as a NOTE.
• Nothing is ever changed in runs/, run_dates.csv or source/. It compares the new files with source/ and lists new, removed and moved specimens.
• Output files are written completely or not at all. When it stops, it also deletes old outputs so nobody copies an out-of-date file.

Reading the result

The last lines say RESULT: READY or RESULT: STOPPED. Every line starts with a label:

Label	Meaning	What you do
[OK]	All good	Nothing
[NOTE]	Information, e.g. new or re-sequenced specimens	Skim it
[FIXED]	Harmless formatting fixed in the output (Windows line endings, unusual text encoding, lower-case bases, gaps and spaces in sequences, text after the header name, jh → JH, exact duplicate records, dates not in YYYY-MM-DD); originals untouched	Nothing
[WARNING]	Handled, but a person should know	Read it (below)
[PROBLEM]	Must be fixed; nothing is written until it is	Follow the HOW TO FIX line

Warnings

Warning starts with	What to do
POSSIBLE CONTAMINATION: ... negative control produced sequence	Tell the sequencing team and Dr. Pekosz. The control is left out, but the run's specimens may need checking before you trust the builds. If a line ends its name says run X, the control is named after a different run than the file it is in: check that the right file is in runs/.
... sequences skipped because the name is not <ID>_<segment number> / segment number is not 1 to 8 / empty / characters that are not DNA bases	Those sequences are left out, the rest is used. Tell the sequencing team if there are many.
... appear(s) twice in this file with different sequences	The longer one is kept. Mention it to the sequencing team.
... specimen IDs do not match the run year	Check that the file in runs/ is the right run.
... specimens in source/ now will NOT be in the new files	Expected the first time (old rows without sequences, controls). Otherwise check every run file is in runs/.
run_dates.csv line N: ... is a 2026 run but its date is ...	Check the date on SharePoint and fix it.
... is listed twice in run_dates.csv (... same date)	Delete one of the rows (on SharePoint too).
... ID(s) does not look like a JH specimen ID or a control	Kept in the output. Check it against the run sheet: a control goes in config/exclude.tsv; a misnamed specimen is renamed by the sequencing team.
source/JHH_metadata.txt has no 'sequence_ID' column	Comparison skipped. Fine the first time; the new files replace it.
Could not remove the old ... Do NOT copy it to source/	Do not copy it. Close any program that has it open, delete it by hand, run again.
Ignoring runs/...	A non-FASTA file or empty folder is in runs/. Move it out. If it is a run saved as .fasta.txt or .fasta.gz, that run is left out: rename or unzip it to .fasta.

Problems (RESULT: STOPPED)

The script says	How to fix it
Run IV26Run8 has a FASTA file but no date in run_dates.csv	Add IV26Run8,YYYY-MM-DD, to run_dates.csv (on SharePoint too).
run_dates.csv line N: the date for ... cannot be read	Write the date as YYYY-MM-DD.
... is in the future / ... is before 2015	Typing mistake; correct it.
... is listed twice in run_dates.csv with different dates	Delete the wrong row.
run_dates.csv line N: '...' is not a run name	Write it like IV26Run8, exactly as in the FASTA name.
The first line of run_dates.csv must name the columns	Make it exactly sequencing_run,date,notes.
Found run_dates.xlsx but not run_dates.csv	Excel: File > Save As, CSV UTF-8 (Comma delimited), name run_dates.csv.
source_prep/run_dates.csv does not exist / run_dates.csv is empty	Copy it from SharePoint into source_prep/.
Cannot find a run name in the file name runs/...	Rename to ConsensusIV##Run#.fasta (how the run name is read). An old combined JHH_sequences.fasta does not belong in runs/.
2 files are for the same run IV26Run8	Keep one file per run. Ask the sequencing team if unsure which.
The folder runs/IV26Run8/ contains FASTA files	You copied the whole run folder. Move the FASTA itself into runs/, remove the folder.
There are no run FASTA files in source_prep/runs/ / The folder source_prep/runs/ did not exist, so it was created (empty)	Copy the run files from SharePoint into source_prep/runs/.
...: the file is empty	Copy it again after it has fully downloaded (no cloud icon).
...: this is not a text FASTA file / does not start with a FASTA header	Another program changed it. Copy the original again.
...: no usable sequences in this file / ... does not look like a JHH consensus FASTA	Wrong file. Copy the consensus FASTA of that run.
The output files did not read back correctly	Check disk space and that output/ is not open elsewhere; run again.
RESULT: UNEXPECTED ERROR	A bug, not your fault. Send everything printed to the maintainer.

Before a production run: checklist

• python scripts/prepare_source.py ended with RESULT: READY, and both files from source_prep/output/ were copied into source/.
• Any POSSIBLE CONTAMINATION warning was reported to the sequencing team and Dr. Pekosz.
• A control given a JH number is listed in config/exclude.tsv.
• vaccine.fasta headers have 7 fields: grep '^>' source/vaccine.fasta | awk -F'|' 'NF != 7' prints nothing.
• The previous build was archived (snakemake --cores 8 snapshot_clean, Step 2).
• Provenance recorded: date received, and for vaccine.fasta the download date, workset and EPI_SET (Best Practices).

For the maintainer: one-time setup and changes

Setting up from scratch, changing settings, how it was validated

One-time setup

1. Create a lab-owned SharePoint folder with a runs/ subfolder, give lab members edit access, and add its link under Getting access.

2. Collect the original consensus FASTA of every past run from the Mostafa Lab folders into SharePoint runs/, named ConsensusIV##Run#.fasta.

3. Create run_dates.csv from the current metadata file (one row per run), check it, then upload it to SharePoint. If the old file has several dates for one run, the earliest is used and all of them are listed in notes. It refuses to overwrite an existing run_dates.csv:

   python scripts/prepare_source.py --bootstrap-dates source/JHH_metadata.txt

4. Run python scripts/prepare_source.py and read the comparison with source/. Expected differences from the old hand-made files: metadata rows that never had sequences disappear; sequenced specimens without a metadata row appear; controls disappear; re-sequenced specimens appear once.

Options (normally not needed): --folder PATH uses another prep folder instead of source_prep/; --bootstrap-dates FILE is the one-time setup above. Exit status: 0 READY, 1 STOPPED, 2 unexpected error. The script uses only the Python standard library (3.8+), so python3 works even outside the conda environment.

Changing settings. Everything adjustable is in the SETTINGS block at the top of scripts/prepare_source.py: folder names, passage_history, the run-name pattern (RUN_NAME), the control-name pattern, the contamination threshold (200 bases), the minimum lengths (keep them in step with min_lengths in the Snakefile) and accepted date formats. Changing what goes into the output files (columns, IDs, which records are kept) changes what the pipeline builds: a Tier 3 change (Best Practices).

Validation (October 2026). The 37 runs IV23Run6 to IV26Run7 were rebuilt from the existing source/JHH_sequences.fasta. All 17,096 output sequences were identical to the input records, and every metadata row matched the old file except the water control JH23609 (sample_ID "water" became its ID; it stays excluded through config/exclude.tsv). Findings at the time: two NTCs (IV26Run5bNTC, IV26Run6NTC) with full B/Victoria genomes, which had been included in the B/Victoria builds; JH25667 listed twice in the old metadata; 28 sequenced specimens without a metadata row; 332 metadata rows without sequences.

Replaces scripts/jhh_unique_strains.py (now in scripts/depreciated/) and the manual Excel copy-paste and cat steps.

Troubleshooting / FAQ

• SharePoint files show a cloud icon. Right-click, Always keep on this device, then copy; otherwise you copy an empty placeholder.
• Which FASTA in the run folder is the consensus? Ask the sequencing team. Do not guess.
• Still stuck? Send source_prep/output/prep_report.txt to the maintainer (inside JHU only).
• New kind of data (another hospital, GenBank, a full GISAID download)? That is a Tier 3 change: it needs its own upload script in fludb/scripts/, a database_origin value, provenance and maintainer review. Never paste it into JHH_sequences.fasta.

Next step → Step 2: Running the Pipeline