Best Practices

Best Practices

What is this?

The lab's standards for changing code, handling data, committing and sharing, as checklists.

Why does it matter?

The builds must stay correct and the lab must stay within its data agreements after the people who built them have moved on.

What do I do?

Read the golden rules once. Before any change, find its tier and follow that checklist.

Golden rules

1. Raw data lives only in source/ (plus the git-ignored source_prep/runs/, source_prep/output/ and source_prep/run_dates.csv). Never copy GISAID or JHH sequences or metadata anywhere else in the repository, and never to the top-level folder.
2. GISAID data is never published, committed, emailed or posted: not on GitHub, a public report, Slack, an issue, a public slide, or an AI chatbot.
3. Never bypass the safety checks (--no-verify, git add -f). If a check blocks you, it is protecting the lab (Data Protection).
4. Nothing goes to Nextstrain automatically. Review, upload privately, publish only with approval (Step 5).
5. One change at a time, tested on the tutorial, approved by the maintainer when it touches filters, references, clock rates, ingest or the environment.
6. Record where external data came from and when.
7. Write commit messages for the person after you: what, why, and the effect on the builds.
8. No Jupyter notebooks. Explore in marimo or Quarto; reusable code goes in scripts/.

Roles

Role	Who	May do	Needs approval for
PI	Dr. Andy Pekosz	Receives reports; decides on data sharing and public uploads	-
Maintainer	Elgin Akin (until handover; update at handover)	Everything below; approves Tier 2-3; uploads; owns the environment and data guard	Public uploads, data sharing (PI)
PhD students	Flu team	Run builds; Tier 0-1 changes; propose Tier 2-3	Tier 2-3 (maintainer)
New users	Rotation students, technicians, new members	Run the tutorial and production builds; read outputs	Any code or config change (pair with a PhD student or the maintainer)

Handover checklist (maintainer leaving): name the successor above and on Home; transfer Nextstrain group admin and GitHub admin rights; walk the successor through a production run, an upload and Tutorial 3; confirm they can explain Data Protection without reading it.

Change tiers

flowchart TD
    A["I want to change something"] --> B{"Only words?<br/>docs, captions"}
    B -->|yes| T0["Tier 0: just do it"]
    B -->|no| C{"Changes what is in the builds<br/>or how it is computed?"}
    C -->|no| T1["Tier 1: test + PR"]
    C -->|yes| D{"Ingest, fludb, input formats,<br/>environment, rule all,<br/>.gitignore, hooks?"}
    D -->|no| T2["Tier 2: maintainer approves first,<br/>tutorial before/after, PR review"]
    D -->|yes| T3["Tier 3: as Tier 2 +<br/>maintainer reviews the code"]

Loading

Find your tier before you write any code; Tier 2-3 need approval before you start, not after.

Tier	Meaning	Examples
0	Words only	README or wiki typo, report caption or note in config/report.yaml
1	Additive, outside the build path	New marimo notebook, new report plot, new standalone script
2	Changes what is in the builds or how	Filter thresholds, config/exclude.tsv, clock rates, references, auspice configs, colors, Nextclade datasets, new rules
3	Changes the foundation	Ingest/fludb scripts and schema, prepare_source.py outputs, input formats, environment versions, rule all, .gitignore, hooks

Checklist: any change

• Tier decided; Tier 2-3 approved by the maintainer before starting.
• Branch: git switch main && git pull && git switch -c yourname/short-description.
• Read the rule(s) and every script involved, end to end.
• Smallest change that does the job; one logical change per branch.
• snakemake -n --cores 1 succeeds, with and without --configfile config/tutorial.yaml.
• Tier 1-3: full Tutorial 1 run on a clean workspace.
• Tier 2-3: before/after comparison (filter logs, tip counts, report tables) written in the PR.
• Environment changed: bash scripts/build_env.sh --check.
• bash scripts/data_guard.sh reports OK.
• Wiki pages and diagrams updated for what changed (Coding with Agents has the page map).
• Descriptive commit, PR listing tier, tests and output changes. Tier 2-3 merged by the maintainer only.

Checklist: adding code

• A new metric, table or transformation = a script in scripts/ + a rule in the matching .smk (or a new .smk for a new feature) + output under results/.
• A new report figure = an item_* function in scripts/report.py, registered in report_layout().
• A new subtype, segment or build target is Tier 2: wildcard constraints, rule all, config/{subtype}/, Nextclade datasets, report and upload scripts. Plan it with the maintainer.
• Conventions followed: Snakemake & Scripts Reference.

Checklist: modifying code

• Not changing what a data file records (columns, IDs, header formats, dates) without asking; every rule and script that reads it listed.
• No renamed rules, outputs or wildcards (other rules, the report and habits depend on them).
• No reformatting or "tidying" of files you are not otherwise changing.
• Attribution in scripts/report.py kept ("Developed by Elgin Akin, 2026"; verify_attribution() intact).
• Docs, diagrams and rulegraph.png updated in the same PR when behaviour changes.

Code style

Minimal, readable, commented, concise. Prefer the obvious way over the clever one.

• Python: standard library, pandas, Biopython; 4-space indents; functions under ~40 lines; constants in UPPER_CASE at the top; comment the why.
• Shell: set -euo pipefail, quoted variables.
• Snakemake: one directive per line, aligned as in existing files; little Python inside .smk files.
• No dead code, no unused options, no copy-pasted variants (_v2, _new), no new dependencies without a Tier 3 environment change.

Notebooks and exploratory analysis

• marimo or Quarto only. No new Jupyter notebooks (the pre-commit hook blocks them; existing ones are legacy). Convert one when you next touch it: marimo convert notebooks/old.ipynb > notebooks/old.py.
• marimo is in the environment: marimo edit notebooks/my_analysis.py. Notebooks are plain Python with no stored outputs, so they diff well and cannot leak data through saved outputs.
• Quarto (.qmd) is not in the environment; install it separately. Never commit rendered HTML that embeds data, or _freeze/.
• Notebooks explore; scripts compute. A function used twice, or by the pipeline, moves to scripts/.
• Read outputs, never raw inputs: load from results/ and auspice/; never write into source/, data/ or results/ from a notebook.
• No sequences or unpublished identifiers in notebook text.
• Name for the question and date: notebooks/2026-10_ha_na_tanglegram.py.

Data provenance

Re-running the same database query later returns different data, so a date is what keeps an analysis interpretable.

• Downloaded data records its source and date: the URL or query in a README beside it, the download date in the filename, folder name or a small tracked date file. Applies to database downloads, files from other repositories, and external releases.
• Prefer a rule over a manual download: the URL lives in a config/ file and the rule writes a tracked date file.
• Track the date even when the data is ignored (too large or restricted to track).
• Lab-produced files need no provenance note (plate layouts, bench records, sample manifests).
• Ask before changing what a data file records (columns, ID, header or date formats).

Source	How it arrives	Provenance today	Record by hand
JHH run FASTAs + run dates	Mostafa Lab folder → lab SharePoint → source_prep/	run_dates.csv (run dates); prep_report.txt (date, who, checksums)	Nothing extra
GISAID vaccine strains	Manual workset download → source/vaccine.fasta	Each run's provenance file: record count, modified time, SHA-256 of the file used	Download date, workset name, EPI_SET ID if one was generated, who (in source/README.md, kept with the data)
Nextclade HA/NA datasets	Rule fetch_hana_datasets, every run (git-ignored)	Tag of every dataset used, in each run's provenance file; each snapshot	Nothing
Custom B/Victoria Nextclade datasets	Built by hand from GenBank	Table in nextclade/README.md (accession, date)	Update the table when changed
References config/*/reference_*.gb, genome.gb	GenBank	Accession inside each file	Note replacements in the commit message
Vaccine selection dates config/*/vaccine.json	WHO recommendations	Dates in the file	Source in the commit message

Sharing analyses

Share the git commit hash, run ID, config used (production or tutorial, any --config overrides) and, internally, the snapshot archive. A figure without its run ID cannot be traced back. Who may receive what: Data Protection.

Git

Branches and pull requests

• main always runs. Work on yourname/short-description; merge by pull request describing the change, tier, tests and output changes.
• Recommended GitHub settings (maintainer): protect main (PR + one approving review, no force pushes); enable secret scanning and push protection.

Commit messages

Be as descriptive as possible; the message is the only record of why. Example (values illustrative):

Raise HA minimum length to 1600 nt for H3N2 builds

Short HA fragments (1400-1600 nt) from IV25Run12 placed
far from their clades and inflated the H3N2 HA tree's
clock variance.

- Snakefile: min_lengths["ha"] 1400 -> 1600
- config/report.yaml: no change (length is not reported)
- Effect on tutorial: H3N2 HA 18 -> 17 tips; production
  run 20261007T101500: H3N2 HA 912 -> 887 tips
- Approved by: E. Akin (maintainer), 2026-10-07

• Summary: imperative ("Add", "Fix", "Raise"), specific, ≤ 72 characters, no period.
• Body: what changed, why, effect on builds/outputs, anything the next person must do (rebuild env, rerun ingest).
• Files, rules, thresholds and old → new values named.
• One logical change per commit; formatting-only changes separate.
• No sequences, GISAID IDs or unpublished JHH identifiers in the message.
• Bad: update, fix stuff, snakemake build, changes.

Never committed

Anything git-ignored (folder map); data at the top level; FASTA or metadata copies anywhere; databases, archives, spreadsheets, slide decks; notebook outputs; credentials, tokens and .env files; files over 10 MB. Stage files by name (git add path/to/file, not git add .), and read git status and git diff --staged before every commit.

Undoing mistakes

Situation	Fix
Staged the wrong file	git restore --staged <file>
Committed but not pushed	git reset --soft HEAD~1, fix, recommit
Pushed a bad code change	git revert <commit> in a new commit
Pushed protected data	Data Protection, today

Troubleshooting / FAQ

• Is my change Tier 1 or 2? If it can change a single tip in any tree or a single number in the report, it is Tier 2.
• The maintainer is unavailable. Tier 0-1 can proceed; Tier 2-3 wait.
• Can I use an AI assistant? Yes, under the Coding with Agents policy.

Next step → Data Protection: GISAID & JHH