Data Protection

Data Protection: GISAID & JHH

What is this?

What data is restricted, where it may live, the automatic checks that keep it off GitHub, and what to do if something goes wrong.

Why does it matter?

The code is public; the data must never be. Breaching the GISAID agreement can cost the lab (and you) GISAID access, which the vaccine-strain step depends on. JHH data is unpublished patient-sample data.

What do I do?

Keep raw data in its allowed folders, keep the git hooks on, and read the guard's message if it ever stops you.

What is protected

Data	Comes from	Rule
vaccine.fasta and anything made from it	GISAID, downloaded by a lab member with their own account	GISAID's Database Access Agreement forbids redistribution. Only people who accepted GISAID's terms with their own account may receive it. Never public.
JHH run FASTAs, run_dates.csv, JHH_sequences.fasta, JHH_metadata.txt	Mostafa Lab, JHH network	Private until released on GISAID by the Mostafa Lab. Internal use only.
fludb.db, data/, results/, snapshots/	Built from the above	Contain both. Never shared outside the lab.
Surveillance report (reports/)	Built from the above	Confidential, named internal recipients only.

The only GISAID data the pipeline ingests today is the vaccine-strain file. fludb/scripts/upload_gisaid.py exists for bulk GISAID downloads but is not part of the pipeline; adding GISAID ingest is a Tier 3 change.

Where restricted data may exist

Only in source/, source_prep/runs/, source_prep/output/, source_prep/run_dates.csv and tutorial/vaccine.fasta, plus the git-ignored folders the pipeline writes (data/, results/, fludb.db, snapshots/). Nowhere else in the repository, and never at the top level.

Repository layout

[tracked] = in git and on GitHub. [ignored] = only on your computer (listed in .gitignore), with the reason.

nextstrain/
├── README.md                    [tracked] short intro; links to this wiki
├── BEST_PRACTICES.md            [tracked] pointer to this wiki
├── .agents/                     [tracked] AGENTS.md and CLAUDE.md: rules for AI coding agents
├── CITATION.cff                 [tracked] how to cite this software
├── Snakefile                    [tracked] pipeline entry point: settings, `rule all`, includes
├── rulegraph.png                [tracked] diagram of the pipeline's rules
├── .gitignore                   [tracked] what git must never pick up (commented)
├── .gitattributes               [tracked] keeps Unix line endings so WSL works
├── .githooks/                   [tracked] pre-commit and pre-push data checks
├── .claude/settings.json        [tracked] permission limits for Claude Code agents
│
├── workflow/
│   ├── snakemake_rules/         [tracked] one .smk file per major feature
│   └── envs/environment.yml     [tracked] the conda environment
├── profiles/default/            [tracked] default Snakemake options
├── scripts/                     [tracked] every script that computes or changes build data
│   ├── flusort/                 [tracked] BLAST typing of JHH segments + BLAST database
│   │   └── flusort_out/         [ignored] BLAST diagnostics, large and regenerated
│   └── depreciated/             [tracked] retired scripts, not used
├── fludb/                       [tracked] the SQLite database scripts and docs
├── config/                      [tracked] build settings
│   ├── h1n1/ h3n2/ vic/         references (.gb), annotations (.gff), auspice_config.json, vaccine.json, colors
│   ├── exclude.tsv              sequence IDs removed from all builds by hand
│   ├── description.md           text shown on every Nextstrain page (acknowledgements)
│   ├── report.yaml              surveillance report settings
│   ├── tutorial.yaml            switches inputs to tutorial/
│   └── snapshot_clean.yaml      archive + clean after a full build
├── nextclade/flu/               [tracked] Nextclade datasets: custom B/Vic sets, H1N1/H3N2 internal segments
│                                [ignored] {h1n1,h3n2,vic}/{ha,na}/: re-downloaded every run
├── notebooks/                   [tracked] exploratory marimo/Quarto work (legacy Jupyter files kept)
├── tutorial/                    [tracked] public teaching dataset (samples released on GISAID)
│   └── vaccine.fasta            [ignored] GISAID data; each user supplies their own
├── source_prep/                 [tracked] README pointer + .gitignore
│   ├── runs/, run_dates.csv     [ignored] raw JHH run FASTAs and run dates (master copy on the lab SharePoint)
│   └── output/                  [ignored] prepared JHH files + prep_report.txt
│
├── source/                      [ignored] RAW INPUTS: JHH sequences + metadata (private), vaccine.fasta (GISAID)
├── data/                        [ignored] per-build inputs made from fludb; include GISAID vaccine sequences
├── results/                     [ignored] Nextclade, filters, alignments, trees
├── fludb.db                     [ignored] database built from source/; contains all sequences
├── auspice/                     [ignored] final Nextstrain JSONs; uploaded by hand after review
├── reports/                     [ignored] surveillance report; CONFIDENTIAL
├── figures/                     [ignored] figures made from outputs; may show unpublished data
├── .run/                        [ignored] logs, benchmarks, provenance, run reports; include user/host names
├── snapshots/                   [ignored] archives of whole runs, including source/
└── .snakemake/                  [ignored] Snakemake's own bookkeeping

Also ignored: data files at the top level (/*.fasta, /*.tsv, /*.csv, /*.xls*), databases (*.db), new Jupyter notebooks (*.ipynb), notebook and Quarto caches, OS clutter (.DS_Store), personal agent settings (.claude/settings.local.json) and .env files.

The automatic checks

flowchart LR
    A["git add"] --> B["git commit"]
    B --> C{"pre-commit hook<br/>data_guard.sh --staged"}
    C -->|protected data| X["BLOCKED<br/>file paths + how to fix"]
    C -->|clean| D["git push"]
    D --> E{"pre-push hook<br/>data_guard.sh --push"}
    E -->|protected data| X
    E -->|clean| F["GitHub"]
    G["every snakemake run"] --> H["data_guard.sh --scan<br/>warns, never stops the run"]

Loading

Three layers: .gitignore hides known data folders, the pre-commit hook checks each commit, and the pre-push hook re-checks everything leaving your computer, including commits made with --no-verify.

The guard (scripts/data_guard.sh) recognises GISAID data even after it was renamed, moved, reformatted or subsampled:

• FASTA headers with GISAID accessions (EPI_ISL_..., EPI...) or GISAID type labels (A_/_H3N2);
• FASTA headers naming a strain or accession from the GISAID file(s) in source/ (catches pipeline copies like data/*/sequences.fasta);
• sequences identical to a GISAID sequence in source/ (JHH records are exempt: a JHH segment can be identical to a vaccine strain's);
• tables (.tsv, .csv, .txt) listing EPI_ISL accessions;
• exact copies of source/ files.

Commits and pushes are also blocked for anything in source/ or the source_prep/ data folders, FASTAs sharing sequence IDs with the JHH data, databases/archives/spreadsheets/slide decks, top-level data files, new Jupyter notebooks and files over 10 MB. The public Nextclade reference datasets in nextclade/flu/ are exempt (Nextstrain publishes them openly).

When it finds something it prints "STOP: GISAID TERMS OF USE VIOLATION" (or the JHH/data equivalent), the full path of every offending file on your computer, and what to do. Every pipeline run also scans at the start and warns if your hooks are off.

bash scripts/data_guard.sh             # scan your copy for GISAID data outside its allowed folders
bash scripts/data_guard.sh --history   # audit every file ever committed
bash scripts/data_guard.sh --help

The hooks are turned on by bash scripts/build_env.sh (Getting Started); check with git config --get core.hooksPath (must print .githooks).

If the guard stops you

• At commit: nothing has left your computer. git restore --staged <file>, move the file back to its allowed folder (or delete the stray copy), commit again.
• At push: the data is in a local commit but not on GitHub. Do not push, and do not "fix" it with another commit (the data stays in the earlier one). Ask the maintainer to help remove the commit (git reset --soft origin/main, unstage, recommit).
• A false alarm (e.g. a public NCBI sequence identical to a vaccine strain): raw sequence data does not belong in git anyway. Keep it out and talk to the maintainer if you think the check is wrong.
• JHH data that is already public (e.g. regenerating the tutorial dataset): only the maintainer, after confirming release, may commit with ALLOW_JHH_DATA=1 git commit .... This overrides the JHH check only; the GISAID checks cannot be switched off.
• Never use --no-verify or git add -f.

If protected data reached GitHub

1. Stop. Do not push anything else.
2. Tell the maintainer and Dr. Pekosz the same day: which files, which commit, when it was pushed.
3. The maintainer makes the repository private if possible, rewrites history to remove the files (git filter-repo), force-pushes, and asks GitHub Support to purge cached views and pull-request references. Deleting the file in a new commit is not enough.
4. Everyone re-clones afterwards.
5. Dr. Pekosz decides whether GISAID or the Mostafa Lab must be informed.

Leaks the checks cannot see

• Emailing or messaging FASTA, metadata, run_dates.csv, fludb.db, snapshots or reports.
• Sharing a OneDrive/Dropbox link to the repository folder or snapshots/.
• Pasting sequences or GISAID metadata into an AI chatbot, a public issue, a forum or Stack Overflow. Describe problems with counts and column names.
• Screenshots of trees or tables with unpublished JHH identifiers in public talks before release.
• Public Nextstrain uploads without the GISAID acknowledgement (Step 5).

Sharing

What	With whom	How
Code, config, docs	Anyone	GitHub (public), through pull requests
Raw JHH data, fludb.db, data/, results/, snapshots/	Lab members working on the builds	Lab SharePoint/OneDrive with named access; never email, Slack, USB to outsiders, or public links
GISAID data (vaccine.fasta and derivatives)	Only people with their own GISAID account	As above; better, each person downloads their own copy (Tutorial 3)
Surveillance report	Named internal recipients (PI, flu team, Mostafa Lab)	Lab OneDrive/Teams; carries a confidentiality notice; never public. Includes vaccine strains' GISAID IDs
prep_report.txt	Maintainer, inside JHU	Contains specimen IDs, no sequences
Trees	Collaborators / public	Private group link / public group after Step 5
Figures and numbers in talks/papers	Public	Released JHH data only; acknowledge GISAID with the EPI_SET; cite the Nextstrain URL and commit/run ID

Troubleshooting / FAQ

• The run-start scan warns about a file. It is on your computer outside the allowed folders. Move it back or delete it; the message gives the path.
• "data-protection git hooks are OFF". Run git config core.hooksPath .githooks.
• Can I keep a FASTA I made for an analysis? Yes, in results/ (ignored), not in a tracked folder.
• Can I show a tree in a lab meeting? Yes; internal audiences are fine.

Next step → Coding with Agents