Skip to content

Preprocessing.md

Jump to bottom
Wenyu (Eddy) Huang edited this page Aug 5, 2026 · 1 revision

Preprocessing

Profiler reports to the CLR feature matrix that discover reads. Two subcommands: build-cohort assembles the tables and decides which samples are usable, preprocess turns the surviving abundances into features.

build-cohort: reports to cohort tables

mbcausal build-cohort -i $REPORTS/ihmp -o $RESULTS \
  --cohort ihmp --depth-from bracken --min-depth 150000000

Merges each tool's reports and joins the cohort's metadata, writing one directory per tool:

<preprocess_dir>/<cohort>-<tool>/
  abundance.csv          samples x species, relative abundance
  abundance.meta.json    profiler, database, parameters, tool version
  sample_table.csv       metadata, aligned to abundance, plus qc_pass
  qc.csv                 the depth measurements qc_pass was computed from

--cohort names the loader that supplies metadata. --tools is a comma list of tool labels; leave it off and the labels are discovered from the <label>-reports/ directories under -i. A label with no registered profiler is skipped with a warning.

qc_pass is computed once for the whole cohort

sample_table.csv carries a boolean qc_pass column, and it is the single place a sample is judged. Two rules compose into it:

  • Technical replicates. An iHMP <sample_id>_TR row is dropped when its original is also present.
  • Depth. A sample below --min-depth sequenced bases fails.

The depth signal comes from one source for the entire cohort, either a tool's qc.csv via --depth-from, or the metadata bases column when that flag is omitted. This matters: the same keep list is applied to every profiler, so a comparison across profilers runs on identical samples and a difference between them cannot be an artefact of different QC.

--depth-from will reuse an earlier run's qc.csv if the tool was built before, so you do not have to rebuild bracken to add a profiler.

preprocess: cohort tables to a CLR feature matrix

mbcausal preprocess -i $RESULTS/ihmp-bracken \
  -o $RESULTS/ihmp-bracken/clr --outcome fecalcal

Two steps, in src/mbcausal/preprocess.py:

  1. Prevalence filter. Prevalence is the fraction of samples where a taxon is present, meaning abundance above zero. Taxa below min_prevalence are dropped. The default is 0.0, so nothing is filtered, and the CLI exposes no flag to change it. The causal branch runs unfiltered on purpose; the filter exists for callers that want it.
  2. CLR. Each sample is centred by the geometric mean of its own abundances, computed as log(x + 1e-6) minus the row mean of those logs. The output is feature_matrix.csv.

The kept taxa and their column order are learned on fit and frozen, so any later split transforms into the identical layout. Taxa missing from a split are filled with zero and taxa unseen at fit time are dropped.

Samples failing qc_pass are dropped by default. --no-qc keeps them.

Always give preprocess its own subdirectory

preprocess writes sample_table.csv as well as feature_matrix.csv, holding the QC filtered and aligned rows. Point -o at the cohort directory with --overwrite and it replaces the cohort's complete record with the filtered one, and the pre QC sample list is gone. Write to a subdirectory such as clr-<outcome> instead. The CLI warns when -o equals -i, but the warning is easy to miss in a log.

Reading the tables downstream

Model code goes through one entry point, which applies qc_pass by default so nothing can train on samples that failed:

from mbcausal.data import load_cohort

cohort = load_cohort("preprocess/ihmp-bracken")     # QC clean
full   = load_cohort("preprocess/ihmp-bracken", qc=False)   # the complete record

What to check before moving on

The preprocess log reports the numbers worth reading:

  • how many samples qc_pass kept, against the total
  • the feature matrix shape, and how many species the prevalence filter dropped
  • coverage of --outcome, which is usually far sparser than the samples
  • how many units have more than one observed outcome, since those are the only ones a lag can use

On iHMP the outcome is the binding constraint rather than the samples. Calprotectin is measured at a fraction of visits, so the panel that reaches discover is much smaller than the cohort.

See also

Clone this wiki locally