Skip to content

Workflow Architecture

Jump to bottom
Charlie Bayne edited this page Mar 30, 2026 · 2 revisions

Workflow Architecture

The Conduit workflow follows a modular design with five main processing stages:

  1. Setup — Copy configs, pull containers
  2. Search Space Definition — Build a protein database via one of five supported methods
  3. DIA-NN Processing — Generate a spectral library and run the DIA-NN search
  4. Protein Annotation — Functional and taxonomic annotation of detected proteins
  5. Conduit Object Construction — Assemble the final .rds object via conduitR

Directory Structure

conduit-ascent/
├── Snakefile                         # Main workflow orchestration
├── VERSION                           # Current version
├── config/                           # Default configuration templates
│   ├── snakemake.yaml
│   ├── generate_diann_spectral_library.cfg
│   ├── run_diann.cfg
│   ├── peptidotyping_firstpass_diann.cfg
│   └── peptidotyping_firstpass_diann_spectral_library.cfg
├── modules/
│   ├── setup/                        # Config copying, image building
│   ├── search_space/
│   │   ├── ncbi_taxonomy/            # Search space from NCBI taxonomy IDs
│   │   ├── uniprot_proteome_ids/     # Search space from UniProt proteome IDs
│   │   ├── peptidotyping/            # First-pass species detection from MS data
│   │   ├── metaphlan/                # Search space from metagenomic profiling
│   │   ├── MAGs/                     # Search space from metagenome-assembled genomes
│   │   └── database_processing/      # Shared post-processing (used by MAGs)
│   ├── diann/                        # DIA-NN identification and quantification
│   ├── annotation/
│   │   ├── uniprot/                  # UniProt-based functional annotation
│   │   ├── MAGs/                     # Bakta + UniProt annotation for MAG proteins
│   │   ├── eggnogmapper/             # eggNOG-mapper sequence-based annotation
│   │   └── external_annotations/     # KEGG, Pfam, CAZy annotations
│   └── build_conduit/                # Builds the final Conduit RDS object
├── containers/                       # Apptainer/Docker container definitions
│   ├── bakta/
│   ├── conduitR/
│   ├── diann/
│   ├── eggnogmapper/
│   ├── metaphlan/
│   └── umgap/
├── experiments/
│   └── {experiment_name}/
│       ├── config/                   # Per-experiment config
│       ├── input/
│       │   ├── raw_files/            # Thermo .raw MS files
│       │   ├── sample_annotation.txt
│       │   ├── ncbi_taxa_ids.txt     # For ncbi_taxonomy_id method
│       │   ├── proteome_ids.txt      # For uniprot_proteome_id method
│       │   ├── fastq_files/          # For metaphlan method
│       │   └── MAG_files/            # For MAGs method
│       └── runs/
│           └── {run_name}/           # All outputs for a given run
│               ├── config/
│               ├── database_resources/
│               ├── logs/
│               └── output_files/
├── tests/
│   └── configs/                      # Integration test configs
└── .github/workflows/                # CI (dry-run and lint tests)

Module Dependency by Search Space Method

Each search_space_method activates a specific set of modules:

Method Modules used
ncbi_taxonomy_id ncbi_taxonomy → uniprot_proteome_ids → diann → uniprot_annotation + eggnogmapper + external_annotations
uniprot_proteome_id uniprot_proteome_ids → diann → uniprot_annotation + eggnogmapper + external_annotations
peptidotyping peptidotyping → ncbi_taxonomy → uniprot_proteome_ids → diann → uniprot_annotation + eggnogmapper + external_annotations
metaphlan metaphlan → ncbi_taxonomy → uniprot_proteome_ids → diann → uniprot_annotation + eggnogmapper + external_annotations
MAGs mags → database_processing → diann → mag_annotation + eggnogmapper + external_annotations

All methods use the setup and build_conduit modules.

Output Structure

All outputs for a run live under experiments/{experiment}/runs/{run_name}/:

Path Contents
database_resources/database.fasta Protein sequences in the search space
database_resources/taxonomy.txt Taxonomy for all organisms in the database
database_resources/protein_info.txt Per-protein metadata
database_resources/taxonomic_tree_of_database.pdf Taxonomic tree visualization
database_resources/database.predicted.speclib DIA-NN predicted spectral library
database_resources/README.md / .html Database summary metrics
database_resources/detected_protein_resources/detected_protein_info.txt Detected proteins
database_resources/detected_protein_resources/detected_protein.fasta Detected proteins in FASTA
database_resources/detected_protein_resources/uniprot_annotated_protein_info.txt Annotated detected proteins
database_resources/detected_protein_resources/conduit_annotations.txt All functional annotations
output_files/{experiment}_{run_name}_conduit.rds Final Conduit R object

Named Runs (run_name)

The run_name parameter namespaces all outputs for a given analysis. This allows multiple runs — e.g., comparing ncbi_taxonomy_id vs peptidotyping — to coexist in the same experiment directory without overwriting each other.

experiment: my_experiment
run_name: ncbi_run     # outputs → experiments/my_experiment/runs/ncbi_run/
experiment: my_experiment
run_name: peptidotyping_run  # outputs → experiments/my_experiment/runs/peptidotyping_run/

Clone this wiki locally