Workflow (running a pipeline)

Running a pipeline

1. Download the latest version of the pipeline

cd /cluster/home/username/git/
git clone https://github.com/pughlab/MismatchResolveDX/

2. Prepare interval files

For targeted-sequencing panels, a bed file containing target regions should be provided (listing at minimum: chromosome, start and end positions). Some variant calling pipelines (ie, Mutect2) will automatically add 100bp of padding to each region provided. For consistency, this padding must be manually added prior to variant calling with other tools (ie,SomaticSniper and VarDict).

module load perl

perl ~/git/MismatchResolveDx/scripts/format_intervals_bed.pl \
-b /path/to/base/intervals.bed \
-r /path/to/reference.fa

3. Prepare your input config files

Prepare your data config files (providing sample information and paths to each of your DNA-, EM- and sWGS fastq files). Prepare your pipeline config files (indicate tool versions to use, paths to reference files and resources required for each step, etc.).

See here for details: https://github.com/pughlab/MismatchResolveDX/wiki/Input-files

4. Run the pipelines

I suggest always first running the pipeline with the --dry-run flag. This will generate the directory structure and output the individual run scripts so that you can verify that there are no errors/missing values/typos in your config files. Once everything looks set, run the pipeline again without --dry-run.

module load perl

# Run the MismatchResolveDX pipeline as a full block
perl ~/git/MismatchResolveDx/pughlab_multimmr_pipeline.pl \
-w /path/to/swgs_pipeline_config.yaml \
-s /path/to/swgs_fastq_config.yaml \
-t /path/to/dna_pipeline_config.yaml \
-d /path/to/dna_fastq_config.yaml \
-f /path/to/em_pipeline_config.yaml \
-e /path/to/em_fastq_config.yaml \
--report_dir /path/to/final/report/directory \
-c slurm \
--remove \
--dry-run { optional }

# Or, run the pipeline in separate stages (ie, preprocessing first, then once that's complete, run the QC and variant calling stages)
perl ~/git/MismatchResolveDx/pughlab_dnaseq_pipeline.pl \
-w /path/to/swgs_pipeline_config.yaml \
-s /path/to/swgs_fastq_config.yaml \
--preprocessing \
--qc \
--variant_calling \
-c slurm \
--remove \
--dry-run { optional }

perl ~/git/MismatchResolveDx/pughlab_dnaseq_pipeline.pl \
-t /path/to/dna_pipeline_config.yaml \
-d /path/to/dna_fastq_config.yaml \
--preprocessing \
--qc \
--variant_calling \
-c slurm \
--remove \
--dry-run { optional }

perl ~/git/MismatchResolveDx/pughlab_emseq_pipeline.pl \
-f /path/to/em_pipeline_config.yaml \
-e /path/to/em_fastq_config.yaml \
--fastq_prep \
--alignment \
--qc \
--analysis \
-c slurm \
--remove \
--dry-run { optional }

# Generate the cohort-level and patient-level summary files and reports (only run this after each of the above pipelines are complete!):
perl ~/git/MismatchResolveDx/scripts/report/make_multimmr_reports.pl \
-w /path/to/swgs_pipeline_config.yaml \
-d /path/to/dna_pipeline_config.yaml \
-e /path/to/em_pipeline_config.yaml \
--output_directory /path/to/final/report/directory \
-c slurm \
--dry-run { optional }

This will generate the directory structure in the output directory (provided in your tool config), including a "logs/run_{DNA|EM}_pipeline_TIMESTAMP/" directory containing a file "run_{DNA|EM}Seq_pipeline.log" which lists the individual tool commands; these can be run separately if "--dry-run" was set, or in the event of a failure at any stage and you don't need to re-run the entire thing (Note: doing so should not regenerate files that already exist).

Resuming a run:

If the initial run is unsuccessful or incomplete, check the logs to identify the problem - it is most likely due to insufficient memory or runtime allocation. In this case, update the necessary parameters for the affected stage in the tool config.

Now, rerun individual tool commands (as found in /path/to/output/directory/logs/TIMESTAMP/run_{D|R}NASeq_pipeline.log).