MIP - Mutation Identification Pipeline
MIP enables identification of potential disease causing variants from sequencing data.
Rapid pulsed whole genome sequencing for comprehensive acute diagnostics of inborn errors of metabolism Stranneheim H, Engvall M, Naess K, Lesko N, Larsson P, Dahlberg M, Andeer R, Wredenberg A, Freyer C, Barbaro M, Bruhn H, Emahazion T, Magnusson M, Wibom R, Zetterström RH, Wirta V, von Döbeln U, Wedell A. BMC Genomics. 2014 Dec 11;15(1):1090. doi: 10.1186/1471-2164-15-1090. PMID:25495354
MIP performs whole genome or target region analysis of sequenced single-end and/or paired-end reads from the Illumina platform in fastq(.gz) format to generate annotated ranked potential disease causing variants.
MIP performs QC, alignment, coverage analysis, variant discovery and annotation, sample checks as well as ranking the found variants according to disease potential with a minimum of manual intervention. MIP is compatible with Scout for visualization of identified variants. MIP analyses snv, indels and SV.
MIP has been in use in the clinical production at the Clinical Genomics facility at Science for Life Laboratory since 2014.
$ mip --family_id [family_id] --pbwa_mem 1 --config_file [mip_config.yaml] --pedigree_file [family_id_pedigree.yaml]
- Simple automated install of all programs using conda/SHELL via supplied install script
- Downloads and prepares references in the installation process
- Handle conflicting tool dependencies
- Checks that all dependencies are fulfilled before launching
- Builds and prepares references and/or files missing before launching
- Decompose and normalise reference(s) and variant vcf(s)
- Splits and merges files/contigs for samples and families when relevant
- A minimal amount of hands-on time
- Tracks and executes all module without manual intervention
- Creates internal queues at nodes to optimize processing
- Minimal IO between nodes and login node
- Design your own workflow by turning on/off relevant modules
- Restart an analysis from anywhere in your workflow
- Process one, or multiple samples using the module(s) of your choice
- Supply parameters on the command line, in a pedigree.yaml file or via config files
- Simulate your analysis before performing it
- Redirect each modules analysis process to a temporary directory (@nodes or @login)
- Limit a run to a specific set of genomic intervals
- Use multiple variant callers for both snv, indels and SV
- Use multiple annotation programs
- Optionally split data into clinical variants and research variants
- Analyses an exome trio in approximately 4 h
- Analyses a genome in approximately 21 h
- Rapid mode analyzes a WGS sample in approximately 4 h using a data reduction and parallelization scheme
- Track the status of each modules through dynamically updated status logs
- Recreate your analysis from the MIP log or generated config files
- Log sample meta-data and sequence meta-data
- Log version numbers of softwares and databases
- Checks sample integrity (sex, contamination, duplications, ancestry, inbreeding and relationship)
- Test data output existens and integrity using automated tests
- Gene annotation
- Summarize over all transcript and output on gene level
- Transcript level annotation
- Separate pathogenic transcripts for correct downstream annotation
- Annotate all alleles for a position
- Split multi-allelic records into single records to ease annotation
- Left align and trim variants to normalise them prior to annotation
- Extracts QC-metrics and stores them in YAML format
- Annotate coverage across genetic regions via Sambamba and Chanjo
- Gene annotation
- Use standard formats whenever possible
- Ranks variants according to pathogenic potential
- Output is directly compatible with Scout
MIP is written in perl and therefore requires that perl is installed on your OS.
We recommend perlbrew for installing and managing perl and cpanm libraries. Installation instructions and setting up specific cpanm libraries can be found here.
Automated Installation (Linux x86_64)
1.Clone the official git repository
$ git clone https://github.com/Clinical-Genomics/MIP.git $ cd MIP
2.Install required modules from cpan
$ cd definitions $ cpanm --installdeps . $ cd -
- Some versions of the cpan module List::MoreUtils has conflicts with certain versions of Perl. Perl v 5.18.2 has been confirmed to be compatible with version 0.4.13 of the List::MoreUtils module.
If you experience issues with that module, try to reinstall it by running:
cpanm --reinstall List::MoreUtils@0.4.13
3.Test conda and mip_installation (optional)
$ cd t; prove mip_install.t $ cd -
4.Create the install instructions for MIP
$ perl mip_install.pl
This will generate a batch script "mip.sh" for the install in your working directory.
The batch script will install the MIP dependencies in Conda's root environment. Often it is beneficial to create a separate environment for each of your applications. In order to create a separate environment for MIP supply the
-env [env_name]flag when running mip_install.pl.
For a full list of available options and parameters, run:
$ perl mip_install.pl --help
For a full list of parameter defaults, run:
$ perl mip_install.pl -ppd
5.Run the bash script
$ bash mip.sh
This will install MIP and most of its dependencies into a conda environment.
- Some references are quite large and will take time to download. You might want to run this using screen or tmux.
6.Test your MIP installation (optional)
Make sure to activate your conda environment if that option was used above.
$ cd t; prove -r $ cd -
7.Install tools with conflicting dependencies
Tools that have conflicting dependencies needs to be installed in separate conda environments. Currently these programs requires separate environments:
- Genmod, Chanjo, Multiqc and Variant_integrity
- requires python 3
- conda installs perl
- Requires access to ROOT which disturbs the normal linking of C libraries
- scipy conflicts
## Python 3 tools $ perl mip_install.pl -env mip_pyv3.6 --python_version 3.6 --select_program genmod --select_program chanjo --select_program variant_integrity --select_program multiqc $ bash mip.sh ## Freebayes $ perl mip_install.pl -env mip_freebayes --select_program freebayes $ bash mip.sh ## Peddy $ perl mip_install.pl -env mip_peddy --select_program peddy $ bash mip.sh ## SVDB $ perl mip_install.pl -env mip_svdb --select_program svdb $ bash mip.sh ## VEP $ perl mip_install.pl -env mip_vep --select_program vep $ bash mip.sh ## TIDDIT $ perl mip_install.pl -env mip_tiddit --select_program tiddit $ bash mip.sh ## CNVnator $ perl mip_install.pl -env mip_cnvnator --select_program cnvnator $ bash mip.sh
In your config yaml file or on the command line you will have to supply the
module_source_environment_command parameter to activate the conda environment specific for the tool. Here is an example with three Python 3 tools in their own environment and Peddy, CNVnator, SVDB and VEP in each own, with some extra initialization:
program_source_environment_command: pgenmod: "source activate mip_pyv3.6" module_source_environment_command: pchanjo_sexcheck: "source activate mip_pyv3.6" pcnvnator: "LD_LIBRARY_PATH=[CONDA_PATH]/lib/:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; source [CONDA_PATH]/envs/mip_cnvnator/root/bin/thisroot.sh; source activate mip_cnvnator" pfreebayes: "source activate mip_freebayes" pmultiqc: "source activate mip_pyv3.6" ppeddy: "source activate mip_peddy" prankvariant: "source activate mip_pyv3.6" psv_rankvariant: "source activate mip_pyv3.6" psv_combinevariantcallsets: "source activate mip_svdb" psv_varianteffectpredictor: "LD_LIBRARY_PATH=[CONDA_PATH]/envs/mip_vep/lib/:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; source activate mip_vep" ptiddit: "source activate mip_tiddit" pvarianteffectpredictor: "LD_LIBRARY_PATH=[CONDA_PATH]/envs/mip_vep/lib/:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; source activate mip_vep" pvariant_integrity: "source activate mip_pyv3.6" source_main_environment_commands: - source - activate - mip
MIP will execute this on the node before executing the program and then revert to the
--source_main_environment_command if set. Otherwise
source deactivate is used to return to the conda root environment.
MIP is called from the command line and takes input from the command line (precedence) or falls back on defaults where applicable.
Lists are supplied as repeated flag entries on the command line or in the config using the yaml format for arrays.
Only flags that will actually be used needs to be specified and MIP will check that all required parameters are set before submitting to SLURM.
Program parameters always begins with "p" followed by a capital letter. Program parameters can be set to "0" (=off), "1" (=on) and "2" (=dry run mode). Any progam can be set to dry run mode and MIP will create sbatch scripts, but not submit them to SLURM. MIP can be restarted from any module, but you need to supply previous dependent programs in dry run mode to ensure proper file handling.
MIP will overwrite data files when reanalyzing, but keeps all "versioned" sbatch scripts for traceability.
You can always supply
perl mip.pl --help to list all available parameters and defaults.
$ mip -f 3 --sample_ids 3-1-1A --sample_ids 3-2-1U --sample_ids 3-2-2U -pfqc 0 --pbwa_mem 2 -c 3_config.yaml
This will analyse family 3 using 3 individuals from that family and begin the analysis with programs after Bwa mem and use all parameter values as specified in the config file except those supplied on the command line, which has precedence.
All references and template files should be placed directly in the reference directory specified by
- Configuration file (YAML-format)
- Gene panel file
- Pedigree file (YAML-format)
- Rank model file (Ini-format; Snv/indel)
- SV rank model file (Ini-format; SV)
- Qc regexp file (YAML-format)
Analyses done per individual is found in each sample_id directory and analyses done including all samples can be found in the family directory.
MIP will create sbatch scripts (.sh) and submit them in proper order with attached dependencies to SLURM. These sbatch script are placed in the output script directory specified by
--outscript_dir. The sbatch scripts are versioned and will not be overwritten if you begin a new analysis. Versioned "xargs" scripts will also be created where possible to maximize the use of the cores processing power.
MIP will place any generated datafiles in the output data directory specified by
--outdata_dir. All data files are regenerated for each analysis. STDOUT and STDERR for each program is written in the program/info directory prior to alignment and in the aligner/info directory post alignment.