Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First Release #19

Merged
merged 98 commits into from
Oct 8, 2019
Merged

First Release #19

merged 98 commits into from
Oct 8, 2019

Conversation

maxulysse
Copy link
Member

@maxulysse maxulysse commented Jul 3, 2019

Major PR with first changes for first nf-core release. Pipeline ported from https://github.com/SciLifeLab/Sarek by @maxulysse

Minor things to do before release:

  • rename wgs_calling_regions_CAW.list for GRCh37
  • ignore QC processes ('--skipQC' or whatever), and change docs accordingly
  • add check for misspelled parameteres (i.e. adding '--pom file.vcf' is silently ignored)
  • make sure results are copied, not only linked -> That needs to be specify with the --publishDirMode that can be added to nf-core/configs, currently I do use 'copy which is the default for other nf-core pipelines
  • dummy PON and germline resource VCF for both GRCh37 and 38 (Create germlineResource for GRCh37 #25)

maxulysse and others added 22 commits April 30, 2019 17:42
- update README
- update `.travis.yml`
- add `Jenkinsfile`
- update conda `environment.yml` file
* update README
* update .travis.yml
* add Jenkinsfile
* update conda environment file
* update software_version collect
* remove lib/SarekUtils.groovy
* add References to .gitignore
* improve software versions gathering
* fix docker owner
* update tests
* sort params
* add preprocessing
* improve multiple TSV
* add parrallelized BaseRecalibrator
* Update allelecount to 4.0.2
* Update GATK to 4.1.2.0
* Update docs
* Add HaplotypeCaller
* Add single Strelka and single Manta
Add somatic Variant Callers
* add containers for annotation
* Add Annotation
* Annotation containers are tagged version.genomeversion
* Better indentation
* Replace `--sampleDir` by `--sample`
* add MultiQC
* update Tests
* Add parallelized testing on Travis
* Add parallelized building of Docker Images for annotation on CircleCI
* Update docs
* improve tests + add some docs
* update tests
* code polishing
* add some docs about QC tools
* better docs
* add docs about annotation
* move helper scripts to scripts
* add helper script to filter FreeBayes output
* add docs about Variant Calling and Preprocessing
* add scripts for ASCAT
* fix publishDir for MergeMpileup
* fix intra link
* completed docs about ASCAT
* add building container for dogs
* add Abstracts & Posters
* improve tests
* update docs
* Update CI scripts
* Add possibility to download singularity images with helper script
* Update Docs
* Improve build.nf
* Use label for processes configuration
* Update configuration files
* Disable Docker in singularity profile
* Disable Singularity in docker profile
* Disable Docker and Singularity in conda profile
* Simplify check_max() function
* Add BamQC, CompressVCFsnpEff and CompressVCFvep processes
* Add Citation documentation
* Fix merge in annotation
* Merge BamQCmapped and BamQCrecalibrated processes into BamQC process
* Removed BamQCmapped, BamQCrecalibrated and CompressVCF processes
* Split CompressVCF process into CompressVCFsnpEff and CompressVCFvep processes
* chmod +x
* add correct path to Rscript
* update CHANGELOG
@nf-core-bot
Copy link
Member

nf-core-bot commented Jul 4, 2019

🚨 Build log output:


Branch indexing
Connecting to https://api.github.com using nf-core-bot/****** (nf-core)
Obtained Jenkinsfile from 10eff4e+7a93b668665336cfd7cd80fbf17a93314e9fa090 (0f07bf2)
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on ship-1 in /share/jenkins_home/workspace/nf-core_sarek_PR-19
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
using credential 3d82ac34-e01a-4592-a628-ce6732c445a3
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/nf-core/sarek.git
> git init /share/jenkins_home/workspace/nf-core_sarek_PR-19 # timeout=10
Fetching upstream changes from https://github.com/nf-core/sarek.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials nf-core
> git fetch --no-tags --progress https://github.com/nf-core/sarek.git +refs/pull/19/head:refs/remotes/origin/PR-19 +refs/heads/master:refs/remotes/origin/master
Fetching without tags
Merging remotes/origin/master commit 7a93b66 into PR head commit 10eff4e
> git config remote.origin.url https://github.com/nf-core/sarek.git # timeout=10
> git config --add remote.origin.fetch +refs/pull/19/head:refs/remotes/origin/PR-19 # timeout=10
> git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
> git config remote.origin.url https://github.com/nf-core/sarek.git # timeout=10
Fetching upstream changes from https://github.com/nf-core/sarek.git
using GIT_ASKPASS to set credentials nf-core
> git fetch --no-tags --progress https://github.com/nf-core/sarek.git +refs/pull/19/head:refs/remotes/origin/PR-19 +refs/heads/master:refs/remotes/origin/master
> git config core.sparsecheckout # timeout=10
> git checkout -f 10eff4e
Merge succeeded, producing 10eff4e
Checking out Revision 10eff4e (PR-19)
Commit message: "Fix R scripts (#16)"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // stage
> git merge 7a93b66 # timeout=10
> git rev-parse HEAD^{commit} # timeout=10
> git config core.sparsecheckout # timeout=10
> git checkout -f 10eff4e
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withCredentials
Masking only exact matches of $JENKINS_API or $JENKINS_API_USR or $JENKINS_API_PSW
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Docker setup)
[Pipeline] sh
+ ./scripts/download_image.sh -n docker -t ALL
+ ENGINE=docker
+ NXF_SINGULARITY_CACHEDIR=work/singularity/.
+ TEST=ALL
+ [[ 4 -gt 0 ]]
+ key=-n
+ case $key in
+ ENGINE=docker
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ key=-t
+ case $key in
+ TEST=ALL
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ [[ ALL,ANNOTATEALL,ANNOTATESNPEFF =~ ALL ]]
+ [[ docker =~ docker ]]
+ docker pull nfcore/sareksnpeff:dev.GRCh37
dev.GRCh37: Pulling from nfcore/sareksnpeff
Digest: sha256:734b4ba5645f8b88a18ea37c33cc4cedef97a53dc389810bfb3d5c4281650f92
Status: Image is up to date for nfcore/sareksnpeff:dev.GRCh37
+ docker tag nfcore/sareksnpeff:dev.GRCh37 nfcore/sareksnpeff:dev.smallGRCh37
+ [[ ALL,ANNOTATEALL,ANNOTATEVEP =~ ALL ]]
+ [[ docker =~ docker ]]
+ docker pull nfcore/sarekvep:dev.GRCh37
dev.GRCh37: Pulling from nfcore/sarekvep
Digest: sha256:0f051af98e8360e8f01f5aad56d1d6dce96c3ad4dc59b3627873acaa0bb729f9
Status: Image is up to date for nfcore/sarekvep:dev.GRCh37
+ docker tag nfcore/sarekvep:dev.GRCh37 nfcore/sarekvep:dev.smallGRCh37
+ [[ ANNOTATEALL,ANNOTATEVEP,ANNOTATESNPEFF != ALL ]]
+ [[ docker =~ docker ]]
+ docker pull nfcore/sarek:dev
dev: Pulling from nfcore/sarek
Digest: sha256:d2ac7fc24278fb0ce21952a7ed7318dde0dda1988423ac8d72b86dff5a338810
Status: Image is up to date for nfcore/sarek:dev
+ docker tag nfcore/sarek:dev nfcore/sarek:dev
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build references)
[Pipeline] sh
+ rm -rf references/
[Pipeline] sh
+ ./scripts/build_reference.sh
+ NXF_SINGULARITY_CACHEDIR=work/singularity/.
+ OFFLINE=
+ PROFILE=docker
+ TEST=ALL
+ TRAVIS_BUILD_DIR=.
+ TRAVIS=false
+ VERBOSE=
+ [[ 0 -gt 0 ]]
+ [[ ALL != ANNOTATESNPEFF ]]
+ [[ ALL != ANNOTATEVEP ]]
+ rm -rf references
+ nextflow run ./build.nf -profile test,docker --build --outdir references
N E X T F L O W ~ version 19.04.0
Launching `./build.nf` [spontaneous_borg] - revision: 712bf0811e
WARN: There's no process matching config selector: BamQC
Run Name : spontaneous_borg
Max Resources : 7 GB memory, 2 cpus, 2d time per job
Container : docker - nfcore/sarek:dev
Output dir : references
Launch dir : /share/****_home/workspace/nf-core_sarek_PR-19
Working dir : /share/****_home/workspace/nf-core_sarek_PR-19/work
Script dir : /share/****_home/workspace/nf-core_sarek_PR-19
User : ****
Config Profile : test,docker
Config Description: Minimal test dataset to check pipeline function
executor > local (1)
[74/b0b54b] process > DecompressFile [ 0%] 0 of 1

executor > local (2)
[b1/4f8c63] process > DecompressFile [ 0%] 0 of 2

executor > local (2)
[74/b0b54b] process > DecompressFile [ 50%] 1 of 2

executor > local (3)
[02/bcc797] process > DecompressFile [ 33%] 1 of 3

executor > local (3)
[b1/4f8c63] process > DecompressFile [ 67%] 2 of 3

executor > local (5)
[fb/2e7000] process > DecompressFile [ 50%] 2 of 4
[f0/8b2f37] process > BuildSAMToolsIndex [ 0%] 0 of 1

executor > local (8)
[fb/2e7000] process > DecompressFile [ 50%] 2 of 4
[f0/8b2f37] process > BuildSAMToolsIndex [ 0%] 0 of 1
[e7/1b54c8] process > BuildBWAindexes [ 0%] 0 of 1
[b7/b92b2b] process > BuildReferenceIndex [ 0%] 0 of 1
[ee/692355] process > BuildVCFIndex [ 0%] 0 of 1

executor > local (9)
[02/bcc797] process > DecompressFile [ 75%] 3 of 4
[f0/8b2f37] process > BuildSAMToolsIndex [ 0%] 0 of 1
[e7/1b54c8] process > BuildBWAindexes [ 0%] 0 of 1
[b7/b92b2b] process > BuildReferenceIndex [ 0%] 0 of 1
[d8/e66ced] process > BuildVCFIndex [ 0%] 0 of 2

executor > local (10)
[fb/2e7000] process > DecompressFile [100%] 4 of 4 ✔
[f0/8b2f37] process > BuildSAMToolsIndex [ 0%] 0 of 1
[e7/1b54c8] process > BuildBWAindexes [ 0%] 0 of 1
[b7/b92b2b] process > BuildReferenceIndex [ 0%] 0 of 1
[6f/0a2b76] process > BuildVCFIndex [ 0%] 0 of 3

executor > local (10)
[fb/2e7000] process > DecompressFile [100%] 4 of 4 ✔
[f0/8b2f37] process > BuildSAMToolsIndex [100%] 1 of 1 ✔
[e7/1b54c8] process > BuildBWAindexes [ 0%] 0 of 1
[b7/b92b2b] process > BuildReferenceIndex [ 0%] 0 of 1
[6f/0a2b76] process > BuildVCFIndex [ 0%] 0 of 3

executor > local (10)
[fb/2e7000] process > DecompressFile [100%] 4 of 4 ✔
[f0/8b2f37] process > BuildSAMToolsIndex [100%] 1 of 1 ✔
[e7/1b54c8] process > BuildBWAindexes [ 0%] 0 of 1
[b7/b92b2b] process > BuildReferenceIndex [ 0%] 0 of 1
[ee/692355] process > BuildVCFIndex [ 33%] 1 of 3

executor > local (10)
[fb/2e7000] process > DecompressFile [100%] 4 of 4 ✔
[f0/8b2f37] process > BuildSAMToolsIndex [100%] 1 of 1 ✔
[e7/1b54c8] process > BuildBWAindexes [100%] 1 of 1 ✔
[b7/b92b2b] process > BuildReferenceIndex [ 0%] 0 of 1
[ee/692355] process > BuildVCFIndex [ 33%] 1 of 3

executor > local (10)
[fb/2e7000] process > DecompressFile [100%] 4 of 4 ✔
[f0/8b2f37] process > BuildSAMToolsIndex [100%] 1 of 1 ✔
[e7/1b54c8] process > BuildBWAindexes [100%] 1 of 1 ✔
[b7/b92b2b] process > BuildReferenceIndex [ 0%] 0 of 1
[d8/e66ced] process > BuildVCFIndex [ 67%] 2 of 3

executor > local (10)
[fb/2e7000] process > DecompressFile [100%] 4 of 4 ✔
[f0/8b2f37] process > BuildSAMToolsIndex [100%] 1 of 1 ✔
[e7/1b54c8] process > BuildBWAindexes [100%] 1 of 1 ✔
[b7/b92b2b] process > BuildReferenceIndex [ 0%] 0 of 1
[6f/0a2b76] process > BuildVCFIndex [100%] 3 of 3 ✔

executor > local (10)
[fb/2e7000] process > DecompressFile [100%] 4 of 4 ✔
[f0/8b2f37] process > BuildSAMToolsIndex [100%] 1 of 1 ✔
[e7/1b54c8] process > BuildBWAindexes [100%] 1 of 1 ✔
[b7/b92b2b] process > BuildReferenceIndex [100%] 1 of 1 ✔
[6f/0a2b76] process > BuildVCFIndex [100%] 3 of 3 ✔

executor > local (10)
[fb/2e7000] process > DecompressFile [100%] 4 of 4 ✔
[f0/8b2f37] process > BuildSAMToolsIndex [100%] 1 of 1 ✔
[e7/1b54c8] process > BuildBWAindexes [100%] 1 of 1 ✔
[b7/b92b2b] process > BuildReferenceIndex [100%] 1 of 1 ✔
[6f/0a2b76] process > BuildVCFIndex [100%] 3 of 3 ✔
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Completed at: 04-Jul-2019 13:00:44
Duration : 14s
CPU hours : (a few seconds)
Succeeded : 10

+ rm -rf .nextflow .nextflow.log references/pipeline_info work
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Germline)
[Pipeline] sh
+ rm -rf data/
[Pipeline] sh
+ git clone --single-branch --branch sarek https://github.com/nf-core/test-datasets.git data
Cloning into 'data'...
[Pipeline] sh
+ ./scripts/run_tests.sh --test GERMLINE
+ CPUS=2
+ LOGS=
+ NXF_SINGULARITY_CACHEDIR=work/singularity/.
+ OFFLINE=false
+ PROFILE=docker
+ TEST=ALL
+ TRAVIS_BUILD_DIR=.
+ TRAVIS=false
+ VERBOSE=
+ [[ 2 -gt 0 ]]
+ key=--test
+ case $key in
+ TEST=GERMLINE
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ [[ ALL,GERMLINE =~ GERMLINE ]]
+ [[ false == false ]]
+ rm -rf data
+ git clone --single-branch --branch sarek https://github.com/nf-core/test-datasets.git data
Cloning into 'data'...
+ run_sarek --tools=false --sample data/testdata/tiny/normal --noReports
+ nextflow run ./main.nf -profile test,docker --monochrome_logs --tools=false --sample data/testdata/tiny/normal --noReports
N E X T F L O W ~ version 19.04.0
Launching `./main.nf` [soggy_heyrovsky] - revision: fb7cd02724
No TSV file
Reading data/testdata/tiny/normal directory
----------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
____ _____ _
.' _ `. / ____| | |
/ |\`-_ \ | (___ ___ _ __ __ | | __
| | \ `-| \___ \/__ \| ´__/ _\| |/ /
\ | \ / ____) | __ | | | __| <
`|____\' |_____/\____|_| \__/|_|\_\

nf-core/sarek v2.5dev
----------------------------------------------------
Run Name : soggy_heyrovsky
Max Resources : 7 GB memory, 2 cpus, 2d time per job
Container : docker - nfcore/sarek:dev
Sample : data/testdata/tiny/normal
Step : mapping
No Reports : true
Nucleotides/s : 1000.0
Output dir : ./results
Launch dir : /share/****_home/workspace/nf-core_sarek_PR-19
Working dir : /share/****_home/workspace/nf-core_sarek_PR-19/work
Script dir : /share/****_home/workspace/nf-core_sarek_PR-19
User : ****
Config Profile : test,docker
Config Description: Minimal test dataset to check pipeline function
----------------------------------------------------
executor > local (2)
[6d/c6e37d] process > CreateIntervalBeds [ 0%] 0 of 1
[cb/4b4feb] process > MapReads [ 0%] 0 of 1

executor > local (6)
[6d/c6e37d] process > CreateIntervalBeds [ 0%] 0 of 1
[b0/fe04e4] process > MapReads [ 0%] 0 of 5

executor > local (6)
[6d/c6e37d] process > CreateIntervalBeds [ 0%] 0 of 1
[cb/4b4feb] process > MapReads [ 20%] 1 of 5

executor > local (6)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[64/0a5a48] process > MapReads [ 40%] 2 of 5

executor > local (6)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[10/1e8516] process > MapReads [ 60%] 3 of 5

executor > local (6)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[66/c1979b] process > MapReads [ 80%] 4 of 5

executor > local (7)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [ 0%] 0 of 1

executor > local (8)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [ 0%] 0 of 1

executor > local (14)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[23/8b006d] process > BaseRecalibrator [ 0%] 0 of 6

executor > local (14)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[c1/c749c3] process > BaseRecalibrator [ 17%] 1 of 6

executor > local (14)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[23/8b006d] process > BaseRecalibrator [ 33%] 2 of 6

executor > local (14)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[ca/bd3099] process > BaseRecalibrator [ 50%] 3 of 6

executor > local (14)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[af/efe42f] process > BaseRecalibrator [ 83%] 5 of 6

executor > local (15)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [ 0%] 0 of 1

executor > local (17)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[15/2c3be8] process > ApplyBQSR [ 0%] 0 of 2

executor > local (21)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[7d/d4b9c4] process > ApplyBQSR [ 0%] 0 of 6

executor > local (21)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[15/2c3be8] process > ApplyBQSR [ 17%] 1 of 6

executor > local (21)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[93/9fedc6] process > ApplyBQSR [ 33%] 2 of 6

executor > local (21)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[54/d6814e] process > ApplyBQSR [ 67%] 4 of 6

executor > local (21)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[7d/d4b9c4] process > ApplyBQSR [ 83%] 5 of 6

executor > local (22)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[24/aa115e] process > ApplyBQSR [100%] 6 of 6 ✔
[a1/85fe1c] process > MergeBamRecal [ 0%] 0 of 1

executor > local (22)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[24/aa115e] process > ApplyBQSR [100%] 6 of 6 ✔
[a1/85fe1c] process > MergeBamRecal [100%] 1 of 1 ✔

executor > local (22)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[24/aa115e] process > ApplyBQSR [100%] 6 of 6 ✔
[a1/85fe1c] process > MergeBamRecal [100%] 1 of 1 ✔
Warning, pipeline completed, but with errored process(es)
Number of ignored errored process(es) : 0
Number of successfully ran process(es) : 22
WARN: [nf-core/sarek] Could not attach MultiQC report to summary email

executor > local (22)
[6d/c6e37d] process > CreateIntervalBeds [100%] 1 of 1 ✔
[b0/fe04e4] process > MapReads [100%] 5 of 5 ✔
[64/59cc75] process > MergeBamMapped [100%] 1 of 1 ✔
[34/c7e2d1] process > MarkDuplicates [100%] 1 of 1 ✔
[f8/a0bc4b] process > BaseRecalibrator [100%] 6 of 6 ✔
[f1/135459] process > GatherBQSRReports [100%] 1 of 1 ✔
[24/aa115e] process > ApplyBQSR [100%] 6 of 6 ✔
[a1/85fe1c] process > MergeBamRecal [100%] 1 of 1 ✔
Warning, pipeline completed, but with errored process(es)
Number of ignored errored process(es) : 0
Number of successfully ran process(es) : 22
WARN: [nf-core/sarek] Could not attach MultiQC report to summary email
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Completed at: 04-Jul-2019 13:01:50
Duration : 55.9s
CPU hours : 0.1
Succeeded : 22

+ run_sarek --tools=false --sample results/Preprocessing/TSV/duplicateMarked.tsv --step recalibrate --noReports
+ nextflow run ./main.nf -profile test,docker --monochrome_logs --tools=false --sample results/Preprocessing/TSV/duplicateMarked.tsv --step recalibrate --noReports
N E X T F L O W ~ version 19.04.0
Launching `./main.nf` [small_wilson] - revision: fb7cd02724
----------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
____ _____ _
.' _ `. / ____| | |
/ |\`-_ \ | (___ ___ _ __ __ | | __
| | \ `-| \___ \/__ \| ´__/ _\| |/ /
\ | \ / ____) | __ | | | __| <
`|____\' |_____/\____|_| \__/|_|\_\

nf-core/sarek v2.5dev
----------------------------------------------------
Run Name : small_wilson
Max Resources : 7 GB memory, 2 cpus, 2d time per job
Container : docker - nfcore/sarek:dev
Sample : results/Preprocessing/TSV/duplicateMarked.tsv
Step : recalibrate
No Reports : true
Nucleotides/s : 1000.0
Output dir : ./results
Launch dir : /share/****_home/workspace/nf-core_sarek_PR-19
Working dir : /share/****_home/workspace/nf-core_sarek_PR-19/work
Script dir : /share/****_home/workspace/nf-core_sarek_PR-19
User : ****
Config Profile : test,docker
Config Description: Minimal test dataset to check pipeline function
----------------------------------------------------
executor > local (1)
[28/a7838b] process > CreateIntervalBeds [ 0%] 0 of 1

executor > local (1)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔

executor > local (7)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔
[4a/e15971] process > ApplyBQSR [ 0%] 0 of 6

executor > local (7)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔
[84/779aec] process > ApplyBQSR [ 17%] 1 of 6

executor > local (7)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔
[4a/e15971] process > ApplyBQSR [ 33%] 2 of 6

executor > local (7)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔
[90/7581aa] process > ApplyBQSR [ 50%] 3 of 6

executor > local (7)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔
[9c/a847c2] process > ApplyBQSR [ 83%] 5 of 6

executor > local (8)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔
[43/9529ba] process > ApplyBQSR [100%] 6 of 6 ✔
[6e/476a8d] process > MergeBamRecal [ 0%] 0 of 1

executor > local (8)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔
[43/9529ba] process > ApplyBQSR [100%] 6 of 6 ✔
[6e/476a8d] process > MergeBamRecal [100%] 1 of 1 ✔

executor > local (8)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔
[43/9529ba] process > ApplyBQSR [100%] 6 of 6 ✔
[6e/476a8d] process > MergeBamRecal [100%] 1 of 1 ✔
Warning, pipeline completed, but with errored process(es)
Number of ignored errored process(es) : 0
Number of successfully ran process(es) : 8
WARN: [nf-core/sarek] Could not attach MultiQC report to summary email

executor > local (8)
[28/a7838b] process > CreateIntervalBeds [100%] 1 of 1 ✔
[43/9529ba] process > ApplyBQSR [100%] 6 of 6 ✔
[6e/476a8d] process > MergeBamRecal [100%] 1 of 1 ✔
Warning, pipeline completed, but with errored process(es)
Number of ignored errored process(es) : 0
Number of successfully ran process(es) : 8
WARN: [nf-core/sarek] Could not attach MultiQC report to summary email
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Completed at: 04-Jul-2019 13:02:13
Duration : 18.7s
CPU hours : (a few seconds)
Succeeded : 8

+ run_sarek --tools HaplotypeCaller --sample results/Preprocessing/TSV/recalibrated.tsv --step variantCalling --noReports
+ nextflow run ./main.nf -profile test,docker --monochrome_logs --tools HaplotypeCaller --sample results/Preprocessing/TSV/recalibrated.tsv --step variantCalling --noReports
N E X T F L O W ~ version 19.04.0
Launching `./main.nf` [soggy_gilbert] - revision: fb7cd02724
----------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
____ _____ _
.' _ `. / ____| | |
/ |\`-_ \ | (___ ___ _ __ __ | | __
| | \ `-| \___ \/__ \| ´__/ _\| |/ /
\ | \ / ____) | __ | | | __| <
`|____\' |_____/\____|_| \__/|_|\_\

nf-core/sarek v2.5dev
----------------------------------------------------
Run Name : soggy_gilbert
Max Resources : 7 GB memory, 2 cpus, 2d time per job
Container : docker - nfcore/sarek:dev
Sample : results/Preprocessing/TSV/recalibrated.tsv
Step : variantCalling
Tools : haplotypecaller
No Reports : true
Nucleotides/s : 1000.0
Output dir : ./results
Launch dir : /share/****_home/workspace/nf-core_sarek_PR-19
Working dir : /share/****_home/workspace/nf-core_sarek_PR-19/work
Script dir : /share/****_home/workspace/nf-core_sarek_PR-19
User : ****
Config Profile : test,docker
Config Description: Minimal test dataset to check pipeline function
----------------------------------------------------
executor > local (1)
[9d/9e292f] process > CreateIntervalBeds [ 0%] 0 of 1

executor > local (2)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[8b/26de48] process > HaplotypeCaller [ 0%] 0 of 1

executor > local (7)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [ 0%] 0 of 6

executor > local (8)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[8b/26de48] process > HaplotypeCaller [ 17%] 1 of 6
[a7/86e87d] process > GenotypeGVCFs [ 0%] 0 of 1

executor > local (9)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[5a/2d6d60] process > HaplotypeCaller [ 33%] 2 of 6
[3b/1bd347] process > GenotypeGVCFs [ 0%] 0 of 2

executor > local (11)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[50/4bfc19] process > HaplotypeCaller [ 67%] 4 of 6
[ff/966091] process > GenotypeGVCFs [ 0%] 0 of 4

executor > local (12)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[d3/7cdb5c] process > HaplotypeCaller [ 83%] 5 of 6
[24/32a2dc] process > GenotypeGVCFs [ 0%] 0 of 5

executor > local (14)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [100%] 6 of 6 ✔
[97/127f2a] process > GenotypeGVCFs [ 0%] 0 of 6
[f6/c87ba1] process > ConcatVCF [ 0%] 0 of 1

executor > local (14)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [100%] 6 of 6 ✔
[97/127f2a] process > GenotypeGVCFs [ 0%] 0 of 6
[f6/c87ba1] process > ConcatVCF [100%] 1 of 1

executor > local (14)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [100%] 6 of 6 ✔
[a7/86e87d] process > GenotypeGVCFs [ 17%] 1 of 6
[f6/c87ba1] process > ConcatVCF [100%] 1 of 1

executor > local (14)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [100%] 6 of 6 ✔
[ff/966091] process > GenotypeGVCFs [ 33%] 2 of 6
[f6/c87ba1] process > ConcatVCF [100%] 1 of 1

executor > local (14)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [100%] 6 of 6 ✔
[7e/773221] process > GenotypeGVCFs [ 83%] 5 of 6
[f6/c87ba1] process > ConcatVCF [100%] 1 of 1

executor > local (15)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [100%] 6 of 6 ✔
[97/127f2a] process > GenotypeGVCFs [100%] 6 of 6 ✔
[5d/19a5ac] process > ConcatVCF [ 50%] 1 of 2

executor > local (15)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [100%] 6 of 6 ✔
[97/127f2a] process > GenotypeGVCFs [100%] 6 of 6 ✔
[5d/19a5ac] process > ConcatVCF [100%] 2 of 2 ✔

executor > local (15)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [100%] 6 of 6 ✔
[97/127f2a] process > GenotypeGVCFs [100%] 6 of 6 ✔
[5d/19a5ac] process > ConcatVCF [100%] 2 of 2 ✔
Warning, pipeline completed, but with errored process(es)
Number of ignored errored process(es) : 0
Number of successfully ran process(es) : 15
WARN: [nf-core/sarek] Could not attach MultiQC report to summary email

executor > local (15)
[9d/9e292f] process > CreateIntervalBeds [100%] 1 of 1 ✔
[47/0cddc1] process > HaplotypeCaller [100%] 6 of 6 ✔
[97/127f2a] process > GenotypeGVCFs [100%] 6 of 6 ✔
[5d/19a5ac] process > ConcatVCF [100%] 2 of 2 ✔
Warning, pipeline completed, but with errored process(es)
Number of ignored errored process(es) : 0
Number of successfully ran process(es) : 15
WARN: [nf-core/sarek] Could not attach MultiQC report to summary email
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.
Completed at: 04-Jul-2019 13:02:53
Duration : 35.3s
CPU hours : 0.1
Succeeded : 15

+ [[ false == false ]]
+ rm -rf data
+ manage_logs
+ [[ -n '' ]]
+ [[ ALL,SOMATIC =~ GERMLINE ]]
+ [[ ALL,TARGETED =~ GERMLINE ]]
+ [[ false == false ]]
+ pathToSample=https://github.com/nf-core/test-datasets/raw/sarek/testdata
+ [[ ALL,ANNOTATEALL,ANNOTATESNPEFF,ANNOTATEVEP =~ GERMLINE ]]
+ [[ MULTIPLE =~ GERMLINE ]]
[Pipeline] sh
+ rm -rf data/
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Somatic)
[Pipeline] sh
+ ./scripts/run_tests.sh --test SOMATIC
+ CPUS=2
+ LOGS=
+ NXF_SINGULARITY_CACHEDIR=work/singularity/.
+ OFFLINE=false
+ PROFILE=docker
+ TEST=ALL
+ TRAVIS_BUILD_DIR=.
+ TRAVIS=false
+ VERBOSE=
+ [[ 2 -gt 0 ]]
+ key=--test
+ case $key in
+ TEST=SOMATIC
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ [[ ALL,GERMLINE =~ SOMATIC ]]
+ [[ ALL,SOMATIC =~ SOMATIC ]]
+ [[ false == false ]]
+ run_sarek --tools FreeBayes,HaplotypeCaller,Manta,Strelka,Mutect2 --noReports
+ nextflow run ./main.nf -profile test,docker --monochrome_logs --tools FreeBayes,HaplotypeCaller,Manta,Strelka,Mutect2 --noReports
N E X T F L O W ~ version 19.04.0
Launching `./main.nf` [infallible_shaw] - revision: fb7cd02724
----------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
____ _____ _
.' _ `. / ____| | |
/ |\`-_ \ | (___ ___ _ __ __ | | __
| | \ `-| \___ \/__ \| ´__/ _\| |/ /
\ | \ / ____) | __ | | | __| <
`|____\' |_____/\____|_| \__/|_|\_\

nf-core/sarek v2.5dev
----------------------------------------------------
Run Name : infallible_shaw
Max Resources : 7 GB memory, 2 cpus, 2d time per job
Container : docker - nfcore/sarek:dev
Sample : https://github.com/nf-core/test-datasets/raw/sarek/testdata/tsv/tiny-manta-https.tsv
Step : mapping
Tools : freebayes, haplotypecaller, manta, strelka, mutect2
No Reports : true
Nucleotides/s : 1000.0
Output dir : ./results
Launch dir : /share/****_home/workspace/nf-core_sarek_PR-19
Working dir : /share/****_home/workspace/nf-core_sarek_PR-19/work
Script dir : /share/****_home/workspace/nf-core_sarek_PR-19
User : ****
Config Profile : test,docker
Config Description: Minimal test dataset to check pipeline function
----------------------------------------------------
executor > local (1)
[a5/c4014e] process > CreateIntervalBeds [ 0%] 0 of 1

[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Targeted)
Stage "Targeted" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Annotation)
Stage "Annotation" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Multiple)
Stage "Multiple" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] script
[Pipeline] {
[Pipeline] sh

@maxulysse
Copy link
Member Author

Relaunched the Jenkins tests, and it worked, I have no clue why it failed the first time...

* Improve cpu usage
* Use same font for nf-core and sarek in ascii art
* Improve syntax
* Add --no-reports option for tests + add snpEff,VEP,merge to MULTIPLE test
* Use --no-reports for TravisCI testing
* Add possibility to download other genome for sareksnpeff and sarekvep containers
* Update docs
* Add --no-reports for all tests but MULTIPLE
* Use sensible label selectors for configs
* Enhance the old --noReports with --skip to skip QC on demand
* Replace --noReports by --skip all
* Update logos with new nf-core/sarek logo
* Add logo to MultiQC report
* Remove createMultiQCconfig()
* Improve docs about offline usage with bianca
* Code polishing
* Update CHANGELOG
* fix: fix logo
* fix: Fix path to reference genomes
* feat: add markdownlint config file
* fix: lint all md files + small typos corrected
* feat: update CHANGELOG
* add tiddit to container

* add process for TIDDIT

* update TIDDIT to 2.7.1

* fix merge issues

* feat: add tests for TIDDIT

* feat: update CHANGELOG

* feat: Add documentation for TIDDIT

* fix: arrange TIDDIT output

* feat: filter out low quality calls from TIDDIT

* feat: complete TIDDIT docs

* fix: syntac error, bad grep
* feat: merge markdownlint config files

* fix: more linting on md files

* feat: add md linting to travisCI tests

* feat: sort tests

* feat: improve travis testing

* feat: improve Travis tests

* feat: sort tests

* feat: use YAML anchors to improve/simplify circle ci docker deployment

* fix: remove with stroke over aple core in nf-core logo

* feat: use direct link to slack sarek channel

* fix: code spacing

* feat: remove ALL from tests and simplify/improve script

* feat: use kraken profile to run tests

* feat: remove unnecessary Channel.create()

* fix: removed links to deleted files

* feat: moved smallGRCh37 to new genomes.config

* feat: update CHANGELOG
@maxulysse maxulysse added the release This PR is for a release label Aug 16, 2019
@maxulysse maxulysse self-assigned this Aug 16, 2019
@apeltzer
Copy link
Member

Bildschirmfoto 2019-09-20 um 10 48 02

Bildschirmfoto 2019-09-20 um 10 52 03

@apeltzer
Copy link
Member

We're on it!
we're reviewing

Copy link
Member

@apeltzer apeltzer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Me after this review (not because of the quality, but because I'm tired of reviewing that much):

tenor

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
docs/containers.md Outdated Show resolved Hide resolved
docs/containers.md Outdated Show resolved Hide resolved
docs/reference.md Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
main.nf Outdated Show resolved Hide resolved
main.nf Show resolved Hide resolved
main.nf Outdated Show resolved Hide resolved
maxulysse and others added 2 commits September 24, 2019 11:36
Co-Authored-By: Alexander Peltzer <apeltzer@users.noreply.github.com>
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
Co-Authored-By: Alexander Peltzer <apeltzer@users.noreply.github.com>
@maxulysse
Copy link
Member Author

By the way, now that I finally have one circleCI job complete (even for VEP GRCh37 and GRCh38), we now have all images built for annotation, and can merge even if circleCI is not complete or failing.

Copy link
Contributor

@szilvajuhos szilvajuhos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks OK now, means I could not find a major problem in 2 hours. Would go through once extensive tests are over.

conf/genomes.config Outdated Show resolved Hide resolved
conf/igenomes.config Show resolved Hide resolved
conf/genomes.config Outdated Show resolved Hide resolved
conf/genomes.config Outdated Show resolved Hide resolved
* enforcing file names to be idsample_idRun pre merging to avoid collision
* more consistency for fastqc and bamqc output
* update CHANGELOG
* fix typo for bwaIndex in params.genomes[params.genome]
* add more logs for reference Files
* fix params.dbsnpIndex
* remove bwaIndex from smallGRCh37
* fix ch_dbsnp
* update CHANGELOG
main.nf Show resolved Hide resolved
nextflow.config Outdated Show resolved Hide resolved
docs/usage.md Show resolved Hide resolved
Copy link
Member

@alneberg alneberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really great job @maxulysse! I have done a few test runs and it looks very good. I will continue my review but if I find something it will have to wait for the next release. 👍

nextflow.config Outdated Show resolved Hide resolved
Copy link
Member

@alneberg alneberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Also removes the strange "issue PR".

@maxulysse maxulysse merged commit c5fc547 into master Oct 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release This PR is for a release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants