Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test data - 2 #5

Closed
denis-yuen opened this issue Sep 2, 2016 · 5 comments
Closed

test data - 2 #5

denis-yuen opened this issue Sep 2, 2016 · 5 comments
Assignees
@briandoconnor

Comments

@denis-yuen
Copy link
Member

Currently attempting to execute the workflow with the test data on the branch https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker/tree/feature/cwl1

Command is

dockstore tool launch --entry Dockstore.cwl --local-entry --json test2.json

which becomes

[job Dockstore.cwl] /tmp/tmp510v1N$ docker \                                                                                                                                                             [1536/1663]
    run \
    -i \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/ef2387e1-00c2-497e-9bae-19d833273d70/9f8f6f03df6d100c90a761164800a42c.bam:/var/lib/cwl/stg86779b72-335
2-4adf-8478-98e3853b440f/9f8f6f03df6d100c90a761164800a42c.bam:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/196da89e-6d8a-49af-9ab2-70d302816e29/GRCh37d5_battenberg.tar.gz:/var/lib/cwl/stg6f32c8f1-139e-43cd-b94
7-427fdd812590/GRCh37d5_battenberg.tar.gz:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/ef2387e1-00c2-497e-9bae-19d833273d70/9f8f6f03df6d100c90a761164800a42c.bam.bai:/var/lib/cwl/stg86779b72
-3352-4adf-8478-98e3853b440f/9f8f6f03df6d100c90a761164800a42c.bam.bai:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/ca7276bf-10fa-45b8-9634-1b6d04270b99/ad4aeedc224c6be5e21e26c038a43df4.bam.bai:/var/lib/cwl/stg78f59412
-5cf5-4473-b80e-1720be3192c4/ad4aeedc224c6be5e21e26c038a43df4.bam.bai:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/ca7276bf-10fa-45b8-9634-1b6d04270b99/ad4aeedc224c6be5e21e26c038a43df4.bam:/var/lib/cwl/stg78f59412-5cf
5-4473-b80e-1720be3192c4/ad4aeedc224c6be5e21e26c038a43df4.bam:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/e1b2a106-3fe8-4e2b-9159-33c5ce9c95d6/GRCh37d5_CGP_refBundle.tar.gz:/var/lib/cwl/stg4d74f530-1cd7-4de3-
835f-7de7820a2dab/GRCh37d5_CGP_refBundle.tar.gz:ro \
    --volume=/tmp/tmp510v1N:/var/spool/cwl:rw \
    --volume=/home/ubuntu/CGP-Somatic-Docker/datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/workinguyrgqy:/tmp:rw \
    --workdir=/var/spool/cwl \
    --read-only=true \
    --user=1000 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/var/spool/cwl \
    quay.io/pancancer/pcawg-sanger-cgp-workflow:feature_cwl1 \
    python \
    /home/seqware/CGP-Somatic-Docker/scripts/run_seqware_workflow.py \
    --tumor \
    /var/lib/cwl/stg78f59412-5cf5-4473-b80e-1720be3192c4/ad4aeedc224c6be5e21e26c038a43df4.bam \
    --normal \
    /var/lib/cwl/stg86779b72-3352-4adf-8478-98e3853b440f/9f8f6f03df6d100c90a761164800a42c.bam \
    --refFrom \
    /var/lib/cwl/stg4d74f530-1cd7-4de3-835f-7de7820a2dab/GRCh37d5_CGP_refBundle.tar.gz \
    --bbFrom \
    /var/lib/cwl/stg6f32c8f1-139e-43cd-b947-427fdd812590/GRCh37d5_battenberg.tar.gz

Looking at
https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker/blob/develop/Dockstore_cwl.json

corresponds to

https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker/blob/feature/cwl1/test2.json

Currently fails with the following messages

[2016/09/02 11:59:54] | Row #7 , Attempt #1 out of 1 : s58_ASCAT_ascat_62,s58_BRASS_cover_101,s58_cgpPindel_pindel_71,s58_renameSampleFile_213,s58_renameSampleFile_214
[2016/09/02 11:59:54] |         Submitting 26480M batch with: s58_ASCAT_ascat_62,s58_BRASS_cover_101,s58_cgpPindel_pindel_71,s58_renameSampleFile_213,s58_renameSampleFile_214
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_renameSampleFile_213-runner.sh
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_BRASS_cover_101-runner.sh
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_renameSampleFile_214-runner.sh
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_ASCAT_ascat_62-runner.sh
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_cgpPindel_pindel_71-runner.sh
[2016/09/02 11:59:59] |         Workflow step succeeded: s58_renameSampleFile_214
[2016/09/02 12:00:15] |         Workflow step succeeded: s58_renameSampleFile_213
[2016/09/02 12:38:13] |         Workflow step failed: s58_ASCAT_ascat_62
[2016/09/02 13:37:31] |         Workflow step succeeded: s58_BRASS_cover_101
[2016/09/02 15:22:18] |         Workflow step succeeded: s58_cgpPindel_pindel_71
[2016/09/02 15:22:18] | Setting workflow-run status to failed for: 58
[--plugin, io.seqware.pipeline.plugins.WorkflowWatcher, --, --workflow-run-accession, 58]
Found local reference archive: /var/lib/cwl/stg4d74f530-1cd7-4de3-835f-7de7820a2dab/GRCh37d5_CGP_refBundle.tar.gz
Found local reference archive: /var/lib/cwl/stg6f32c8f1-139e-43cd-b947-427fdd812590/GRCh37d5_battenberg.tar.gz
Workflow run 58 is currently failed


Job Name: s58_ASCAT_ascat_62
Job ID:   817594229
File:     /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_ASCAT_ascat_62.o817594229
Updated:  Fri Sep 02 12:38:13 UTC 2016
Contents Excerpt:
Calls: ascat.plotRawData -> png
In addition: Warning message:
In png(filename = paste(ASCATobj$samples[i], ".tumour.png", sep = ""),  :
  unable to open connection to X11 display ''
Execution halted
"cd /var/spool/cwl/0/ascat/tmpAscat/ascat; /opt/wtsi-cgp/bin/Rscript /opt/wtsi-cgp/lib/perl5/auto/share/module/Sanger-CGP-Ascat-Implement/ascat/runASCAT.R /opt/wtsi-cgp/lib/perl5/auto/share/module/Sanger-CGP-Asca
t-Implement/ascat /var/spool/cwl/reference_files/ascat/SnpPositions.tsv /var/spool/cwl/reference_files/ascat/SnpGcCorrections.tsv fc969f27-bdc3-cd03-e040-11ac0c48577e fc969f27-bdc3-cd03-e040-11ac0c48577e.count fc
969f27-bdc6-cd03-e040-11ac0c48577e fc969f27-bdc6-cd03-e040-11ac0c48577e.count XX /var/spool/cwl/0/ascat/tmpAscat/ascat/fc969f27-bdc3-cd03-e040-11ac0c48577e.Rdata" unexpectedly returned exit value 1 at (eval 294) 
line 13.
 at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 190
Command exited with non-zero status 2
597.47user 7.46system 38:15.68elapsed 26%CPU (0avgtext+0avgdata 6048784maxresident)k
662816inputs+209168outputs (130major+435707minor)pagefaults 0swaps
Calls: ascat.plotRawData -> png
In addition: Warning message:
In png(filename = paste(ASCATobj$samples[i], ".tumour.png", sep = ""),  :
  unable to open connection to X11 display ''
Execution halted
"cd /var/spool/cwl/0/ascat/tmpAscat/ascat; /opt/wtsi-cgp/bin/Rscript /opt/wtsi-cgp/lib/perl5/auto/share/module/Sanger-CGP-Ascat-Implement/ascat/runASCAT.R /opt/wtsi-cgp/lib/perl5/auto/share/module/Sanger-CGP-Asca
t-Implement/ascat /var/spool/cwl/reference_files/ascat/SnpPositions.tsv /var/spool/cwl/reference_files/ascat/SnpGcCorrections.tsv fc969f27-bdc3-cd03-e040-11ac0c48577e fc969f27-bdc3-cd03-e040-11ac0c48577e.count fc
969f27-bdc6-cd03-e040-11ac0c48577e fc969f27-bdc6-cd03-e040-11ac0c48577e.count XX /var/spool/cwl/0/ascat/tmpAscat/ascat/fc969f27-bdc3-cd03-e040-11ac0c48577e.Rdata" unexpectedly returned exit value 1 at (eval 294) 
line 13.
 at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 190
Command exited with non-zero status 2
597.47user 7.46system 38:15.68elapsed 26%CPU (0avgtext+0avgdata 6048784maxresident)k
662816inputs+209168outputs (130major+435707minor)pagefaults 0swaps
The method 'do_run' exited abnormally so the Runner will terminate here!
Return value was: 1

Full output: /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_ASCAT_ascat_62.stdout
@denis-yuen
Copy link
Member Author

denis-yuen commented Sep 2, 2016 •
edited

Warning, this takes around 24 hours to run to failure and a pretty beefy machine

@adamstruck
Copy link
Contributor

I think I have encountered this error before when I was developing the WDL version... It occurred when the .Rprofile file wasn't being loaded by R because I had changed the home directory.

@denis-yuen
Copy link
Member Author

I wonder if this means the --user=1000 parameter that cwltool adds to the docker run invocation is messing with something. I'll poke around.

This was referenced Sep 2, 2016
Closed
Merged
@adamstruck
Copy link
Contributor

what was the resolution to the issue?

@denis-yuen
Copy link
Member Author

It turns out cwltool replaces the contents of the working directory when running a docker container. Since that is where the .Rprofile file lives, it gets wiped out resulting in the error above. So we simply create that file anew in the working directory which seems to do the trick. Using a bigger and beefier machine handled the other issue which I suspect was just an out of space error that one of the underlying tools reacted to as a division by zero error.

Otherwise, the rest of the changes are just minor syntax changes for CWL 1.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@briandoconnor briandoconnor

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@briandoconnor @denis-yuen @adamstruck