Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cwltool --singularity failure with `ERROR : Could not create /dev/loopX: Permission denied` #764

Closed
pvanheus opened this issue May 21, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@pvanheus
Copy link
Contributor

commented May 21, 2018

Expected Behavior

cwltool --singularity should work the same as without --singularity

(Surprisingly (to me) cwltoil version 3.15.0 works correctly despite seeming to use the same version of cwltool)

Actual Behavior

cwltool --singularity reverse/reverse.cwl input.json (CWL and input JSON below) fails with an error:
ERROR : Could not create /dev/loop10: Permission denied

Workflow Code

cwlVersion: v1.0
class: CommandLineTool

hints:
  DockerRequirement:
    dockerPull: pvanheus/reverse:latest

baseCommand: reverse.py

inputs:
  dnafile:
    type: File
    inputBinding:
      position: 1

stdout: $(inputs.dnafile.nameroot)_reversed$(inputs.dnafile.nameext)

outputs:
  rev_dnafile:
    type: stdout

with input.json:

dnafile:
  class: File
  path: data/dna.txt

where dna/dna.txt is simply a text file with the string GATACCA.

Full Traceback

$ cwltool --singularity --debug reverse/reverse.cwl input.json
/home/pvh/anaconda3/bin/cwltool 1.0.20180518123035
Resolved 'reverse/reverse.cwl' to 'file:///home/pvh/Documents/SANBI/biocourse2017/cwltutorial/reverse/reverse.cwl'
[job reverse.cwl] initializing from file:///home/pvh/Documents/SANBI/biocourse2017/cwltutorial/reverse/reverse.cwl
[job reverse.cwl] {
    "dnafile": {
        "class": "File",
        "location": "file:///home/pvh/Documents/SANBI/biocourse2017/cwltutorial/data/dna.txt",
        "size": 8,
        "basename": "dna.txt",
        "nameroot": "dna",
        "nameext": ".txt"
    }
}
[job reverse.cwl] path mappings is {
    "file:///home/pvh/Documents/SANBI/biocourse2017/cwltutorial/data/dna.txt": [
        "/home/pvh/Documents/SANBI/biocourse2017/cwltutorial/data/dna.txt",
        "/var/lib/cwl/stgf6ec8e0d-317b-4ea4-947f-39cb93d45be7/dna.txt",
        "File",
        true
    ]
}
[job reverse.cwl] command line bindings is [
    {
        "datum": "reverse.py",
        "position": [
            -1000000,
            0
        ]
    },
    {
        "position": [
            1,
            "dnafile"
        ],
        "datum": {
            "class": "File",
            "location": "file:///home/pvh/Documents/SANBI/biocourse2017/cwltutorial/data/dna.txt",
            "size": 8,
            "basename": "dna.txt",
            "nameroot": "dna",
            "nameext": ".txt",
            "path": "/var/lib/cwl/stgf6ec8e0d-317b-4ea4-947f-39cb93d45be7/dna.txt",
            "dirname": "/var/lib/cwl/stgf6ec8e0d-317b-4ea4-947f-39cb93d45be7"
        }
    }
]
Using local copy of Singularity image found in /home/pvh/Documents/SANBI/biocourse2017/cwltutorial
[job reverse.cwl] /tmp/tmp7og2_8lt$ singularity \
    --quiet \
    exec \
    --contain \
    --pid \
    --ipc \
    --userns \
    --bind \
    /tmp/tmp7og2_8lt:/var/spool/cwl:rw \
    --bind \
    /tmp/tmpzrruu5fi:/tmp:rw \
    --bind \
    /home/pvh/Documents/SANBI/biocourse2017/cwltutorial/data/dna.txt:/var/lib/cwl/stgf6ec8e0d-317b-4ea4-947f-39cb93d45be7/dna.txt:ro \
    --pwd \
    /var/spool/cwl \
    /home/pvh/Documents/SANBI/biocourse2017/cwltutorial/pvanheus-reverse-latest.img \
    reverse.py \
    /var/lib/cwl/stgf6ec8e0d-317b-4ea4-947f-39cb93d45be7/dna.txt > /tmp/tmp7og2_8lt/dna_reversed.txt
ERROR  : Could not create /dev/loop10: Permission denied
ABORT  : Retval = 255
[job reverse.cwl] completed permanentFail
[job reverse.cwl] {
    "rev_dnafile": {
        "http://commonwl.org/cwltool#generation": 0,
        "nameext": ".txt",
        "class": "File",
        "location": "file:///tmp/tmp7og2_8lt/dna_reversed.txt",
        "nameroot": "dna_reversed",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "basename": "dna_reversed.txt",
        "size": 0
    }
}
[job reverse.cwl] Removing input staging directory /tmp/tmpfrhy46i2
[job reverse.cwl] Removing temporary directory /tmp/tmpzrruu5fi
Moving /tmp/tmp7og2_8lt/dna_reversed.txt to /home/pvh/Documents/SANBI/biocourse2017/cwltutorial/dna_reversed.txt
Removing intermediate output directory /tmp/tmp7og2_8lt
{
    "rev_dnafile": {
        "class": "File",
        "path": "/home/pvh/Documents/SANBI/biocourse2017/cwltutorial/dna_reversed.txt",
        "location": "file:///home/pvh/Documents/SANBI/biocourse2017/cwltutorial/dna_reversed.txt",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "basename": "dna_reversed.txt",
        "size": 0
    }
}
Final process status is permanentFail

Your Environment

  • cwltool version: 1.0.20180518123035
  • singularity version: 2.5.1-dist
  • operating system version: Ubuntu 16.04.4 LTS
  • kernel version: 4.4.0-124-generic #148-Ubuntu
@pvanheus

This comment has been minimized.

Copy link
Contributor Author

commented May 25, 2018

The problems turns out to be an underlying singularity one, as illustrated by this gist. When run with --userns singularity is running through the list of loopback devices and hitting permission denied on each. The test in https://github.com/common-workflow-language/cwltool/blob/master/cwltool/singularity.py#L26 but the command run singularity exec --userns /etc true succeeds because it doesn't try and mount an image. The failure that causes this issue can be illustrated using a simple test:

singularity pull docker://hello-world && singularity exec --userns hello-world.simg - the "hello-world.simg" is small, only 4K, so perhaps it can be included in the code for testing purposes (the imagine can be built from this Dockerfile).

@mr-c

This comment has been minimized.

Copy link
Member

commented May 25, 2018

Thank you @pvanheus for tracking this down. 4k is small enough for me; can you send a PR?

@pvanheus

This comment has been minimized.

Copy link
Contributor Author

commented May 25, 2018

Thanks @mr-c, see #771

@mr-c

This comment has been minimized.

Copy link
Member

commented Jun 27, 2018

Fixed in #771

@mr-c mr-c closed this Jun 27, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.