Tool fails with "path is outside the designated output directory" if input has secondaryFiles and using InitialWorkDirRequirement

[michael-kotliar]

Expected Behavior

When staging input file to the output directory the secondaryFiles should be also staged into the same directory, otherwise tools fails to execute with "path is outside the designated output directory" error when mounting files to the docker container.

Actual Behavior

Fails with "path is outside the designated output directory"

Workflow Code

Failure example

git clone https://github.com/Barski-lab/workflows.git --recursive
cd workflows
git checkout c2591a0d550001f12ecc97c1d54f3659e20eff82
cd ..
cwltool --debug ./workflows/tools/samtools-rmdup.cwl ./workflows/tests/samtools-rmdup-1.json

Successful example

git clone https://github.com/Barski-lab/workflows.git --recursive
cd workflows
git checkout c2591a0d550001f12ecc97c1d54f3659e20eff82
cd ..
cwltool --debug ./workflows/tools/samtools-rmdup.cwl ./workflows/tests/samtools-rmdup-2.json

Full Traceback

cwltool --debug ~/workspaces/cwl_ws/workflows/tools/samtools-rmdup.cwl ~/workspaces/cwl_ws/workflows/tests/samtools-rmdup-1.json 
INFO /Users/tester/venv/cwl_airflow_devel/bin/cwltool 3.0.20200807132242
INFO Resolved '/Users/tester/workspaces/cwl_ws/workflows/tools/samtools-rmdup.cwl' to 'file:///Users/tester/workspaces/cwl_ws/workflows/tools/samtools-rmdup.cwl'
WARNING http://schema.org/docs/!DOCTYPE html does not look like a valid URI, trying to serialize this will break.
WARNING html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" does not look like a valid URI, trying to serialize this will break.
/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/rdflib/plugins/parsers/structureddata.py:29: UserWarning: html5lib not found! RDFa and Microdata parsers will not be available.
  'html5lib not found! RDFa and Microdata ' +
Could not load extension schema http://schema.org/docs/schema_org_rdfa.html: html5lib is not installed, cannot use RDFa and Microdata parsers.
DEBUG [job samtools-rmdup.cwl] initializing from file:///Users/tester/workspaces/cwl_ws/workflows/tools/samtools-rmdup.cwl
DEBUG [job samtools-rmdup.cwl] {
    "bam_file": {
        "class": "File",
        "location": "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam",
        "secondaryFiles": [
            {
                "class": "File",
                "location": "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai",
                "size": 1368,
                "basename": "chr4_100_mapped_reads.Aligned.out.bam.bai",
                "nameroot": "chr4_100_mapped_reads.Aligned.out.bam",
                "nameext": ".bai"
            }
        ],
        "size": 11034,
        "basename": "chr4_100_mapped_reads.Aligned.out.bam",
        "nameroot": "chr4_100_mapped_reads.Aligned.out",
        "nameext": ".bam"
    },
    "single_end": false,
    "bash_script": "#!/bin/bash\nif [ \"$0\" = \"true\" ]\nthen\n  samtools rmdup \"${@:1}\"\nelse\n  echo \"Skip samtools rmdup \" ${@:1}\n  mv ${@: -2}\nfi\n",
    "output_filename": "",
    "trigger": true,
    "force_single_end": null
}
DEBUG [job samtools-rmdup.cwl] path mappings is {
    "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam": [
        "/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam",
        "/VUCcrH/input_file_backup",
        "WritableFile",
        false
    ],
    "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai": [
        "/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai",
        "/VUCcrH/chr4_100_mapped_reads.Aligned.out.bam.bai",
        "File",
        false
    ]
}
DEBUG [job samtools-rmdup.cwl] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "bash"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "-c"
    },
    {
        "position": [
            1,
            "bash_script"
        ],
        "datum": "#!/bin/bash\nif [ \"$0\" = \"true\" ]\nthen\n  samtools rmdup \"${@:1}\"\nelse\n  echo \"Skip samtools rmdup \" ${@:1}\n  mv ${@: -2}\nfi\n"
    },
    {
        "position": [
            2,
            "trigger"
        ],
        "valueFrom": "${ return self ? \"true\" : \"false\" }\n",
        "datum": true
    },
    {
        "position": [
            3,
            "single_end"
        ],
        "prefix": "-s",
        "datum": false
    },
    {
        "position": [
            10,
            "bam_file"
        ],
        "datum": {
            "class": "File",
            "location": "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam",
            "secondaryFiles": [
                {
                    "class": "File",
                    "location": "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai",
                    "size": 1368,
                    "basename": "chr4_100_mapped_reads.Aligned.out.bam.bai",
                    "nameroot": "chr4_100_mapped_reads.Aligned.out.bam",
                    "nameext": ".bai",
                    "path": "/VUCcrH/chr4_100_mapped_reads.Aligned.out.bam.bai",
                    "dirname": "/VUCcrH"
                }
            ],
            "size": 11034,
            "basename": "input_file_backup",
            "nameroot": "input_file_backup",
            "nameext": "",
            "path": "/VUCcrH/input_file_backup",
            "dirname": "/VUCcrH"
        }
    },
    {
        "position": [
            11,
            "output_filename"
        ],
        "valueFrom": "${\n    if (self == \"\" || inputs.trigger == false){\n      return default_output_filename();\n    } else {\n      return self;\n    }\n}\n",
        "datum": ""
    },
    {
        "valueFrom": "${\n  if (inputs.output_filename == \"\" || inputs.trigger == false){\n    return \" > \" + default_output_filename().split('.').slice(0,-1).join('.') + \"_samtools_rmdup_report.txt 2>&1\";\n  } else {\n    return \" > \" + inputs.output_filename.split('.').slice(0,-1).join('.') + \"_samtools_rmdup_report.txt 2>&1\";\n  }\n}\n",
        "position": [
            100000,
            0
        ],
        "shellQuote": false
    }
]
DEBUG [job samtools-rmdup.cwl] initial work dir {
    "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam": [
        "/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam",
        "/VUCcrH/input_file_backup",
        "WritableFile",
        true
    ],
    "file:///Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai": [
        "/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai",
        "/var/lib/cwl/stg4f8de60a-bcda-4f9e-9d35-e83ba82755c1/chr4_100_mapped_reads.Aligned.out.bam.bai",
        "WritableFile",
        true
    ]
}
ERROR Workflow error:
No mandatory DockerRequirement, yet path is outside the designated output directory, also know as $(runtime.outdir): MapperEnt(resolved='/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai', target='/var/lib/cwl/stg4f8de60a-bcda-4f9e-9d35-e83ba82755c1/chr4_100_mapped_reads.Aligned.out.bam.bai', type='WritableFile', staged=True)
Traceback (most recent call last):
  File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/main.py", line 1132, in main
    tool, initialized_job_order_object, runtimeContext, logger=_logger
  File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/executors.py", line 59, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
  File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/executors.py", line 150, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/executors.py", line 257, in run_jobs
    job.run(runtime_context)
  File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/job.py", line 828, in run
    (runtime, cidfile) = self.create_runtime(env, runtimeContext)
  File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/docker.py", line 373, in create_runtime
    tmpdir_prefix=runtimeContext.tmpdir_prefix,
  File "/Users/tester/venv/cwl_airflow_devel/lib/python3.7/site-packages/cwltool/job.py", line 700, in add_volumes
    "$(runtime.outdir): {}".format(vol)
cwltool.errors.WorkflowException: No mandatory DockerRequirement, yet path is outside the designated output directory, also know as $(runtime.outdir): MapperEnt(resolved='/Users/tester/workspaces/cwl_ws/workflows/tests/data/dm3_chr4/generated_inputs/bam_bai/chr4_100_mapped_reads.Aligned.out.bam.bai', target='/var/lib/cwl/stg4f8de60a-bcda-4f9e-9d35-e83ba82755c1/chr4_100_mapped_reads.Aligned.out.bam.bai', type='WritableFile', staged=True)

Your Environment

cwltool==3.0.20200807132242
schema-salad==7.0.20200811075006

[michael-kotliar]

It seems to be broken starting from cwltool==3.0.20200720165847

[michael-kotliar]

More specifically this commit
eba8091

[pcodyhe]

I also experienced this error.

[mr-c]

Possible problematic line: https://github.com/Barski-lab/workflows/blob/c2591a0d550001f12ecc97c1d54f3659e20eff82/tools/samtools-rmdup.cwl#L117 (use of a fully qualified path for the secondaryfile, but the container is incorrect)

[michael-kotliar]

For cwltool==3.1.20210628163208 it still fails with the same error, however, if I put docker in the requirements section instead of hints, it works fine :)