Test [99/128] Test secondaryFiles on array of files

[tiborsimko]

Following up the recent changes about absolute path file upload, there is one CWL test case that fails:

Test [99/128] Test secondaryFiles on array of files.
Test failed: /home/simko/.virtualenvs/reana-client-py27/bin/reana-cwl-runner --outdir=/tmp/tmpNhdiFC --quiet v1.0/docker-array-secondaryfiles.cwl v1.0/docker-array-secondaryfiles-job.json
Test secondaryFiles on array of files.
Returned non-zero
/home/simko/.virtualenvs/reana-client-py27/bin/reana-cwl-runner 1.0.20180403145700
INFO:cwltool:/home/simko/.virtualenvs/reana-client-py27/bin/reana-cwl-runner 1.0.20180403145700
Resolved 'v1.0/docker-array-secondaryfiles.cwl' to 'file:///home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0/v1.0/docker-array-secondaryfiles.cwl'
INFO:cwltool:Resolved 'v1.0/docker-array-secondaryfiles.cwl' to 'file:///home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0/v1.0/docker-array-secondaryfiles.cwl'
ERROR:root:{u'message': u'Workflow workspace created', u'workflow_id': u'c668a2e0-010f-4b69-902b-8b4948868204', u'workflow_name': u'cwl-runner.99'}
ERROR:root:{u'message': u'File successfully transferred'}
ERROR:root:Transferred file: /home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0/v1.0/ref.fasta.fai
ERROR:root:{u'message': u'File successfully transferred'}
ERROR:root:Transferred file: /home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0/v1.0/ref2.fasta.fai
ERROR:root:{u'message': u'File successfully transferred'}
ERROR:root:Transferred file: /home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0/v1.0/ref.fasta
ERROR:root:{u'message': u'File successfully transferred'}
ERROR:root:Transferred file: /home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0/v1.0/ref2.fasta
ERROR:root:{u'status': u'running', u'workflow_name': u'cwl-runner.99', u'workflow_id': u'c668a2e0-010f-4b69-902b-8b4948868204', u'user': u'00000000-0000-0000-0000-000000000000', u'organization': u'default', u'message': u'Workflow successfully launched'}
ERROR:root:Polling workflow logs
ERROR:root:/usr/local/bin/celery 0.0.1.dev20170912 with cwltool 1.0.20180326152342Search path is [u'/data/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/workflow.json#main', u'/root/.local/share/commonwl/workflow.json#main', u'/usr/local/share/commonwl/workflow.json#main', u'/usr/share/commonwl/workflow.json#main']Resolved 'workflow.json#main' to 'file:///data/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/workflow.json#main'
ERROR:root:Polling workflow logs
ERROR:root:[job workflow.json_49] initializing from file:///data/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/workflow.json[job workflow.json_49] {
    "fasta_path": [
        {
            "class": "File",
            "location": "file:///data/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/ref.fasta",
            "size": 12010,
            "basename": "ref.fasta",
            "nameroot": "ref",
            "nameext": ".fasta"
        },
        {
            "class": "File",
            "location": "file:///data/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/ref2.fasta",
            "size": 12010,
            "basename": "ref2.fasta",
            "nameroot": "ref2",
            "nameext": ".fasta"
        }
    ]
}Workflow error, try again with --debug for more information:
inputs.json:1:35: Traceback (most recent call last):

                    File "/usr/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 271,
                    in visit
                      self.visitlisting(obj.get("secondaryFiles", []), stagedir, basedir,
                      copy=copy, staged=staged)

                    File "/usr/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 234,
                    in visitlisting
                      self.visit(ld, stagedir, basedir, copy=ld.get("writable", copy),
                      staged=staged)

                    File "/usr/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 271,
                    in visit
                      self.visitlisting(obj.get("secondaryFiles", []), stagedir, basedir,
                      copy=copy, staged=staged)

                    File "/usr/local/lib/python2.7/site-packages/schema_salad/sourceline.py", line
                    155, in __exit__
                      raise self.makeError("\n".join(traceback.format_exception(exc_type,
                      exc_value, tb)))

                  ValidationException: Traceback (most recent call last):

                    File "/usr/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 263,
                    in visit
                      st = os.lstat(deref)

                  OSError: [Errno 2] No such file or directory:
                  '/data/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/ref.fasta.fai'

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cwltool/main.py", line 602, in main
    **vars(args))
  File "/code/reana_workflow_engine_cwl/pipeline.py", line 93, in executor
    raise WorkflowException(str(e))
WorkflowException: inputs.json:1:35: Traceback (most recent call last):
inputs.json:1:35:
inputs.json:1:35:   File "/usr/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 271, in visit
inputs.json:1:35:     self.visitlisting(obj.get("secondaryFiles", []), stagedir, basedir, copy=copy, staged=staged)
inputs.json:1:35:
inputs.json:1:35:   File "/usr/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 234, in visitlisting
inputs.json:1:35:     self.visit(ld, stagedir, basedir, copy=ld.get("writable", copy), staged=staged)
inputs.json:1:35:
inputs.json:1:35:   File "/usr/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 271, in visit
inputs.json:1:35:     self.visitlisting(obj.get("secondaryFiles", []), stagedir, basedir, copy=copy, staged=staged)
inputs.json:1:35:
inputs.json:1:35:   File "/usr/local/lib/python2.7/site-packages/schema_salad/sourceline.py", line 155, in __exit__
inputs.json:1:35:     raise self.makeError("\n".join(traceback.format_exception(exc_type, exc_value, tb)))
inputs.json:1:35:
inputs.json:1:35: ValidationException: Traceback (most recent call last):
inputs.json:1:35:
inputs.json:1:35:   File "/usr/local/lib/python2.7/site-packages/cwltool/pathmapper.py", line 263, in visit
inputs.json:1:35:     st = os.lstat(deref)
inputs.json:1:35:
inputs.json:1:35: OSError: [Errno 2] No such file or directory: '/data/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/ref.fasta.fai'
inputs.json:1:35:
ERROR:root:Workflow execution failed

Indeed:

$ find /reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/code
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/outputs
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/inputs.json
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/workflow.json
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/ref2.fasta
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/ref.fasta
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/home
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/home/simko
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/home/simko/.virtualenvs
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/home/simko/.virtualenvs/reana-client-py27
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/home/simko/.virtualenvs/reana-client-py27/common-workflow-language
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0/v1.0
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0/v1.0/ref2.fasta.fai
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/inputs/home/simko/.virtualenvs/reana-client-py27/common-workflow-language/v1.0/v1.0/ref.fasta.fai
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/workspace
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/workspace/cwl
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/workspace/cwl/outdir
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/workspace/cwl/outdir/IfH1w4
/reana/default/00000000-0000-0000-0000-000000000000/analyses/c668a2e0-010f-4b69-902b-8b4948868204/workspace/cwl/tmpdir

Note the location of ref.fasta.fai...

The other tests seem to be passing still OK.

Let's see whether we can fix this quickly?