WIP Fix resolver for imports

[denis-yuen]

• resolve imports properly for TRS

test with commands like the following for command-line tools (previously working)

$ cat test.json
{
  "input_file": {
        "class": "File",
        "path": "md5sum.input"
    }
}
(venv) $ cwltool --make-template quay.io/briandoconnor/dockstore-tool-md5sum:1.0.4 test.json
/home/dyuen/cwltool/venv/bin/cwltool 1.0.20190323055837
Resolved 'quay.io/briandoconnor/dockstore-tool-md5sum:1.0.4' to 'https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-md5sum/versions/1.0.4/plain-CWL/descriptor/Dockstore.cwl'
input_file:  # type "File"
    class: File
    path: a/file/path
(venv) $ cwltool quay.io/briandoconnor/dockstore-tool-md5sum:1.0.4 test.json
/home/dyuen/cwltool/venv/bin/cwltool 1.0.20190323055837
Resolved 'quay.io/briandoconnor/dockstore-tool-md5sum:1.0.4' to 'https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-md5sum/versions/1.0.4/plain-CWL/descriptor/Dockstore.cwl'
[job Dockstore.cwl] Skipping Docker software container '--memory' limit despite presence of ResourceRequirement with ramMin and/or ramMax setting. Consider running with --strict-memory-limit for increased portability assurance.
... (snip)
Final process status is success

and for workflows (what was broken) since workflows have multiple files (e.g. https://dockstore.org/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fdockstore-testing%252Fmd5sum-checker/versions/develop/CWL/files )

$ cwltool --make-template \#workflow/github.com/dockstore-testing/md5sum-checker:develop
/home/dyuen/cwltool/venv/bin/cwltool 1.0.20190323055837
Resolved '#workflow/github.com/dockstore-testing/md5sum-checker:develop' to 'https://dockstore.org/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fdockstore-testing%2Fmd5sum-checker/versions/develop/plain-CWL/descriptor/md5sum-workflow.cwl'
input_file:  # type "File"
    class: File
    path: a/file/path
(venv) $ cwltool \#workflow/github.com/dockstore-testing/md5sum-checker:develop test.json
/home/dyuen/cwltool/venv/bin/cwltool 1.0.20190323055837
Resolved '#workflow/github.com/dockstore-testing/md5sum-checker:develop' to 'https://dockstore.org/api/api/ga4gh/v2/tools/%23workflow%2Fgithub.com%2Fdockstore-testing%2Fmd5sum-checker/versions/develop/plain-CWL/descriptor/md5sum-workflow.cwl'
[workflow ] start
... (snip)
Final process status is success

[mr-c]

Thank you @denis-yuen! Can we get a test for the workflow case?

[denis-yuen]

@mr-c I'll give it a shot, but I may need help (not very familiar with Python)
Do you know of a good example test I can work off of? And does this repo normally use mocking for external resources (like a TRS) ?

[codecov]

Codecov Report

Merging #1093 into master will increase coverage by 0.24%.
The diff coverage is 89.47%.

@@            Coverage Diff             @@
##           master    #1093      +/-   ##
==========================================
+ Coverage   79.68%   79.92%   +0.24%     
==========================================
  Files          30       30              
  Lines        6137     6152      +15     
  Branches     1534     1537       +3     
==========================================
+ Hits         4890     4917      +27     
+ Misses        870      854      -16     
- Partials      377      381       +4

Impacted Files	Coverage Δ
cwltool/resolver.py	77.04% <89.47%> (+33.57%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e9f82a...4f1572d. Read the comment docs.

[mr-c]

Thanks for the integration tests, though judging from Travis CI they either don't work with Py3.4 & 3.5 (unlikely?) or are flaky.

Maybe refactor so that the network query is pulled out from the main logic so it can be mocked out in unit tests?

[denis-yuen]

@mr-c
I'm thinking flaky.

[mr-c]

The test is failing under MS Windows: https://ci.appveyor.com/project/mr-c/cwltool/builds/23430620/job/908j222j6snpasj9/tests

[denis-yuen]

For reviewers, it looks like remaining errors are false positives since they occur on a different branch too https://ci.commonwl.org/job/cwltool-pr-conformance-multiver/2400/

[tetron]

jenkins, test this please