bidirectional mapping of inputs and outputs between original WDL and dx

[notestaff]

For the purpose of comparing the same workflow run locally on Cromwell, vs remotely on dx, one needs to know the correspondence of input and output names . I know dxWDL has the -inputs option. But (1) it cannot be run without creating new files on dx; (2) it does not intermediate or final outputs. Would it be possible to add an option to output a file giving the full correspondence?

[orodeh]

Could you give an example?

[notestaff]

If you have a dx analysis that used a WDL workflow compiled by dxWDL, and now want to re-run it locally (maybe with some parameter change), what would be the steps? I have written some ad-hoc code to convert the json output from dx describe analysis-xxxxx to corresponding Cromwell input file, but it e.g. relies on knowing that stage-0/common has the workflow-level arguments. It would be better if dxWDL had more direct support for doing this.

[orodeh]

I see what you mean. It is somewhat like the --input flag which takes a JSON file of Cromwell inputs, but for workflow outputs. It is an interesting enhancement idea. I can think about it for the next compiler version.

[notestaff]

@orodeh Thanks for considering it. Most ideally, dxWDL would provide a command to map the output of 'dx describe --json analysis-xxxxxx' to the corresponding metadata.json from Cromwell, and vice versa, for a workflow compiled by dxWDL. I've been doing it using heuristics, but that of course is fragile as it relies on assumptions about dxWDL's inner workings.

[jdidion]

I have actually already implemented this in pytest-wdl, in the dxWDL executor. I will write a stand-alone command-line tool that uses that code. Later we can look at re-implementing it in dxWDL.

[notestaff]

Great, thanks!

One other possible place to put this info, is in the details field, where womSourceCode now goes; or add it as workflow metadata to the WDL stored in womSourceCode. Then the mapping will be retrievable even if later versions of dxWDL change how mapping is done.

[jdidion]

I have started on this here: https://github.com/dnanexus/dxWDL/tree/feat/191-input-mapping/contrib/io_mapping. For now it is a separate tool written in python.

[jdidion]

The tool now works for booth input mapping (cromwell -> DNAnexus) and output mapping (DNAnexus -> cromwell). @notestaff please test.