Abstract operation

[stain]

Implements common-workflow-language/common-workflow-language#337 for abstract operations.

I've done this as v1.2.0-dev2 to avoid conflict with @kaushik-work on conditionals - in a sense this one should be easier to merge in as Operation is a new class.

Adds a new definition for Operation - basically like CommandLineTool with inputs and outputs - but no run

(TODO: should it permit dummy run and steps which are ignored? I think requirements/hints already would work)

Example standalone use in operation.cwl:

class: Operation
cwlVersion: v1.2.0-dev2
# are very important.
id: reverse
doc: "Reverse each line"
inputs:
  input:
    type: File
outputs:
  output:
    type: File

Example inline use (perhaps more typical) where the last step is not yet implemented in revsort-abstract.cwl:

#
# This is a variant of revsort.cwl where the "sorted"
# step has not yet been implemented, but has a placeholder
# Operation.
#
class: Workflow
doc: "Reverse the lines in a document, then sort those lines."
cwlVersion: v1.2.0-dev2

inputs:
  input:
    type: File
    doc: "The input file to be processed."
  reverse_sort:
    type: boolean
    default: true
    doc: "If true, reverse (decending) sort"

outputs:
  output:
    type: File
    outputSource: sort/sorted
    doc: "The output with the lines reversed and sorted."

steps:
  rev:
    in:
      input: input
    out: [output]
    run: revtool.cwl

  sort:
    in:
      input: rev/output
      reverse: reverse_sort
    out: [sorted]
    run:
      class: Operation
      id: "sort"
      doc: "Sort the lines of the file"
      inputs:
        input:
          type: File
          doc: "The input file to be sorted."
        reverse:
          type: boolean
          default: true
          doc: "If true, reverse (decending) sort"
      outputs:
        sorted:
          type: File
          doc: "The sorted file"

I added support in cwltool now on corresponding abstract-operation branch, which just throws warning and returns null - there is no way to bind the Operation to a CommandLineTool at a moment, but that was raised in common-workflow-language/common-workflow-language#337 as desired.

(cwltool) stain@biggie:~/src/cwltool/cwltool/schemas/v1.2.0-dev2/tests$ cwltool --enable-dev --validate  revsort-abstract.cwl 
INFO /home/stain/miniconda3/envs/cwltool/bin/cwltool 1.0.20191119100023
INFO Resolved 'revsort-abstract.cwl' to 'file:///home/stain/src/cwltool/cwltool/schemas/v1.2.0-dev2/tests/revsort-abstract.cwl'
revsort-abstract.cwl is valid CWL.

(cwltool) stain@biggie:~/src/cwltool/cwltool/schemas/v1.2.0-dev2/tests$ cwltool --enable-dev  revsort-abstract.cwl revsort-job.json 
INFO /home/stain/miniconda3/envs/cwltool/bin/cwltool 1.0.20191119100023
INFO Resolved 'revsort-abstract.cwl' to 'file:///home/stain/src/cwltool/cwltool/schemas/v1.2.0-dev2/tests/revsort-abstract.cwl'
INFO [workflow ] start
INFO [workflow ] starting step rev
INFO [step rev] start
INFO [job rev] /tmp/y3mfib74$ rev \
    /tmp/tmpfuwkei1t/stgee4b18d8-1410-49b7-b08e-4ed3e5206cc6/whale.txt > /tmp/y3mfib74/output.txt
INFO [job rev] completed success
INFO [step rev] completed success
INFO [workflow ] starting step sort
INFO [step sort] start
WARNING Can't execute abstract Operation file:///home/stain/src/cwltool/cwltool/schemas/v1.2.0-dev2/tests/revsort-abstract.cwl#sort/run/sort
WARNING [step sort] completed permanentFail
INFO [workflow ] completed permanentFail
{
    "output": null
}
WARNING Final process status is permanentFail
(cwltool) stain@biggie:~/src/cwltool/cwltool/sc

Crucially both cwltool --validate and cwltool --print-rdf works - so CWL Viewer would be able to render these workflows. In addition, the validation will still complain if the outputs are not wired correctly in the rest of the workflow.

[stain]

Can we merge this? Is v1.2.0-dev2 ok?

[tetron]

I'm working on 1.2 this week so let me give it a review

[tetron]

Rebased on master. Reverted the -dev2 version to -dev1 to avoid the need for a migration.