New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cwl: run full conformance test suite #9

Closed
tiborsimko opened this Issue Oct 10, 2017 · 3 comments

Comments

Projects
None yet
4 participants
@tiborsimko
Member

tiborsimko commented Oct 10, 2017

Once we have the first implementation ready, see CWL-First-Example milestone, it would be good to run the full CWL conformance test suite containing ~49 tests.

@tiborsimko tiborsimko added this to the Someday milestone Oct 10, 2017

@mr-c

This comment has been minimized.

Show comment
Hide comment
@mr-c

mr-c Oct 10, 2017

Member

How to run the conformance tests:

  1. Provide a program with the cwl-runner interface (perhaps reana-cwl-runner?): https://github.com/common-workflow-language/common-workflow-language/pull/278/files#diff-ee814a9c027fc9750beb075c283a973c

  2. Run https://github.com/common-workflow-language/common-workflow-language/blob/master/run_test.sh like ./run_test.sh RUNNER=reana-cwl-runner

  3. Tell me about it and I'll add an a Jenkins job to http://ci.commonwl.org/ & a badge to http://www.commonwl.org/#Implementations

Member

mr-c commented Oct 10, 2017

How to run the conformance tests:

  1. Provide a program with the cwl-runner interface (perhaps reana-cwl-runner?): https://github.com/common-workflow-language/common-workflow-language/pull/278/files#diff-ee814a9c027fc9750beb075c283a973c

  2. Run https://github.com/common-workflow-language/common-workflow-language/blob/master/run_test.sh like ./run_test.sh RUNNER=reana-cwl-runner

  3. Tell me about it and I'll add an a Jenkins job to http://ci.commonwl.org/ & a badge to http://www.commonwl.org/#Implementations

@anton-khodak

This comment has been minimized.

Show comment
Hide comment
@anton-khodak

anton-khodak Oct 30, 2017

Member

To be run with /.run_test.sh cwl-runner interface must also have --version option

Member

anton-khodak commented Oct 30, 2017

To be run with /.run_test.sh cwl-runner interface must also have --version option

@anton-khodak anton-khodak modified the milestones: Someday, Full CWL conformance Nov 20, 2017

@anton-khodak

This comment has been minimized.

Show comment
Hide comment
@anton-khodak

anton-khodak Dec 13, 2017

Member

Current conformance tests situation (live updates):

  • [1] General test of command line generation
  • [2] Test nested prefixes with arrays
  • [3] Test nested command line bindings
  • [4] Test command line with optional input (missing)
  • [5] Test command line with optional input (provided)
  • [6] Test InitialWorkDirRequirement ExpressionEngineRequirement.engineConfig feature
  • [7] Test command execution in Docker with stdout redirection
  • [8] Test command execution in Docker with simplified syntax stdout redirection
  • [9] Test command execution in Docker with stdout redirection
  • [10] Test command line with stderr redirection
  • [11] Test command line with stderr redirection, brief syntax
  • [12] Test command line with stderr redirection, named brief syntax
  • [13] Test command execution in Docker with stdin and stdout redirection
  • [14] Test default usage of Any in expressions.
  • [15] Test explicitly passing null to Any type inputs with default values.
  • [16] Testing the string 'null' does not trip up an Any with a default value.
  • [17] Testing the string 'null' does not trip up an Any without a default value.
  • [18] Test command execution in with stdin and stdout redirection
  • [19] Test ExpressionTool with Docker-based expression engine
  • [20] Test outputEval to transform output
  • [21] Test two step workflow with imported tools
  • [22] Test two step workflow with inline tools
  • [23] Test single step workflow with Scatter step
  • [24] Test single step workflow with Scatter step and two data links connected to
    same input, default merge behavior
  • [25] Test single step workflow with Scatter step and two data links connected to
    same input, nested merge behavior
  • [26] Test single step workflow with Scatter step and two data links connected to
    same input, flattened merge behavior
  • [27] Test workflow with default value for input parameter (missing)
  • [28] Test workflow with default value for input parameter (provided)
  • [29] Test EnvVarRequirement
  • [30] Test workflow scatter with single scatter parameter
  • [31] Test workflow scatter with two scatter parameters and nested_crossproduct join method
  • [32] Test workflow scatter with two scatter parameters and flat_crossproduct join method
  • [33] Test workflow scatter with two scatter parameters and dotproduct join method
  • [34] Test workflow scatter with single empty list parameter
  • [35] Test workflow scatter with two scatter parameters and nested_crossproduct join method with second list empty
  • [36] Test workflow scatter with two scatter parameters and nested_crossproduct join method with first list empty
  • [37] Test workflow scatter with two scatter parameters, one of which is empty and flat_crossproduct join method
  • [38] Test workflow scatter with two empty scatter parameters and dotproduct join method
  • [39] Test Any type input parameter
  • [40] Test nested workflow
  • [41] Test requirement priority
  • [42] Test requirements override hints
  • [43] Test requirements on workflow steps
  • [44] Test default value on step input parameter
  • [45] Test use default value on step input parameter with empty source
  • [46] Test use default value on step input parameter with null source
  • [47] Test default value on step input parameter overridden by provided source
  • [48] Test simple workflow
  • [49] Test unknown hints are ignored.
  • [50] Test InitialWorkDirRequirement linking input files and capturing secondaryFiles
    on input and output.
  • [51] Test InitialWorkDirRequirement with expression in filename.
  • [52] Test inline expressions
  • [53] Test SchemaDefRequirement definition used in tool parameter
  • [54] Test SchemaDefRequirement definition used in workflow parameter
  • [55] Test parameter evaluation, no support for JS expressions
  • [56] Test parameter evaluation, with support for JS expressions
  • [57] Test metadata
  • [58] Test simple format checking.
  • [59] Test format checking against ontology using subclassOf.
  • [60] Test format checking against ontology using equivalentClass.
  • [61] Test optional output file and optional secondaryFile on output.
  • [62] Test valueFrom on workflow step.
  • [63] Test valueFrom on workflow step with multiple sources
  • [64] Test valueFrom on workflow step referencing other inputs
  • [65] Test record type output binding.
  • [66] Test support for reading cwl.output.json when running in a Docker container
    and just 'path' is provided.
  • [67] Test support for reading cwl.output.json when running in a Docker container
    and just 'location' is provided.
  • [68] Test support for returning multiple glob patterns from expression
  • [69] Test workflow scatter with single scatter parameter and valueFrom on step input
  • [70] Test workflow scatter with two scatter parameters and nested_crossproduct join method and valueFrom on step input
  • [71] Test workflow scatter with two scatter parameters and flat_crossproduct join method and valueFrom on step input
  • [72] Test workflow scatter with two scatter parameters and dotproduct join method and valueFrom on step input
  • [73] Test workflow scatter with single scatter parameter and valueFrom on step input
  • [74] Test valueFrom eval on scattered input parameter
  • [75] Test workflow two input files with same name.
  • [76] Test directory input with parameter reference
  • [77] Test directory input in Docker
  • [78] Test directory output
  • [79] Test directories in secondaryFiles
  • [80] Test dynamic initial work dir
  • [81] Test writable staged files.
  • [82] Test file literal as input
  • [83] Test expression in InitialWorkDir listing
  • [84] Test nameroot/nameext expression in arguments, stdout
  • [85] Test directory input with inputBinding
  • [86] Test command line generation of array-of-arrays
  • [87] Test $HOME and $TMPDIR are set correctly
  • [88] Test $HOME and $TMPDIR are set correctly in Docker
  • [89] Test that expressionLib requirement of individual tool step overrides expressionLib of workflow.
  • [90] Test output of InitialWorkDir
  • [91] Test embedded subworkflow
  • [92] Test secondaryFiles on array of files.
  • [93] Test directory literal output created by ExpressionTool
  • [94] Test file literal output created by ExpressionTool
  • [95] Test dockerOutputDirectory
  • [96] Test hints with $import
  • [97] Test warning instead of error when default path is not found
  • [98] Test InlineJavascriptRequirement with multiple expressions in the same tool
  • [99] Test if a writable input directory is recursivly copied and writable
  • [100] Test that missing parameters are null (not undefined) in expression
  • [101] Test that provided parameter is not null in expression
  • [102] Test compound workflow document
  • [103] Test that nameroot and nameext are generated from basename at execution time by the runner
  • [104] Test that file path in $(inputs) for initialworkdir is in $(outdir).
  • [105] Test single step workflow with Scatter step and two data links connected to
    same input, flattened merge behavior. Workflow inputs are set as list
  • [106] Test step input with multiple sources with multiple types
  • [107] Test that shell directives are not interpreted.
  • [108] Test that shell directives are quoted.
  • [109] Test empty writable dir with InitialWorkDirRequirement
  • [110] Test dynamic resource reqs referencing inputs
  • [111] Test file literal as input without Docker

1 test failing

Member

anton-khodak commented Dec 13, 2017

Current conformance tests situation (live updates):

  • [1] General test of command line generation
  • [2] Test nested prefixes with arrays
  • [3] Test nested command line bindings
  • [4] Test command line with optional input (missing)
  • [5] Test command line with optional input (provided)
  • [6] Test InitialWorkDirRequirement ExpressionEngineRequirement.engineConfig feature
  • [7] Test command execution in Docker with stdout redirection
  • [8] Test command execution in Docker with simplified syntax stdout redirection
  • [9] Test command execution in Docker with stdout redirection
  • [10] Test command line with stderr redirection
  • [11] Test command line with stderr redirection, brief syntax
  • [12] Test command line with stderr redirection, named brief syntax
  • [13] Test command execution in Docker with stdin and stdout redirection
  • [14] Test default usage of Any in expressions.
  • [15] Test explicitly passing null to Any type inputs with default values.
  • [16] Testing the string 'null' does not trip up an Any with a default value.
  • [17] Testing the string 'null' does not trip up an Any without a default value.
  • [18] Test command execution in with stdin and stdout redirection
  • [19] Test ExpressionTool with Docker-based expression engine
  • [20] Test outputEval to transform output
  • [21] Test two step workflow with imported tools
  • [22] Test two step workflow with inline tools
  • [23] Test single step workflow with Scatter step
  • [24] Test single step workflow with Scatter step and two data links connected to
    same input, default merge behavior
  • [25] Test single step workflow with Scatter step and two data links connected to
    same input, nested merge behavior
  • [26] Test single step workflow with Scatter step and two data links connected to
    same input, flattened merge behavior
  • [27] Test workflow with default value for input parameter (missing)
  • [28] Test workflow with default value for input parameter (provided)
  • [29] Test EnvVarRequirement
  • [30] Test workflow scatter with single scatter parameter
  • [31] Test workflow scatter with two scatter parameters and nested_crossproduct join method
  • [32] Test workflow scatter with two scatter parameters and flat_crossproduct join method
  • [33] Test workflow scatter with two scatter parameters and dotproduct join method
  • [34] Test workflow scatter with single empty list parameter
  • [35] Test workflow scatter with two scatter parameters and nested_crossproduct join method with second list empty
  • [36] Test workflow scatter with two scatter parameters and nested_crossproduct join method with first list empty
  • [37] Test workflow scatter with two scatter parameters, one of which is empty and flat_crossproduct join method
  • [38] Test workflow scatter with two empty scatter parameters and dotproduct join method
  • [39] Test Any type input parameter
  • [40] Test nested workflow
  • [41] Test requirement priority
  • [42] Test requirements override hints
  • [43] Test requirements on workflow steps
  • [44] Test default value on step input parameter
  • [45] Test use default value on step input parameter with empty source
  • [46] Test use default value on step input parameter with null source
  • [47] Test default value on step input parameter overridden by provided source
  • [48] Test simple workflow
  • [49] Test unknown hints are ignored.
  • [50] Test InitialWorkDirRequirement linking input files and capturing secondaryFiles
    on input and output.
  • [51] Test InitialWorkDirRequirement with expression in filename.
  • [52] Test inline expressions
  • [53] Test SchemaDefRequirement definition used in tool parameter
  • [54] Test SchemaDefRequirement definition used in workflow parameter
  • [55] Test parameter evaluation, no support for JS expressions
  • [56] Test parameter evaluation, with support for JS expressions
  • [57] Test metadata
  • [58] Test simple format checking.
  • [59] Test format checking against ontology using subclassOf.
  • [60] Test format checking against ontology using equivalentClass.
  • [61] Test optional output file and optional secondaryFile on output.
  • [62] Test valueFrom on workflow step.
  • [63] Test valueFrom on workflow step with multiple sources
  • [64] Test valueFrom on workflow step referencing other inputs
  • [65] Test record type output binding.
  • [66] Test support for reading cwl.output.json when running in a Docker container
    and just 'path' is provided.
  • [67] Test support for reading cwl.output.json when running in a Docker container
    and just 'location' is provided.
  • [68] Test support for returning multiple glob patterns from expression
  • [69] Test workflow scatter with single scatter parameter and valueFrom on step input
  • [70] Test workflow scatter with two scatter parameters and nested_crossproduct join method and valueFrom on step input
  • [71] Test workflow scatter with two scatter parameters and flat_crossproduct join method and valueFrom on step input
  • [72] Test workflow scatter with two scatter parameters and dotproduct join method and valueFrom on step input
  • [73] Test workflow scatter with single scatter parameter and valueFrom on step input
  • [74] Test valueFrom eval on scattered input parameter
  • [75] Test workflow two input files with same name.
  • [76] Test directory input with parameter reference
  • [77] Test directory input in Docker
  • [78] Test directory output
  • [79] Test directories in secondaryFiles
  • [80] Test dynamic initial work dir
  • [81] Test writable staged files.
  • [82] Test file literal as input
  • [83] Test expression in InitialWorkDir listing
  • [84] Test nameroot/nameext expression in arguments, stdout
  • [85] Test directory input with inputBinding
  • [86] Test command line generation of array-of-arrays
  • [87] Test $HOME and $TMPDIR are set correctly
  • [88] Test $HOME and $TMPDIR are set correctly in Docker
  • [89] Test that expressionLib requirement of individual tool step overrides expressionLib of workflow.
  • [90] Test output of InitialWorkDir
  • [91] Test embedded subworkflow
  • [92] Test secondaryFiles on array of files.
  • [93] Test directory literal output created by ExpressionTool
  • [94] Test file literal output created by ExpressionTool
  • [95] Test dockerOutputDirectory
  • [96] Test hints with $import
  • [97] Test warning instead of error when default path is not found
  • [98] Test InlineJavascriptRequirement with multiple expressions in the same tool
  • [99] Test if a writable input directory is recursivly copied and writable
  • [100] Test that missing parameters are null (not undefined) in expression
  • [101] Test that provided parameter is not null in expression
  • [102] Test compound workflow document
  • [103] Test that nameroot and nameext are generated from basename at execution time by the runner
  • [104] Test that file path in $(inputs) for initialworkdir is in $(outdir).
  • [105] Test single step workflow with Scatter step and two data links connected to
    same input, flattened merge behavior. Workflow inputs are set as list
  • [106] Test step input with multiple sources with multiple types
  • [107] Test that shell directives are not interpreted.
  • [108] Test that shell directives are quoted.
  • [109] Test empty writable dir with InitialWorkDirRequirement
  • [110] Test dynamic resource reqs referencing inputs
  • [111] Test file literal as input without Docker

1 test failing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment