Implement clarifications and new conformance tests for 1.2.1

[mr-c]

requests to fix any of the below should go against https://github.com/common-workflow-language/cwl-v1.2/tree/1.2.1_proposed You can preview all the implemented changes for v1.2.1 so far but know that they have not been ratified by the CWL Leadership Team; additional comments are requested.

Anyone is welcome to take one of these and start adding the needed conformance tests and specification clarifications! Please just drop a note in the linked issue when you start.

If you have never written a CWL conformance test or edited the text of the CWL standards before, that's okay! Just ask for instructions on what to do in the linked issue of your choice.

Candidates issues

Rough consensus with a known solution

New conformance tests

See https://github.com/common-workflow-language/cwl-v1.2/blob/main/CONFORMANCE_TESTS.md#format-of-the-conformance-test-file for information on how to write a new conformance test

ToDo

• Conformance tests for renaming output files with basename
  • confirm that fixing the above also covers everything in Clarify the description of basename modification in the spec
• Add tests for itemSeparator
• Conformance test for optional record input
• need conformance test for workflow output + linkMerge: merge_flattened
• There are no specific conformance tests about the behavior for nested inputBinding
• Add conformance test for ECMAScript 5.1 strict mode
• create conformance test for InitialWorkDirRequirement w/ & w/o Docker and reference to input file paths
• Continue with Audit the objects available for every place we allow parameter references/expressions producing new conformance tests for each category
• new conformance test: symlinks using absolute paths in a Directory
• two directory inputs with same basename but different paths
• dockerLoad, dockerFile, and dockerImport lack conformance tests
• workflow in subdirectory, tools in another subdirectory, run contains relative paths like run: ../tools/my_tool.cwl. Example; source: Run CWL conformance tests via WES DataBiosphere/toil#4052 (comment)
• test for parameter references in CommandLineTool.arguments (pull request)
• a proper test for WorkReuse (not possible with cwltest)
• potential secondaryFiles conformance test candidate (cannot reproduce with cwltool)

Specification clarification

changes to wording of the specification, may require new conformance test(s) to clarify or confirm

ToDo

• Specification is unclear on whether array typeDefs are allowed in SchemaDefRequirement
• Document that Directory loadListing behavior in v1.1 is not backwards compatible
• Clarify records with multiple inputBindings
• Several terms are used for the value of record type: objects, maps, and records
• Define decoding method for loadContents and cwl.output.json
• clarify use of parameter references / expressions in ResourceRequirement fields
• Does CommandLineBinding.loadContents apply to secondary files?
• Expected behavior of outputBinding on array of file
• be explicit about how to format an input object file via the optional cwl-runner CLI
• Improve explanations about array level and per-item inputBinding
• better document the 'default' and 'valueFrom' fields on WorkflowStepInput
• clarify type checking rules
• WorkflowOutputParameter outputSource should not be optional
• Examples for linkMerge merge_flattened
• specify YAML version, which schema
• fields not specified in an input object get a value of 'null'
• clarify sort order for globs
• clarify LoadListingEnum
• [prose] Confirm that final output objects must have the listings of all directories recursively expanded
• explain how to render a type: enum on the command line
• clarify evaluation policy of Workflow level requirements
• Explain the concept of a Abstract CWL Workflow, a class: Workflow with one or more class: Operation steps (or sub-steps). An Abstract CWL Workflow description is not (fully) executable, as Operation processes are themselves abstract; but it does allow describing a workflow and providing a skeleton to attach metadata about the steps, authorship, tools used, and more. Abstract CWL Workflows are used in the RO-Crate workflows and workflow runs profiles; the former used by workflowhub.eu.

Bonus issues

These are worthy, but we don't have a solution ready or obvious plan yet

Pending Discussion:

• directory with executable script inside → dir with executable file #263

ToDo

• JS expression returning Undefined
• document format schema bridging technique
• Definition of hints is inconsistent
• Clarify, make explicit, and add conformance test if CommandLineTool requires a baseCommand vs being a no-op
• default values with secondaryfiles defined by extension needs a conformance test
• "Edit This Section" button for the rendered CWL standards

Done

• outputEval: "time passed" syntax error
• Allow [] ? in types under ArraySchema.list
• conformance tests: add docker.io/ to dockerPull entries that lack an explicit registry
• clarify $(..) vs ${...}
• conformance test requires location: Any what does it mean ? (in progress at Document writing a new conformance test #124)
• testing InitialWorkDirRequirement with directories as output and more (in progress at test for nested directories and initial workdir requirement #126 )
• Conflict between the conformance test 61 and the spec for parameter references (in progress at Clarify parameter reference: null and array length #64)
• new conformance test: $(runtime.outdir) for outputBinding glob Add conformance test that verifies we can use runtime outdir with output glob #164
• Unclear specification of run field in the workflow description
• mention cwl-upgrader; explain version mixing Add note about cwl-upgrader in the Workflow changelog #172
• fix the broken InitialWorkDirRequirement link
• remove the initial_workdir_output_glob_duplicate test Typo fixes common-workflow-language#160
• The ID numbers are off by one at 220 or before Irrelvant due to Move label to id cwltest#110
• conformance test for input record + JS Add a conformance test for InlineJavascriptRequirement with an input record #170
• New SchemaDefRequirement conformance test candidate Add test for SchemaDefRequirement with a workflow and the types containing an $import statement #171
• conformance test needed: MultipleInputFeatureRequirement on workflow outputs Add a conformance test for MultipleInputFeatureRequirement on Workflow outputs #169
• Define the concept of an "opaque string" Add glossary and clarification on opaque strings and identifiers #162
• affirm that WorkflowOutputParameter.outputSource can reference workflow level inputs Mention that we can refer to workflow level inputs in output parameter outputSource #178
• Stdout behavior with chained commands Add new test for chained commands, update the specification wording #190
• clarify meaning of Process.id Clarify process id field ($graph only) #166
• Why conformance test 63 expect empty result ? Update the doc and use "true" as base command for the metadata test #184
• complicated IWDRs & symlinks #193
• need conformance test for multiple entries in a WorkflowOutputParameter.outputSource #239
• Prose for location in File/Directory mentions URIs instead of IRIs
• About specification when output class is File
• reparent file specific fields from SchemaBase to Parameter common-workflow-language#232

[cwl-bot]

This issue has been mentioned on Common Workflow Language Discourse. There might be relevant details there:

https://cwl.discourse.group/t/dockerfile-contents-vs-dockerfile-file/221/6

[smyja]

ItemSeparator test has been written already.

[mr-c]

ItemSeparator test has been written already.

Hello @smyja ; if you think common-workflow-language/common-workflow-language#834 has been fixed, you should comment in that issue. However, I don't see the referenced tests merged into this repository, just simple usage of itemSeparator in two other tests