- The
metaandparameter_metablocks are now valid withinworkflowblocks, not justtask - The JES backend configuration now has an option
genomics-api-queries-per-100-secondsto help tune the rate of batch polling against the JES servers. Users with quotas larger than default should make sure to set this value. - Added an option
call-caching.invalidate-bad-cache-results(default:true). If true, Cromwell will invalidate cached results which have failed to copy as part of a cache hit. - Timing diagrams and metadata now receive more fine grained workflow states between submission and Running.
- Support for the Pair WDL type (e.g.
Pair[Int, File] floo = (3, "gs://blar/blaz/qlux.txt")) - Added support for new WDL functions:
zip: (Array[X], Array[Y]) => Array[Pair[X, Y]]- align items in the two arrays by index and return them as WDL pairscross: (Array[X], Array[Y]) => Array[Pair[X, Y]]- create every possible pair from the two input arrays and return them all as WDL pairstranspose: (Array[Array[X]]) => Array[Array[X]]compute the matrix transpose for a 2D array. Assumes each inner array has the same length.
- Support for sub workflows (see Annex A)
- Support for WDL imports through an additional imports.zip parameter
- Corrected file globbing in JES to correctly report all generated files. Additionally, file globbing in JES now uses bash-style glob syntax instead of python style glob syntax
- Support declarations as graph nodes
- Added the ability to override the default service account that the compute VM is started with via the configuration option
JES.config.genomics.compute-service-accountor through the workflow options parametergoogle_compute_service_account. More details can be found in the README.md - By default,
system.abort-jobs-on-terminateis false when runningjava -jar cromwell.jar server, and true when runningjava -jar cromwell.jar run <wdl> <inputs>. - Bug fix which allows Cromwell to successfully exit once a workflow completes in Single Workflow Runner Mode.
Annex A - Workflow outputs
The WDL specification has changed regarding workflow outputs to accommodate sub workflows.
This change is backward compatible in terms of runnable WDLs (WDL files using the deprecated workflow outputs syntax will still run the same).
The only visible change lies in the metadata (as well as the console output in single workflow mode, when workflow outputs are printed out at the end of a successful workflow).
TL;DR Unless you are parsing or manipulating the "key" by which workflow outputs are referenced in the metadata (and/or the console output for single workflow mode), you can skip the following explanation.
Metadata Response
{
...
outputs {
"task_output_1": "hello",
"task_output_2": "world"
^
If you don't manipulate this part of the metadata, then skip this section
}
}
In order to maintain backward compatibility, workflow outputs expressed with the deprecated syntax are "expanded" to the new syntax. Here is an example:
task t {
command {
#do something
}
output {
String out1 = "hello"
String out2 = "world"
}
}
workflow old_syntax {
call t
output {
t.*
}
}
workflow new_syntax {
call t
output {
String wf_out1 = t.out1
String wf_out2 = t.out2
}
}
The new syntax allows for type checking of the outputs as well as expressions. It also allows for explicitly naming to the outputs.
The old syntax doesn't give the ability to name workflow outputs. For consistency reasons, Cromwell will generate a "new syntax" workflow output for each task output, and name them.
Their name will be generated using their FQN, which would give
output {
String w.t.out1 = t.out1
String w.t.out2 = t.out2
}
However as the FQN separator is ., the name itself cannot contain any ..
For that reason, . are replaced with _ :
Old syntax expanded to new syntax
output {
String w_t_out1 = t.out1
String w_t_out2 = t.out2
}
The consequence is that the workflow outputs section of the metadata for old_syntax would previously look like
outputs {
"w.t.out1": "hello",
"w.t.out2": "hello"
}
but it will now look like
outputs {
"w_t_out1": "hello",
"w_t_out2": "hello"
}
The same applies for the console output of a workflow run in single workflow mode.