@rsasch rsasch released this Oct 17, 2018 · 68 commits to develop since this release

Assets 4

36 Release Notes

Extra configuration options

The value exit-code-timeout-seconds can now set in a backend configuration.
Details here

Extra configuration options for Firecloud

In addition to a "PAPI" status key, the FC custom health monitor now supports "PAPIV1" and "PAPIV2" status keys.

AWS S3 file transfers are now encrypted

Bug fixes

Metadata Request Coalescing

Coalesce metadata requests to eliminate expensive and redundant queries and metadata construction.

Eliminate redundant SFS logging and metadata

Eliminate superfluous logging and metadata publishing in the shared filesystem backend on poll intervals where there was not a state change.

AWS region configuration respected throughout

Previously US-EAST-1 was hardcoded in places.

Server jar back under 200MB

Removed a large number of unused AWS dependencies.

Sub-workflows logging to the correct file

Sub-workflow logs are always written to the root-workflow's log instead of a separate log file that used to leak file pointers.


@aednichols aednichols released this Oct 2, 2018 · 127 commits to develop since this release

Assets 4

35 Release Notes

Submit workflow using URL

Cromwell now allows for a user to submit the URL pointing to workflow file to run a workflow.
More details on how to use it in:

  • Server mode can be found here.
  • Run mode can be found here.


  • Added an opt-in namespace cache for the WDL Draft 2 language factory. Please see the Cromwell example configuration for details. NOTE: if upgrading from a hotfix version of Cromwell
    that relied upon this cache, the cache is now opt-in and must be turned on explicitly in config.
  • To maintain conformance with the OpenWDL spec, Cromwell drops support for the version draft-3 identifier in this release. In the rare case where end users may have been using version draft-3, version 1.0 is a drop-in replacement with no effect on functionality.

HTTP Workflow Inputs for Shared File System and Google Pipelines API Version 2 Backends

http and https workflow inputs are now supported for shared filesystem and Google Pipelines API (PAPI) version 2
backends. Configuration details are described here.

Call cache hint support

More efficient cache hit copying in multi-user environments is now supported through the call_cache_hit_path_prefixes workflow option.
Details here

Extra configuration options

The value dockerRoot can now be set in a backend configuration.
This will set the execution folder in the container (default: /cromwell-executions).

Bug Fixes


  • The releaseHold endpoint will now return 404 Not Found for an unrecognized workflow ID and 400 Bad Request for a malformed or invalid workflow ID.


  • Fixed a bug that allowed values to be "auto-boxed" into a single-element Array of that type, which is not allowed in the WDL spec (Closes #3478).

PAPI version 1

  • Restored standard output and error streaming for jobs.

@mcovarr mcovarr released this Jul 19, 2018 · 253 commits to develop since this release

Assets 4

34 Release Notes

Query API

  • Fixes a bug which stopped includeSubworkflow=false from paging correctly and subworkflows from being discounted correctly from totalResultsCount.
  • Query results will now be returned in reverse chronological order, with the most-recently submitted workflows returned first.

Requester Pays on GCS

Access of Google Cloud Storage buckets with Requester Pays enabled is now supported.
Please read the relevant documentation for information on how to enable it and the consequences.

Private Docker Support on Pipelines API v2

Support for private Docker Hub images is now included in the Google Pipelines API v2 backend. PAPI v2 private Docker support is
equivalent to that in PAPI v1 but the configuration differs, please see
Docker configuration for more details.

Updated MySQL client with 8.0 support

Updated the MySQL connector client from 5.1.42 to 5.1.46 which adds support for connecting to MySQL 8.0. See the
documentation on Changes in MySQL Connector/J for
more information.


@Horneth Horneth released this Jun 25, 2018 · 302 commits to develop since this release

Assets 4

33 Release Notes

Query endpoint

Exclude workflows based on Labels

This gives the ability to filter out workflows based on labels. Two new parameters called excludeLabelAnd and excludeLabelOr can be used for this purpose.
More details on how to use them can be found here.

Include/Exclude subworkflows

Cromwell now supports excluding subworkflows from workflow query results using the includeSubworkflows parameter. By default they are included in the results.
More information can be found at REST API.

Query workflows by Submission time

Cromwell now supports querying workflows by submission time. This will help find workflows that are submitted but not started yet (i.e. workflows which are
in On Hold state). More information can be found here.

Submission time in Workflow Query Response

Submission time of a workflow is now included in WorkflowQueryResult, which is part of the response for workflow query.

File Localization (NIO) Hint

Cromwell now allows tasks in WDL 1.0 can now specify an optimization in their parameter_meta that some File inputs do not need to be localized for the task to run successfully.
Full details are available in the documentation page for this optimization.

Bug Fixes

Workflows which are in 'On Hold' state can now be fetched using the query endpoint.


@kshakir kshakir released this May 25, 2018 · 365 commits to develop since this release

Assets 4

32 Release Notes


Pipelines API V2

Initial support for Google Pipelines API version 2.
Expect feature parity except for private dockerhub images which are not supported at the moment, but will be in the near future.
Additionally, the "refresh token" authentication mode is NOT supported on PAPI V2.

In addition, the following changes are to be expected:

  • Error messages for failed jobs might differ from V1
  • The Pipelines API log file content might differ from V1

Important (If you're running Cromwell with a Google backend, read this):
The actor-factory value for the google backend (cromwell.backend.impl.jes.JesBackendLifecycleActorFactory) is being deprecated.
Please update your configuration accordingly.

PAPI Version actor-factory
V1 cromwell.backend.google.pipelines.v1alpha2.PipelinesApiLifecycleActorFactory
V2 cromwell.backend.google.pipelines.v2alpha1.PipelinesApiLifecycleActorFactory

If you don't update the actor-factory value, you'll get a deprecation warning in the logs, and Cromwell will default back to PAPI V1

Task Retries

Cromwell now supports retrying failed tasks up to a specified count by declaring a value for the maxRetries key through the WDL runtime attributes.


  • Cromwell has removed most of the formatting restrictions from custom labels. Please check the README for more detailed documentation.
  • Custom labels won't be submitted to Google backend as they are now decoupled from Google's default labels.
  • Cromwell now publishes the labels as soon as the workflow is submitted (whether started or on hold). If the labels are invalid, the workflow will not be submitted and request will fail.

Scala 2.11 Removed

From version 32 onwards we will no longer be publishing build artifacts compatible with Scala 2.11.

  • If you don't import the classes into your own scala project then this should have no impact on you.
  • If you are importing the classes into your own scala project, make sure you are using Scala 2.12.

Input Validation

Cromwell can now validate that your inputs files do not supply inputs with no impact on the workflow. Strict validation will be disabled by default in WDL draft 2 and CWL but enabled in WDL draft 3. See the 'Language Factory Config' below for details.

Language Factory Config

All language factories can now be configured on a per-language-version basis. All languages and versions will support the following options:

  • enabled: Defaults to true. Set to false to disallow workflows of this language and version.
  • strict-validation: Defaults to true for WDL draft 3 and false for WDL draft 2 and CWL. Specifies whether workflows fail if the inputs JSON (or YAML) file contains values which the workflow did not ask for (and will therefore have no effect). Additional strict checks may be added in the future.


  • More accurately returns 503 instead of 500 when Cromwell can not respond in a timely manner
  • Cromwell now allows a user to submit a workflow but in a state where it will not automatically be picked up for execution. This new state is called 'On Hold'. To do this you need to set the parameter workflowOnHold to true while submitting the workflow.
  • API end point 'releaseHold' will allow the user to send a signal to Cromwell to allow a workflow to be startable, at which point it will be picked up by normal execution schemes.


The PAPI backend now supports specifying GPU through WDL runtime attributes:

runtime {
    gpuType: "nvidia-tesla-k80"
    gpuCount: 2
    zones: ["us-central1-c"]

The two types of GPU supported are nvidia-tesla-k80 and nvidia-tesla-p100

Important: Before adding a GPU, make sure it is available in the zone the job is running in: https://cloud.google.com/compute/docs/gpus/

Job Shell

Cromwell now allows for system-wide or per-backend job shell configuration for running user commands rather than always
using the default /bin/bash. To set the job shell on a system-wide basis use the configuration key system.job-shell or on a
per-backend basis with <config-key-for-backend>.job-shell. For example:

# system-wide setting, all backends get this
# override for just the Local backend

For the Config backend the value of the job shell will be available in the ${job_shell} variable. See Cromwell's reference.conf for an example
of how this is used for the default configuration of the Local backend.

Bug Fixes

The imports zip no longer unpacks a single (arbitrary) internal directory if it finds one (or more). Instead, import statements should now be made relative to the base of the import zip root.

Reverting Custom Labels

Reverting to a prior custom label value now works.

"Retrieves the current labels for a workflow"
will return the most recently summarized custom label value.

The above endpoint may still return the prior value for a short period of time after using
"Updated labels for a workflow"
until the background metadata summary process completes.

Deleting Duplicate Custom Label Rows

If you never used the REST API to revert a custom label back to a prior value you will not be affected. This only applies to workflows previously updated using
"Updated labels for a workflow".

The database table storing custom labels will delete duplicate rows for any workflow label key. For efficiency purposes
the values are not regenerated automatically from the potentially large metadata table.

In rare cases where one tried to revert to a prior custom label value you may continue to see different results
depending on the REST API used. After the database update
"Retrieves the current labels for a workflow"
will return the most-recent-unique value while
"Get workflow and call-level metadata for a specified workflow"
will return the up-to-date value. For example, if one previously updated a value from "value-1" > "value-2" >
"value-3" > "value-2" then the former REST API will return value-3 while the latter will return value-2.

Workflow options google_project output in metadata

Workflow metadata for jobs run on a Google Pipelines API backend will report the google_project specified via a
workflow options json.

@mcovarr mcovarr released this May 19, 2018

Assets 4

Bug fix for $HOME variable in job script, performance improvements for metrics.


@Horneth Horneth released this Mar 12, 2018 · 552 commits to develop since this release

Assets 4

31 Release Notes

  • Cromwell server
    The Cromwell server source code is now located under server/src. sbt assembly will build the runnable Cromwell JAR in
    server/target/scala-2.12/ with a name like cromwell-<VERSION>.jar.

  • Robustness

    • The rate at which jobs are being started can now be controlled using the system.job-rate-control configuration stanza.
    • A load controller service has been added to allow Cromwell to self-monitor and adjust its load accordingly.
      The load controller is currently a simple on/off switch controlling the job start rate. It gathers metrics from different parts of the system
      to inform its decision to stop the creation of jobs.
      You can find relevant configuration in the services.LoadController section of the cromwell.examples.conf file,
      as well as in the load-control section in reference.conf.
      The load level of the monitored sub-systems are instrumented and can be found under the cromwell.load statsD path.
    • The statsD metrics have been re-shuffled a bit. If you had a dashboard you might find that you need to update it.
      Changes include:
      • Removed artificially inserted "count" and "timing" the path
      • Added a load section
      • Metrics were prefixed twice with cromwell (cromwell.cromwell.my_metric), now they're only prefixed once
      • Added processed and queue metrics under various metrics monitoring the throughput and amount of queued work respectively
      • Added a memory metric representing an estimation of the free memory Cromwell thinks it has left
  • Added a configuration option under docker.hash-lookup.enabled to disable docker hash lookup.
    Disabling it will also disable call caching for jobs with floating docker tags.

  • Rest API

    • Updated the /query response to include the total number of query results returned. See here for more information.
  • Language APIs

    • The WDL library import from Cromwell 30 has split in two and its scala packages have changed.
    • The WDL draft 2 parser is now in cromwell-wdl-model-draft2 and its classes have moved from the wdl4s.parser package to wdl.draft2.parser.
    • The WDL object model is now in cromwell-wdl-model-draft2 and its classes have moved from the wdl package to wdl.draft2.model.
    • The WDL to WOM transform functions are now in cromwell-wdl-transforms-draft2. The functions were removed from their object model classes and are now found in their own objects in wdl.draft2.transforms.wdlom2wom.

@kshakir kshakir released this Jan 25, 2018 · 1 commit to 30_hotfix since this release

Assets 4

A set of bug fixes following the migration of Cromwell to WOM (the Workflow Object Model) in version 30.

@cjllanwarne cjllanwarne released this Dec 20, 2017 · 9 commits to 30_hotfix since this release

Assets 4

A set of bug fixes following the migration of Cromwell to WOM (the Workflow Object Model) in version 30.