Removed deadlock workaround.
Minimal updates to fix depracated usage.
Old Slick 3.1 style names are still used in scala, ex: `val driver: JdbcProfile`.
Added notes about SQL converters handling empty LOBs.

.idea/inspectionProfiles/Project_Default.xml

@@ -3,4 +3,4 @@
     <option name="myName" value="Project Default" />
     <inspection_tool class="TypeAnnotation" enabled="false" level="WARNING" enabled_by_default="false" />
   </profile>
-</component>
+</component>

.pullapprove.yml

@@ -1,19 +1,28 @@
-approve_by_comment: true
-approve_regex: ':\+1:'
-reset_on_push: false
-author_approval: ignored
-reviewers:
+# enabling version 2 turns github reviews on by default
+version: 2
+group_defaults:
+  approve_by_comment:
+    enabled: true
+    approve_regex: ':\+1:'
+  reset_on_push:
+    enabled: false
+groups:
+  reviewers:
     required: 2
-    members:
+    github_reviews:
+      enabled: true
+    author_approval:
+      ignored: true
+    users:
         - Horneth
         - cjllanwarne
         - francares
         - gauravs90
         - geoffjentry
-        - jainh
         - jsotobroad
         - katevoss
         - kcibul
         - kshakir
         - mcovarr
         - ruchim
+        - danbills

.travis.yml

@@ -1,8 +1,10 @@
 sudo: required
 dist: trusty
+services:
+  - docker
 language: scala
 scala:
-  - 2.11.8
+  - 2.12.2
 jdk:
   - oraclejdk8
 cache:
@@ -16,10 +18,12 @@ before_cache:
   - find $HOME/.ivy2 -name "ivydata-*.properties" -delete
   - find $HOME/.sbt -name "*.lock" -delete
 before_install:
-  - openssl aes-256-cbc -K "$encrypted_5ebd3ff04788_key" -iv "$encrypted_5ebd3ff04788_iv" -in src/bin/travis/resources/jesConf.tar.enc -out jesConf.tar -d || true
+  # https://github.com/travis-ci/travis-ci/issues/7940#issuecomment-310759657
+  - sudo rm -f /etc/boto.cfg
 env:
   global:
     - CENTAUR_BRANCH=develop
+    - INTEGRATION_TESTS_DIR=src/main/resources/integrationTestCases
   matrix:
     # Setting this variable twice will cause the 'script' section to run twice with the respective env var invoked
     - BUILD_TYPE=sbt
@@ -36,3 +40,12 @@ deploy:
   script: src/bin/travis/publishRelease.sh
   on:
     tags: true
+notifications:
+  slack:
+    rooms:
+      - secure: B5KYcnhk/ujAUWlHsjzP7ROLm6MtYhaGikdYf6JYINovhMbVKnZCTlZEy7rqT3L2T5uJ25iefD500VQGk1Gn7puQ1sNq50wqjzQaj20PWEiBwoWalcV/nKBcQx1TyFT13LJv8fbFnVPxFCkC3YXoHedx8qAhDs8GH/tT5J8XOC8=
+    template:
+      - "Build <%{build_url}|#%{build_number}> (<%{compare_url}|%{commit}>) of %{repository}@%{branch} by %{author} %{result} in %{duration}"
+    on_success: change
+    on_failure: change
+    on_pull_requests: false

CHANGELOG.md

@@ -1,5 +1,246 @@
 # Cromwell Change Log
 
+## 29
+
+### Breaking Changes
+
+* Request timeouts for HTTP requests on the REST API now return a 503 status code instead of 500. The response for a request timeout is no longer in JSON format.
+* The metadata endpoint no longer returns gzipped responses by default. This now needs to be explicitly requested with an `Accept-Encoding: gzip` header 
+
+* Command line usage has been extensively revised for Cromwell 29.  Please see the
+[README](https://github.com/broadinstitute/cromwell#command-line-usage) for details.
+
+* The engine endpoints are now served under `/engine`. Previousely engine endpoints were available under
+`/api/engine`. Workflow endpoints are still served under `/api/workflows`. The setting `api.routeUnwrapped` has been
+retired at the same time.
+
+* The response format of the [callcaching/diff](https://github.com/broadinstitute/cromwell#get-apiworkflowsversioncallcachingdiff) endpoint has been updated.
+
+### Cromwell Server
+
+* Cromwell now attempts to gracefully shutdown when running in server mode and receiving a `SIGINT` (`Ctrl-C`) or `SIGTERM` (`kill`) signal. This includes waiting for all pending Database writes before exiting.
+A detailed explanation and information about how to configure this feature can be found in the [Cromwell Wiki](https://github.com/broadinstitute/cromwell/wiki/DevZone#graceful-server-shutdown).
+
+## 28
+
+### Bug Fixes
+
+#### WDL write_* functions add a final newline
+
+The following WDL functions now add a newline after the final line of output (the previous behavior of not adding this
+newline was inadvertent):
+- `write_lines`
+- `write_map`
+- `write_object`
+- `write_objects`
+- `write_tsv`
+
+For example:
+
+```
+task writer {
+  Array[String] a = ["foo", "bar"]
+  command {
+    # used to output: "foo\nbar"
+    # now outputs: "foo\nbar\n"
+    cat write_lines(a)
+  }
+}
+```
+
+#### `ContinueWhilePossible`
+
+A workflow utilizing the WorkflowFailureMode Workflow Option `ContinueWhilePossible` will now successfully reach a terminal state once all runnable jobs have completed.
+#### `FailOnStderr` 
+When `FailOnStderr` is set to false, Cromwell no longer checks for the existence of a stderr file for that task. 
+
+### WDL Functions
+
+#### New functions: floor, ceil and round:
+
+Enables the `floor`, `ceil` and `round` functions in WDL to convert floating point numbers to integers.
+
+For example we can now use the size of an input file to influence the amount of memory the task is given. In the example below a 500MB input file will result in a request for a VM with 2GB of memory:
+
+```
+task foo {
+    File in_file
+    command { ... }
+    runtime {
+      docker: "..."
+      memory: ceil(size(in_file)) * 4 
+    }
+}
+```
+
+### Call Caching
+
+* Hash values calculated by Cromwell for a call when call caching is enabled are now published to the metadata.
+It is published even if the call failed. However if the call is attempted multiple times (because it has been preempted for example),
+since hash values are strictly identical for all attempts, they will only be published in the last attempt section of the metadata for this call.
+If the hashes fail to be calculated, the reason is indicated in a `hashFailures` field in the `callCaching` section of the call metadata.
+*Important*: Hashes are not retroactively published to the metadata. Which means only workflows run on Cromwell 28+ will have hashes in their metadata.
+
+See the [README](https://github.com/broadinstitute/cromwell#get-apiworkflowsversionidmetadata) for an example metadata response.
+
+* New endpoint returning the hash differential for 2 calls. 
+
+`GET /api/workflows/:version/callcaching/diff`
+
+See the [README](https://github.com/broadinstitute/cromwell#get-apiworkflowsversioncallcachingdiff) for more details.
+
+### Workflow Submission
+
+* The workflow submission parameters `wdlSource` and `wdlDependencies` have been deprecated in favor of `workflowSource` and
+`workflowDependencies` respectively.  The older names are still supported in Cromwell 28 with deprecation warnings but will
+be removed in a future version of Cromwell.
+
+### Labels
+* A new `/labels` endpoint has been added to update labels for an existing workflow. See the [README](README.md#patch-apiworkflowsversionidlabels) for more information.
+* Label formatting requirements have been updated, please check the [README](README.md#label-format) for more detailed documentation.
+
+
+### JES Backend
+
+The JES backend now supports a `filesystems.gcs.caching.duplication-strategy` configuration entry.
+It can be set to specify the desired behavior of Cromwell regarding call outputs when a call finds a hit in the cache.
+The default value is `copy` which will copy all output files to the new call directory.
+A second value is allowed, `reference`, that will instead point to the original output files, without copying them.
+
+
+```hocon
+filesystems {
+  gcs {
+    auth = "application-default"
+    
+    caching {
+      duplication-strategy = "reference"
+    }
+  }
+}
+```
+
+A placeholder file will be placed in the execution folder of the cached call to explain the absence of output files and point to the location of the original ones.
+
+
+### Metadata Write Batching
+
+Metadata write batching works the same as in previous versions of Cromwell, but the default batch size has been changed from 1 to 200.  It's possible that 200 is too high in some environments, but 200 is more likely to be an appropriate value
+than the previous default.
+
+
+## 27
+
+### Migration
+
+* Call Caching has been improved in this version of Cromwell, specifically the time needed to determine whether or not a job can be cached
+ has drastically decreased. To achieve that the database schema has been modified and a migration is required in order to preserve the pre-existing cached jobs.
+ This migration is relatively fast compared to previous migrations. To get an idea of the time needed, look at the size of your `CALL_CACHING_HASH_ENTRY` table.
+ As a benchmark, it takes 1 minute for a table with 6 million rows.
+ The migration will only be executed on MySQL. Other databases will lose their previous cached jobs.
+ In order to run properly on MySQL, **the following flag needs to be adjusted**: https://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_group_concat_max_len
+ The following query will give you a minimum to set the group_concat_max_len value to:
+ 
+ ```sql
+SELECT MAX(aggregated) as group_concat_max_len FROM
+      (
+            SELECT cche.CALL_CACHING_ENTRY_ID, SUM(LENGTH(CONCAT(cche.HASH_KEY, cche.HASH_VALUE))) AS aggregated
+            FROM CALL_CACHING_HASH_ENTRY cche
+            GROUP BY cche.CALL_CACHING_ENTRY_ID
+      ) aggregation
+ ```
+
+ Here is the SQL command to run to set the group_concat_max_len flag to the proper value:
+ 
+ ```sql
+SET GLOBAL group_concat_max_len = value
+ ```
+ 
+ Where `value` is replaced with the value you want to set it to.
+ 
+ Note that the migration will fail if the flag is not set properly.
+ 
+### Breaking Changes
+
+* The update to Slick 3.2 requires a database stanza to
+[switch](http://slick.lightbend.com/doc/3.2.0/upgrade.html#profiles-vs-drivers) from using `driver` to `profile`.
+
+```hocon
+database {
+  #driver = "slick.driver.MySQLDriver$" #old
+  profile = "slick.jdbc.MySQLProfile$"  #new
+  db {
+    driver = "com.mysql.jdbc.Driver"
+    url = "jdbc:mysql://host/cromwell?rewriteBatchedStatements=true"
+    user = "user"
+    password = "pass"
+    connectionTimeout = 5000
+  }
+}
+```
+
+### Call Caching
+
+Cromwell now supports call caching with floating Docker tags (e.g. `docker: "ubuntu:latest"`). Note it is still considered
+a best practice to specify Docker images as hashes where possible, especially for production usages.
+
+Within a single workflow Cromwell will attempt to resolve all floating tags to the same Docker hash, even if Cromwell is restarted
+during the execution of a workflow. In call metadata the `docker` runtime attribute is now the same as the
+value that actually appeared in the WDL:
+
+```
+   "runtimeAttributes": {
+     "docker": "ubuntu:latest",
+     "failOnStderr": "false",
+     "continueOnReturnCode": "0"
+   }
+```
+
+Previous versions of Cromwell rewrote the `docker` value to the hash of the Docker image.
+
+There is a new call-level metadata value `dockerImageUsed` which captures the hash of the Docker image actually used to
+run the call:
+
+```
+   "dockerImageUsed": "library/ubuntu@sha256:382452f82a8bbd34443b2c727650af46aced0f94a44463c62a9848133ecb1aa8"
+```
+
+### Docker
+
+* The Docker section of the configuration has been slightly reworked 
+An option to specify how a Docker hash should be looked up has been added. Two methods are available.
+    "local" will try to look for the image on the machine where cromwell is running. If it can't be found, Cromwell will try to `pull` the image and use the hash from the retrieved image.
+    "remote" will try to look up the image hash directly on the remote repository where the image is located (Docker Hub and GCR are supported)
+Note that the "local" option will require docker to be installed on the machine running cromwell, in order for it to call the docker CLI.
+* Adds hash lookup support for public [quay.io](https://quay.io/) images.
+
+### WDL Feature Support
+* Added support for the new WDL `basename` function. Allows WDL authors to get just the file name from a File (i.e. removing the directory path)
+* Allows coercion of `Map` objects into `Array`s of `Pair`s. This also allows WDL authors to directly scatter over WDL `Map`s.
+
+### Miscellaneous
+* Adds support for JSON file format for google service account credentials. As of Cromwell 27, PEM credentials for PAPI are deprecated and support might be removed in a future version.
+
+```
+google {
+
+  application-name = "cromwell"
+
+  auths = [
+    {
+      name = "service-account"
+      scheme = "service_account"
+      json-file = "/path/to/file.json"
+    }
+  ]
+}
+```
+
+### General Changes
+
+* The `/query` endpoint now supports querying by `label`. See the [README](README.md#get-apiworkflowsversionquery) for more information.
+* The `read_X` standard library functions limit accepted filesizes.  These differ by type, e.g. read_bool has a smaller limit than read_string.  See reference.conf for default settings.
+
 ## 26
 
 ### Breaking Changes
@@ -43,17 +284,6 @@ system.io {
 }
 ```
 
-* Added a `script-epilogue` configuration option to adjust the logic that runs at the end of the scripts which wrap call executions.
-  This option is adjustable on a per-backend basis.  If unspecified, the default value is `sync`.
-
-### WDL Features
-
-With Cromwell 26, Cromwell will support `if x then y else z` expressions (see: https://github.com/broadinstitute/wdl/blob/develop/SPEC.md#if-then-else). For example: 
-```
-Boolean b = true
-String s = if b then "value if True" else "value if False"
-```
-
 ## 25
 
 ### External Contributors

NOTICE

@@ -1,4 +0,0 @@
-cromwell.webservice/PerRequest.scala (https://github.com/NET-A-PORTER/spray-actor-per-request)
-is distributed with this software under the Apache License, Version 2.0 (see the LICENSE-ASL file). In accordance
-with that license, that software comes with the following notices:
-    Copyright (C) 2011-2012 Ian Forsey