[GH-1634] Release WFL v0.17.0 #592
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Snapshot creation uses datetime rather than date * Making the name of the column used for datetime interval variable Co-authored-by: rhiananthony <ranthony@broadinstitute.org>
…ultiple times (#358) [GH-1278] Fix SG update-workload! implementation from trying to update the clio BAM records for each Succeeded workflow repeatedly by restoring the :finished guard for register-workload-in-clio. Introducted in https://broadinstitute.atlassian.net/browse/GH-1277.
…Release (#351) * Restore inputs processing hacks. * GDCWholeGenomeSomaticSingleSample moved. * Run off of /develop/ branch instead. * clojure -M:format * Use the GDCWholeGenomeSomaticSingleSample_v1.1.0 release. * Remove the Rich Comment.
* Draft doc module for SG. * Update navigation Camel^H^H^H^H^HYAML.
cherry-pick from main
The `Dockerfile` is being invoked in the root directory - we don't need to add the parent directory, just the current one.
Catch all throwables to make sure a workload failure doesn't bring down workflow-launcher.
* util/do-or-nil-silently should move to build.clj * Fixed indentation to match standard
* First pass: create-submissions generates one submission per entity. * Cleaned up TODOs for first pass attempt, reformatted tests to pass lint step in build. * Second pass: support entity set creation when specifying >1 entity for a submission. Refactored bigquery table dump to pull out now-common code. * Address PR feedback To simplify, will generate an entity set even for the singleton input. Removed firecloud/consolidate-entities-to-set as a result. Need to pass in 'expression' to submission creation payload when specifying an entity set. Supplemented integration and unit tests.
* Create an interface all hornets work on top of. * Apply suggestions from code review * Update readme.md Build Boad.
…ty (#368) * Boad -=> Board * Still doesn't round-trip on doubles and values like "19A". * Unit test wfl.tsv. * Ensure TSV mappability. * Ensure TSV is tabulatable. * Fix bug in assert-mapulatable!.
We create the snapshot in the TDR, and link to it in the workspace via Rawls.
…ema (#375) * Add snapshot_reference_id to Sarscov2IlluminaFull table schema Because schema update hasn't run, altering existing table definition in place. * Documentation changes from local Postgres debugging, onboarding - Clarified order of operations when installing, using local Postgres - Fixed broken intra-doc links - Added docs readme with instruction for launching local documentation site * Add instruction for recreating wfl DB to development docs * Remove references to undefined documentation files
Simply getting the submission again is not sufficient to guarantee that the workflow has been queued for execution. Poll instead.
* Update documentation and infrastructure for release candidates - creating release candiates - bashing release candidates Restore version override in cli.py. Re-tag latest docker images in cli.py so that version can be overwritten. Update IMAGES target in Makefile to add the :latest tag. Clean up :latest images on distclean target. * fix whitespace
Bumps [ssri](https://github.com/npm/ssri) from 6.0.1 to 6.0.2. - [Release notes](https://github.com/npm/ssri/releases) - [Changelog](https://github.com/npm/ssri/blob/v6.0.2/CHANGELOG.md) - [Commits](npm/ssri@v6.0.1...v6.0.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* add interactive script ahead of covid-19 demo tomorrow * fix rawls test failure * reformat test * share demo remources with cdc-covid-surveillance firecloud group * corrections from review * format for consistency * clone a dev workspace instead of the prod one * fix test-import-snapshot - workspace changed * fix arrays test failures
* changelog changes * Link workload with source, sink and executor tables. * Update the p-keys to bigint. * SQL has reserved table. * Initial creat covid workload commit * Verifies dataset as well * Update DB schema. * Address comments. * v0.6.0 Release (#359) * [GH-1278] Fix SG update-workload! implementation from updating Clio multiple times (#358) [GH-1278] Fix SG update-workload! implementation from trying to update the clio BAM records for each Succeeded workflow repeatedly by restoring the :finished guard for register-workload-in-clio. Introducted in https://broadinstitute.atlassian.net/browse/GH-1277. * GH-1188: Bump GDCWholeGenomeSomaticSingleSample Version When Lantern Release (#351) * Restore inputs processing hacks. * GDCWholeGenomeSomaticSingleSample moved. * Run off of /develop/ branch instead. * clojure -M:format * Use the GDCWholeGenomeSomaticSingleSample_v1.1.0 release. * Remove the Rich Comment. * GitHub `develop` infrastructure + release changes (#356) Update test actions to only run on PRs. Update release action to only run on new commits into main. Update + format docs about branching off develop and releasing into main. Restore tag-and-push-images in cli.py * GH-1282: Document WFL's support for Somatic Genomes. (#361) * Draft doc module for SG. * Update navigation Camel^H^H^H^H^HYAML. * Update CHANGELOG.md for v0.6.0, including previous patches Co-authored-by: Tom Lyons <tbl3rd@gmail.com> Co-authored-by: Tom Lyons <tbl@broadinstitute.org> * Resolve merge conflicts. * Fix issues generated from rebasing. * Fix the failing integration tests. Co-authored-by: rhiananthony <ranthony@broadinstitute.org> Co-authored-by: Edmund Higham <ehigham@users.noreply.github.com> Co-authored-by: Tom Lyons <tbl3rd@gmail.com> Co-authored-by: Tom Lyons <tbl@broadinstitute.org>
* Add multimethods for source, executor and sink operations Add skeleton implementations for tdr-source * make use of `!` more consistent? * add create operations for source, executor and sink
- Data model change: a snapshot_reference_id is linked to a workspace and should be associated with the executor, not the source (TDR in our use case) - Added covid/get-imported-snapshot-reference - nil or snapshot reference from Rawls for snapshot_reference_id in executor details instance (to be created from TerraExecutorDetails) - Added covid/import-snapshot! - import snapshot to workspace, writing to DB if successful - Added integration tests (incomplete coverage)
RR: https://broadinstitute.atlassian.net/browse/GH-1295 firecloud's flexibleImportEntities has a size limit on the TSV file you POST. Unfortunately, one sarscov2_illumina_full workflow's outputs exceeds this. We can work around this issue by using Rawl's batchUpsert. This is slightly different as it takes a list of operations on how to construct the entity rather than the entity serialised to TSV. In this PR, I've demonstrated how we can use Rawls to import the workflow's outputs into the workspace. I've also updated the demo to do this for a workflow that has already passed.
…them (#372) * Simplify day intervals. * Document wfl.tools.snapshots. * Map name over keyword arguments. * Force the production Rawls. * Checkpoint groups getters. * Patch rebase conflict. * Implement start-covid-workload! kinda. * Checkpoint start-covid-workload! test. * Add covid-workload-request to support unit. * clojure -M:format * Move Rich comment into a unit test. * Respond to comments and tidy up a little * Remove unit test now covered by integration. * Document clojure.test/test-vars to remind me.
Bumps [url-parse](https://github.com/unshiftio/url-parse) from 1.4.7 to 1.5.1. - [Release notes](https://github.com/unshiftio/url-parse/releases) - [Commits](unshiftio/url-parse@1.4.7...1.5.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update db schemas. * Make the names more clear. * Address comments. * Switch to SQL syntax from XML. * Address more comments. * one more update.
Bumps [y18n](https://github.com/yargs/y18n) from 4.0.0 to 4.0.1. - [Release notes](https://github.com/yargs/y18n/releases) - [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md) - [Commits](https://github.com/yargs/y18n/commits) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ur (#386) RR: https://broadinstitute.atlassian.net/browse/GH-1301 Introduce wfl.util.UserVisibleException - a new type of exception that we should handle to throw and handle errors that users are meant to see. Re-wire the background loop to handle UserVisibleException and also be more exception safe. Start to plumb in email notifications.
To enable Slack notifications, set to "enabled": all other values disable them.
* Rebase on develop. * fix call to find-and-snapshot-new-rows * Maybe fix what the rebase broke. * Reformat docstring for readability. * Hide some GCP details. * Fix what I broke. * Re-format a bit for readability. * Checkpoint until I can get back to this. * Checkpoint before more refactoring. * Checkpoint before refinement. * Checkpoint more experiments. * Delete cromwell/release-hold. * Everything compiles again. * Enable all log levels and fix bugs. * Simplify most JDBC macros back to functions. * clojure -M:format ... :jeez: * Skip labels and uuid for now. * Fix rebase botches, clean up, and abandon some logs for now. * Simplify. * clojure -M:format ... * Checkpoint tests passing again. * Consistency. * Back off the log level. * Compute log severity at compile-time. * Parse Clio responses more carefully. * Clean up and fix bugs discovered along the way. * OK's review comments. * Give up on value-fn in favor of extend. * Add a JSON handler for ExceptionInfo. * Template out the log macros. * Simplify by removing redundant options. * Update documentation. Co-authored-by: Tom Lyons <tbl3rd@gmail.com>
* Bump version to 0.15.0. * Update the CHANGELOG.
* Set the API's image pullPolicy to Always. * Bump another random "semantic" version number.
* Update reitit. * Reduce cleverness. * Expose the logging_level API. * Add logging_level endpoints. * Fix bug in level-string? ... jeez. * Add logging_level endpoint test. * clojure -M:format` in this directory. * Respond to OK's comments.
And remove unnecessary module imports in `wfl.api.handlers`: previously WFL would complain about invalid pipelines when creating workloads without these imports, but this no longer seems to be the case.
Define concise, consistent workload representations for logging. Integrate into server, slack, and source (for staged workloads) logs.
…-filters. (#583) * Look at only the first 7 workloads returned. * Simplify.
Slack token now stored in environment variable, and propagated from Vault to GitHub Secrets via Atlantis.
…oud (#585) And log warning instead of error on method config version mismatch.
GH-1629 TerraExecutor gets method configuration version from Firecloud (#585) GH-1553 GH-1604 Add error handling to Slack service (#584) GH-1628 Look at the first 7 workloads returned in test-workflows-by-filters. (#583) GH-1617 Consistent workload representation for logs (#579) GH-1624 Rename covid -> staged in code, docs (#580)
Now Slack notifications are always enabled.
#591) And fix failing Firecloud integration tests by referencing a new unarchived workflow.
tbl3rd
reviewed
Mar 21, 2022
tbl3rd
approved these changes
Mar 21, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Changes
System Tests
Passed: