Core consonance utilities for scheduling, reporting on, and provisioning VMs for workflows
Java Shell Python Other
achave11 Bugfix/wes (#167)
* Added repository

* Added swagger generated WES client to the consonance project, the project builds with maven, and should pass travis integration testing.

* Auto-generated files, classes, mathods from swagger-codegen

* Exclude travis.yml

*  Builds locally, debuging Travis CI

* Added the client, debuggin tests

* Added WES server side

* Server endpoints for WES implemented, except for delete. Adapted to integration-testing, expect failures.

*  Working implementation of ga4gh server with succesffull responses

*  Added unitOfWork to endpointnt, fixes some runtime errors. Temporarely stoped testing WES-integration tests, can only be invoked through client. Not yet available, should be available in next update.

* Added WES client to the swagger-java maven project repo

*  Pushing lattest changes, server side fully implemented and working, delete funct not yet avalable.

*  Added WES implementation to consonnce, delete functionality not yet implemented, need to be added to the consonance infrastructure

* Deprecated changes to delete functionality

*  Added info to key_values under get_service-info for defining a flavour

* Removed non-functional tests, will include testing infrastructure with ga4gh-wes-client, when implemented.

* New dependency checker, failing on license test, skipping test

*  Modified tidelift script

*  Modified tidelift script

*  surefire test failure fix

* Updated dependency versions, and fix maven-surefire tests activated fail on warning

* Implemented changes suggested by @david4096. Most of his suggestions were implemented.

* Improvements to the postgres request, returns results in order.

* Changes to integrating dockstore call effectively.
 * Provision dockstore plugins to actuate on s3 buckets.
 * Function to determine cwl tool or workflow. Or if WDL

* Temp reaper bug for the ContainerProvisionerThreads.java by
commenting out the code block that reaps instances.

Modified pom to include more resent version of the jersey-client
compatible with dockstore but not compatible with youxia.

FUTURE:
Synchronize versioning scheme through consonance codebase.

* Fixed format for calling function to identify descriptor type.

* Added exclusions to avoid conflict with transitive dependencies mismatch.
Checked with maven-surefire. Ready for merge.

* Hot fix for release. Fixed issues with returning correct id, when scheduling a job.
Latest commit 69e683c Jun 28, 2018
Permalink
Failed to load latest commit information.
.travis Generify consonance config for travis Sep 29, 2015
consonance-arch
consonance-client
consonance-common Modified porm to arch, added version to youxia-common Dec 6, 2017
consonance-integration-testing Feature/abraham wes support (#161) Jun 26, 2018
consonance-reporting Feature/abraham wes support (#161) Jun 26, 2018
consonance-server-common Feature/abraham wes support (#161) Jun 26, 2018
consonance-webservice Bugfix/wes (#167) Jun 27, 2018
container-admin Looks like OpenStack Mitaka support is broken Mar 31, 2017
swagger-ga4gh-client Feature/abraham wes support (#161) Jun 26, 2018
swagger-java-client Feature/abraham wes support (#161) Jun 26, 2018
.gitignore
.tidelift.yml Feature/abraham wes support (#161) Jun 26, 2018
.travis.yml Feature/more cleanup (#133) Mar 7, 2017
LICENSE Add license Feb 24, 2016
README.md
checkstyle.xml Add license Feb 24, 2016
findbugs-exclude.xml Add license Feb 24, 2016
mvnvm.properties adding mvnvm prop Aug 19, 2017
pom.xml Feature/abraham wes support (#161) Jun 26, 2018

README.md

Consonance

Build Status Coverage Status

About

Consonance is a cloud orchestration tool for running Docker-based tools and CWL/WDL workflows available at Dockstore on fleets of VMs running in clouds. It allows you to schedule a set of Dockstore job orders, spinning up the necessary VMs on AWS, Microsoft Azure, or OpenStack via the Youxia library for provisioning cloud-based VMs, and then tearing them down after all work is complete.

We are currently at work on Consonance 2.0 which supports anything from Dockstore and allows users to submit a variety of jobs intended for a variety of instance types.

The latest unstable releases on develop support running tools/workflows from Dockstore.

Consonance 1.0 is currently in maintenance mode and has also been integrated into a simplified workflow launcher Pancancer Workflow Launcher for AWS for external users and a more flexible but less well documented Pancancer Workflow Launcher for users internal to OICR.

The latest stable releases on master support the pancancer project https://icgc.org/working-pancancer-data-aws

See the Consonance wiki for more information on this project.

Dependencies

I'm showing how to install dependencies on a Mac so adapt to your system/OS.

PostgreSQL

You need a PostgreSQL server to run the tests, on the Mac try Postgres.app.

Once you have this setup (using whatever technique is appropriate for your system) create the queue_status database.

$bash> "/Applications/Postgres.app/Contents/Versions/10/bin/psql" -p5432 -d "postgres"
postgres=# create database queue_status;
postgres=# create user queue_user;
postgres=# grant all privileges  on database queue_status to queue_user;

Now load the schema:

$bash> cat ~/gitroot/Consonance/consonance/consonance-arch/sql/schema.sql | "/Applications/Postgres.app/Contents/Versions/10/bin/psql" -p5432 -d "queue_status"

RabbitMQ

The integration tests require RabbitMQ, here's how to install via home brew:

$bash> brew install rabbitmq
$bash> /usr/local/sbin/rabbitmq-server

Consonance Config File

By default the client looks for ~/.consonance/config. You need this file for tests to pass:

$bash> mkdir ~/.consonance/ && cp ./consonance-client/src/test/resources/config ~/.consonance/config

Dockstore CLI

The integration tests (see below for how to trigger) will actually simulate the full lifecycle of a WDL workflow run using the Dockstore CLI. See http://dockstore.org for how to install, you need the dockstore command dependencies in your path and properly configured in order for the full integration tests to pass. Consonance is using the Dockstore CLI library so you only need to install cwltool in your path if you want to test CWL workflows. Cromwell for WDL workflows is baked into the Dockstore CLI library.

Docker

Related to the Dockstore CLI above, one of the WDL workflows used for testing requires Docker. Make sure the system you are using for testing has Docker installed if you run the full integration tests.

Building & Testing

The build uses maven (3.2.3, look at using MVNVM), just run:

$bash> mvn clean install

To avoid tests (probably a bad idea!):

$bash> mvn -Dmaven.test.skip=true clean install

Skip tests and use parallel build (see more info here):

$bash> mvn -Dmaven.test.skip=true -T 1C install -pl consonance-integration-testing -am

This gives me a build time of 36 seconds vs. 1:21 min for mvn clean install

Now run the full integration tests (assumes you have RabbitMQ and PostgreSQL installed):

$bash> cd consonance-integration-testing
# run a particular test class
$bash> mvn -Dtest=SystemMainIT test
# or even a speicifc test method!
$bash> mvn -Dtest=SystemMainIT#testGetConfiguration test
# or all ITs
$bash> cd ..
$bash> mvn -B clean install -DskipITs=false

Monitoring Integration Tests

RabbitMQ

See http://localhost:15672/#/queues for your queues, username and password is guest by default.

Releasing

See the developer page on our wiki.

Installation

See the container-admin README for information on quickly setting up Consonance via Docker-compose and an interactive bootstrap configuration script.

Using

See the quickstart guide and the non-quickstart guide on our wiki.

TODO

  • test container-admin with pre-release locally with provisioning on AWS
  • review and merge the pull request
  • perform release
  • test container-admin with release on AWS, should be a safe harbor
  • then review/try WES support branch from Abraham