healthomics-workflows

UltimaGenomics repository for workflows compatible with AWS HealthOmics

Table of Contents

1. Introduction
2. Terraform Modules for Environment Setup
3. Deploying Private Workflow
4. Running Private Workflow
5. Deploying & Running Private workflow (manual)
6. Support Tools

Introduction

1. This repository provides Ultima Genomics pipelines that have been adapted to run as private workflows on AWS HealthOmics. Follow the directions here to deploy and run private workflows, including cases where you need flexibility in resource configuration, references, and workflow inputs.

2. Each workflow folder contains the following:

   • required wdl file\s
   • HowTo documentation that details the workflow flow and how to run it externally of wdl
   • documentation of the wdl inputs and outputs
   • json that list the parameters for creating workflow
   • folder with optional input templates with default parameters for the wdl
   • folder with the different tasks the wdl is running

3. The instructions below include localizing resources, deploying workflow and creating a run.

4. Warning: the current Ready2Run workflows on AWS HealthOmics are associated with pipelines that are not relevant for use with this repository. Do not use Ready2Run as the recommended path for these workflows.

5. For more questions about these workflows, please contact healthomics.support@ultimagen.com.

Terraform Modules for Environment Setup

This repository includes Terraform modules to simplify the deployment of AWS resources required to run Ultima Genomics WDL workflows on AWS HealthOmics. These modules automate the creation of necessary infrastructure, such as S3 buckets, IAM roles, and permissions, ensuring a consistent and secure environment for workflow execution. We strongly recommend using it for building your AWS omics env. Feel free to make customizations as per your request

Key features:

• Automated provisioning of AWS resources for HealthOmics workflows
• Easy integration with existing workflow deployment steps and version management

Refer to the terraform/healthomics-wdl-env/README.md for detailed usage instructions and module configuration options.

Deploying Private Workflow

To localize workflow resources and create a private workflow in AWS HealthOmics:

• use this script that localizes the necessary resources and create a private workflow on AWS HealthOmics. See scripts/healthomics_wf/README.md for detailed usage instructions. After running it your workflow is ready to run.
• Alternatively, follow this section to do it manually.

Running Private Workflow

If you built your environment with the healthomics-wdl-env terraform module and deployed the workflow using the script with --use-dynamodb, you can run your workflow using invoke_healthomics_run.py that invokes the StartOmicsRun lambda. See scripts/healthomics_wf/README.md for detailed usage instructions.

Alternatively, follow this section to do it manually.

Deploying and Running Private workflow (manual)

Deploying Private workflow (manual way)

To localize workflow resources to create a private workflow in AWS HealthOmics, follow the steps below: i. Pre requisites:

1. Ultima Genomics workflows uses several ECR containers, they are listed under each workflow tasks\globals.wdl.

2. Pull and push the required public containers to your private ECR by following the steps:

   a. Pull from docker hub or broad gcr into your local ecr

   docker pull <hub_username>/<image_name>:<tag> #the docker as it appear on globals.wdl
   docker tag <hub_username>/<image_name>:<tag> <your_aws_account_id>.dkr.ecr.<region>.amazonaws.com/<repository_name>:<tag>
   aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <your_aws_account_id>.dkr.ecr.<region>.amazonaws.com
   docker push <your_aws_account_id>.dkr.ecr.<region>.amazonaws.com/<repository_name>:<tag> #if repository doesn't exist, you will need to create it first
   

   b. Grant AWS HealthOmics permission to access your private ECR by following the instructions here.

3. Import your input files into a S3 bucket.

4. Create an OmicsService role to access your resources by following the instructions here.

ii. Download the workflow folder as a zipped file, this should include main wdl file on the top level folder, tasks folders and <workflow_name>_params.json . You can save this zipped file locally or in a S3 bucket.

iii. Download locally the parameter template for your desired use case from input_templates folder.

iv. Modify and save the workflow scripts and parameter templates to meet your needs:

• Update tasks/globals.wdl with the urls of you private ECR images.
• Update the template's parameter values with their local s3 paths. Current links are to public buckets and transfer cost might incur by using them

Once the workflow resources have been deployed into locally (see instructions per workflow), user can create private workflow on AWS HealthOmics

Create a private workflow in HealthOmics by following one of the two options below:

i. From the CLI:

$ aws omics create-workflow \
   --name <workflow_name> \
   --main <main_wdl_file> \  # in case there is more than one wdl file, the main one is the one named after the directory
   --definition-zip fileb://<path_to_local_zip> \
   --parameter-template file://<path_to_parameters_definition_json> \
   --accelerators GPU

ii. From the console:

a. Click on **Private Workflows** from the left pane.

b. Click on **Create Workflow** on the Workflows list.

c. Follow the instructions on the console to create your workflow.
   - Define "Main workflow definition file path" as <workflow_name>.wdl file

Running Private workflow (manual way)

Run your workflow by following one of the two options below:

i. From the CLI:

$ aws omics start-run \
   --workflow-id <workflow_id> \
   --role-arn <service_role_arn> \
   --output-uri <s3_uri_for_output_folder> \
   --parameters file://<path_to_local_parameters_file> \
   --name <run_name> \
   --retention-mode REMOVE

ii. From the console (current omics version doesn't work well with wdl scoped parameters, cli is preferred):

a. Click Private Workflows from the left pane.

b. Click the Workflow ID from the Workflows list.

c. Click Create Run and enter the run information.

Support Tools

In case your private workflow's run failed, you can use this script to extract information and logs from AWS HealthOmics run to ease failures debugging. Please attach the tar file generated by the script in any support call.