The Amazon Genomics CLI project has entered its End Of Life (EOL) phase. The code is no longer actively maintained and the Github repository will be archived on May 31 2024. During this time, we encourage customers to migrate to AWS HealthOmics to run their genomics workflows on AWS, or reach out to their AWS account team for alternative solutions. While the source code of AGC will still be available after the EOL date, we will not make any updates inclusive of addressing issues or accepting Pull Requests.
The Amazon Genomics CLI is a tool to simplify the processes of deploying the AWS infrastructure required to run genomics workflows in the cloud, to submit those workflows to run, and to monitor the logs and outputs of those workflows.
To get an introduction to Amazon Genomics CLI refer to the Quick Start Guide in our wiki.
For full documentation, please refer to our docs.
All releases can be accessed on our releases page.
The latest nightly build can be accessed here: s3://healthai-public-assets-us-east-1/amazon-genomics-cli/nightly-build/amazon-genomics-cli.zip
To build from source you will need to ensure the following prerequisites are met.
There are a few prerequisites you'll need to install on your machine before you can start developing.
Once you've installed all the dependencies listed here, run make init to install the rest.
The Amazon Genomics CLI is written in Go.
To manage and install Go versions, we use goenv. Follow the installation instructions here.
Once goenv is installed, use it to install the version of Go required by the
Amazon Genomics CLI build process, so that it will be available when the build
process invokes goenv's go shim:
goenv installYou will need to do this step again whenever the required version of Go is changed.
Amazon Genomics CLI makes use of the AWS CDK to deploy infrastructure into an AWS account. Our CDK code is written in TypeScript. You'll need Node to ensure the appropriate dependencies are installed at build time.
To manage and install Node versions, we use nvm.
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.38.0/install.sh | bash
echo 'export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"' >> ~/.bashrc
echo '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm' >> ~/.bashrc
source ~/.bashrc
nvm installNote: If you are using Zsh, replace ~/.bashrc with ~/.zshrc.
OSX uses an outdated version of sed. If you are on a Mac, you will need to use a newer version of sed to ensure script compatibility.
brew install gnu-sed
echo 'export PATH="$(brew --prefix gnu-sed)/libexec/gnubin:$PATH"' >> ~/.bashrc
source ~/.bashrcNote: If you are using Zsh, replace ~/.bashrc with ~/.zshrc.
Once you've installed all the dependencies listed here, run make init to automatically install all remaining dependencies.
We use make to build, test and deploy artifacts. To build and test issue the make command from the project root.
If you're experiencing build issues, try running go clean --cache in the project root to clean up your local go build cache. Then try to run make init then make again. This should ideally resolve it.
Unlike running 'run-dev.sh' script, this option will build and install a new version of Amazon Genomics CLI, replacing
the one installed. To run a release version of Amazon Genomics CLI from your local build, first build your changes and then run make release.
This will create a release bundle dist/ at this package root directory. Run the install.sh script in the dist folder
to install your local release version of Amazon Genomics CLI. After installing, you should be able to run agc on the terminal.
To run against a development version of Amazon Genomics CLI, first build your relevant changes and then run ./scripts/run-dev.sh. This will
set the required environment variables and then enter into an Amazon Genomics CLI command shell.
- Update dependencies and build code with
make init && make. At this point the compiled binary will be found atpackages/cli/bin/local/agc. - Optionally, you may build the install package and install the binary and CDK libraries.
make release && (cd dist/amazon-genomics-cli && ./install.sh) - Before creating any contexts, ensure you have the relevant environment variables set to point to the ECR repository holding the images of the engines you wish to test. Leave these values unset to test against production images.
export ECR_CROMWELL_ACCOUNT_ID=<some-value>
export ECR_CROMWELL_REGION=<some-value>
export ECR_CROMWELL_TAG=<some-value>
export ECR_CROMWELL_REPOSITORY=<some-value>
export ECR_NEXTFLOW_ACCOUNT_ID=<some-value>
export ECR_NEXTFLOW_REGION=<some-value>
export ECR_NEXTFLOW_TAG=<some-value>
export ECR_NEXTFLOW_REPOSITORY=<some-value>
export ECR_MINIWDL_ACCOUNT_ID=<some-value>
export ECR_MINIWDL_REGION=<some-value>
export ECR_MINIWDL_TAG=<some-value>
export ECR_MINIWDL_REPOSITORY=<some-value>
export ECR_TOIL_ACCOUNT_ID=<some-value>
export ECR_TOIL_REGION=<some-value>
export ECR_TOIL_TAG=<some-value>
export ECR_TOIL_REPOSITORY=<some-value>These environment variables point to the ECR account, region, repository and tags of the Cromwell, Nextflow, MiniWDL, and Toil engine respectively that will be deployed for your contexts. They are used when you create a context using the corresponding engine types.
The ./scripts/run-dev.sh contains logic to determine the current
dev versions of the images which you would typically use. You may also use production images, the current values of which will
be written when you activate an account with the production version of Amazon Genomics CLI. If you have customized containers that you
want to develop against you can specify these however you will need to make these available if you wish to make pull requests
with code that depends on them.
This package is buildable with AWS CodeBuild. You can use the AWS CodeBuild agent to run CodeBuild builds on a local machine.
You only need to set up the build image the first time you run the agent, or when the image has changed. To set up the build image, use the following commands:
git clone https://github.com/aws/aws-codebuild-docker-images.git
cd aws-codebuild-docker-images/ubuntu/standard/5.0
docker build -t aws/codebuild/standard:5.0 .
docker pull amazon/aws-codebuild-local:latest --disable-content-trust=falseCreate an environment file (e.g. env.txt) with the appropriate entries depending on which image tags you want to use.
CROMWELL_ECR_TAG=2021-06-17T23-48-54Z
ECR_NEXTFLOW_TAG=2021-06-17T23-48-54Z
WES_ECR_TAG=2021-06-17T23-48-54ZIn the root directory for this package, download and run the CodeBuild build script:
wget https://raw.githubusercontent.com/aws/aws-codebuild-docker-images/master/local_builds/codebuild_build.sh
chmod +x codebuild_build.sh
./codebuild_build.sh -i aws/codebuild/standard:5.0 -a ./output -c -e env.txtThe default values for all variables are placeholders (e.g. 'WES_ECR_TAG_PLACEHOLDER'). It is replaces by the actual value during a build process.
Local environment variables:
ECR_WES_ACCOUNT_IDECR_WES_REGIONECR_WES_TAG
The corresponding AWS Systems Manager Parameter Store property names:
- /agc/_common/wes/ecr-repo/account
- /agc/_common/wes/ecr-repo/region
- /agc/_common/wes/ecr-repo/tag
Local environment variables:
ECR_CROMWELL_ACCOUNT_IDECR_CROMWELL_REGIONECR_CROMWELL_TAG
The corresponding AWS Systems Manager Parameter Store property names:
- /agc/_common/cromwell/ecr-repo/account
- /agc/_common/cromwell/ecr-repo/region
- /agc/_common/cromwell/ecr-repo/tag
Local environment variables:
ECR_NEXTFLOW_ACCOUNT_IDECR_NEXTFLOW_REGIONECR_NEXTFLOW_TAG
The corresponding AWS Systems Manager Parameter Store property names:
- /agc/_common/nextflow/ecr-repo/account
- /agc/_common/nextflow/ecr-repo/region
- /agc/_common/nextflow/ecr-repo/tag
See Reporting Bugs/Feature Requests for more information. For a list of open bugs and feature requests, please refer to our issues page.
See Contributing via Pull Requests
See Security Issue Notification for more information.
This project is licensed under the Apache-2.0 License.