-
Notifications
You must be signed in to change notification settings - Fork 585
How to release GATK4
In order to be able to release GATK4, you first need to perform the following one-time setup tasks:
-
-
If you don't already have a sonatype account, create one by going to https://issues.sonatype.org/secure/Signup!default.jspa.
-
Check whether your sonatype account already has
Deploy and UI Upload
permissions for repo targetorg.broadinstitute
by going to https://oss.sonatype.org/#stagingProfiles while logged in. -
If you don't have
Deploy and UI Upload
permissions fororg.broadinstitute
, you need to request them. Ask someone who already has deployment permission to open a ticket on your behalf at https://issues.sonatype.org requesting permissions for you, and comment on the ticket yourself from your sonatype account once it's opened.
-
-
Set up your signing keys for publishing to sonatype
-
(These instructions are adapted from https://central.sonatype.org/publish/requirements/gpg/)
-
Install GPG 1.x (gpg1) via homebrew or your system's package manager. GPG 2.x does not work with our build system.
-
Run
gpg1 --gen-key
to generate a new keypair. Select the default values when asked for the kind and the size of the keys, and be sure to specify a passphrase for the keypair. -
Run
gpg1 --list-keys
andgpg1 --list-secret-keys
. The output should look something like this:$ gpg1 --list-keys /Users/YOUR_USERNAME/.gnupg/pubring.gpg --------------------------------- pub 4096R/4C1B2A36 2016-05-17 uid YOUR NAME <YOUR@EMAIL> sub 4096R/26A37141 2016-05-17
-
Take note of the path to your secret keyring file (eg.,
/Users/YOUR_USERNAME/.gnupg/secring.gpg
) and the public key id (4C1B2A36
in this case). -
Publish your public key by running
gpg1 --keyserver keyserver.ubuntu.com --send-keys YOUR_PUBLIC_KEY_ID
, replacingYOUR_PUBLIC_KEY_ID
with the key ID for your public key noted above (4C1B2A36
in the example).- If this fails with "Server indicated a failure" or "Network is unreachable", try using the IP address of the keyserver instead.
- If this still fails, try
keys.openpgp.org
as the keyserver.
-
-
Set up your credentials in gradle
-
Open
~/.gradle/gradle.properties
in a text editor (creating it if necessary). -
Add the following lines:
signing.keyId=ID_FOR_YOUR_PUBLIC_KEY signing.password=PASSPHRASE_FOR_YOUR_KEYPAIR signing.secretKeyRingFile=PATH_TO_YOUR_SECRET_KEYRING_FILE sonatypeUsername=YOUR_SONATYPE_USERNAME sonatypePassword=YOUR_SONATYPE_PASSWORD
-
Save the file.
-
-
Set up docker and your dockerhub account
-
Install docker from docker.com, and start the docker daemon.
-
Test that you can pull and run the latest GATK4 docker image:
docker pull broadinstitute/gatk:latest docker run -it broadinstitute/gatk:latest
-
If you don't already have a
dockerhub
account, create one by going to hub.docker.com -
Make sure that you have push access to the
https://hub.docker.com/r/broadinstitute/gatk/
repository on dockerhub, as well as theus.gcr.io/broad-gatk/gatk
repository on GCR. Ask to join thegatk-release
google group, which is used to manage these permissions (you may need to verify that you have been granted access). -
Enable "Experimental Features" in your Docker settings (Docker -> Preferences -> Daemon).
-
-
Set up your gcloud credentials
-
Follow the instructions in the GATK README to install the Google Cloud SDK and authenticate with gcloud.
-
Make sure your gcloud project is set to
broad-dsde-methods
by running the command:gcloud config set project broad-dsde-methods
-
Run
gcloud auth configure-docker
to allow docker to push to Google Cloud repositories
-
After you've performed the one-time setup steps above, here's what you need to do each time you want to release GATK4:
-
-
Check with other GATK developers to confirm that it's a good time to release. In particular:
-
Tools and arguments not ready for production use should be marked as
@BetaFeature
-
If a developer has added unstable functionality to a previously-stable tool, their new functionality should be isolated into a separate code path that's off by default and activated using an argument marked as experimental.
-
-
Make sure that all tests are passing on master in travis!
-
-
-
Write release notes in advance and save them in a text editor, otherwise you'll introduce a significant delay in the middle of the release process.
-
Use a markdown-formatted bulleted list, and include a bullet point for each new feature or bug fix in the release, with a brief summary of the major changes at the top. Consult with other GATK developers if necessary to understand new features/patches. Try to include links to the relevant github issue(s) for each item.
-
You can get a list of the commits since the last release by going to the tag for the previous release in https://github.com/broadinstitute/gatk/releases and clicking the
X commits to master since this release
link. This can form the basis for your release notes. -
A good way to quickly auto-generate a skeleton for markdown-formatted release notes is by running this command from your GATK4 clone with the latest master checked out:
git log --pretty=oneline PREVIOUS_VERSION..HEAD | grep -v "Merge pull request" | awk '{ $1=""; print "* " $0; }'
Replacing
PREVIOUS_VERSION
with the tag for the previous releaseThe following does the same but looks up the last tag automatically.
git log --pretty=oneline --no-merges $(git describe --abbrev=0)..HEAD | awk '{ $1=""; print "* " $0; }'
This version of the command also groups the commits by author:
git log --no-merges '--pretty=format:%an %h %s' $(git describe --abbrev=0)..HEAD | sort | awk '{ print "* " $0; }'
You'll have to tweak the output of these commands to produce the final release notes, as not all commit messages will be sufficiently descriptive, and you'll need to add links to the appropriate github issues.
This template may be helpful in formatting the notes.
**Download release:** [gatk-VERSION.zip](https://github.com/broadinstitute/gatk/releases/download/VERSION/gatk-VERSION.zip) **Docker image:** [https://hub.docker.com/r/broadinstitute/gatk/](https://hub.docker.com/r/broadinstitute/gatk/) **Highlights of the VERSION release:** -------------------------------------- * **Full list of changes:** ------------------------- * **New Tools** * * **HaplotypeCaller** * * **Mutect2** * * **Joint Calling** * * **GenomicsDB** * * **Funcotator** * * **Mitochondrial pipeline** * * **CNV Calling** * * **SV Calling** * * **Notable Enhancements** * * **Bug Fixes** * * **Miscellaneous Changes** * * **Documentation** * * **Dependencies** *
-
-
-
In your git clone of GATK4, save any uncommitted work and run
git checkout -f master
. Runninggit diff HEAD
should produce no output. -
Run
git pull --ff-only
from your master branch to bring it up to date. Check the commit at the top of thegit log
output and make sure that it matches the most recent commit athttps://github.com/broadinstitute/gatk/commits/master
. -
Run
./gradlew clean
to clean your build directory.
-
-
-
Decide on a version number for the new GATK4 release (for example,
4.beta.3
). -
Be sure that you still have the latest master branch commit checked out, as instructed above.
-
Tag the release by running
git tag -a VERSION_NUMBER
, whereVERSION_NUMBER
is the version you've chosen for the new release (eg.,4.beta.3
). Your text editor will pop up and ask you to write a commit message for the new tag (this can just be the release number for now). After doing so, save and then exit.git describe
should now print the new version. -
Push the new tag to github by running
git push origin VERSION_NUMBER
, whereVERSION_NUMBER
is the tag you just created (for example,git push origin 4.beta.3
). -
Run
./gradlew clean bundle -Drelease=true
. This will create a zip for the release inbuild/
-
Go to https://github.com/broadinstitute/gatk/releases, click on the new tag you just pushed, then click "Edit Release".
-
Attach the GATK4 release zip file you just created by clicking on the
Attach binaries by dropping them here or selecting them
link, and navigating to the zip file in thebuild/
subdirectory within your GATK4 clone. -
Paste the release notes you wrote earlier into the text box. Look at the markdown preview and make sure that they look reasonable.
-
Once the zip file has finished uploading to github, click the
Publish Release
button. -
Verify that the new release looks ok in https://github.com/broadinstitute/gatk/releases, and has an attached (non-source-code) GATK4 zip file.
-
-
-
Option 1 (do this if possible): build the docker image in the cloud using Google Cloud Build:
-
Confirm that you've set your gcloud project to
broad-dsde-methods
via the commandgcloud config set project broad-dsde-methods
-
From the root of your GATK clone, run:
bash build_docker_remote.sh -e ${GITHUB_TAG} -r -d ${STAGING_DIR}
Replace
${GITHUB_TAG}
with the tag for your new release that you created above (eg.,4.0.0.0
), and${STAGING_DIR}
with the name of the directory that should be used to stage the image. This directory should either not exist or be empty! -
The script will push the new image to
us.gcr.io/broad-dsde-methods/broad-gatk-snapshots/gatk-remote-builds:${YOUR_USERNAME}-${GITHUB_TAG}-${GIT_HASH_FOR_TAG}
by default, though this can be changed by running with the-t
option -
Once the image is built you should pull it and test it out locally before releasing it:
docker pull ${CLOUD_IMAGE_NAME} docker run -it ${CLOUD_IMAGE_NAME}
Replacing
${CLOUD_IMAGE_NAME}
with the actual image location output bybuild_docker_remote.sh
-
Once you've verified that the cloud image runs and looks ok, push it to the official release repositories by running:
bash scripts/docker/release_prebuilt_docker_image.sh "${CLOUD_IMAGE_NAME}" "${GITHUB_TAG}"
Where
${CLOUD_IMAGE_NAME}
is the cloud-built image pulled above, and${GITHUB_TAG}
is the tag for your new release (eg.,4.0.0.0
). You may need to rundocker login
to authenticate before running this script.
-
-
Option 2 (backup option for when the cloud build doesn't work): build the docker image locally:
-
Ensure that the docker daemon is running, and that you still have the latest master checked out, as instructed above. You'll also need to have done the steps for the github release already.
-
If you've done a docker release in the past, be sure to delete any old docker staging directories(s) left over in your GATK4 clone. You'll have to use
sudo
to do the deletion. -
From the root of your GATK clone, run:
bash build_docker.sh -e ${GITHUB_TAG} -p -u -d ${STAGING_DIR}
Replace
${GITHUB_TAG}
with the tag for your new release that you created above (eg.,4.beta.3
), and${STAGING_DIR}
with the name of the directory that should be used to build the image. This directory should either not exist or be empty! -
When prompted by the script, enter your dockerhub username and password.
-
After the script completes (may take 15-20 minutes depending on your upload speed), go to https://hub.docker.com/r/broadinstitute/gatk/tags/ and confirm that the new image is there.
-
Add a link to the new image in the release notes you wrote on github.
-
-
-
-
Be sure that you still have the latest master branch commit checked out, as instructed above. You'll also need to have done the steps for the github release already.
-
Check GATK4's dependencies in
build.gradle
. If there are any dependencies on a SNAPSHOT of a library, you can't release to maven central, and should skip the remainder of this section. -
Run
./gradlew clean publish -Drelease=true
. This builds the release and uploads it to sonatype's staging area. -
Go to https://oss.sonatype.org/#stagingRepositories, logging in if necessary. If you don't see anything, click "refresh".
-
Find the release you just uploaded. It will probably be at the bottom with a name like
orgbroadinstitute-1027
, with your user ID listed as owner. -
Check the box next to your release, then select "close". Press the refresh button repeatedly until it updates and its status says "closed".
-
Select your release again and click "release". Select the box to "automatically drop" in the pop-up confirmation dialog.
-
Wait ~30-180 minutes for the maven central release to happen. Once it happens, it will show up on http://maven.org. Due to caching on maven's website it's often available by directly navigating to the coordinates sooner than it shows up in search.
-
-
-
Publish the GATK WDLs to the gatk-tool-wdls repository by running the following command from the root of your GATK clone:
bash scripts/publish_gatk_tool_wdls.sh GATK_VERSION
where GATK_VERSION is the tag for the GATK release you just published
-
After publishing the GATK WDLs, you'll need to go into the my workflows page on Dockstore and manually publish any unpublished WDLs from the
gatk-tool-wdls
repository. This step might become possible to automate if https://github.com/dockstore/dockstore/issues/3876 is ever implemented. -
Generate GATKDocs via
./gradlew gatkDoc
, then upload the GATKDocs to the GATK website- TODO: add website upload instructions
-