Release process

Introduction

This document covers the release process. We distinguish between two types of releases:

testing:
- alpha/beta/release-candidate releases
- frequent releases, small to medium testing durations
- release notes under testing
- not recommended for production use
production or stable:
- major/minor/patch releases
- rare releases, long testing durations
- release nodes under production
- accompanying versioned documentation
- recommended for production use

Naming scheme: from development branch to stable release

We follow the semver versioning scheme.

Version naming and relation to branches is as follows: example for a 1.1 release.

1.1.0-alpha.20170601:
- type: testing
- branch: master
- the YYYYMMDD string is the release date
1.1.0-beta.20170901:
- type: testing
- branch: release-1.1.0. Branch is cut for the first beta release
- the YYYYMMDD string is the release date
1.1.0-rc.1:
- type: testing
- branch: release-1.1.0
1.1.0:
- type: production
- branch: release-1.1.0

Be sure to always include the trailing .0, if present! Tags should be named, e.g., v2.1.0, never v2.1.

Release tracking

When starting the process for a new release, you should file an issue to track progress.

Title: release: <release string>. eg: release: 1.1.2 Copy the following todo list into the issue:

Candidate SHA: XXX EDIT ME XXX.
Deployment status: XXX EDIT ME XXX.

[Release process checklist](https://github.com/cockroachdb/cockroach/wiki/Release-Process)

Prep date: `XXX fill in date to roll out: usually Monday, a week before the release XXX`
- [ ] [Pick a sha](https://github.com/cockroachdb/cockroach/wiki/Release-Process#pick-sha), fill in `Candidate SHA` above.
- [ ] [Deploy to test clusters and start nightly suite](https://github.com/cockroachdb/cockroach/wiki/Release-Process#deploy-to-test-clusters)
- [ ] fill in `Deployment status` above with clusters and link to Nightly TeamCity Job
- [ ] [Verify node crash reports](https://github.com/cockroachdb/cockroach/wiki/Release-Process#verify-crash-reports)
- Keep an eye on clusters until release date. Do not proceed below until the release date.

Release date: `XXX fill in date: usually Monday XXX`
- [ ] [Check cluster status](https://github.com/cockroachdb/cockroach/wiki/Release-Process#check-cluster)
- [ ] [Tag release](https://github.com/cockroachdb/cockroach/wiki/Release-Process#tag-release)
- [ ] [Trigger release build](https://github.com/cockroachdb/cockroach/wiki/Release-Process#trigger-build)
- [ ] [Check binaries](https://github.com/cockroachdb/cockroach/wiki/Release-Process#check-binaries)

- For production or stable releases in the latest major release series
  - [ ] Update the [Homebrew Formula](https://github.com/Homebrew/homebrew-core/blob/master/Formula/cockroach.rb)
  - [ ] Update the image tag in our [Orchestrator configurations](https://github.com/cockroachdb/cockroach/wiki/Release-Process#orchestrator-configurations)
  - [ ] Update the version label in our [Red Hat Atomic Docker image](https://github.com/cockroachdb/cockroach/blob/master/build/deploy/rhel/Dockerfile). For major/minor releases, also update the documentation link in the [image's help file](https://github.com/cockroachdb/cockroach/blob/master/build/deploy/rhel/help.1)
  - [ ] Announce version to the [registration cluster](https://github.com/cockroachlabs/production#publishing-a-new-release-version)
  - [ ] Upgrade the registration cluster (reg runs latest stable version)

- [ ] Update docs
- [ ] External communications for release
- [ ] File issue for next release and assign to release coordinator

Picking a sha

Alpha releases use commits from the master branch; all other releases use a branch named release-<major>.<minor>.<patch>. eg: release-1.1.0. In general, the release should use the last commit before noon (NYC time) on the appropriate branch.

IMPORTANT: Be sure to pick a SHA that has a green "Publish Bleeding Edge" build in TeamCity. Sometimes broken commits can end up on master or a release branch due to merge skew. It's also possible for a commit to pass the PR build, but not the release build--this happens whenever a commit breaks only the Windows build, for example.

On the teamcity bleeding edge page look for the latest green build on the appropriate branch. For example, the most recent 2.0 point release (e.g., 2.0.4) will look for a branch that says 2.0. Then click the drop down arrow under changes and select view source revisions. The candidate sha will be the top listing under revision and look like d7c99735249c963177eadbc5fd57fc2094de6823

You can also doublecheck this by looking on https://github.com/cockroachdb/cockroach/, select the appropriate branch (e.g., release-2.0) and then looking at the latest commit.

Backport procedure

All development starts on the master branch. In most cases, to add a commit to a patch release (or a beta/rc after a release branch has been created), you should submit a PR to master first and get it reviewed and merged there. After the PR is merged to master, follow these steps to cherry-pick it into the release branch.

(These steps are automated by https://github.com/benesch/backport)

checkout the release branch: git checkout origin/release-2.0
create a branch for your backport: git checkout -b <username>/backport_<pr number> (the branch name doesn't matter)
cherry-pick the commits from the PR: git cherry-pick <commit sha> (if the PR contained multiple commits, do this for each one in order)
send a pull-request for your backported changes
1. prepend release-2.x: to the title
2. choose the appropriate release-2.0 branch as the target for the PR
3. assign @cockroachdb/release and the author or reviewer of the original master PR as reviewers
4. If there were any non-trivial merge conflicts, be sure to call those out in the PR message so reviewers can pay closer attention to the diff

Deploy to test clusters

Trigger a Nightly Suite build for the chosen SHA.

Note, you might need permissions within teamcity to use the run button necessary to trigger a nightly test. Once you have permissions, click the run options, then the changes tab, then select the appropriate release. For example, select release 2.0. Then in the include changes tab pick the same sha you identified previously.

For testing we use "portable" binaries. Portable binaries are built against a more compatible (read: older) libc that is expected to be available across more linux distributions. These binaries are built automatically from merges to the master/release branch. Check that a binary for your sha exists:

https://edge-binaries.cockroachdb.com/cockroach/cockroach.linux-gnu-amd64.<candidate SHA for release>

As of 2017-11-06, you deploy to adriatic and then omega 24 hours later if all goes well. See the production wiki for the definitive list of clusters to use.

Make sure you have a current version of the cockroachlabs/production repository.
You'll use crl-prod tools in some of the next steps. If you haven't already, add the relevant directory to your PATH so you can run the tools from anywhere:
```
export PATH=${PATH}:${GOPATH}/src/github.com/cockroachlabs/production/crl-prod
```

Perform a rolling upgrade on adriatic:

COCKROACH_SHA=<candidate SHA for release> continuous_deployment/rolling_restart_cluster.sh adriatic

You'll need to be in the production folder when you run that command. Also, you should delete <> as well.

You may also need to run this with a wipe of the cluster if it doesn't work such as

COCKROACH_SHA=<candidate SHA for release>./continuous_deployment/wipe_cluster.sh omega

If that doesn't work, manually stage the binaries:
```
COCKROACH_SHA=<candidate SHA for release> crl-stage-binaries adriatic all cockroach
```

Then do the rolling restart by hand.

Wait for 24 hours. If all looks good, tell everyone on #release and #production that you are pushing the binary to omega.

Repeat the rolling upgrade process on omega:

COCKROACH_SHA=<candidate SHA for release> continuous_deployment/rolling_restart_cluster.sh omega

Note that we tend to use clusters for alpha and the patch releases, which means that you may run into cluster version errors. In such a case, simply wipe the offending cluster after verifying that the error is expected. For instance, if a cluster previously ran the latest alpha and you deploy a patch release, you may see nodes crash at startup with:

cockroach server exited with error: inspecting engines: engine =/mnt/data1 requires at least v1.1-3, but running version is 1.1

Verify node crash reports appear in sentry.io

Warn in #monitoring that you're about to intentionally crash a node.
Connect to a node (e.g., crl-prod/crl-sql adriatic 1)
Ask it to crash with select crdb_internal.force_panic('testing');
Verify that the crash report appears in sentry.io. You may have to dig on sentry.io, intentional panics get suppressed and don't make it to the slack channel.
Resolve the crash report in sentry.io.
Restart the node (e.g., crl-prod/crl-supervisor adriatic 1 start cockroach).

Check cluster status

Look at the health of the cluster
Document any known problems in the issue.
Go/nogo decision: this is very subjective. Discuss whether to proceed or not.

Tag the release

Wait until the release date before proceeding.
Make sure you are in cockroachdb/cockroach
Run git fetch, if you haven't recently.
Create a name for the tag: $ export TAG=v<version string>, where <version string> is the version string from the title of the issue.
Add the git tag: $ git tag ${TAG} ${SHA}
Push the git tag: $ git push origin ${TAG}
Verify that the tag was pushed successfully. It should be at the top of this list.
Click on the tag and make sure the most recent commit has the SHA slated for release.

If you get an unexpected prompt for your GitHub credentials when pushing the tag to origin, try this command to switch your git remote from https to ssh:

$ git remote set-url origin git@github.com:cockroachdb/cockroach.git

Otherwise, make sure you're pushing to upstream.

Trigger the release build

Pushing a tag matching does not automatically trigger a build. We'll trigger a build that will publish the new binaries to the well-known locations that our Download links point to.

NB: it has happened that the first time the release binaries were built, teamcity hadn't picked up the tag yet. When this happens (you'll find out during verification), try to rebuild but be aware that the S3 bucket serving the binaries may cache the old binary. You'll need to find someone with cloudfront (not cloudflare) access to clear the cache.

Go here in teamcity.
In the Active branches drop-down, select the branch that matches the tag.
Click Run in the upper right.

Wait for the TC build to complete successfully before the next step.

Check binaries

Follow the instructions at cockroachdb/bincheck. Tests are only run once a PR is made! If the tests are green, merge and continue below. If you run into trouble, carry out the steps in bincheck manually.

Next only for production or stable release,

Check the docker image.

$ docker run cockroachdb/cockroach:${TAG} version
Status: Downloaded newer image for cockroachdb/cockroach:beta-20170209
Build Tag:    beta-20170209 // verify tag!
Build Time:   2017/02/09 16:11:47
Distribution: CCL
Platform:     linux amd64
Go Version:   go1.7.5
C Compiler:   gcc 4.9.3
Build SHA-1:  3271ce4097dcb2d7ad74b877b011dd6ee62f04d5 // verify!
Build Type:   release // verify! This is release-gnu for 1.1 releases.

If you're doing an alpha or beta release, you want to check an "unstable" image:

docker run cockroachdb/cockroach-unstable:${TAG} version

Update version numbers

For stable releases only.

Brew recipe

If you have Homebrew, just run (with TAG set. For example, TAG=v1.1.1):

$ brew bump-formula-pr --url=https://binaries.cockroachdb.com/cockroach-${TAG}.src.tgz --version=${TAG#v} cockroach

Otherwise, something like the following should do the trick:

Check out the Homebrew formula repository and make a branch.

$ git clone https://github.com/homebrew/homebrew-core.git
$ git checkout -b cockroach-${TAG#v}

Open Formula/cockroach.rb.
Update the url and version lines with the new version. You'll need to update the sha256 line with the SHA-256 hash of the source archive. Something like the following should do the trick:
```
curl https://binaries.cockroachdb.com/cockroach-${TAG}.src.tgz | shasum -a 256
```
Send PR.

Homebrew's CI will automatically build the cockroach binary and run a basic test. Circle back in about ten minutes. If Homebrew's CI is happy, that's it! You're done. Someone from Homebrew will come along and merge the PR later.

Orchestrator configurations

For stable releases in the last major version:

Update the Docker image tag version in our Kubernetes config files to the new version number. To replace all of them at once, run this command from the root of the repo with the appropriate old/new version numbers:

find ./cloud/kubernetes/ -type f -exec sed -i '' -e 's/cockroach:v2.0.0/cockroach:v2.0.1/g' {} \;

Update the version label in our Red Hat Atomic Docker image:

https://github.com/cockroachdb/cockroach/blob/master/build/deploy/rhel/Dockerfile

Send out a PR with these changes.

Update the version numbers in our Mesosphere DC/OS configuration files to the new version number. See the below PR for an example:

https://github.com/cockroachdb/dcos-cockroachdb-service/pull/39

We also maintain configuration files outside our own repo. It's ok if these lag behind by a couple patch versions, but if you'd like to contribute to them it's welcome. Assign @a-robinson to any such reviews.

Update the registration cluster

Once Docs has merged their PR and the URL for the version docs is visible, add the version to the registration server.

Upgrade reg cluster

The reg cluster runs the latest stable release.

e.g. RELEASE_VERSION=v2.0 crl-stage-binaries register all release
For each node id [1..4]: crl-supervisor register <node id> stop cockroach && sleep 1 && crl-supervisor register <node id> start cockroach

Make sure to do this one at a time and verify within the webui that it is back up and with the new version number. Note that the registration cluster is fragile and needs to be done one at a time.

Update docs

For now, this is handled by the Docs team.

Generate release notes

In the cockroach repo, make sure master is up-to-date and you have the latest tags:
```
$ git checkout master
$ git pull
$ git fetch --tags
```
Use scripts/release-notes.py to auto-generate draft release notes:
```
python3 scripts/release-notes.py --from=<tag of last release> --until=<candidate SHA> --hide-unambiguous-shas --hide-per-contributor-section > draft-release-notes.md
```
You can get the tag of the last release from the Tags page and the candidate SHA from the cockroach issue for the release.

Update the docs site

Create a new branch in docs.
In the releases directory, add a new <version>.md file for the release notes, e.g., v1.1.3.md.
- Use the content of the previous release notes file as a starting point.
- Be sure to update the version in the title and summary front-matter fields, the h1 header, and the binary download links.
Copy the contents of draft-release-notes.md into the release notes file and copy edit.
Update the version in the front-matter and download links.
In _config.yml, edit the release_info section:
- Update the version: line with the new tag.
- Update the build_time: line with the new tag and build date.
- Update the start_time: line to any time after the build_time.
In the _data directory, add the release to either the Production Releases or Testing Releases section of releases.yml.
- If it's a production release, be sure to move latest: true to the new version.
If it's a production release (e.g., v1.1.2 not v1.2-alpha.20171026), in the _includes directory, add the new release notes file to sidebar-releases.json.

Edit release notes

Copy edit the generated text. Some guidelines:
- Use the past tense (e.g., "Fixed xyz") or use "now" to describe how the feature works currently (e.g., "The Node Map now shows").
- Add links to docs where relevant.
Ask @nate for summary bullets for the release and add at the top of the doc.
If there are new docs or doc changes worth calling out, summarize the changes in the Doc Updates section; otherwise, delete that section. Ask @jseldess for this summary.
Open the PR and get it reviewed by somebody on the Docs team.
Once the release binaries are ready, click on their links to make sure they download, rerun TeamCity, and then merge when green.
Once the PR is successfully merged, tell @swati in the #release channel that the release notes are live and send the link.

Send external communications

Send release notes to users.
- Create styled email in HubSpot.
- Send styled email to "release notes" list.
- Send copy of styled email [less the "unsubscribe" options] to the User Group and Developers Group.

Home
Jargon
Roadmap
Git commit messages
Issue labels
Working as reviewer
Release Process
Building and running tests
Code Coverage
Continuous Integration
- TeamCity
- Bors merge bot
Nightly tests
Large Scale Testing
Productivity
Life of a transaction
Readings
Misc (ccache)

cockroachdb/cockroach

Release process

Introduction

Naming scheme: from development branch to stable release

Release tracking

Picking a sha

Backport procedure

Deploy to test clusters

Verify node crash reports appear in sentry.io

Check cluster status

Tag the release

Trigger the release build

Check binaries

Check the docker image.

Update version numbers

Brew recipe

Orchestrator configurations

Update the registration cluster

Upgrade reg cluster

Update docs

Generate release notes

Update the docs site

Edit release notes

Send external communications

Pages 34

Clone this wiki locally

cockroachdb/cockroach

Join GitHub today

Release process

Introduction

Naming scheme: from development branch to stable release

Release tracking

Picking a sha

Backport procedure

Deploy to test clusters

Verify node crash reports appear in sentry.io

Check cluster status

Tag the release

Trigger the release build

Check binaries

Check the docker image.

Update version numbers

Brew recipe

Orchestrator configurations

Update the registration cluster

Upgrade reg cluster

Update docs

Generate release notes

Update the docs site

Edit release notes

Send external communications

Clone this wiki locally