Permalink
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
237 lines (179 sloc) 10.5 KB

Bazel Release Playbook

Status: Work in progress

This is the guide to conducting a Bazel release. This is especially relevant for release managers, but will be of interest ti anyone who is curious about the release process.

Each release has a tracking bug (see the list). The bug includes a "Target RC date". On that day, create a new release candidate.

Creating a new release candidate

Setup

Do these steps once per release.

Create a Candidate

Create candidates with the release.sh script.

  1. If it's the first candidate for this version, run:

    RELEASE_NUMBER=<CURRENT RELEASE NUMBER x.yy.z>
    BASELINE_COMMIT=01234567890abcdef # From the Setup phase
    git clone https://github.com/bazelbuild/bazel.git ~/bazel-release-$RELEASE_NUMBER
    cd ~/bazel-release-$RELEASE_NUMBER
    scripts/release/release.sh create $RELEASE_NUMBER $BASELINE_COMMIT [CHERRY_PICKS...]

    Note that the three-digit version is important: "0.19.0". not "0.19".

  2. For cherry-picks, you need --force_rc=N where N is the number of the release candidate of $RELEASE_NUMBER. For example, the first time you do a cherry-pick (after the initial candidate), N will be 2.

    scripts/release/release.sh create --force_rc=2 $RELEASE_NUMBER $BASELINE_COMMIT [CHERRY_PICKS...]
  3. If you already did some cherry-picks and you want to add more, use "git log" to find the latest commit (this corresponds to the last cherry-pick commit). Use that as the new baseline and list the new cherry-picks to add on top. Or simply re-use the same baseline and cherrypicks from the previous candidate, and add the new cherrypicks.

    scripts/release/release.sh create --force_rc=3 $RELEASE_NUMBER NEW_BASELINE_COMMIT [NEW_CHERRY_PICKS...]
  4. Resolve conflicts if there are any, type exit when you are done, then the script will continue.

    • WARNING: release.sh create handles conflicts in a subshell (which is why you need to type exit).
  5. Check/edit release notes.

  6. Run release.sh push. This uploads the candidate and starts the release process on BuildKite.

    scripts/release/release.sh push
  7. Update GitHub issue with the command that was run and the new candidate name (ie, 0.19.1rc3).

  8. Check BuildKite results at https://buildkite.com/bazel-trusted/bazel-release. You should see the release-$RELEASE_NUMBER branch here and a new build running for your release.

    • If the build fails with "Build creator not allowed", simply start a new one by clicking on the "New build" button in the top right corner (Issue #281).
  9. Check the postsubmit test run for the release branch to ensure that all tests on all platforms pass with the version you're about to release.

  10. When it all looks good, go back to the job in the release pipeline, click "Unblock step" for the deployment step. This will upload the release candidate binaries to GitHub, https://releases.bazel.build and our apt-get repository.

    • If you don't have the permission, ask one of the Buildkite org admins to add you to the bazel-sheriffs group.

    • If that worked, click "Unblock step" for the "Generate Announcement" step.

  11. Prepare the release announcement on https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit.

    • Create a new section for the release. Populate using the generated text (from the "generate announcement" step).
    • Reorganize the notes per category (C++, Java, etc.)
    • Add a comment with "+spomorski@google.com" so that he takes a look.
    • Send an email to bazel-dev asking for reviewers.
  12. Copy & paste the generated text into a new e-mail and send it.

    • The first line is the recipient address.
    • The second line is the subject.
    • The rest is the body of the message.
  13. Trigger a new pipeline in BuildKite to test the release candidate with all the downstream projects.

  14. Look for failing projects in red.

  15. Once issues are fixed, create a new candidate with the relevant cherry-picks.

Push a release

  1. Verify that the following conditions all apply:

    1. at least 2 weeks passed since you pushed RC1, and
    2. at least 2 business days passed since you pushed the last RC, and
    3. there are no open "Release blocking" Bazel bugs on GitHub.
  2. Generate a new identifier: https://bazel.googlesource.com/new-password (and paste the code in your shell). This is only necessary the first time you handle a release.

  3. Push the final release (do not cancel midway):

    scripts/release/release.sh release
  4. A CI job is uploading the release artifacts to GitHub. Look for the release workflow on https://buildkite.com/bazel/release/. Unblock the steps.

  5. Ensure all binaries were uploaded to GitHub properly.

    1. Why? Sometimes binaries are uploaded incorrectly.
    2. How? Go to the GH releases page, click "Edit", see if there's a red warning sign next to any binary. You need to manually upload those; get them from https://storage.googleapis.com/bazel/$RELEASE_NUMBER/release/index.html.
  6. Update the release bug:

    1. State the fact that you pushed the release
    2. Ask the package maintainers to update the package definitions: @vbatts @petemounce
    3. Example: [https://github.com/bazelbuild/bazel/issues/3773#issuecomment-352692144]
  7. Publish versioned documentation

    1. Fetch the git tag for the release: git fetch --tags

    2. Do a checkout to that tag: git checkout $RELEASE_NUMBER

      1. You should see this message (e.g. for 0.21.0):
      $ git checkout 0.21.0
      Note: checking out '0.21.0'.
      
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by performing another checkout.
      
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -b with the checkout command again. Example:
      
      git checkout -b <new-branch-name>
      
      HEAD is now at defd737761 Release 0.21.0 (2018-12-19)
      
    3. Install gsutil and ensure you have access to the bazel-public GCP project.

    4. Run scripts/docs/generate_versioned_docs.sh. If you get interrupted, it is safe to re-run the script. This script will build the web assets for the documentation, generate a tarball from them, and push the tarball to Google Cloud Storage.

      • The script will fail to run if you're not in a git checkout of a release.
      • If the tarball has already been pushed to GCS, this script will not overwrite the existing tarball.
    5. Add $RELEASE_NUMBER to site/_config.yml and scripts/docs/doc_versions.bzl, and submit these changes. After ~30 minutes to an hour, the new release will show up on the documentation site.

  8. Publish blog post (https://docs.google.com/document/d/1wDvulLlj4NAlPZamdlEVFORks3YXJonCjyuQMUQEmB0/edit).

    1. Use versioned links whenever possible: /versions/0.21.0/foo.html instead of /versions/master/foo.html.

Updating the Homebrew recipe

Homebrew is a package manager for OS X. This section assumes that you are on a Mac OS machine with homebrew installed.

To update the bazel recipe on Homebrew, you must send a pull request to https://github.com/bazelbuild/homebrew-tap

Updating the Chocolatey package

As of November 2016, this is done by an external contributor, @petemounce on GitHub. Ping him when there's a new release coming out.

Updating the Fedora package

This is done by an external contributor, @vbatts on GitHub. Ping him when there's a new release coming out.