Skip to content

Latest commit

 

History

History
287 lines (220 loc) · 17.7 KB

RELEASE_PROCESS.md

File metadata and controls

287 lines (220 loc) · 17.7 KB
Raw

Chef Infra Server Release Process

Document Purpose

The purpose of this document is to describe the current release process such that any member of the team can do a release. As we improve the automation around the release process, the document should be updated such that it always has the exact steps required to release Chef Infra Server.

This document is NOT aspirational. We have a number of automation tools that we intend to use to improve the release release process; however, many are not fully integrated with our process yet. Do not add them to this document until they are ready to be part of the release process and can be used by any team member to perform a release.

Pre-requisites

In order to release, you will need the following accounts/permissions:

  • Chef Software Inc. Slack account
  • VPN account for Chef Software Inc.
  • Access to automate.chef.co

⚠️ If it is a Friday -- do we really need that release today? ⚠️

THE PROCESS

Upgrade Erlang Dependencies

Upgrade Erlang dependencies via automated script. See https://github.com/chef/chef-server/blob/main/dev-docs/FrequentTasks.md#updating-erlang-dependencies-using-rebar3 .

Update documentation

Ensure documentation in docs-site is updated for changes that would be included in the release.

Update Release Notes

Pending Release Notes In Wiki

https://github.com/chef/chef-server/wiki/Pending-Release-Notes

Getting the build to be released into current with a minor/major version bump

Pre-release, bump the major or minor version anytime a PR is being merged if the upcoming release is being designed around that and possibly other features.

  1. Apply the label Expeditor: Bump Version Minor or Expeditor: Bump Version Major to your PR to bump the version number of the release candidate, as applicable.
  2. DOUBLE-CHECK the labels to confirm that they are correct. A mistake here is not easy to revert.

After a commit is merged to main, expeditor automatically bumps the patch version, and a build is automatically kicked-off on the Chef / [chef/chef-server:main] omnibus/release pipeline. After the build and tests in buildkite pass, expeditor should put the build artifact in Artifactory's current channel. Monitor the build's progress at chef-server-notify slack channel.

  1. Confirm that the omnibus build to be promoted is present in Artifactory's current channel. One approach is to enter the following into a bash shell, where version is the version number of the new release:
$ mixlib-install download chef-server -c current -a x86_64 -p ubuntu -l 18.04 -v <version>
Starting download https://packages.chef.io/files/current/chef-server/14.2.23/ubuntu/18.04/chef-server-core_14.2.23-1_amd64.deb

Sometimes it takes a while for mixlib to realize a new artifact is there, so you can also look here:

https://www.chef.io/downloads/tools/infra-server/current

Testing the Release

Integration Testing

Every merge to chef-server main must be built, and the various upgrade paths for the build must be tested with the full Umbrella automated integration test pipeline at https://buildkite.com/chef/chef-umbrella-main-chef-server-full (use https://buildkite.com/chef/chef-umbrella-main-chef-server for testing the 12.17.15 -> YOUR-RELEASE upgrade path). [NOTE: Every merge to main automatically runs through adhoc, and umbrella is run nightly for all changes made that day]. The integration test run for the tag being shipped must be successful.

Any Chef Infra Server release 12.17.15 or later should be able to upgrade directly to the latest release of 14. The nightly builds test upgrades to the latest current artifact from 12.17.15 and 13.2.0. Releases prior to 12.17.15 must perform a stepped upgrade. See: https://docs.chef.io/server/upgrades/#upgrade-matrix

Using the step-by-step Umbrella testing process detailed below...

  1. Test that the 12.17.15 release can successfully upgrade to the new release (for this, use the pipeline at https://buildkite.com/chef/chef-umbrella-main-chef-server). This is the 12.17.15 -> YOUR-RELEASE upgrade path.
  2. Test that the 13.2.0 release can successfully upgrade to the new release. This is the 13.2.0 -> YOUR-RELEASE upgrade path.
  3. Test that the latest stable release can successfully upgrade to the new release. This is the LATEST-STABLE -> YOUR-RELEASE upgrade path.

Past stable releases: https://downloads.chef.io/tools/infra-server

Umbrella Testing Step-by-Step:

  1. Navigate your web browser to https://buildkite.com/chef/chef-umbrella-main-chef-server-full (or for 12.17.15 -> YOUR-RELEASE, use https://buildkite.com/chef/chef-umbrella-main-chef-server).
  2. Select 'New Build'.
  3. Leave 'Branch' set to 'main'.
  4. Select 'Options' to expand the 'Environment Variables' field.
  5. Enter INSTALL_VERSION=<version number> into the 'Options | Environment Variables' field, where version number is the stable channel version number of the release you wish to test upgrading FROM (for example 12.17.15).
  6. Enter UPGRADE_VERSION=<version number> into the 'Options | Environment Variables' field, where version number is the current channel version number of the release candidate you wish to test upgrading TO (for example 14.6.32). This release candidate version number should come from the release pipeline build for the release you are doing.
  7. Optionally, fill-in the 'Message' field with something descriptive.
  8. Select 'Create Build'.

Chef Manage Testing

Currently (07/21), the Umbrella pipeline does not perform a test login to Chef Manage, so this should be done manually. A successful test login should be verified on both IPV4 and IPV6 setups, and must use the latest stable version of Manage. One method is to run representative AWS and Azure Umbrella scenarios from your local box to get running server instances to test. For umbrella you should specify the Chef Manage version to install when creating the Umbrella scenario (see below), or it will pick the latest unstable version (that's generally not what you want). For local non-Umbrella setups, chef-server-ctl install chef-manage will install the latest stable release of chef-manage. See document here for setting up and obtaining Umbrella credentials: https://chefio.atlassian.net/wiki/spaces/Policy/pages/2514386984/SOP-003+-+Gaining+access+to+AWS+accounts+via+the+AWS+API+CLI

Chef Manage releases: https://downloads.chef.io/tools/manage

Typical Umbrella scenario for AWS, where version is the version number of the release candidate you are testing, and manage version is the latest stable version of Chef Manage:

$ cd umbrella/chef-server/scenarios/aws
$ saml2aws login --profile chef-engineering
Fetching credentials for: chef-engineering
...
$ make create-vpc
Initializing modules...
Downloading terraform-aws-modules/vpc/aws 2.77.0 for vpc...
...
IPV4 Scenario:
$ PLATFORM=ubuntu-18.04 INSTALL_VERSION=<version> UPGRADE_VERSION=<version> SCENARIO=standalone-fresh-install ENABLE_ADDON_PUSH_JOBS=false ENABLE_GATHER_LOGS_TEST=false ENABLE_PEDANT_TEST=false ENABLE_PSQL_TEST=false ENABLE_SMOKE_TEST=false MANAGE_VERSION=<manage_version> ENABLE_IPV6=false make apply

IPV6 Scenario:
$ PLATFORM=ubuntu-18.04 INSTALL_VERSION=<version> UPGRADE_VERSION=<version> SCENARIO=standalone-fresh-install ENABLE_ADDON_PUSH_JOBS=false ENABLE_GATHER_LOGS_TEST=false ENABLE_PEDANT_TEST=false ENABLE_PSQL_TEST=false ENABLE_SMOKE_TEST=false MANAGE_VERSION=<manage_version> ENABLE_IPV6=true make apply
  1. Obtain the DNS name of the ephemeral machine by observing the output of the boot-up. A sample output is shown below:
null_resource.chef_server_config: Provisioning with 'remote-exec'...
null_resource.chef_server_config (remote-exec): Connecting to remote host via SSH...
null_resource.chef_server_config (remote-exec):   Host: ec2-34-212-122-231.us-west-2.compute.amazonaws.com
null_resource.chef_server_config (remote-exec):   User: ec2-user
null_resource.chef_server_config (remote-exec):   Password: false
null_resource.chef_server_config (remote-exec):   Private key: false
null_resource.chef_server_config (remote-exec):   Certificate: false
null_resource.chef_server_config (remote-exec):   SSH Agent: true
null_resource.chef_server_config (remote-exec):   Checking Host Key: false
null_resource.chef_server_config (remote-exec): Connected!
null_resource.chef_server_config (remote-exec): echo -e '
null_resource.chef_server_config (remote-exec): BEGIN INSTALL CHEF SERVER
  1. Navigate to http://<hostname> via web browser where hostname is the DNS name of the emphemeral machine obtained in the previous step.
  2. Enter the username and password to test the login. The username and password are stored in the following script: https://github.com/chef/umbrella/blob/main/chef-server/common/files/add_user.sh.
  3. Verify that the login is successful.
  4. Clean-up:
PLATFORM=ubuntu-18.04 INSTALL_VERSION=<version> UPGRADE_VERSION=<version> SCENARIO=standalone-fresh-install ENABLE_ADDON_PUSH_JOBS=false ENABLE_GATHER_LOGS_TEST=false ENABLE_PEDANT_TEST=false ENABLE_PSQL_TEST=false ENABLE_SMOKE_TEST=false MANAGE_VERSION=<manage_version> ENABLE_IPV6=true make destroy
make destroy-vpc

Typical scenario for Azure, where version is the version number of the release candidate you are testing:

$ cd umbrella/chef-server/scenarios/azure
$ ARM_DEPT=Eng ARM_CONTACT=your_login_here make create-resource-group
...
IPV4 Scenario:
$ PLATFORM=ubuntu-18.04 INSTALL_VERSION=<version> UPGRADE_VERSION=<version> SCENARIO=external-postgresql ENABLE_ADDON_PUSH_JOBS=false ENABLE_GATHER_LOGS_TEST=false ENABLE_PEDANT_TEST=false ENABLE_PSQL_TEST=false ENABLE_SMOKE_TEST=false MANAGE_VERSION=<manage_version> ENABLE_IPV6=false make apply

IPV6 Scenario:
$ PLATFORM=ubuntu-18.04 INSTALL_VERSION=<version> UPGRADE_VERSION=<version> SCENARIO=external-postgresql ENABLE_ADDON_PUSH_JOBS=false ENABLE_GATHER_LOGS_TEST=false ENABLE_PEDANT_TEST=false ENABLE_PSQL_TEST=false ENABLE_SMOKE_TEST=false MANAGE_VERSION=<manage_version> ENABLE_IPV6=true make apply
  1. Perform the same login processes specified above for AWS.
  2. Clean-up:
IPV4 Scenario:
PLATFORM=ubuntu-18.04 INSTALL_VERSION=<version> UPGRADE_VERSION=<version> SCENARIO=external-postgresql ENABLE_ADDON_PUSH_JOBS=false ENABLE_GATHER_LOGS_TEST=false ENABLE_PEDANT_TEST=false ENABLE_PSQL_TEST=false ENABLE_SMOKE_TEST=false MANAGE_VERSION=<manage_version> ENABLE_IPV6=false make destroy
make destroy-resource-group

IPV6 Scenario:
PLATFORM=ubuntu-18.04 INSTALL_VERSION=<version> UPGRADE_VERSION=<version> SCENARIO=external-postgresql ENABLE_ADDON_PUSH_JOBS=false ENABLE_GATHER_LOGS_TEST=false ENABLE_PEDANT_TEST=false ENABLE_PSQL_TEST=false ENABLE_SMOKE_TEST=false MANAGE_VERSION=<manage_version> ENABLE_IPV6=true make destroy
make destroy-resource-group

  1. Any failures must be fixed before shipping a release, unless they are "known failures" or expected. A document at an unknown state of updatedness tracking known failures can be found at: https://docs.google.com/spreadsheets/d/10LZszYuAIlrk1acy0GRhrZMd0YohLTQWmcNIdY6XuZU/edit#gid=0 Use chef.io account credentials to access it. Please update the spreadsheet as you resolve failures, especially on the resolution as to whether the issue is release-blocking or not. This will save time in the future when the error is encountered again and the team has to investigate whether the issue is release-blocking. Note that no changes other than CHANGELOG/RELEASE_NOTES changes should land on main between testing and releasing since we typically tag HEAD of main. If something large does land on main, the release tag you create should point specifically at the build that you tested. The git SHA of the build you are testing can be found in /opt/opscode/version-manifest.json.

  2. Make sure the Habitat builds for main are passing. These are kicked-off automatically on every merge. Chef / [chef/chef-server:main] habitat/build / main https://buildkite.com/chef/chef-chef-server-main-habitat-build

  3. Test the chef-server version with Automate if there are any schema changes. Follow the below document for testing the chef-server version in Automate environment. https://github.com/chef/automate/blob/main/dev-docs/DEV_ENVIRONMENT.md

Software Versions Check

If any software was bumped or updated (including any CVEs addressed), load up a dev VM and verify that the software is the correct version.

For example, after an OpenJRE bump:

# /opt/opscode/embedded/open-jre/bin/java --version
openjdk 11.0.18 2023-01-17
OpenJDK Runtime Environment Temurin-11.0.18+10 (build 11.0.18+10)
OpenJDK 64-Bit Server VM Temurin-11.0.18+10 (build 11.0.18+10, mixed mode)

Special Testing

Do any special testing specific to the particular release you are doing, as applicable.

Other

Verify that the manifest displays the correct version number of the release.

http://packages.chef.io/manifests/current/chef-server/latest.json

Building and Releasing the Release

  1. Select a version from the current channel that you wish to promote to stable. Make sure that this version has gone through the upgrade testing.

  2. Make sure you are part of the slack group release-promoters. If not added, make a request for the same in the who-do-i-ask-about channel.

  3. Use expeditor to promote the build. The expeditor command is of the form:

     /expeditor promote ORG/REPO:BRANCH VERSION

In practice it will look like:

    /expeditor promote chef/chef-server:main VERSION

Example:

/expeditor promote chef/chef-server:main 14.2.2

Some other projects use the same expeditor promote method. Here is an example from Knife EC Backup:

/expeditor promote chef/knife-ec-backup:main 3.0.0

NOTE: As an aside, with respect to the above example, to update Chef Server to point to the Knife EC Backup gem you just released, update src/chef-server-ctl/Gemfile.lock. You can confirm the existence of the gem at https://rubygems.org/gems/knife-ec-backup/versions/3.0.0.

Please do expeditor promotes in the #chef-server channel. Once this is done, the release is available to the public via the APT and YUM repositories and https://downloads.chef.io/tools/infra-server.

Announce the release post-promote to the following channels:

- #a2-release-coordinate
- #chef-server
- #cft-announce

Copying/pasting a discourse link to the post-promote channels should suffice. You can find the link here. Note that this is NOT the link to copy/paste, this is a link where you can find the link to copy/paste: https://discourse.chef.io/c/chef-release/9

Verifying Release Success

  1. Confirm the existence of the notification in the #chef-server-notify channel.
  2. Confirm that the release appears at https://downloads.chef.io/tools/infra-server. This usually takes a while.
  3. Confirm that the release notes from Pending Release Notes are automatically posted on discourse. Sample post: https://discourse.chef.io/t/chef-infra-server-14-10-23-released/20438
  4. Confirm that the data for the Pending Release Notes at https://github.com/chef/chef-server/wiki/Pending-Release-Notes is automatically deleted by expeditor, and only the titles remain. Notify releng at https://github.com/chef/release-engineering/issues if this does not automatically happen on promote.
  5. Confirm that the release notes appear at https://docs.chef.io/release_notes_server/ . This should happen automatically via expeditor, but if it does not you need to perform the steps manually and create an issue with releng at https://github.com/chef/release-engineering/issues. For instructions on how to proceed with a manual edit, contact docs-support.

In case of release failure, consult appropriate documentation to assist with troubleshooting and correcting the issue. https://expeditor.chef.io/dashboard/?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2NDYxODMwMTgsImlhdCI6MTY0NjE2ODYxOCwiaXNzIjoiaHR0cHM6Ly9leHBlZGl0b3IuY2hlZi5pbyIsInVzZXJfbG9naW4iOiJQcmFqYWt0YVB1cm9oaXQiLCJ1c2VyX25hbWUiOm51bGwsImFjY2Vzc190b2tlbiI6ImdodV95R3o4MkE5dWg5TzdnVXhOMHhEbXQ5bHU1YXM0YUE0TVZuVjYiLCJhZG1pbiI6dHJ1ZX0.5EZ1GvD_ufWmXPzfosGbs11RXYnHNm7gABgc-TpLgkQ#/

Automate

Notify the Automate team that chef server has been released, and give them the version. Confirm that the hab packages are available for them to consume:

https://bldr.habitat.sh/#/pkgs/chef/oc_erchef/latest https://bldr.habitat.sh/#/pkgs/chef/oc_bifrost/latest https://bldr.habitat.sh/#/pkgs/chef/bookshelf/latest https://bldr.habitat.sh/#/pkgs/chef/chef-server-nginx/latest https://bldr.habitat.sh/#/pkgs/chef/chef-server-ctl/latest

Chef Infra Server is now released.

A sample release checklist depicting a release in progress:

RELEASE CHECKLIST
- updated release notes                     DONE
- omnibus build in current channel          DONE
- umbrella pipeline full
    12.17.15 -> 14.11.21                    FAILURES [EXTERNAL ELASTICSEARCH]
https://buildkite.com/chef/chef-umbrella-main-chef-server/builds/955#02f24277-5696-44a6-89d6-e9ae10c27568
    13.2     -> 14.11.21                    FAILURES
https://buildkite.com/chef/chef-umbrella-main-chef-server-full/builds/98#d250c5f4-882c-481b-915c-70c14a37b6a9
    14.11.15 -> 14.11.21                    FAILURES
https://buildkite.com/chef/chef-umbrella-main-chef-server-full/builds/97#20ad56a9-2161-4d0c-bcce-3490d4d060ec
- update known failure sheet                PENDING
- login to chef manage
    AWS
        ipv4                                EMAIL NOT GREYED-OUT
        ipv6                                EMAIL NOT GREYED-OUT
    AZURE
        ipv4                                TERRAFORM ERROR
        ipv6                                TERRAFORM ERROR
- check that hab builds are successful      PASSED
- automate                                  PENDING
- special testing                           PENDING
- verify that manifest contains correct     PASSED
  release number
- build and release the release             PENDING
- notify automate                           PENDING