release-tasks.md

# Release Tasks

This document details the responsibilities and tasks for each role in the release team.

**Notes**:

* The examples in this document are based on the v1.6 release cycle.
* This document focuses on tasks that are done for every release. One-time improvement tasks are out of scope.
* If a task is prefixed with `[Track]` it means it should be ensured that this task is done, but the folks with
  the corresponding role are not responsible to do it themselves.

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

- [Release Lead](#release-lead)
  - [Responsibilities](#responsibilities)
  - [Tasks](#tasks)
    - [Finalize release schedule and team](#finalize-release-schedule-and-team)
    - [Add/remove release team members](#addremove-release-team-members)
    - [Prepare main branch for development of the new release](#prepare-main-branch-for-development-of-the-new-release)
    - [Create a new GitHub milestone for the next release](#create-a-new-github-milestone-for-the-next-release)
    - [[Track] Remove previously deprecated code](#track-remove-previously-deprecated-code)
    - [[Track] Bump dependencies](#track-bump-dependencies)
    - [Set a tentative release date for the next minor release](#set-a-tentative-release-date-for-the-next-minor-release)
    - [Assemble next release team](#assemble-next-release-team)
    - [Update milestone applier and GitHub Actions](#update-milestone-applier-and-github-actions)
    - [[Continuously] Maintain the GitHub release milestone](#continuously-maintain-the-github-release-milestone)
    - [[Continuously] Bump the Go version](#continuously-bump-the-go-version)
    - [[Repeatedly] Cut a release](#repeatedly-cut-a-release)
    - [[Optional] Public release session](#optional-public-release-session)
    - [[Optional] [Track] Bump the Cluster API apiVersion](#optional-track-bump-the-cluster-api-apiversion)
    - [[Optional] [Track] Bump the Kubernetes version](#optional-track-bump-the-kubernetes-version)
    - [[Optional] Track Release and Improvement tasks](#optional-track-release-and-improvement-tasks)
- [Communications/Docs/Release Notes Manager](#communicationsdocsrelease-notes-manager)
  - [Responsibilities](#responsibilities-1)
  - [Tasks](#tasks-1)
    - [Add docs to collect release notes for users and migration notes for provider implementers](#add-docs-to-collect-release-notes-for-users-and-migration-notes-for-provider-implementers)
    - [Update supported versions](#update-supported-versions)
    - [Ensure the book for the new release is available](#ensure-the-book-for-the-new-release-is-available)
    - [Generate weekly PR updates to post in Slack](#generate-weekly-pr-updates-to-post-in-slack)
    - [Create PR for release notes](#create-pr-for-release-notes)
    - [Change production branch in Netlify to the new release branch](#change-production-branch-in-netlify-to-the-new-release-branch)
    - [Update clusterctl links in the quickstart](#update-clusterctl-links-in-the-quickstart)
    - [Continuously: Communicate key dates to the community](#continuously-communicate-key-dates-to-the-community)
    - [Communicate beta to providers](#communicate-beta-to-providers)
- [CI Signal/Bug Triage/Automation Manager](#ci-signalbug-triageautomation-manager)
  - [Responsibilities](#responsibilities-2)
  - [Tasks](#tasks-2)
    - [Setup jobs and dashboards for a new release branch](#setup-jobs-and-dashboards-for-a-new-release-branch)
    - [[Continuously] Monitor CI signal](#continuously-monitor-ci-signal)
    - [[Continuously] Reduce the amount of flaky tests](#continuously-reduce-the-amount-of-flaky-tests)
    - [[Continuously] Bug triage](#continuously-bug-triage)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Release Lead

### Responsibilities

* Coordination:
  * Take ultimate accountability for all release tasks to be completed on time
  * Coordinate release activities
  * Create and maintain the GitHub release milestone
  * Track tasks needed to add support for new Kubernetes versions in upcoming releases
  * Ensure a retrospective happens
  * Ensure one of the [maintainers](https://github.com/kubernetes-sigs/cluster-api/blob/main/OWNERS_ALIASES) is available when a release needs to be cut.
* Staffing:
  * Assemble the release team for the next release cycle
  * Ensure a release lead for the next release cycle is selected and trained
  * Set a tentative release date for the next release cycle
* Cutting releases:
  * Release patch releases for supported previous releases at least monthly or more often if needed
  * Create beta, RC and GA releases for the minor release of the current release cycle
* Release lead should keep an eye on what is going on in the project to be able to react if necessary

### Tasks

#### Finalize release schedule and team

1. Finalize release schedule and team in the [docs/release/releases](../../docs/release/releases), e.g. [release-1.6.md](../../docs/release/releases/release-1.6.md).
2. Update @cluster-api-release-team Slack user group and GitHub team accordingly.
   <br>Prior art `org`: https://github.com/kubernetes/org/pull/4353
   <br>Prior art `community`: https://github.com/kubernetes/community/pull/7423
3. Update @cluster-api-release-lead and @cluster-api-release-team aliases in root OWNERS_ALIASES file with Release Team members.
   <br>Prior art: https://github.com/kubernetes-sigs/cluster-api/pull/9111/files#diff-4985b733677adf9dda6b5187397d4700868248ef646d64aecfb66c1ced575499
4. Announce the _release team_ and _release schedule_ to the mailing list.

#### Add/remove release team members

If necessary, the release lead can adjust the release team during the cycle to handle unexpected changes in staffing due to personal/professional issues, no-shows, or unplanned work spikes. Adding/removing members can be done by opening a PR to update the release team members list for the release cycle in question.

#### Prepare main branch for development of the new release

The goal of this issue is to bump the versions on the main branch so that the upcoming release version
is used for e.g. local development and e2e tests. We also modify tests so that they are testing the previous release.

This comes down to changing occurrences of the old version to the new version, e.g. `v1.5` to `v1.6`:

1. Setup E2E tests for the new release:
   1. Goal is that we have clusterctl upgrade tests for the latest stable versions of each contract / for each supported branch. For `v1.6` this means:
      * v1beta1: `v1.0`, `v1.4`, `v1.5` (will change with each new release)
   2. Update providers in `docker.yaml`:
       1. Add a new `v1.6.0` entry.
       2. Remove providers that are not used anymore in clusterctl upgrade tests.
       3. Change `v1.5.99` to `v1.6.99`.
   3. Adjust `metadata.yaml`'s:
      1. Create a new `v1.6` `metadata.yaml` (`test/e2e/data/shared/v1.6/metadata.yaml`) by copying
   `test/e2e/data/shared/main/metadata.yaml`
      2. Add the new release to the main `metadata.yaml` (`test/e2e/data/shared/main/metadata.yaml`).
      3. Add the new release to the root level `metadata.yaml`
      4. Remove old `metadata.yaml`'s that are not used anymore in clusterctl upgrade tests.
   4. Adjust cluster templates in `test/e2e/data/infrastructure-docker`:
      1. Create a new `v1.6` folder. It should be created based on the `main` folder and only contain the templates we use in the clusterctl upgrade tests (as of today `cluster-template` and `cluster-template-topology`).
      2. Remove old folders that are not used anymore in clusterctl upgrade tests.
   5. Modify the test specs in `test/e2e/clusterctl_upgrade_test.go` (according to the versions we want to test described above).
      Please note that both `InitWithKubernetesVersion` and `WorkloadKubernetesVersion` should be the highest mgmt cluster version supported by the respective Cluster API version.
2. Update `create-local-repository.py` and `tools/internal/tilt-prepare/main.go`: `v1.5.99` => `v1.6.99`.
3. Make sure all tests are green (also run `pull-cluster-api-e2e-full-main` and `pull-cluster-api-e2e-workload-upgrade-1-27-latest-main`).
4. Remove an unsupported release version of Cluster API from the Makefile target that generates e2e templates. For example, remove `v1.3` while working on `v1.6`.

Prior art: 

* 1.5 - https://github.com/kubernetes-sigs/cluster-api/pull/8430/files
* 1.6 - https://github.com/kubernetes-sigs/cluster-api/pull/9097/files

#### Create a new GitHub milestone for the next release

The goal of this task is to create [a new GitHub milestone](https://github.com/kubernetes-sigs/cluster-api/milestones) for the next release, so that we can already move tasks
out of the current milestone if necessary.

1. Create the milestone for the new release via GitHub UI.

#### [Track] Remove previously deprecated code

The goal of this task is to remove all previously deprecated code that can be now removed.

1. Check for deprecated code and remove it.
    * We can't just remove all code flagged with `Deprecated`. In some cases like e.g. in API packages
      we have to keep the old code.

Prior art: [Remove code deprecated in v1.6](https://github.com/kubernetes-sigs/cluster-api/pull/9136)

#### [Track] Bump dependencies

The goal of this task is to ensure that we have relatively up-to-date dependencies at the time of the release.
This reduces the risk that CVEs are found in outdated dependencies after our release.

We should take a look at the following dependencies:

* Go dependencies in `go.mod` files.
* Tools used in our Makefile (e.g. kustomize).

#### Set a tentative release date for the next minor release

1. Set a tentative release date for the next minor release and document it by creating a `release-X.Y.md` in [docs/release/releases](../../docs/release/releases).
   <br>Prior art: https://github.com/kubernetes-sigs/cluster-api/pull/9635

#### Assemble next release team

There is currently no formalized process to assemble the release team.
As of now we ask for volunteers in Slack and office hours.

#### Update milestone applier and GitHub Actions

Once release branch is created by GitHub Automation, the goal of this task would be to ensure we have the milestone
applier that applies milestones accordingly and to update GitHub actions to work with new release version.
From this point forward changes which should land in the release have to be cherry-picked into the release branch.
  
1. Update the [milestone applier config](https://github.com/kubernetes/test-infra/blob/0b17ef5ffd6c7aa7d8ca1372d837acfb85f7bec6/config/prow/plugins.yaml#L371) accordingly (e.g. `release-1.5: v1.5` and `main: v1.6`)
   <br>Prior art: [cluster-api: update milestone applier config for v1.5](https://github.com/kubernetes/test-infra/pull/30058)

2. Update the GitHub Actions to work with the new release version.
   <br>Prior art: [Update actions for v1.7](https://github.com/kubernetes-sigs/cluster-api/pull/10357)

#### [Continuously] Maintain the GitHub release milestone

The goal of this task is to keep an overview over the current release milestone and the implementation
progress of issues assigned to the milestone.

This can be done by:

1. Regularly checking in with folks implementing an issue in the milestone.
2. If nobody is working on an issue in the milestone, drop it from the milestone.
3. Ensuring we have a plan to get `release-blocking` issues implemented in time.

#### [Continuously] Bump the Go version

The goal of this task is to ensure we are always using the latest Go version for our releases.

1. Keep track of new Go versions
2. Bump the Go version in supported branches if necessary
   <br>Prior art: [Bump to Go 1.19.5](https://github.com/kubernetes-sigs/cluster-api/pull/7981)

Note: If the Go minor version of one of our supported branches goes out of supported, we should consider bumping
to a newer Go minor version according to our [backport policy](./../../CONTRIBUTING.md#backporting-a-patch).

#### [Repeatedly] Cut a release

1. Ensure CI is stable before cutting the release (e.g. by checking with the CI manager)
   Note: special attention should be given to image scan results, so we can avoid cutting a release with CVE or document known CVEs in release notes.
2. Ask the [Communications/Docs/Release Notes Manager](#communicationsdocsrelease-notes-manager) to [create a PR with the release notes](#create-pr-for-release-notes) for the new desired tag and review the PR. Once the PR merges, it will trigger a [GitHub Action](https://github.com/kubernetes-sigs/cluster-api/actions/workflows/release.yaml) to create a release branch, push release tags, and create a draft release. This will also trigger a [ProwJob](https://prow.k8s.io/?repo=kubernetes-sigs%2Fcluster-api&job=post-cluster-api-push-images) to publish images to the staging repository.
3. Promote images from the staging repository to the production registry (`registry.k8s.io/cluster-api`):
    1. Wait until images for the tag have been built and pushed to the [staging repository](https://console.cloud.google.com/gcr/images/k8s-staging-cluster-api) by the [post push images job](https://prow.k8s.io/?repo=kubernetes-sigs%2Fcluster-api&job=post-cluster-api-push-images).
    2. If you don't have a GitHub token, create one by going to your GitHub settings, in [Personal access tokens](https://github.com/settings/tokens). Make sure you give the token the `repo` scope.
    3. Create a PR to promote the images to the production registry:

       ```bash
       # Export the tag of the release to be cut, e.g.:
       export RELEASE_TAG=v1.0.1
       export GITHUB_TOKEN=<your GH token>
       make promote-images
       ```

       **Notes**:
        * `make promote-images` target tries to figure out your Github user handle in order to find the forked [k8s.io](https://github.com/kubernetes/k8s.io) repository.
          If you have not forked the repo, please do it before running the Makefile target.
        * if `make promote-images` fails with an error like `FATAL while checking fork of kubernetes/k8s.io` you may be able to solve it by manually setting the USER_FORK variable i.e.  `export USER_FORK=<personal GitHub handle>`
        * `kpromo` uses `git@github.com:...` as remote to push the branch for the PR. If you don't have `ssh` set up you can configure
          git to use `https` instead via `git config --global url."https://github.com/".insteadOf git@github.com:`.
        * This will automatically create a PR in [k8s.io](https://github.com/kubernetes/k8s.io) and assign the CAPI maintainers.
    4. Merge the PR (/lgtm + /hold cancel) and verify the images are available in the production registry:
         * Wait for the [promotion prow job](https://prow.k8s.io/?repo=kubernetes%2Fk8s.io&job=post-k8sio-image-promo) to complete successfully. Then test the production images are accessible:

         ```bash
         docker pull registry.k8s.io/cluster-api/clusterctl:${RELEASE_TAG} &&
         docker pull registry.k8s.io/cluster-api/cluster-api-controller:${RELEASE_TAG} &&
         docker pull registry.k8s.io/cluster-api/kubeadm-bootstrap-controller:${RELEASE_TAG} &&
         docker pull registry.k8s.io/cluster-api/kubeadm-control-plane-controller:${RELEASE_TAG}
         ```

4. Publish the release in GitHub:
   1. Reach out to one of the maintainers over the Slack to publish the release in GitHub.
      * **NOTE:** clusterctl will have issues installing providers between the time the release tag is cut and the Github release is published. See [issue 7889](https://github.com/kubernetes-sigs/cluster-api/issues/7889) for more details.
      * The draft release should be automatically created via the [Create Release GitHub Action](https://github.com/kubernetes-sigs/cluster-api/actions/workflows/release.yaml) with release notes previously committed to the repo by the release team. Ensure by reminding the maintainer that release is flagged as `pre-release` for all `beta` and `rc` releases or `latest` for a new release in the most recent release branch.
   
5. Publish `clusterctl` to Homebrew by bumping the version in [clusterctl.rb](https://github.com/Homebrew/homebrew-core/blob/master/Formula/c/clusterctl.rb).
   <br>**Notes**:
    * This is only done for new latest stable releases, not for beta / RC releases and not for previous release branches.
    * Check if homebrew already has a PR to update the version (homebrew introduced automation that picks it up). Open one if no PR exists.
      * To open a PR, you need two things: `tag` (i.e v1.5.3 & v1.4.8 releases are being published, where release-1.5 is the latest stable release branch, so tag would be v1.5.4) and `revision` (it is a commit hash of the tag, i.e if the tag is v1.5.3, it can be found by looking for commit id in [v1.5.3 tag page](https://github.com/kubernetes-sigs/cluster-api/releases/tag/v1.5.3)).
      * Once the PR is open, no action should be needed. Homebrew bot should push a second commit (see an example [here](https://github.com/Homebrew/homebrew-core/pull/129986/commits/0da6edddf1143aa50033f7e8ae1ebd07ecdd0941)) to the same PR to update the binary hashes automatically.
      * For an example please see: [PR: clusterctl 1.5.3](https://github.com/Homebrew/homebrew-core/pull/152279).
      * Homebrew has [conventions for commit messages](https://docs.brew.sh/Formula-Cookbook#commit) usually
        the commit message for us should look like: `clusterctl 1.5.3`.
6. **For minor releases** Set EOL date for previous release and update Cluster API support and guarantees in CONTRIBUTING.md (prior art: https://github.com/kubernetes-sigs/cluster-api/pull/9817/files).
7. **For latest stable releases** Index the most recent CRDs in the release by navigating to `https://doc.crds.dev/github.com/kubernetes-sigs/cluster-api@<CURRENT_RELEASE>` 

Additional information:

* [Versioning documentation](./../../CONTRIBUTING.md#versioning) for more information.
* Cutting a release as of today requires permissions to:
  * Create a release tag on the GitHub repository.
  * Create/update/publish GitHub releases.

#### [Optional] Public release session
   1. Host a release session over a public zoom meeting.
   2. Record the session for future reference and transparency.
   3. Use release process-related waiting periods as a forum for discussing issues/questions.
   4. Publish the recording on YouTube channel.

#### [Optional] [Track] Bump the Cluster API apiVersion

**Note** This should only be done when we have to bump the apiVersion of our APIs.

1. Add new version of the types:
    1. Create new api packages by copying existing packages.
    2. Make sure webhooks only exist in the latest apiVersion (same for other subpackages like `index`).
    3. Add conversion and conversion tests.
    4. Adjust generate targets in the Makefile.
    5. Consider dropping fields deprecated in the previous apiVersion.
2. Update import aliases in `.golangci.yml`.
3. Switch other code over to the new version (imports across the code base, e.g. controllers).
    1. Add all versions to the schema in the `main.go` files.
4. Add types to the `PROJECT` files of the respective provider.
5. Add test data for the new version in `test/e2e/data/{infrastructure-docker,shared}` (also update top-level `.gitignore`).
6. Update `docker.yaml`, make sure all tests are successful in CI.

#### [Optional] [Track] Bump the Kubernetes version

1. Create an issue for the new Kubernetes version via: [New Issue: Kubernetes bump](https://github.com/kubernetes-sigs/cluster-api/issues/new/choose).
2. Track the issue to ensure the work is completed in time.

#### [Optional] Track Release and Improvement tasks

1. Create an issue for easier tracking of all the tasks for the release cycle in question.
   <br>Prior art: [Tasks for v1.6 release cycle](https://github.com/kubernetes-sigs/cluster-api/issues/9094)
2. Create a release improvement tasks [GitHub Project Board](https://github.com/orgs/kubernetes-sigs/projects/55) to track 
   the current status of all improvement tasks planned for the release, their priorities, status (i.e `Done`/`In Progress`)
   and to distribute the work among the Release Team members.

   **Notes**:
   * At the beginning of the cycle, Release Team Lead should prepare the improvement tasks board for the ongoing release cycle.
     The following steps can be taken:
      - Edit improvement tasks board name for current cycle (e.g. `CAPI vX.Y release improvement tasks`)
      - Add/move all individual missing issues to the board

## Communications/Docs/Release Notes Manager

### Responsibilities

* Communication:
  * Communicate key dates to the community
* Documentation:
  * Improve release process documentation
  * Ensure the book and provider upgrade documentation are up-to-date
  * Maintain and improve user facing documentation about releases, release policy and release calendar
* Release Notes:
  * Create PR with release notes

### Tasks

#### Add docs to collect release notes for users and migration notes for provider implementers

The goal of this task is to initially create the docs so that we can continuously add notes going forward.
The release notes doc will be used to collect release notes during the release cycle and will be eventually
used to write the final release notes. The provider migration doc is part of the book and contains instructions
for provider authors on how to adopt to the new Cluster API version.

1. Add a new migration doc for provider implementers.
   <br>Prior art: [Add v1.5 -> v1.6 migration doc](part of: https://github.com/kubernetes-sigs/cluster-api/pull/8996) - see changes to [SUMMARY.md](https://github.com/kubernetes-sigs/cluster-api/pull/8996/files#diff-72d1da5cbeb1afbe684444ec598fbe1815dd2ddc6aa99078ab577cefb9e279ac) and addition of [v1.5-to-v1.6.md](https://github.com/kubernetes-sigs/cluster-api/pull/8996/files#diff-135e34a16773fd40a82b4adbb265444a4fed6c1a973f48d621082b957e7ef93f)

#### Update supported versions

1. Update supported versions in versions.md.
   <br>Prior art: [Update supported versions for v1.6](https://github.com/kubernetes-sigs/cluster-api/pull/9119)

#### Ensure the book for the new release is available

The goal of this task to make the book for the current release available under e.g. `https://release-1-6.cluster-api.sigs.k8s.io`.

1. Add a DNS entry for the book of the new release (should be available under e.g. `https://release-1-6.cluster-api.sigs.k8s.io`).
   <br>Prior art: [Add DNS for CAPI release-1.6 release branch](https://github.com/kubernetes/k8s.io/pull/6114)
2. Open `https://release-1-6.cluster-api.sigs.k8s.io/` and verify that the certificates are valid.  If they are not, reach out to CAPI maintainers and request someone with access Netlify [click the `renew certificate` button](https://app.netlify.com/sites/kubernetes-sigs-cluster-api/settings/domain#https) in the Netlify UI.
    - To add new subdomains to the certificate config, checkout the email snippet [template](https://github.com/kubernetes-sigs/cluster-api/issues/6017#issuecomment-1823306891) for reference.
3. Update references in introduction.md only on the main branch (drop unsupported versions, add the new release version).
   <br>Prior art: [Add release 1.5 book link](https://github.com/kubernetes-sigs/cluster-api/pull/9767)

#### Generate weekly PR updates to post in Slack
The goal of this task is to keep the CAPI community updated on recent PRs that have been merged. This is done by using the weekly update tool in `hack/tools/release/weekly/main.go`. Here is how to use it:
1. Build the release weekly update tools binary.
   ```bash
   make release-weekly-update-tool
   ```
2. Generate the weekly update with the following command for desired branches:
   ```bash
   ./bin/weekly --since <YYYY-MM-DD> --until <YYYY-MM-DD> --branch <branch_name>
   ```
   **Note:** the `GITHUB_TOKEN` environment variable can be set to prevent/reduce GitHub rate limiting issues.
3. Paste the output into a new Slack message in the [`#cluster-api`](https://kubernetes.slack.com/archives/C8TSNPY4T) channel. Currently, we post separate messages in a thread for the branch `main` - which corresponds to the active milestone - as well as the two most recent release branches (e.g. `release-1.6` and `release-1.5`).

   Example commands to run for the weekly update during ongoing development towards release 1.7:
   ```bash
   # Main branch changes, which correspond to actively worked on release 1.7
   ./bin/weekly --since 2024-01-22 --until 2024-01-28 --branch main

   # Previous 2 release branches
   ./bin/weekly --since 2024-01-22 --until 2024-01-28 --branch release-1.6
   ./bin/weekly --since 2024-01-22 --until 2024-01-28 --branch release-1.5
   ```

#### Create PR for release notes
1. Checkout the `main` branch.
1. Generate release notes with:

   ```bash
   # RELEASE_TAG should be the new desired tag (note: at this point the tag does not yet exist).
   # Can be also used for pre-releases. The warning banner for RC and beta releases will be determined automatically.
   RELEASE_TAG=v1.6.x make release-notes
   ```

1. This will generate a new release notes file at `CHANGELOG/<RELEASE_TAG>.md`. Finalize the release notes:
    - [ ] Look for any `MISSING_AREA` entries. Add the corresponding label to the PR and regenerate the notes.
    - [ ] Look for any `MULTIPLE_AREAS` entries. If the PR does indeed guarantee multiple areas, just remove the `MULTIPLE_AREAS` prefix and just leave the areas. Otherwise, fix the labels in the PR and regenerate the notes.
    - [ ] Review that all areas are correctly assigned to each PR. If not, correct the labels and regenerate the notes.
    - [ ] Update the `Kubernetes version support section`. If this is a patch release you can most probably copy the same values from the previous patch release notes. Except if this is the release where a new Kubernetes version support is added.
       <br>**Note**: Check our [Kubernetes support policy](https://cluster-api.sigs.k8s.io/reference/versions.html#supported-kubernetes-versions) in the CAPI book. In case of doubt, reach out to the current release lead.
    - [ ] If this is a `vX.X.0` release, fill in the content for the `Highlights` section. Otherwise, remove the section altogether.
    - [ ] If there a deprecations in this release (for example, a CAPI API version drop), add them, to the `Deprecation Warning` section. Otherwise, remove the section altogether.
    - [ ] Look for area duplications in PR title. Sometimes authors add a prefix in their PR title that matches the area label. When the notes are generated, the area is as a prefix to the PR title, which can create redundant information. Remove the one from the PR title and just leave the area. Make sure you capitalize the title after this.
    - [ ] Check that all entries are in the right section. Sometimes the wrong emoji prefix is added to the PR title, which drives the section in which the entry is added in the release notes. Manually move any entry as needed. Note that fixing the PR title won't fix this even after regenerating the notes, since the notes tool reads this info from the commit messages and these don't get rewritten.
    - [ ] Sort manually all entries if you made any manual edits that might have altered the correct order.
    - [ ] **For minor releases:** Modify `Changes since v1.x.y` to `Changes since v1.x`
       <br>**Note**: The release notes tool includes all merges since the previous release branch was branched of.
1. Checkout `main`, branch out from it and add `CHANGELOG/<RELEASE_TAG>.md`.
1. Open a pull request **against the main branch** with all manual edits to `CHANGELOG/<RELEASE_TAG>.md` which is used for the new release notes. The commit and PR title should be `🚀 Release v1.x.y`.
       <br>**Note**: Important! The commit should only contain the release notes file, nothing else, otherwise automation will not work.


#### Change production branch in Netlify to the new release branch

The goal of this task to make the book for the current release available under `https://cluster-api.sigs.k8s.io`.

Reach out to the CAPI maintainers to request someone with access to Netlify perform the following steps:

1. Change production branch in Netlify the current release branch (e.g. `release-1.6`) to make the book available under `https://cluster-api.sigs.k8s.io`. It's done under [production branch settings](https://app.netlify.com/sites/kubernetes-sigs-cluster-api/settings/deploys#branches-and-deploy-contexts)
2. [Trigger a redeploy](https://app.netlify.com/sites/kubernetes-sigs-cluster-api/deploys).

#### Update clusterctl links in the quickstart

The goal of this task is to ensure the quickstart has links to the latest `clusterctl` binaries.

Update clusterctl links in the quickstart (on main and cherry-pick onto release-1.6).
<br>Prior art: [Update clusterctl version to v1.6.x in quick start](https://github.com/kubernetes-sigs/cluster-api/pull/9801)

**Note**: The PR for this should be merged after the minor release has been published. Recommended to create it before
the release but with `/hold`. This will allow maintainers to review and approve before the release. When the release is
done just remove the hold to merge it.

#### Continuously: Communicate key dates to the community

The goal of this task is to ensure all stakeholders are informed about the current release cycle. For example announcing
upcoming code freezes etc based on the [release timeline (1.6 example)](./releases/release-1.6.md).

Templates for all types of communication can be found in the [release-templates page](./release-templates.md).

Information can be distributed via:

* `sig-cluster-lifecycle` mailing list
  * Note: The person sending out the email should ensure that they are first part of the mailing list. If the email is sent out is not received by the community, reach out to the maintainers to unblock and approve the email.
* #cluster-api Slack channel
* Office hours
* Release Team meetings
* Cluster API book
* [Github Issue](https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/release/release-tasks.md#communicate-beta-to-providers) (when communicating beta release to providers)

Relevant information includes:

* Beta, RC, GA and patch release
* Start of code freeze
* Implementation progress
* Release delays and changes if applicable

Stakeholders are:

* End users of Cluster API
* Contributors to core Cluster API
* Provider implementers

#### Communicate beta to providers

The goal of this task is to inform all providers that a new beta.0 version a release is out and that it should be tested. We want to prevent issues where providers don't have enough time to test before a new version of CAPI is released. This stems from a previous issue we are trying to avoid: https://github.com/kubernetes-sigs/cluster-api/issues/8498

We should inform at least the following providers via a new issue on their respective repos that a new version of CAPI is being released (provide the release date) and that the beta.0 version is ready for them to test.

* Addon provider helm: https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm/issues/new
* AWS: https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/new
* Azure: https://github.com/kubernetes-sigs/cluster-api-provider-azure/issues/new
* Cloudstack: https://github.com/kubernetes-sigs/cluster-api-provider-cloudstack/issues/new
* Digital Ocean: https://github.com/kubernetes-sigs/cluster-api-provider-digitalocean/issues/new
* GCP: https://github.com/kubernetes-sigs/cluster-api-provider-gcp/issues/new
* Kubemark: https://github.com/kubernetes-sigs/cluster-api-provider-kubemark/issues/new
* Kubevirt: https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/issues/new
* IBMCloud: https://github.com/kubernetes-sigs/cluster-api-provider-ibmcloud/issues/new
* Metal3: https://github.com/metal3-io/cluster-api-provider-metal3/issues/new
* Nested: https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/new
* OCI: https://github.com/oracle/cluster-api-provider-oci/issues/new
* Openstack: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/issues/new
* Operator: https://github.com/kubernetes-sigs/cluster-api-operator/issues/new
* Packet: https://github.com/kubernetes-sigs/cluster-api-provider-packet/issues/new
* vSphere: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/issues/new

To create GitHub issues at the Cluster API providers repositories and inform about a new minor beta release, use ["provider_issues.go"](../../hack/tools/release/internal/update_providers/provider_issues.go) go utility.
- Ensure that the [provider repos pre-requisites](../../hack/tools/release/internal/update_providers/README.md#pre-requisites) are completed.
- From the root of this repository, run `make release-provider-issues-tool` to create git issues at the provider repositories.

## CI Signal/Bug Triage/Automation Manager

### Responsibilities

* Signal:
  * Responsibility for the quality of the release
  * Continuously monitor CI signal, so a release can be cut at any time
  * Add CI signal for new release branches
* Bug Triage:
  * Make sure blocking issues and bugs are triaged and dealt with in a timely fashion
* Automation:
  * Maintain and improve release automation, tooling & related developer docs

### Tasks

#### Setup jobs and dashboards for a new release branch

The goal of this task is to have test coverage for the new release branch and results in testgrid.
While we add test coverage for the new release branch we will also drop the tests for old release branches if necessary.

1. Create new jobs based on the jobs running against our `main` branch:
    1. Copy the `main` branch entry as `release-1.6` in the `cluster-api-prowjob-gen.yaml` file in [test-infra](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/cluster-api/).
    2. Modify the following at the `release-1.6` branch entry:
            * Change intervals (let's use the same as for `release-1.5`).
2. Create a new dashboard for the new branch in: `test-infra/config/testgrids/kubernetes/sig-cluster-lifecycle/config.yaml` (`dashboard_groups` and `dashboards`).
3. Remove old release branches and unused versions from the `cluster-api-prowjob-gen.yaml` file in [test-infra](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-sigs/cluster-api/) according to our policy documented in [Support and guarantees](../../CONTRIBUTING.md#support-and-guarantees). For example, let's assume we just added `release-1.6`, then we can now drop test coverage for the `release-1.3` branch.
4. Regenerate the prowjob configuration running `make generate-test-infra-prowjobs` command from cluster-api repository. Before running this command, ensure to export the `TEST_INFRA_DIR` variable, specifying the location of the [test-infra](https://github.com/kubernetes/test-infra/) repository in your environment. For further information, refer to this [link](https://github.com/kubernetes-sigs/cluster-api/pull/9937).

 ```sh
  TEST_INFRA_DIR=../../k8s.io/test-infra make generate-test-infra-prowjobs
  ```
5. Verify the jobs and dashboards a day later by taking a look at: `https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api-1.6`
6. Update `.github/workflows/weekly-security-scan.yaml` - to setup Trivy and govulncheck scanning - `.github/workflows/weekly-md-link-check.yaml` - to setup link checking in the CAPI book - and `.github/workflows/weekly-test-release.yaml` - to verify the release target is working - for the currently supported branches.
7. Update the [PR markdown link checker](https://github.com/kubernetes-sigs/cluster-api/blob/main/.github/workflows/pr-md-link-check.yaml) accordingly (e.g. `main` -> `release-1.6`).
   <br>Prior art: [Update branch for link checker](https://github.com/kubernetes-sigs/cluster-api/pull/9206)


Prior art:

* [Add jobs for CAPI release 1.6](https://github.com/kubernetes/test-infra/pull/31208)

#### [Continuously] Monitor CI signal

The goal of this task is to keep our tests running in CI stable.

**Note**: To be very clear, this is not meant to be an on-call role for Cluster API tests.

1. Add yourself to the [Cluster API alert mailing list](https://github.com/kubernetes/k8s.io/blob/151899b2de933e58a4dfd1bfc2c133ce5a8bbe22/groups/sig-cluster-lifecycle/groups.yaml#L20-L35)
    <br\>**Note**: An alternative to the alert mailing list is manually monitoring the [testgrid dashboards](https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api)
    (also dashboards of previous releases). Using the alert mailing list has proven to be a lot less effort though.
2. Subscribe to `CI Activity` notifications for the Cluster API repo.
3. Check the existing **failing-test** and **flaking-test** issue templates under `.github/ISSUE_TEMPLATE/` folder of the repo, used to create an issue for failing or flaking tests respectively. Please make sure they are up-to-date and if not, send a PR to update or improve them.
4. Check if there are any existing jobs that got stuck (have been running for more than 12 hours) in a ['pending'](https://prow.k8s.io/?repo=kubernetes-sigs%2Fcluster-api&state=pending) state:
   - If that is the case, notify the maintainers and ask them to manually cancel and re-run the stuck jobs.   
5. Triage CI failures reported by mail alerts or found by monitoring the testgrid dashboards:
    1. Create an issue using an appropriate template (failing-test) in the Cluster API repository to surface the CI failure.
    2. Identify if the issue is a known issue, new issue or a regression.
    3. Mark the issue as `release-blocking` if applicable.
6. Triage periodic GitHub actions failures, with special attention to image scan results;
   Eventually open issues as described above.
7. Run periodic deep-dive sessions with the CI team to investigate failing and flaking tests. Example session recording: https://www.youtube.com/watch?v=YApWftmiDTg

#### [Continuously] Reduce the amount of flaky tests

The Cluster API tests are pretty stable, but there are still some flaky tests from time to time.

To reduce the amount of flakes please periodically:

1. Take a look at recent CI failures via `k8s-triage`:
    * [main: e2e, e2e-mink8s, test, test-mink8s](https://storage.googleapis.com/k8s-triage/index.html?job=.*cluster-api.*(test%7Ce2e)-(mink8s-)*main&xjob=.*-provider-.*)
2. Open issues using an appropriate template (flaking-test) for occurring flakes and ideally fix them or find someone who can.
   **Note**: Given resource limitations in the Prow cluster it might not be possible to fix all flakes.
   Let's just try to pragmatically keep the amount of flakes pretty low.

#### [Continuously] Bug triage

The goal of bug triage is to triage incoming issues and if necessary flag them with `release-blocking`
and add them to the milestone of the current release.

We probably have to figure out some details about the overlap between the bug triage task here, release leads
and Cluster API maintainers.