- Branch Manager Handbook
- Content Notice
- Overview
- Prerequisites
- Releases Management
- Branch Management
- Staging Repositories
- Debugging
- Search past builds
- References
- Background information
- Visual Release Cut Process
The Patch Release Team and Branch Manager roles have been consolidated into a single Release Managers group.
This means that several areas of this document may currently be out of date. While we work to update these documents, please reach out to Release Managers directly for any clarifications on Release Engineering processes.
This notice will be removed when the documentation is no longer under construction.
In this handbook, we will make several references to Kubernetes releases, milestones, and semantic versioning.
For the purposes of this handbook, we'll assume that:
- the current release in development is Kubernetes 1.18
- the previous release is Kubernetes 1.17
- the next release is Kubernetes 1.19
- the release no longer in support is Kubernetes 1.14
To simplify certain instructions, we will make the following connections:
| Text | SemVer | Reference Release |
|---|---|---|
| "current release", "current milestone", "in development" | x.y |
Kubernetes 1.18 |
| "previous release", "previous milestone" | x.y-1 |
Kubernetes 1.17 |
| "next release", "next milestone" | x.y+1 |
Kubernetes 1.19 |
| "release no longer in support" | x.y-4 |
Kubernetes 1.14 |
As an editor of this content, Branch Managers should periodically update these conventions and the examples contained within this handbook.
This is a collection of requirements and conditions to fulfill when taking on the role as branch manager.
Before we can grant Release Manager access to new Branch Managers, a Release Manager onboarding issue MUST be opened in this repo. Please take a moment to do that before executing the tasks contained in this handbook.
There's a small amount of effort to ensure our release tools are supported on multiple platforms, please note that only the following systems are supported:
- Debian-like (Debian, Ubuntu)
- Fedora-like (Fedora, RHEL, CentOS)
- MacOS
Windows is not supported by release tools.
While our tooling may not support every platform, you may find success running within a container image.
See "Cutting v1.15.0-alpha.2" under References for an example Dockerfile.
If you notice that release tools are not working as expected, please file an issue in kubernetes/release.
To leverage/contribute to our release tools, Release Managers will need to fork and clone the kubernetes/release repo.
Building and publishing releases requires the latest revision of the release tools. The release tools can be compiled by running the following command from the kubernetes/release repository:
make release-toolsRelease Managers primarily use an SSH key to authenticate to GitHub.
GitHub has documentation to assist in:
Additionally, kubernetes/community has a great overview of the GitHub workflow we use across several Kubernetes org repositories.
Please take a moment to review the above documentation before continuing.
Kubernetes release artifacts are stored on Google Cloud Platform (GCP).
Release Managers will need to use the Google Cloud SDK to interact with release artifacts.
Google Cloud has documentation on installing and configuring the Google Cloud SDK CLI tools.
To authenticate the Google Cloud SDK tool with your GCP account, you can run:
gcloud auth loginIt might ask you to define the default project, region, and similar settings. The project can be set to kubernetes-release-test, while other settings can be ignored.
In addition to gcloud, gsutil is required as well. By default, it comes with Google Cloud SDK.
At the end of a release, Release Managers will need to announce the new release to the community.
NOTE: ALL OF THE FOLLOWING
krelCOMMANDS RUN AS MOCK (NO CHANGES) BY DEFAULT. No mock (--nomock) must be specified for command to execute / take action. This is clear in the output based on the targeted email test groups, but not obvious before running the command.
This can be done in one of two ways:
- The
krel announcesub command -- ASENDGRID_API_KEYwill need to be configured correctly on your environment for this to work- If you haven't used SendGrid before, SendGrid might require you to go through the Sender Identity Verification process before you can send emails/announcements
- Manually -- Send the email notification manually to kubernetes-announce and kubernetes-dev. You can take contents of the announcement in one of the following ways:
- By taking the contents from the Release Cloud Bucket:
https://dl.k8s.io/archive/anago-vX.Y.0-{alpha,beta,rc}.z/{announcement-subject.txt,announcement.html}: - By using
krel announcecommand with the--print-onlyflag - Example subject
- Example body
- By taking the contents from the Release Cloud Bucket:
# Only for the official release: Inform the Google team to complete the corresponding Deb and RPM builds and confirm with them whether Debian and RPM repositories have the packages before sending the email
export SENDGRID_API_KEY=<API_KEY>
krel announce send --tag vX.Y.0-{alpha,beta,rc}.Z --name "<Your Name>" --name <Your Email Address>See the Release Commands Cheat Sheet for example commands.
Permissions to send mail to the kubernetes-announce are managed in Google Groups. To request access please reach out to the list owners in slack. This is only needed if you plan on sending mails manually (not through sendgrid).
Skopeo is a command line utility that performs various operations on container images and image repositories. Skopeo is not required for performing branch management tasks (if needed, Docker and other relevant tools can be used instead), however, it might be referred to by other guides. If you want to install Skopeo, you can follow the official installation guide.
General overview:
Public build artifacts are published and an email notification goes out to the community. You will become very familiar with the following commands over the course of the 3 month release cycle:
krel stage/releasefor creating releaseskrel announceto send the announcement email notification.
There are more examples of the release process under the References section.
It is also highly recommended to glance over the Release Commands Cheat Sheet.
Prior to cutting a release version, open a "Cut a Release" issue on kubernetes/sig-release.
On the issue template, there are comments describing the predefined items that need to be completed.
For the item Screenshot unhealthy release branch testgrid boards...:
krel testgridshot takes care of generating screenshots of
Testgrid to keep as a reference of the state
it was in before cutting a release. This subcommand will generate images
of the boards and upload them to a Cloud Storage bucket, as well as the
Markdown code.
To invoke the subcommand, run it with the branch you are working on:
krel testgridshot --branch 1.20
You can include other testgrid states in the output and even have krel autocomment the issue for you:
krel testgridshot --branch 1.20 --github-issue 12345 --states=FLAKY
Once the script generates the Markdown table, post it as a comment on the created issue. You can take a look at the following comment as an example.
When running a release cut, you should open a thread in the #release-management Slack channel and include links to the GCP build console. You can take a look at the following thread as an example.
Once mock and nomock runs are complete, data about the jobs launched must be
collected in the issue. These are assembled in a table and correspond to the
Collect metrics, links... check mark.
After the release process has been completed, get the data table by using the
krel history subcommand. It will output a markdown table with the options used
to run the jobs, links to the GCB logs, and the result of each run.
krel history --branch release-1.xy --date-from <date-of-release>The generated table is then appended to the release issue, as it can be seen in the following issue for the v1.20.0-alpha.1 release.
Note: krel history works using the Default Application
Credentials
set in your environment. While you may be logged as a user with one or more Google
accounts in the GCP SDK (which are used in gsutil, gcloud, etc), you need to
make sure your user identity is set as DAC as software using the Google Cloud
libraries uses it to authenticate.
If you have not set any Default Application Credentials previously, krel will notify you. If you have another user or a service account you will simply get an authentication error.
Use the following command to start the authorization flow to set your default credentials:
gcloud auth application-default loginAfter having thoroughly read the section on cutting a release version of the handbook, and that all items on the checklist have completed (you may include notes on events that was unique to cutting that release version as comments), close the issue with /close as a comment the issue thread.
To stage a new alpha release, simply run krel stage. The default command line
parameters will already default to the correct --branch and --type.
The output of the command should provide a link to GCP, where you can track the
progress of the build. The build duration takes approximately 1 hour. krel stage is building all the bits for a bunch of target operating systems and
hardware architectures.
The actual releasing may be intentionally delayed after staging. For example, the branch manager may stage a build from the head of the release branch late in the release cycle, doing so in the morning so that it is fully built and would be releasable in the afternoon (pending CI tests run against the head of the branch).
If the CI results are good and the release team approves, you can initiate the publishing portion of the release process. If staging the build only happened after the receipt of clean CI tests results, this will delay completing the entire release process for a release version (alpha,beta,rc,...).
This presumes reproducible builds and that the CI tests are meaningful relative to the release builds. There is always a risk that these diverge, and this needs to be managed broadly by the project and the release team.
Before running the release step, please refer to the Image Promotion documentation.
To finish the previously staged release, run krel release --build-version=….
Use the --build-version= as specified in the output when krel stage is done.
This copies the staged build to public GCP buckets at well-known urls for the truncated release version string. The unique build staging identifier will subsequently be “v1.16.0-alpha.1”, even though the staged build had an “alpha.0” in its name.
This can be confusing. The v1.16.0-alpha.0 tag was created automatically in the past when the v1.15 branch was forked, but it wasn’t actually built.
Any krel stage/release command without the --nomock flag is a dry run. It is highly encouraged to dry run first before letting krel stage/release take any actual impact on the release process. Mock building/releasing can help you verify that you have a working setup!
To get more information on krel stage/release, please refer to their
corresponding help (-h) output.
Note: This run may fail. Mock builds can only be mock released. A nomock release requires a nomock build to be staged.
Before run the official release step please refer to the Image Promotion documentation.
To stage a new beta release, simply run krel stage --type=beta. The same
applies to krel release --build-version=… --type=beta.
Builds against a release-x.y branch are implicitly the next RC (release candidate). krel automatically finds and increments the current build number.
Note: If this is the first release (rc.0), there are additional tasks to complete. Please review them COMPLETELY in the Branch Creation section, before continuing.
Before run the official release step please refer to the Image Promotion documentation.
To stage a new RC release, simply run krel stage --type=rc --branch=release-x.y. The same
applies to krel release --build-version=… --type=rc --branch=release-x.y.
In a perfect world, rc.1 and the official release are the same commit. To get as close to that perfect state as we can, the following things should be considered:
-
PRs tagged with the release cycle milestone should have all merged onto the
masterbranch:For example, there should be no open PRs tagged with the v1.18 milestone,
is:pr is:open milestone:v1.18.You and the release team should push for PRs to get merged before the code freeze is lifted or determine if the open PRs can be removed from the milestone and be merged in the next release cycle.
-
When to stage and release
rc.1:Make sure that all the changes that have been merged onto
master, make it onto the release branch. Runkrel ffto see the state of the two branches and pull any remaining PRs frommasteronto the release branch.At this point in time,
masterand the release branch should have the same commits. Nothing new gets merged ontomaster, since code freeze is not lifted. Therefore, it is safe to cut the release candidate. -
Factors that determine if Code Freeze can be lifted:
- Zero pending PRs and no open issues tagged with the release cycle milestone.
- No failing X.Y-blocking tests, for example the sig-release-1.18-blocking dashboard should not have any red failing test builds.
Note: If there are non-zero pending PRs, open issues or failing tests, there is high confidence that the issue is understood and the solution can be cherry picked over to the release branch between the release candidate(s) and official release.
Technically we can keep code freeze in place after rc.1 was cut. However, we should aim at lifting code freeze relatively quickly after rc.1.
Otherwise we might have a mix of PRs against master, some have been merged in code freeze and for the milestone, just after rc.1, and others have been merged when code freeze has been lifted. We might miss this specific PR in the plethora of PRs that tide merges after code thaw, and we might miss that this PR actually needs to be cherry-picked into the release branch.
Before run the official release step please refer to the Image Promotion documentation.
To stage a new official release, simply run krel stage --type=official --branch=release-x.y. The same applies to krel release --build-version=… --type=official --brancH=release-x.y.
In addition to v1.18.n this will also build and stage the subsequent patch's
rc.0, in this example v1.18.(n+1)-rc.0. Similar to creating a new branch, the staging step will take about twice as long, the
release step will also take a couple of minutes more.
However, there is an embargo policy which requires the nomock release publication happens after 4 pm Pacific (see Release Team Lead Handbook for details), make sure to coordinates with other roles and follow the timeline.
To better prepare and see what to expect, this is a sequence of events that took place on past official release days.
The Security Response Committee (SRC) may contact you via the Security Release Team mailing list if there are security fixes needed on the release branch.
Once the release has been confirmed to contain security fixes, the Branch Manager must inform the current Release Team Lead and Lead Shadows. Information pertaining to these fixes is considered need-to-know and should not be disseminated to anyone else on the Release Team.
You must not make any public announcements regarding these fixes unless the SRC tells you to.
See the Security Release Process doc for more details.
The security fixes are usually announced upon publishing releases by:
- including information about security fixes in the release announcement sent to kubernetes-announce and dev mailing lists
- including information about security fixes in CHANGELOGs for affected releases
- sending an email announcement for each security fix to the relevant mailing lists (done by the Security Response Committee)
The first two are handled by Release Managers in coordination with the Security Response Committee (SRC). Before starting the release process, the Release Managers need to reach out to the SRC and ask them to provide a CVE data map that contains information about security fixes.
CVE data map file is created for each CVE and it looks like this:
---
pr: <cherry-pick PR number>
datafields:
cve:
id: CVE-2022-0000
title: "<title for given CVE>"
description: |
<description of vulnerability>
**Affected Versions**:
- kube-apiserver v1.25.0
<list of affected versions>...
**Fixed Versions**:
- kube-apiserver v1.25.1
<list of fixed versions>...
<credits and acknowledgements (optional)>
trackingissue: "[https://github.com/kubernetes/kubernetes/issues/TODO](https://github.com/kubernetes/kubernetes/issues/TODO)"
vector: "<cvss vector string>"
score: <cvss score>
rating: "<CVE rating (Low/Medium/High/Critical)>"
linkedprs: []
The CVE data map can also be created by using krel. Running the following command will open a text editor with a sample data map. Populating and saving the data map will automatically upload it to the appropriate (private) bucket (this requires access to the said bucket, therefore only the Release Managers can use this command).
krel cve edit CVE-2022-0000If multiple releases affected releases are affected by the same CVE, you must put a data map for each release in the same data map file, such as:
---
pr: <cherry-pick PR number for first release branch>
datafields:
cve:
id: CVE-2022-0000
...
---
pr: <cherry-pick PR number for second release branch>
datafields:
cve:
id: CVE-2022-0000
...
---
pr: <cherry-pick PR number for third release branch>
datafields:
cve:
id: CVE-2022-0000
...If CVE data map YAML files are provided by the SRC (instead of using krel cve command), those files can be uploaded to the appropriate private bucket using krel such as:
krel cve edit CVE-2022-0000 -f ./path-to-data-map.yamlThis command will also take care of validating the provided data map file. Once the data maps are uploaded, krel will (automatically) take those data maps into account when generating the changelog and the release announcement emails.
The following are some ways to determine if the release process was successful:
-
The build tag and release artifacts become visible on GitHub at https://github.com/kubernetes/kubernetes/releases
-
The release is logged automatically by k8s-release-robot in k/sig-release
-
CHANGELOG-X.Y.md is automatically loaded into the kubernetes/kubernetes repo: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.16.md
Set the K8S_RELEASE marker for the current release variant to stable-x.y which was currently latest-x.y in the variants.yaml file for kubekins-e2e. (reference PR review comment)
Recall that an alpha.0 of the next minor release was created during branch creation.
The previously created alpha.0 is now several commits behind master.
As an example, see the comparison between the v1.18.0-alpha.0 (after 1.17 branch creation) and v1.18.0-alpha.1 (after 1.17.0 release) tags.
To assist downstream consumers of Kubernetes, a new alpha should be cut to bring our next release tag to the tip of master.
Begin the release process with:
krel stageProceed with the alpha release steps.
Example 1.18.0-alpha.1 release issue template: kubernetes#928
This section discusses the methods in managing commits on the release-x.y branch.
During a rc.0 release our release tooling creates a new release branch named release-x.y, where x and y are the major and minor versions of the next release, respectively.
Behind the scenes krel is doing a git branch create and git push. The branch is created in the staging phase and is pushed to the repository in the release phase.
prow’s branchprotector runs every hour at 54 minutes past the hour and automatically adds branch protection to any new branch in the kubernetes/kubernetes repo.
New release branch creation (for example: release-1.18) also automatically triggers an alpha.0 build for the subsequent release (for example: v1.19.0-alpha.0).
This means that the staging step will take about twice as long, as it will stage both versions v1.18.0-rc.0 and v1.19.0-alpha.0. The release step will also be extended, but not substantially longer in time.
Before the nomock release job is started, run through the following tasks, putting an explicit hold on any PRs (to be removed once the release branch has been created):
Once the new release-x.y branch is created, the release jobs and dashboards should be generated and merged.
Before getting started, Branch Managers should:
- Fork the test-infra repository
- Clone their fork of
kubernetes/test-infra:
git clone git@github.com:<username>/test-infra.git- Install Bazel or run Bazel inside a container
- Running Bazel in a container is recommended over installing Bazel locally, as Bazel has many dependencies
The milestone applier plugin automatically applies a GitHub milestone to pull requests after they have merged.
This only applies to repos that have the milestone applier configured and for pull requests that do not already have a milestone.
Update the milestoneapplier plugin configs for kubernetes/kubernetes:
- Remove configs for the unsupported release branches, if present
- Add config for the current release branch
Here's an example PR.
If the code freeze was enabled before creating the release branch, the milestone requirements wouldn't include the newest release branch.
Find the query config for kubernetes/kubernetes (in config.yaml file) with the code freeze enabled and add the newest release release branch.
Here's an example PR.
Images used in Kubernetes e2e tests are generated via our GCB Builder tool.
The variants.yaml config file, used in conjunction with the GCB Builder, allows us to target various branches or branch combinations during CI tests. The K8S_VERSION variable maps to the version marker latest-x.y.txt file viewable in the kubernetes-release GCS bucket for example, latest-1.17.txt.
Update the variants for the kubekins-e2e image.
- Remove the release variants for the unsupported releases
- Add an entry for the newest release variant
- Copy the
mastervariant, rename it to the newest release and ensure the following:- The
K8S_RELEASEmarker forexperimentalmatchesmaster - The
CONFIGmarker for the newest release matches the variant/release name - The
K8S_RELEASEmarker for the newest release variant islatest-x.y - The
K8S_RELEASEmarker for every other release variant isstable-x.y
- The
Create a PR with this change and wait for it to be merged (example PR).
Wait for the test-infra-push-kubekins-e2e presubmit to finish (you can check on prow).
git pull latest from master before continuing.
To minimize drift with tooling changes, this section has been moved to kubernetes/test-infra.
Once we have cut the branch, it is a good time to start producing a new kube-cross image. To do this, we first need to add the variant to kube-cross variants.yaml file and update the dependencies.yaml file. Here is an example PR.
Once that is merged, wait for this prow job to complete and then do an image promotion PR. Here is an example.
To create a new k8s-cloud-builder image with the new kube-cross image, a similar PR is created to update the variants.yaml for k8s-cloud-builder. Here is an older example pr.
Finally, we can bump the kube-cross builder image in kubernetes/kubernetes. Here is an example PR.
The Kubernetes Publishing Bot is responsible for:
- ensuring that the master and release branches in the staging repositories are in-sync with the appropriate branches in
kubernetes/kubernetes - creating tags in the staging repositories for each Kubernetes release
It's required to create the appropriate publishing-bot rules for the publishing-bot to work with the release branches. Once a new release branch is created in kubernetes/kubernetes, the Release Manager needs to update the publishing-bot rules as described in the k/publishing-bot repository. This best way to do this is by using the update-rules CLI tool.
Here's an example PR, but please generate the changes with the update-rules tooling.
Between the Code Freeze and lifting Code Freeze (Code Thaw) period, merging new code is restricted. The main focus is on fixing existing code and getting green test builds on Testgrid. Preventing new code is implemented by config changes for tide. The master and current release cycle branch (release-x.y) are the only branches affected during this period.
Code freeze initiates additional merge requirements, while Code thaw marks the switch back to the development phase. Look at the release cycle timeline for the exact dates for code freeze and code thaw. Usually the the date for code thaw is flexible depending on pending PRs.
As Branch Manager, coordinate with the Release Lead on checking the exact config changes required to enable and disable merge restrictions.
Tide automates merges and is configured via a config.yaml file. Tide identifies PRs that are mergeable using GitHub queries that correspond to the configuration. Here is an example of what the query config for kubernetes/kubernetes looks like without additional constraints related to the release cycle:
- repos:
- kubernetes/kubernetes
labels:
- lgtm
- approved
- "cncf-cla: yes"
missingLabels:
- do-not-merge
- do-not-merge/blocked-paths
- do-not-merge/cherry-pick-not-approved
- do-not-merge/hold
- do-not-merge/invalid-owners-file
- do-not-merge/release-note-label-needed
- do-not-merge/work-in-progress
- do-not-merge/needs-kind
- do-not-merge/needs-sig
- needs-rebaseDuring code freeze, two queries are used instead of just one for the kubernetes/kubernetes repo. One query handles the master and current release branches while the other query handles all other branches. The partition is achieved with the includedBranches and excludedBranches fields.
Code freeze means the code is "frozen" and there will not be any further modifications from the developers.
Branch managers create the "freeze" by altering the Tide merge requirements for the master and current release-x.y branch from the other branches (enforced by Tide with two queries).
We only add additional merge requirements for PRs to these two branches for code freeze:
- PRs must be in the GitHub milestone for the current release.
Milestone requirements are configured by adding milestone: vX.Y to a query config.
It is also helpful to remind #sig-testing when code freeze starts so they know not to do any major changes.
- repos:
- kubernetes/kubernetes
milestone: v1.16
includedBranches:
- master
- release-1.16
labels:
- lgtm
- approved
- "cncf-cla: yes"
missingLabels:
# as above...
- repos:
- kubernetes/kubernetes
excludedBranches:
- master
- release-1.16
labels:
- lgtm
- approved
- "cncf-cla: yes"
missingLabels:
...Example PR:
Code Thaw removes the release cycle merge restrictions and replaces the two queries with one single query. We remain in this state until the next Code Freeze.
- repos:
- kubernetes/kubernetes
labels:
- lgtm
- approved
- "cncf-cla: yes"
missingLabels:
...Update the milestoneapplier plugin configs for the following repos to the next milestone:
kubernetes/enhancementskubernetes/kuberneteskubernetes/releasekubernetes/sig-releasekubernetes/test-infra
Example PRs:
We now run the branch fast forward automatically to even the branches.
Noted that no need for manual cherry-pick against current release and the job would be promoted to release-blocking.
Once code freeze is lifted (code thaw occurred), PRs that need to be merged onto the release branch are cherry-picked over from master.
During code freeze it is especially important to first look at the list of commits on master since the prior fast forward, scanning their content and issues/PRs to ensure they are changes expected for this milestone.
The merge-blocking mechanisms are relatively weak.
It is possible still for some people to write directly to the repo (bypassing blocking mechanisms) as well as for unintentional milestone maintainers to approve a merge incorrectly. The branch manager is the last line of defense.
If code incorrectly merges onto master it should be reverted in master. Alternatively, release branch management must go to all cherry picks, picking around the incorrectly added commit.
Once code freeze is lifted, and for the post-release patch management process, commits are cherry picked from master.
It's important to take note of what is considered as a release blocking issue or PR and identify PRs and issues as soon as possible. Ensure that there's work being done to release blocking PRs completed prior to the official release day.
Let the release lead know if there's not enough attention placed on the release blocking PR. This way we can prevent from prolonging the official release.
The current documentation in the contributor guide for cherry picks should be generally sufficient. There are a couple prerequisites to running the script which are outlined in that guide.
The cherry pick script is also fairly self documenting in terms of how to invoke the command.
Running cherry_pick_pull.sh creates a cherry pick PR against the specified release-x.y branch. The content of the cherry pick PR is taken from a PR that was made against master. Once the appropriate approval is given to the cherry pick PR and has met the requirements set by the k8s-ci-robot, this will trigger test re-runs on the PR. As soon as these tests all pass, add the cherry-pick-approved label.
For example, #79044 is the cherry pick PR of #78999. Once #79044 is tagged with the cherry-pick-approved label, Prow begins the merge onto the release branch.
There has been quite a bit of recent discussion (see: 1, 2) around improving both the cherry pick process process and its documentation.
After the official release has been published, the Release Managers will take over in handling cherry picks. In the time between code thaw and the official release, cherry picks are the responsibility of the branch management team.
Consider the following when assessing the cherry-picks:
- Check regularly if there are new cherry picks with
is:open is:pr base:release-1.18 label:do-not-merge/cherry-pick-not-approved - Consider that each cherry-pick diverges the latest release candidate that has been cut from the bits to be released as the official release
- Engage with the cherry pick requester: How important is that cherry-pick, can it be pushed to a later release (patch or even minor), ... ?
- Discuss (especially controversial) cherry-picks in [#sig-release] or at the burndown meeting if you are unsure
- If certain cherry-picks merge, does this mean we want another release candidate, more time for the release candidate to soak (e.g. over the weekend)?
The publishing-bot is responsible for publishing the code in staging to their own repositories.
The bot also syncs the Kubernetes version tags to the published repos, prefixed with kubernetes-. For example, if you check out the kubernetes-1.16.0 tag in client-go, the code you get is exactly the same as if you check out the v1.16.0 tag in Kubernetes, and change the directory to staging/src/k8s.io/client-go.
client-go follows semver and has its own release process. This release process and the publishing-bot are maintained by SIG API Machinery. In case of any questions related to the published staging repos, please ask someone listed in the following OWNERS file.
The bot runs every four hours, so it might take sometime for a new tag to appear on a published repository.
The client-go major release (e.g. v1.18.0) is released manually a day after the main Kubernetes release.
To debug krel stage/release you can set the log level to debug by doing
krel --log-level debug [args]. There is also a trace log level for maximum
output verbosity. The log level is passed correctly to the Google Cloud console
and can be inspected there.
To search past cloudbuilds to make any kind of analysis and check metrics you can use the Google Cloud Console for that.
You can access Google Cloud Build History
There you can see all the cloudbuilds that already ran or if there is any one running in the moment you can see that as well.
To filter for a specific build or set of builds you can use the Filter Builds textbox. When you click there it opens a dropdown box to you select which kind of keys you want to filter.
For example you can select Tags and then add the tags you want to filter, like, Tags: release-1.18 Status: Successful. This query in the Google Cloud Build
Looks like in the Google Cloud Build History you cannot set more than one tag when using the AND operator. The query does not bring anything if you add, for example, Tags: release-1.18 Tags: STAGE.
Besides the fact that exists.
- Release Tools Documentation
- Generic Release Steps
- Detailed overview (Cutting v1.15.0-alpha.2)
- If you prefer a video walk-through, see cutting release candidate v1.15.0-rc.1.
Concerns and questions can be directed to #testing-ops and #sig-testing. For urgent matters, please contact the user group @test-infra-oncall on Slack.
- Prow the Kubernetes-based CI/CD system
- Gubernator displays Kubernetes test results stored in Google Cloud Storage (GCS)
- PR Dashboard
- Hound a Kubernetes Codebase Search
- Kubernetes DevStats displays Kubernetes Developer Productivity
- Kubernetes On-call Rotation displays the current Test Infra person on call
- APISnoop snoops on the Kubernetes OpenAPI communications
- TestGrid displays Kubernetes CI tests results in grids
- Triage Dashboard displays Kubernetes aggregated failures
You may have seen reference to ./anago when cutting releases but there's no mention of using anago in this handbook, see:
Note: To access this forum, you will need to join the kubernetes-milestone-burndown Google group.
See the branch management process prior to v1.12 when anago was still used.
Note: To view this document, you will need to join the kubernetes-dev Google group.
The diagram below shows the actions needed to cut a Kubernetes release
flowchart TD
Start(["Start"]) --> issue["Create a release cut issue on GitHub"] -->|"update along the way"| issue
issue --> thread["Create a Slack Thread in #release-management \n and cc release-managers"] -->|"update along the way"| thread
thread --> build_admins["Contact Google Build Admins \n for their availability to plan when to cut the release"]
build_admins --> mock["Mock run (stage and release)"]
mock --> staging["Nomock stage"]
staging --> artifact_promotion["Image promotion"]
artifact_promotion --> release["Nomock release"]
release --> publish_pkgs["Contact Google Build Admins to publish the deps/rpms"]
publish_pkgs --> announcement_chat["Notify Slack channel #release-management \n about the new release"]
announcement_chat --> announcement_email["Notify Community by Email \n using Krel"]
announcement_email --> close["Close release cut GitHub Issue"]
close --> done(["End"])
classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
class Start,done plain;
class issue,thread,build_admins,mock,staging,artifact_promotion,release,publish_pkgs,announcement_chat,announcement_email,close k8s;