Skip to content

Commit

Permalink
Squashed 'release-tools/' changes from 3041b8a..4aff857
Browse files Browse the repository at this point in the history 
4aff857 Merge pull request #109 from pohly/alpha-test-defaults
0427289 Merge pull request #110 from pohly/kind-0.9-bazel-build-workaround
9a370ab prow.sh: work around "kind build node-image" failure
522361e prow.sh: only run alpha tests for latest Kubernetes release
22c0395 Merge pull request #108 from bnrjee/master
b5b447b Add go ldflags using LDFLAGS at the time of compilation
16f4afb Merge pull request #107 from pohly/kind-update
7bcee13 prow.sh: update to kind 0.9, support Kubernetes 1.19
df518fb prow.sh: usage of Bazel optional
c3afd42 Merge pull request #104 from xing-yang/snapshot
dde93b2 Update to snapshot-controller v3.0.0
a0f195c Merge pull request #106 from msau42/fix-canary
7100c12 Only set staging registry when running canary job
b3c65f9 Merge pull request #99 from msau42/add-release-process
e53f3e8 Merge pull request #103 from msau42/fix-canary
d129462 Document new method for adding CI jobs are new K8s versions
e73c2ce Use staging registry for canary tests
2c09846 Add cleanup instructions to release-notes generation
60e1cd3 Merge pull request #98 from pohly/kubernetes-1-19-fixes
0979c09 prow.sh: fix E2E suite for Kubernetes >= 1.18
3b4a2f1 prow.sh: fix installing Go for Kubernetes 1.19.0
1fbb636 Merge pull request #97 from pohly/go-1.15
82d108a switch to Go 1.15
d8a2530 Merge pull request #95 from msau42/add-release-process
843bddc Add steps on promoting release images
0345a83 Merge pull request #94 from linux-on-ibm-z/bump-timeout
1fdf2d5 cloud build: bump timeout in Prow job
41ec6d1 Merge pull request #93 from animeshk08/patch-1
5a54e67 filter-junit: Fix gofmt error
0676fcb Merge pull request #92 from animeshk08/patch-1
36ea4ff filter-junit: Fix golint error
f5a4203 Merge pull request #91 from cyb70289/arm64
43e50d6 prow.sh: enable building arm64 image
0d5bd84 Merge pull request #90 from pohly/k8s-staging-sig-storage
3df86b7 cloud build: k8s-staging-sig-storage
c5fd961 Merge pull request #89 from pohly/cloud-build-binfmt
db0c2a7 cloud build: initialize support for running commands in Dockerfile
be902f4 Merge pull request #88 from pohly/multiarch-windows-fix
340e082 build.make: optional inclusion of Windows in multiarch images
5231f05 build.make: properly declare push-multiarch
4569f27 build.make: fix push-multiarch ambiguity
17dde9e Merge pull request #87 from pohly/cloud-build
bd41690 cloud build: initial set of shared files
9084fec Merge pull request #81 from msau42/add-release-process
6f2322e Update patch release notes generation command
0fcc3b1 Merge pull request #78 from ggriffiths/fix_csi_snapshotter_rbac_version_set
d8c76fe Support local snapshot RBAC for pull jobs
c1bdf5b Merge pull request #80 from msau42/add-release-process
ea1f94a update release tools instructions
152396e Merge pull request #77 from ggriffiths/snapshotter201_update
7edc146 Update snapshotter to version 2.0.1
4cf843f Merge pull request #76 from pohly/build-targets
3863a0f build for multiple platforms only in CI, add s390x
8322a7d Merge pull request #72 from pohly/hostpath-update
7c5a89c prow.sh: use 1.3.0 hostpath driver for testing
b8587b2 Merge pull request #71 from wozniakjan/test-vet
fdb3218 Change 'make test-vet' to call 'go vet'
d717c8c Merge pull request #69 from pohly/test-driver-config
a1432bc Merge pull request #70 from pohly/kubelet-feature-gates
5f74333 prow.sh: also configure feature gates for kubelet
84f78b1 prow.sh: generic driver installation
3c34b4f Merge pull request #67 from windayski/fix-link
fa90abd fix incorrect link
ff3cc3f Merge pull request #54 from msau42/add-release-process
ac8a021 Document the process for releasing a new sidecar
23be652 Merge pull request #65 from msau42/update-hostpath
6582f2f Update hostpath driver version to get fix for connection-timeout
4cc9174 Merge pull request #64 from ggriffiths/snapshotter_2_version_update
8191eab Update snapshotter to version v2.0.0
3c463fb Merge pull request #61 from msau42/enable-snapshots
8b0316c Fix overriding of junit results by using unique names for each e2e run
5f444b8 Merge pull request #60 from saad-ali/updateHostpathVersion
af9549b Update prow hostpath driver version to 1.3.0-rc2
f6c74b3 Merge pull request #57 from ggriffiths/version_gt_kubernetes_fix
fc80975 Fix version_gt to work with kubernetes prefix
9f1f3dd Merge pull request #56 from msau42/enable-snapshots
b98b2ae Enable snapshot tests in 1.17 to be run in non-alpha jobs.
9ace020 Merge pull request #52 from msau42/update-readme
540599b Merge pull request #53 from msau42/fix-make
a4e6299 fix syntax for ppc64le build
771ca6f Merge pull request #49 from ggriffiths/prowsh_improve_version_gt
d7c69d2 Merge pull request #51 from msau42/enable-multinode
4ad6949 Improve snapshot pod running checks and improve version_gt
53888ae Improve README by adding an explicit Kubernetes dependency section
9a7a685 Create a kind cluster with two worker nodes so that the topology feature can be tested. Test cases that test accessing volumes from multiple nodes need to be skipped
4ff2f5f Merge pull request #50 from darkowlzz/kind-0.6.0
80bba1f Use kind v0.6.0
6d674a7 Merge pull request #47 from Pensu/multi-arch
8adde49 Merge pull request #45 from ggriffiths/snapshot_beta_crds
003c14b Add snapshotter CRDs after cluster setup
a41f386 Merge pull request #46 from mucahitkurt/kind-cluster-cleanup
1eaaaa1 Delete kind cluster after tests run.
83a4ef1 Adding build for ppc64le
4fcafec Merge pull request #43 from pohly/system-pod-logging
f41c135 prow.sh: also log output of system containers
ee22a9c Merge pull request #42 from pohly/use-vendor-dir
8067845 travis.yml: also use vendor directory
23df4ae prow.sh: use vendor directory if available
a53bd4c Merge pull request #41 from pohly/go-version
c8a1c4a better handling of Go version
5e773d2 update CI to use Go 1.13.3
f419d74 Merge pull request #40 from msau42/add-1.16
e0fde8c Add new variables for 1.16 and remove 1.13
adf00fe Merge pull request #36 from msau42/full-clone
f1697d2 Do full git clones in travis. Shallow clones are causing test-subtree errors when the depth is exactly 50.
2c81919 Merge pull request #34 from pohly/go-mod-tidy
518d6af Merge pull request #35 from ddebroy/winbld2
2d6b3ce Build Windows only for amd64
c1078a6 go-get-kubernetes.sh: automate Kubernetes dependency handling
194289a update Go mod support
0affdf9 Merge pull request #33 from gnufied/enable-hostpath-expansion
6208f6a Enable hostpath expansion
6ecaa76 Merge pull request #30 from msau42/fix-windows
ea2f1b5 build windows binaries with .exe suffix
2d33550 Merge pull request #29 from mucahitkurt/create-2-node-kind-cluster
a8ea8bc create 2-node kind cluster since topology support is added to hostpath driver
df8530d Merge pull request #27 from pohly/dep-vendor-check
35ceaed prow.sh: install dep if needed
f85ab5a Merge pull request #26 from ddebroy/windows1
9fba09b Add rule for building Windows binaries
0400867 Merge pull request #25 from msau42/fix-master-jobs
dc0a5d8 Update kind to v0.5.0
aa85b82 Merge pull request #23 from msau42/fix-master-jobs
f46191d Kubernetes master changed the way that releases are tagged, which needed changes to kind. There are 3 changes made to prow.sh:
1cac3af Merge pull request #22 from msau42/add-1.15-jobs
0c0dc30 prow.sh: tag master images with a large version number
f4f73ce Merge pull request #21 from msau42/add-1.15-jobs
4e31f07 Change default hostpath driver name to hostpath.csi.k8s.io
4b6fa4a Update hostpath version for sidecar testing to v1.2.0-rc2
ecc7918 Update kind to v0.4.0. This requires overriding Kubernetes versions with specific patch versions that kind 0.4.0 supports. Also, feature gate setting is only supported on 1.15+ due to kind.sigs.k8s.io/v1alpha3 and kubeadm.k8s.io/v1beta2 dependencies.
a6f21d4 Add variables for 1.15
db8abb6 Merge pull request #20 from pohly/test-driver-config
b2f4e05 prow.sh: flexible test driver config
0399988 Merge pull request #19 from pohly/go-mod-vendor
066143d build.make: allow repos to use 'go mod' for vendoring
0bee749 Merge pull request #18 from pohly/go-version
e157b6b update to Go 1.12.4
88dc9a4 Merge pull request #17 from pohly/prow
0fafc66 prow.sh: skip sanity testing if component doesn't support it
bcac1c1 Merge pull request #16 from pohly/prow
0b10f6a prow.sh: update csi-driver-host-path
0c2677e Merge pull request #15 from pengzhisun/master
ff9bce4 Replace 'return' to 'exit' to fix shellcheck error
c60f382 Merge pull request #14 from pohly/prow
7aaac22 prow.sh: remove AllAlpha=all, part II
6617773 Merge pull request #13 from pohly/prow
cda2fc5 prow.sh: avoid AllAlpha=true
546d550 prow.sh: debug failing KinD cluster creation
9b0d9cd build.make: skip shellcheck if Docker is not available
aa45a1c prow.sh: more efficient execution of individual tests
f3d1d2d prow.sh: fix hostpath driver version check
31dfaf3 prow.sh: fix running of just "alpha" tests
f501443 prow.sh: AllAlpha=true for unknown Kubernetes versions
95ae9de Merge pull request #9 from pohly/prow
d87eccb prow.sh: switch back to upstream csi-driver-host-path
6602d38 prow.sh: different E2E suite depending on Kubernetes version
741319b prow.sh: improve building Kubernetes from source
29545bb prow.sh: take Go version from Kubernetes source
429581c prow.sh: pull Go version from travis.yml
0a0fd49 prow.sh: comment clarification
2069a0a Merge pull request #11 from pohly/verify-shellcheck
55212ff initial Prow test job
6c7ba1b build.make: integrate shellcheck into "make test"
b2d25d4 verify-shellcheck.sh: make it usable in csi-release-tools
3b6af7b Merge pull request #12 from pohly/local-e2e-suite
104a1ac build.make: avoid unit-testing E2E test suite
34010e7 Merge pull request #10 from pohly/vendor-check
e6db50d check vendor directory
fb13c51 verify-shellcheck.sh: import from Kubernetes
94fc1e3 build.make: avoid unit-testing E2E test suite
849db0a Merge pull request #8 from pohly/subtree-check-relax
cc564f9 verify-subtree.sh: relax check and ignore old content
33d58fd Merge pull request #5 from pohly/test-enhancements
be8a440 Merge pull request #4 from pohly/canary-fix
b0336b5 build.make: more readable "make test" output
09436b9 build.make: fix pushing of "canary" image from master branch
147892c build.make: support suppressing checks
154e33d build.make: clarify usage of "make V=1"

git-subtree-dir: release-tools
git-subtree-split: 4aff857
  • Loading branch information
@pohly
pohly committed Oct 27, 2020
1 parent 3041b8a commit 91bc0c8
Show file tree
Hide file tree
Showing 16 changed files with 2,312 additions and 17 deletions.
7 changes: 7 additions & 0 deletions .prow.sh
View file
@@ -0,0 +1,7 @@
#! /bin/bash -e
#
# This is for testing csi-release-tools itself in Prow. All other
# repos use prow.sh for that, but as csi-release-tools isn't a normal
# repo with some Go code in it, it has a custom Prow test script.

./verify-shellcheck.sh "$(pwd)"
115 changes: 115 additions & 0 deletions README.md
View file
Expand Up @@ -49,3 +49,118 @@ Cheat sheet:
- `git subtree add --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once)
- `git subtree pull --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes)
- edit, `git commit`, `git subtree push --prefix=release-tools git@github.com:<user>/csi-release-tools.git <my-new-or-existing-branch>` - push to a new branch before submitting a PR

verify-shellcheck.sh
--------------------

The [verify-shellcheck.sh](./verify-shellcheck.sh) script in this repo
is a stripped down copy of the [corresponding
script](https://github.com/kubernetes/kubernetes/blob/release-1.14/hack/verify-shellcheck.sh)
in the Kubernetes repository. It can be used to check for certain
errors shell scripts, like missing quotation marks. The default
`test-shellcheck` target in [build.make](./build.make) only checks the
scripts in this directory. Components can add more directories to
`TEST_SHELLCHECK_DIRS` to check also other scripts.

End-to-end testing
------------------

A repo that wants to opt into testing via Prow must set up a top-level
`.prow.sh`. Typically that will source `prow.sh` and then transfer
control to it:

``` bash
#! /bin/bash -e

. release-tools/prow.sh
main
```

All Kubernetes-CSI repos are expected to switch to Prow. For details
on what is enabled in Prow, see
https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi

Test results for periodic jobs are visible in
https://testgrid.k8s.io/sig-storage-csi-ci

It is possible to reproduce the Prow testing locally on a suitable machine:
- Linux host
- Docker installed
- code to be tested checkout out in `$GOPATH/src/<import path>`
- `cd $GOPATH/src/<import path> && ./.prow.sh`

Beware that the script intentionally doesn't clean up after itself and
modifies the content of `$GOPATH`, in particular the `kubernetes` and
`kind` repositories there. Better run it in an empty, disposable
`$GOPATH`.

When it terminates, the following command can be used to get access to
the Kubernetes cluster that was brought up for testing (assuming that
this step succeeded):

export KUBECONFIG="$(kind get kubeconfig-path --name="csi-prow")"

It is possible to control the execution via environment variables. See
`prow.sh` for details. Particularly useful is testing against different
Kubernetes releases:

CSI_PROW_KUBERNETES_VERSION=1.13.3 ./.prow.sh
CSI_PROW_KUBERNETES_VERSION=latest ./.prow.sh

Dependencies and vendoring
--------------------------

Most projects will (eventually) use `go mod` to manage
dependencies. `dep` is also still supported by `csi-release-tools`,
but not documented here because it's not recommended anymore.

The usual instructions for using [go
modules](https://github.com/golang/go/wiki/Modules) apply. Here's a cheat sheet
for some of the relevant commands:
- list available updates: `GO111MODULE=on go list -u -m all`
- update or add a single dependency: `GO111MODULE=on go get <package>`
- update all dependencies to their next minor or patch release:
`GO111MODULE=on go get ./...` (add `-u=patch` to limit to patch
releases)
- lock onto a specific version: `GO111MODULE=on go get <package>@<version>`
- clean up `go.mod`: `GO111MODULE=on go mod tidy`
- update vendor directory: `GO111MODULE=on go mod vendor`

`GO111MODULE=on` can be left out when using Go >= 1.13 or when the
source is checked out outside of `$GOPATH`.

`go mod tidy` must be used to ensure that the listed dependencies are
really still needed. Changing import statements or a tentative `go
get` can result in stale dependencies.

The `test-vendor` verifies that it was used when run locally or in a
pre-merge CI job. If a `vendor` directory is present, it will also
verify that it's content is up-to-date.

The `vendor` directory is optional. It is still present in projects
because it avoids downloading sources during CI builds. If this is no
longer deemed necessary, then a project can also remove the directory.

Conversion of a repository that uses `dep` to `go mod` can be done with:

GO111MODULE=on go mod init
release-tools/go-get-kubernetes.sh <current Kubernetes version from Gopkg.toml>
GO111MODULE=on go mod tidy
GO111MODULE=on go mod vendor
git rm -f Gopkg.toml Gopkg.lock
git add go.mod go.sum vendor

### Updating Kubernetes dependencies

When using packages that are part of the Kubernetes source code, the
commands above are not enough because the [lack of semantic
versioning](https://github.com/kubernetes/kubernetes/issues/72638)
prevents `go mod` from finding newer releases. Importing directly from
`kubernetes/kubernetes` also needs `replace` statements to override
the fake `v0.0.0` versions
(https://github.com/kubernetes/kubernetes/issues/79384). The
`go-get-kubernetes.sh` script can be used to update all packages in
lockstep to a different Kubernetes version. Example usage:
```
$ ./release-tools/go-get-kubernetes.sh 1.16.4
```
106 changes: 106 additions & 0 deletions SIDECAR_RELEASE_PROCESS.md
View file
@@ -0,0 +1,106 @@
# Sidecar Release Process

This page describes the process for releasing a kubernetes-csi sidecar.

## Prerequisites

The release manager must:

* Be a member of the kubernetes-csi organization. Open an
[issue](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-membership&template=membership.md&title=REQUEST%3A+New+membership+for+%3Cyour-GH-handle%3E) in
kubernetes/org to request membership
* Be a top level approver for the repository. To become a top level approver,
the candidate must demonstrate ownership and deep knowledge of the repository
through active maintainence, responding to and fixing issues, reviewing PRs,
test triage.
* Be part of the maintainers or admin group for the repository. admin is a
superset of maintainers, only maintainers level is required for cutting a
release. Membership can be requested by submitting a PR to kubernetes/org.
[Example](https://github.com/kubernetes/org/pull/1467)

## Updating CI Jobs
Whenever a new Kubernetes minor version is released, our kubernetes-csi CI jobs
must be updated.

[Our CI jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) have the
naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.

1. Jobs should be actively monitored to find and fix failures in sidecars and
infrastructure changes early in the development cycle. Test failures are sent
to kubernetes-sig-storage-test-failures@googlegroups.com.
1. "-on-master" jobs are the closest reflection to the new Kubernetes version.
1. Fixes to our prow.sh CI script can be tested in the [CSI hostpath
repo](https://github.com/kubernetes-csi/csi-driver-host-path) by modifying
[prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/release-tools/prow.sh)
along with any overrides in
[.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/.prow.sh)
to mirror the failing environment. Once e2e tests are passing (verify-unit tests
will fail), then the prow.sh changes can be submitted to [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools).
1. Changes can then be updated in all the sidecar repos and hostpath driver repo
by following the [update
instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating).
1. New pull and CI jobs are configured by adding new K8s versions to the top of
[gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh).
New pull jobs that have been unverified should be initially made optional by
setting the new K8s version as
[experimental](https://github.com/kubernetes/test-infra/blob/a1858f46d6014480b130789df58b230a49203a64/config/jobs/kubernetes-csi/gen-jobs.sh#L40).
1. Once new pull and CI jobs have been verified, and the new Kubernetes version
is released, we can make the optional jobs required, and also remove the
Kubernetes versions that are no longer supported.

## Release Process
1. Identify all issues and ongoing PRs that should go into the release, and
drive them to resolution.
1. Download v2.8+ [K8s release notes
generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes)
1. Generate release notes for the release. Replace arguments with the relevant
information.
* Clean up old cached information (also needed if you are generating release
notes for multiple repos)
```bash
rm -rf /tmp/k8s-repo
```
* For new minor releases on master:
```bash
GITHUB_TOKEN=<token> release-notes --discover=mergebase-to-latest
--github-org=kubernetes-csi --github-repo=external-provisioner
--required-author="" --output out.md
```
* For new patch releases on a release branch:
```bash
GITHUB_TOKEN=<token> release-notes --discover=patch-to-latest --branch=release-1.1
--github-org=kubernetes-csi --github-repo=external-provisioner
--required-author="" --output out.md
```
1. Compare the generated output to the new commits for the release to check if
any notable change missed a release note.
1. Reword release notes as needed. Make sure to check notes for breaking
changes and deprecations.
1. If release is a new major/minor version, create a new `CHANGELOG-<major>.<minor>.md`
file. Otherwise, add the release notes to the top of the existing CHANGELOG
file for that minor version.
1. Submit a PR for the CHANGELOG changes.
1. Submit a PR for README changes, in particular, Compatibility, Feature status,
and any other sections that may need updating.
1. Check that all [canary CI
jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) are passing,
and that test coverage is adequate for the changes that are going into the release.
1. Make sure that no new PRs have merged in the meantime, and no PRs are in
flight and soon to be merged.
1. Create a new release following a previous release as a template. Be sure to select the correct
branch. This requires Github release permissions as required by the prerequisites.
[external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new)
1. If release was a new major/minor version, create a new `release-<minor>`
branch at that commit.
1. Check [image build status](https://k8s-testgrid.appspot.com/sig-storage-image-build).
1. Promote images from k8s-staging-sig-storage to k8s.gcr.io/sig-storage. From
the [k8s image
repo](https://github.com/kubernetes/k8s.io/tree/master/k8s.gcr.io/images/k8s-staging-sig-storage),
run `./generate.sh > images.yaml`, and send a PR with the updated images.
Once merged, the image promoter will copy the images from staging to prod.
1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar
and feature pages with the new released version.
1. After all the sidecars have been released, update
CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
and [k/k
in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)

0 comments on commit 91bc0c8

Please sign in to comment.