contrib: Support prereleases in release prep scripts

.github/ISSUE_TEMPLATE/release_template.md

@@ -24,12 +24,16 @@ assignees: ''
 - [ ] Push a PR including the changes necessary for the new release:
   - [ ] Pull latest changes from the branch being released
   - [ ] Run `contrib/release/start-release.sh`
-        **Note**: For `RCs` that are not in a stable branch, you need to follow
-                  the RC release guide manually.
   - [ ] Run `Documentation/check-crd-compat-table.sh vX.Y` and if needed, follow the
         instructions.
+        **Note**: For `RCs` that are not in a stable branch, you need to follow
+                  bump the CRD version manually and skip this step.
   - [ ] Commit all changes with title `Prepare for release vX.Y.Z`
+    - [ ] For RC versions, commit the `AUTHORS` file changes separately.
   - [ ] Submit PR (`contrib/release/submit-release.sh`)
+  - [ ] For a new RC version on the master branch:
+    - [ ] Allow the CI to sanity-check the PR and get review
+    - [ ] Revert the release commit and re-push
   - [ ] For a new minor version:
     - [ ] Add the 'stable' tag as part of the GitHub workflow and remove the
           'stable' tag from the last stable branch.
@@ -41,14 +45,17 @@ assignees: ''
           branch.
 - [ ] Merge PR
 - [ ] Create and push *both* tags to GitHub (`vX.Y.Z`, `X.Y.Z`)
-  - Pull latest branch locally and run `contrib/release/tag-release.sh`
+  - Pull latest branch locally and run `contrib/release/tag-release.sh`.
+  - For RCs, this means checking out the commit before the revert and running
+    `tag-release.sh` against that commit.
 - [ ] Ask a maintainer to approve the build in the following links (keep the URL
       of the GitHub run to be used later):
   - [Cilium Image Release builds](https://github.com/cilium/cilium/actions?query=workflow:%22Image+Release+Build%22)
   - Check if all docker images are available before announcing the release
     `make -C install/kubernetes/ check-docker-images`
 - [ ] Get the image digests from the build process and make a commit and PR with
       these digests.
+  - For RCs on the master branch, skip this step.
   - [ ] Run `contrib/release/post-release.sh` to fetch the image
         digests and submit a PR to update these, use the URL of the GitHub run here.
   - [ ] Merge PR
@@ -70,8 +77,10 @@ assignees: ''
           [docsearch-scraper-webhook].
 - [ ] Check draft release from [releases] page
   - [ ] Update the text at the top with 2-3 highlights of the release
-  - [ ] Copy the text from `digest-vX.Y.Z.txt` (previously generated with
-        `contrib/release/post-release.sh`) to the end of the release.
+  - [ ] Copy the text from `digest-vX.Y.Z.txt` to the end of the release text.
+        This text was previously generated with
+        `contrib/release/post-release.sh`, or is otherwise available in the
+        GitHub workflow run that built the images.
   - [ ] Publish the release
 - [ ] Announce the release in #general on Slack (only [@]channel for vX.Y.0)
 - [ ] Update Grafana dashboards (only for vX.Y.0)

Documentation/contributing/release/rc.rst

@@ -18,12 +18,32 @@ If you intent to release a new feature release, see the
           used in the Cilium development process. See :ref:`dev_env` for
           detailed instructions about setting up said VM.
 
+GitHub template process
+~~~~~~~~~~~~~~~~~~~~~~~
+
+#. File a `new release issue <https://github.com/cilium/cilium/issues/new?assignees=&labels=kind%2Frelease&template=release_template.md&title=vX.Y.Z+release>`_
+   on GitHub, updating the title to reflect the version that will be released.
+
+#. Follow the steps in the issue template to prepare the release.
+
+Reference steps for the template
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note::
+
+   These instructions will generate a CHANGELOG based on the time when the
+   ``start-release.sh`` script is run, so you should aim to coordinate with
+   the core maintainers to complete these steps up until merging the PR
+   before merging additional PRs from contributors.
+
 #. Ensure that the necessary features and fixes have been completed and merged
    into the branch for which the release candidate will happen.
 
    #. Update GitHub project and create ``vX.Y.Z-rcW+1`` project if applicable.
    #. Update PRs / issues that were added to the ``vX.Y.Z-rcW`` project, but didn't
       make it into this release into the ``vX.Y.Z-rcW+1`` project.
+   #. The `Cilium Release Tool <https://github.com/cilium/release>`__ tool can
+      help to manage these for you.
 
 #. Checkout the desired stable branch (can be master branch if stable branch was
    not created) and pull it:
@@ -32,18 +52,11 @@ If you intent to release a new feature release, see the
 
        git checkout v1.0; git pull
 
-#. Create a branch for the release pull request:
+#. Run the release preparation script:
 
    .. code-block:: shell-session
 
-       git checkout -b pr/prepare-v1.0.3
-
-#. Update the ``AUTHORS file``
-
-   .. code-block:: shell-session
-
-       make update-authors
-
+       contrib/release/start-release.sh
 
    .. note::
 
@@ -70,27 +83,21 @@ If you intent to release a new feature release, see the
    there were changes to the CRD, then ensure it is incremented at most by 1
    patch version.
 
-#. If there is a branch for this RC, prepare the changelog now.
-   If this is the first RC, ``X.Y.Z`` should be the last, stable version,
-   otherwise ``X.Y.Z`` should point to the last released RC ``contrib/release/prep-changelog.sh X.Y.Z X.Y.0-rcX``.
+#. If this release will be based on the ``master`` branch rather than a stable
+   branch, add the AUTHORS modifications to a new commit.
 
-#. Add all modified files using ``git add`` and create a commit with the title
-   ``Prepare for release vX.Y.Z-rcW+1``.
+#. Add all other modified files using ``git add`` and create a commit with the
+   title ``Prepare for release vX.Y.Z-rcW+1``.
 
-#. If a branch for ``vX.Y`` doesn't exist yet, we need to modify the VERSION
-   file temporarily so that ``--version`` returns the right RC version.
+#. Prepare a pull request for the changes:
 
-   Change the file ``VERSION`` with ``vX.Y.Z-rcW+1``.
+   .. code-block:: shell-session
 
-#. Run ``make -C install/kubernetes all USE_DIGESTS=false``
+      contrib/release/submit-release.sh
 
-#. If there is not branch for this RC, prepare the changelog now
-   (as it was previously skipped).
-   If this is the first RC, ``X.Y.Z`` should be the last, stable version,
-   otherwise ``X.Y.Z`` should point to the last released RC ``contrib/release/prep-changelog.sh X.Y.Z X.Y.0-rcX``.
+#. Ensure that the CI smoke tests and reviews are in for the pull request.
 
-#. Add all modified files using ``git add`` and create a pull request with the
-   title ``Create helm chart release vX.Y.Z-rcW+1``.
+#. Revert the top commit on the branch and push the branch again to GitHub.
 
 #. Follow standard procedures to get the aforementioned PR merged into the
    desired stable branch. See :ref:`submit_pr` for more information about this
@@ -102,12 +109,11 @@ If you intent to release a new feature release, see the
 
        git checkout v1.0; git pull
 
-#. Create release tags:
+#. Check out the "Prepare for release" commit and create release tags:
 
    .. code-block:: shell-session
 
-       git tag -a v1.0.3 -m 'Release v1.0.3'
-       git tag -a 1.0.3 -m 'Release 1.0.3'
+      git checkout NNNN && contrib/release/tag-release.sh
 
    .. note::
 
@@ -117,64 +123,30 @@ If you intent to release a new feature release, see the
        ``x.y.z`` For more information about how ReadTheDocs does versioning, you can
        read their `Versions Documentation <https://docs.readthedocs.io/en/latest/versions.html>`_.
 
-#. Push the git release tag
-
-   .. code-block:: shell-session
-
-       git push --tags
-
-
-#. `Create a GitHub release <https://github.com/cilium/cilium/releases/new>`_:
+#. Approve the release from the `Release Image build UI <https://github.com/cilium/cilium/actions?query=workflow:%22Image+Release+Build%22>`_.
 
-   #. Choose the correct target branch, e.g. ``v1.0``
-   #. Choose the correct target tag, e.g. ``v1.0.3``
-   #. Title: ``1.0.3``
-   #. Check the ``This is a pre-release`` box if you are releasing a release
-      candidate.
-   #. Fill in the release description:
+#. Once the release images are pushed, fetch the digests from the workflow.
 
-      ::
+#. Prepare Helm changes for the release using the `Cilium Helm Charts Repository <https://github.com/cilium/charts/>`__
+   and push the changes into that repository (not the main cilium repository):
 
-           Summary of Changes
-           ------------------
-
-           **Important Bug Fixes**
-
-           * Fix dropped packets upon agent bootstrap when iptables rules are installed (@ianvernon)
+   .. code-block:: shell-session
 
-           **Enhancements**
+      ./prepare_artifacts.sh /path/to/cilium/repository/checked/out/to/release/commit
+      git push
 
-           **Documentation**
+#. Wait for the `Cilium Helm Charts Workflow <https://github.com/cilium/charts/actions>`__
+   to successfully deploy a cluster using the new Helm charts.
 
-           Changes
-           -------
+#. `Publish a GitHub release <https://github.com/cilium/cilium/releases/>`_:
 
-           ```
-           << contents of NEWS.rst for this release >>
-           ```
+   Following the steps above, the release draft will already be prepared.
 
+   #. Check the ``This is a pre-release`` box.
+   #. Copy the official docker manifests for the release from the previous step
+      into the end of the Github release announcement.
    #. Preview the description and then publish the release
 
-#. Get the digests for the ``vX.Y.Z-rcN`` and make a commit to the helm charts
-   repository to include those digests.
-
-#. Follow standard procedures to get the aforementioned PR merged into the
-   desired stable branch. See :ref:`submit_pr` for more information about this
-   process.
-
-#. Checkout out the stable branch and pull your merged changes:
-
-   .. code-block:: shell-session
-
-       git checkout v1.0; git pull
-
-#. Publish the helm charts for this RC.
-
-#. **If there isn't a stable branch available** we need to revert the changes
-   made in the commit "Create helm chart release vX.Y.Z-rcW+1" as the master
-   should not point to this RC. Make a commit reverting the changes and push
-   those changes as a PR to be merged into master.
-
 #. Announce the release in the ``#general`` channel on Slack. Sample text:
 
    ::

Documentation/contributing/release/stable.rst

@@ -108,14 +108,6 @@ Reference steps for the template
    This will leave a file with the format ``digest-vX.Y.Z.txt`` in the local
    directory which can be used to prepare the release in the next step.
 
-#. `Publish a GitHub release <https://github.com/cilium/cilium/releases/>`_:
-
-   Following the steps above, the release draft will already be prepared.
-   Preview the description and then publish the release.
-
-   #. Copy the official docker manifests for the release from the previous step
-      into the end of the Github release announcement.
-
 #. Prepare Helm changes for the release using the `Cilium Helm Charts Repository <https://github.com/cilium/charts/>`__
    and push the changes into that repository (not the main cilium repository):
 
@@ -143,6 +135,17 @@ Reference steps for the template
    After pushing you can revert all the changes made in the local branch
    ``x.y-dev`` from ``cilium/cilium``.
 
+#. Wait for the `Cilium Helm Charts Workflow <https://github.com/cilium/charts/actions>`__
+   to successfully deploy a cluster using the new Helm charts.
+
+#. `Publish a GitHub release <https://github.com/cilium/cilium/releases/>`_:
+
+   Following the steps above, the release draft will already be prepared.
+   Preview the description and then publish the release.
+
+   #. Copy the official docker manifests for the release from the previous step
+      into the end of the Github release announcement.
+
 #. Announce the release in the ``#general`` channel on Slack. Sample text:
 
    ::

contrib/backporting/common.sh

@@ -16,6 +16,8 @@
 
 set -e
 
+RELEASE_REGEX="[0-9]\+\.[0-9]\+\.[0-9]\+\(-\(\(rc\)\|\(snapshot\)\)\(\.\)\?[0-9]\+\)\?$"
+
 get_remote () {
   local remote
   local org=${1:-cilium}
@@ -61,3 +63,12 @@ commit_in_upstream() {
     local branches="$(git branch -q -r --contains $commit $remote/$branch 2> /dev/null)"
     echo "$branches" | grep -q ".*$remote/$branch"
 }
+
+get_branch_from_version() {
+    local remote="$1"
+    local branch="$(echo $2 | sed 's/.*\(v[0-9]\+\.[0-9]\+\).*/\1/')"
+    if [ -z "$(git ls-remote --heads $remote $branch)" ]; then
+        branch="master"
+    fi
+    echo "$branch"
+}

contrib/release/prep-changelog.sh

@@ -29,12 +29,12 @@ handle_args() {
         common::exit 0
     fi
 
-    if ! echo "$1" | grep -q "[0-9]\+\.[0-9]\+\.[0-9]\+"; then
+    if ! echo "$1" | grep -q "$RELEASE_REGEX"; then
         usage 2>&1
-        common::exit 1 "Invalid OLD-VERSION ARG \"$1\"; Expected X.Y.Z"
+        common::exit 1 "Invalid OLD-VERSION ARG \"$1\"; Expected X.Y.Z[-rcW]"
     fi
 
-    if ! echo "$2" | grep -q "[0-9]\+\.[0-9]\+\.[0-9]\+[-rc0-9]*"; then
+    if ! echo "$2" | grep -q "$RELEASE_REGEX"; then
         usage 2>&1
         common::exit 1 "Invalid NEW-VERSION ARG \"$1\"; Expected X.Y.Z[-rcW]"
     fi

contrib/release/start-release.sh

@@ -6,6 +6,7 @@ DIR=$(dirname $(readlink -ne $BASH_SOURCE))
 source $DIR/lib/common.sh
 source $DIR/../backporting/common.sh
 
+VERSION_GLOB='v[0-9]*\.[0-9]*\.[0-9]*'
 PROJECTS_REGEX='s/.*projects\/\([0-9]\+\).*/\1/'
 ACTS_YAML=".github/maintainers-little-helper.yaml"
 REMOTE="$(get_remote)"
@@ -18,6 +19,18 @@ usage() {
     logecho "--help     Print this help message"
 }
 
+# $1 - VERSION
+version_is_prerelease() {
+    case "$1" in
+        *rc*|*snapshot*)
+            return 0
+            ;;
+        *)
+            return 1
+            ;;
+    esac
+}
+
 handle_args() {
     if ! common::argc_validate 2; then
         usage 2>&1
@@ -29,12 +42,12 @@ handle_args() {
         common::exit 0
     fi
 
-    if ! echo "$1" | grep -q "[0-9]\+\.[0-9]\+\.[0-9]\+"; then
+    if ! echo "$1" | grep -q "$RELEASE_REGEX"; then
         usage 2>&1
         common::exit 1 "Invalid VERSION ARG \"$1\"; Expected X.Y.Z"
     fi
 
-    if ! echo "$2" | grep -q "^[0-9]\+"; then
+    if ! echo "$2" | grep -q "^[0-9]\+" && ! version_is_prerelease "$1"; then
         usage 2>&1
         common::exit 1 "Invalid GH-PROJECT ID argument. Expected [0-9]+"
     fi
@@ -54,25 +67,38 @@ main() {
 
     local ersion="$(echo $1 | sed 's/^v//')"
     local version="v$ersion"
-    local branch="v$(echo $ersion | sed 's/[^0-9]*\([0-9]\+\.[0-9]\+\).*/\1/')"
+    local branch="$(get_branch_from_version $REMOTE $version)"
     local new_proj="$2"
-
-    git fetch $REMOTE
-    git checkout -b pr/prepare-$version $REMOTE/$branch
-    local old_version="$(cat VERSION)"
+    local old_version=""
+
+    git fetch -q $REMOTE
+    if [ "$branch" = "master" ]; then
+        git checkout -b pr/prepare-$version $REMOTE/$branch
+        if echo "$version" | grep -q 'rc'; then
+            old_version="$(git tag -l "$VERSION_GLOB" | grep -v 'snapshot' | sort -V | tail -n 1)"
+        else
+            old_version="$(git tag -l "$VERSION_GLOB" | sort -V | tail -n 1)"
+        fi
+    else
+        git checkout -b pr/prepare-$version $REMOTE/$branch
+        old_version="$(cat VERSION)"
+    fi
 
     logecho "Updating VERSION, AUTHORS.md, $ACTS_YAML, helm templates"
     echo $ersion > VERSION
     sed -i 's/"[^"]*"/""/g' install/kubernetes/Makefile.digests
     logrun make -C install/kubernetes all USE_DIGESTS=false
     logrun make update-authors
-    old_proj=$(grep "projects" $ACTS_YAML | sed "$PROJECTS_REGEX")
-    sed -i 's/\(projects\/\)[0-9]\+/\1'$new_proj'/g' $ACTS_YAML
+    if ! version_is_prerelease "$version"; then
+        old_proj=$(grep "projects" $ACTS_YAML | sed "$PROJECTS_REGEX")
+        sed -i 's/\(projects\/\)[0-9]\+/\1'$new_proj'/g' $ACTS_YAML
+    fi
 
     $DIR/prep-changelog.sh "$old_version" "$version"
 
     logecho "Next steps:"
     logecho "* Check all changes and add to a new commit"
+    logecho "  * If this is a prerelease, create a revert commit"
     logecho "* Push the PR to Github for review ('submit-release.sh')"
     logecho "* Close https://github.com/cilium/cilium/projects/$old_proj"
     logecho "* (After PR merge) Use 'tag-release.sh' to prepare tags/release"