From 55ccd273de31fc3236b65f1cfa229a19fac31e07 Mon Sep 17 00:00:00 2001 From: Pablo Date: Thu, 19 Apr 2018 16:21:42 -0700 Subject: [PATCH 01/13] Updating release guide to use Gradle commands. --- src/contribute/release-guide.md | 55 +++++++++++++++++---------------- 1 file changed, 29 insertions(+), 26 deletions(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index e48981c5126..e166fdf80fc 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -189,30 +189,35 @@ Check out the version of the codebase from which you start the release. For a ne Set up a few environment variables to simplify Maven commands that follow. (We use `bash` Unix syntax in this guide.) - VERSION="1.2.3" - NEXT_VERSION="1.2.4" - BRANCH_NAME="release-${VERSION}" - DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT" + RELEASE_VERSION=2.5.0 + NEXT_VERSION_IN_BASE_BRANCH=2.6.0 + BRANCH=release-${RELEASE} Version represents the release currently underway, while next version specifies the anticipated next version to be released from that branch. Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4. -Use Maven release plugin to create the release branch and update the current branch to use the new development version. This command applies for the new major or minor version. (Warning: this command automatically pushes changes to the code repository.) +**NOTE**: Only if you are doing an incremental/hotfix releasei (e.g. 2.5.1), please check out the previous release tag, before running the following instructions: - mvn release:branch \ - -DbranchName=${BRANCH_NAME} \ - -DdevelopmentVersion=${DEVELOPMENT_VERSION} + BASE_RELEASE=2.5.0 + RELEASE=2.5.1 + NEXT_VERSION_IN_BASE_BRANCH=2.5.0 + git checkout tags/${BASE_RELEASE} -However, if you are doing an incremental/hotfix release, please run the following command after checking out the release tag of the release being patched. +Create a new branch, and update version files in the master branch. - mvn release:branch \ - -DbranchName=${BRANCH_NAME} \ - -DupdateWorkingCopyVersions=false \ - -DupdateBranchVersions=true \ - -DreleaseVersion="${VERSION}-SNAPSHOT" + git branch ${BRANCH} + + # Now change the version in existing gradle files, and Python files + sed -i 's/'$RELEASE'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g' build_rules.gradle + sed -i 's/'$RELEASE'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g' gradle.properties + sed -i 's/'$RELEASE'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g' sdks/python/apache_beam/version.py + + # Save changes in master branch + git add gradle.properties build_rules.gradle + git commit -m "Moving to ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master branch." Check out the release branch. - git checkout ${BRANCH_NAME} + git checkout ${BRANCH} The rest of this guide assumes that commands are run in the root of a repository on `${BRANCH_NAME}` with the above environment variables set. @@ -258,22 +263,20 @@ The core of the release process is the build-vote-fix cycle. Each cycle produces ### Build and stage Java artifacts with Maven -Set up a few environment variables to simplify Maven commands that follow. This identifies the release candidate being built. Start with `RC_NUM` equal to `1` and increment it for each candidate. +Set up a few environment variables to simplify the commands that follow. These identify the release candidate being built, and the branch where you will stage files. Start with `RC_NUM` equal to `1` and increment it for each candidate. + + RC_NUM=1 - RC_NUM="1" - RC_TAG="v${VERSION}-RC${RC_NUM}" -Use Maven release plugin to build the release artifacts, as follows: +Use Gradle release plugin to build the release artifacts, as follows: - mvn release:prepare \ - -Dresume=false \ - -DreleaseVersion=${VERSION} \ - -Dtag=${RC_TAG} \ - -DupdateWorkingCopyVersions=false + ./gradlew release -Prelease.newVersion=${RELEASE_VERSION}-SNAPSHOT \ + -Prelease.releaseVersion=${RELEASE_VERSION}-RC${RC_NUM} \ + -Prelease.useAutomaticVersion=true --info --no-daemon -Use Maven release plugin to stage these artifacts on the Apache Nexus repository, as follows: +Use Gradle publish plugin to stage these artifacts on the Apache Nexus repository, as follows: - mvn release:perform + ./gradlew publish -Prelease Review all staged artifacts. They should contain all relevant parts for each module, including `pom.xml`, jar, test jar, source, test source, javadoc, etc. Artifact names should follow [the existing format](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22) in which artifact name mirrors directory structure, e.g., `beam-sdks-java-io-kafka`. Carefully review any new artifacts. From ec3eb5ad454cb269825b4b31b6d2ddf8499a861f Mon Sep 17 00:00:00 2001 From: Pablo Date: Fri, 20 Apr 2018 10:50:56 -0700 Subject: [PATCH 02/13] Addressing comments. Improving commands. --- src/contribute/release-guide.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index e166fdf80fc..de8c9a91dd3 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -180,12 +180,13 @@ Javadoc to the Javadoc for other modules that Beam depends on. ### Create a release branch -Release candidates are built from a release branch. As a final step in preparation for the release, you should create the release branch, push it to the code repository, and update version information on the original branch. +Release candidates are built from a release branch. As a final step in preparation for the release, you should create the release branch, push it to the Apache code repository, and update version information on the original branch. Check out the version of the codebase from which you start the release. For a new minor or major release, this may be `HEAD` of the `master` branch. To build a hotfix/incremental release, instead of the `master` branch, use the release tag of the release being patched. (Please make sure your cloned repository is up-to-date before starting.) git checkout +**NOTE**: If you are doing an incremental/hotfix release (e.g. 2.5.1), please check out the previous release tag, rather than the master branch. Set up a few environment variables to simplify Maven commands that follow. (We use `bash` Unix syntax in this guide.) @@ -195,7 +196,7 @@ Set up a few environment variables to simplify Maven commands that follow. (We u Version represents the release currently underway, while next version specifies the anticipated next version to be released from that branch. Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4. -**NOTE**: Only if you are doing an incremental/hotfix releasei (e.g. 2.5.1), please check out the previous release tag, before running the following instructions: +**NOTE**: Only if you are doing an incremental/hotfix release (e.g. 2.5.1), please check out the previous release tag, before running the following instructions: BASE_RELEASE=2.5.0 RELEASE=2.5.1 @@ -207,9 +208,9 @@ Create a new branch, and update version files in the master branch. git branch ${BRANCH} # Now change the version in existing gradle files, and Python files - sed -i 's/'$RELEASE'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g' build_rules.gradle - sed -i 's/'$RELEASE'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g' gradle.properties - sed -i 's/'$RELEASE'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g' sdks/python/apache_beam/version.py + sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" build_rules.gradle + sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" gradle.properties + sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" sdks/python/apache_beam/version.py # Save changes in master branch git add gradle.properties build_rules.gradle From 68ad3d68704e8256e25e93cbf5d656e8e32e9883 Mon Sep 17 00:00:00 2001 From: Pablo Date: Fri, 20 Apr 2018 11:13:19 -0700 Subject: [PATCH 03/13] Addressing comments. Improving commands. --- src/contribute/release-guide.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index de8c9a91dd3..ad4c7fc0d1b 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -190,7 +190,7 @@ Check out the version of the codebase from which you start the release. For a ne Set up a few environment variables to simplify Maven commands that follow. (We use `bash` Unix syntax in this guide.) - RELEASE_VERSION=2.5.0 + RELEASE=2.5.0 NEXT_VERSION_IN_BASE_BRANCH=2.6.0 BRANCH=release-${RELEASE} @@ -271,8 +271,8 @@ Set up a few environment variables to simplify the commands that follow. These i Use Gradle release plugin to build the release artifacts, as follows: - ./gradlew release -Prelease.newVersion=${RELEASE_VERSION}-SNAPSHOT \ - -Prelease.releaseVersion=${RELEASE_VERSION}-RC${RC_NUM} \ + ./gradlew release -Prelease.newVersion=${RELEASE}-SNAPSHOT \ + -Prelease.releaseVersion=${RELEASE}-RC${RC_NUM} \ -Prelease.useAutomaticVersion=true --info --no-daemon Use Gradle publish plugin to stage these artifacts on the Apache Nexus repository, as follows: From 0ac546bd235bf01b4c33ad190f23e1d9d04c4df2 Mon Sep 17 00:00:00 2001 From: Scott Wegner Date: Tue, 1 May 2018 14:36:52 -0700 Subject: [PATCH 04/13] Update release guide for publishing source zip from GitHub --- src/contribute/release-guide.md | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index ad4c7fc0d1b..ba3b4bf9111 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -279,7 +279,7 @@ Use Gradle publish plugin to stage these artifacts on the Apache Nexus repositor ./gradlew publish -Prelease -Review all staged artifacts. They should contain all relevant parts for each module, including `pom.xml`, jar, test jar, source, test source, javadoc, etc. Artifact names should follow [the existing format](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22) in which artifact name mirrors directory structure, e.g., `beam-sdks-java-io-kafka`. Carefully review any new artifacts. +Review all staged artifacts. They should contain all relevant parts for each module, including `pom.xml`, jar, test jar, javadoc, etc. Artifact names should follow [the existing format](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22) in which artifact name mirrors directory structure, e.g., `beam-sdks-java-io-kafka`. Carefully review any new artifacts. Close the staging repository on Apache Nexus. When prompted for a description, enter “Apache Beam, version X, release candidate Y”. @@ -293,24 +293,22 @@ Copy the source release to the dev repository of `dist.apache.org`. 1. Make a directory for the new release: - mkdir beam/${VERSION} - cd beam/${VERSION} + mkdir beam/${RELEASE} + cd beam/${RELEASE} -1. Copy and rename the Beam source distribution, hashes, and GPG signature: +1. Download source zip from GitHub: - cp ${BEAM_ROOT}/target/apache-beam-${VERSION}-source-release.zip . - cp ${BEAM_ROOT}/target/apache-beam-${VERSION}-source-release.zip.asc . - cp ${BEAM_ROOT}/sdks/python/target/apache-beam-${VERSION}.zip apache-beam-${VERSION}-python.zip + wget https://github.com/apache/beam/archive/release-${RELEASE}.zip \ + -O apache-beam-${RELEASE}-source-release.zip -1. Create hashes for source files and sign the python source file file +1. Create hashes and sign the source distribution: - sha512sum apache-beam-${VERSION}-source-release.zip > apache-beam-${VERSION}-source-release.zip.sha512 - gpg --armor --detach-sig apache-beam-${VERSION}-python.zip - sha512sum apache-beam-${VERSION}-python.zip > apache-beam-${VERSION}-python.zip.sha512 + gpg --armor --detach-sig apache-beam-${RELEASE}-source-release.zip + sha512sum apache-beam-${RELEASE}-source-release.zip > apache-beam-${RELEASE}-source-release.zip.sha512 1. Add and commit all the files. - svn add beam/${VERSION} + svn add beam/${RELEASE} svn commit 1. Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam). @@ -338,14 +336,14 @@ candidate into the source tree of the website. Add the new Javadoc to [SDK API Reference page]({{ site.baseurl }}/documentation/sdks/javadoc/) page, as follows: * Unpack the Maven artifact `org.apache.beam:beam-sdks-java-javadoc` into some temporary location. Call this `${JAVADOC_TMP}`. -* Copy the generated Javadoc into the website repository: `cp -r ${JAVADOC_TMP} src/documentation/sdks/javadoc/${VERSION}`. +* Copy the generated Javadoc into the website repository: `cp -r ${JAVADOC_TMP} src/documentation/sdks/javadoc/${RELEASE}`. * Set up the necessary git commands to account for the new and deleted files from the javadoc. * Update the Javadoc link on this page to point to the new version (in `src/documentation/sdks/javadoc/current.md`). #### Create Pydoc Add the new Pydoc to [SDK API Reference page]({{ site.baseurl }}/documentation/sdks/pydoc/) page, as follows: -* Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} src/documentation/sdks/pydoc/${VERSION}`. +* Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} src/documentation/sdks/pydoc/${RELEASE}`. * Remove `.doctrees` directory. * Update the Pydoc link on this page to point to the new version (in `src/documentation/sdks/pydoc/current.md`). @@ -454,9 +452,9 @@ Use the Apache Nexus repository to release the staged binary artifacts to the Ma ### Deploy Python artifacts to PyPI 1. Create a new release and upload the Python zip file for the new release using the [PyPI UI] (https://pypi.python.org/pypi/apache-beam) -1. Alternatively, use the command line tool to upload the new release `twine upload apache-beam-${VERSION}.zip` +1. Alternatively, use the command line tool to upload the new release `twine upload apache-beam-${RELEASE}.zip` -Note: It is important to rename `apache-beam-${VERSION}-python.zip` to `apache-beam-${VERSION}.zip` before uploading, because PyPI expects a filename in the `-` format. +Note: It is important to rename `apache-beam-${RELEASE}-python.zip` to `apache-beam-${RELEASE}.zip` before uploading, because PyPI expects a filename in the `-` format. #### Deploy source release to dist.apache.org @@ -466,7 +464,7 @@ Copy the source release from the `dev` repository to the `release` repository at Create and push a new signed tag for the released version by copying the tag for the final release candidate, as follows: - VERSION_TAG="v${VERSION}" + VERSION_TAG="v${RELEASE}" git tag -s "$VERSION_TAG" "$RC_TAG" git push github "$VERSION_TAG" From 372c93ecbafbf3a1440492df1e12050dfe939e91 Mon Sep 17 00:00:00 2001 From: Pablo Date: Tue, 1 May 2018 15:08:48 -0700 Subject: [PATCH 05/13] Addressing TODOs --- src/contribute/release-guide.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index ba3b4bf9111..e1580362c8a 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -158,12 +158,11 @@ Adjust any of the above properties to the improve clarity and presentation of th ### Verify that a Release Build Works -Run `mvn -Prelease clean install` to ensure that the build processes that are specific to that -profile are in good shape. +Run `./gradlew -PisRelease clean build` to ensure that the build processes that are specific to that profile are in good shape. ### Update and Verify Javadoc -The build with `-Prelease` creates the combined Javadoc for the release in `sdks/java/javadoc`. +The build with `-PisRelease` creates the combined Javadoc for the release in `sdks/java/javadoc`. The file `sdks/java/javadoc/ant.xml` file contains a list of modules to include in and exclude, plus a list of offline URLs that populate links from Beam's @@ -209,11 +208,11 @@ Create a new branch, and update version files in the master branch. # Now change the version in existing gradle files, and Python files sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" build_rules.gradle - sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" gradle.properties - sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" sdks/python/apache_beam/version.py + sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" gradle.properties + sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" sdks/python/apache_beam/version.py # Save changes in master branch - git add gradle.properties build_rules.gradle + git add gradle.properties build_rules.gradle sdks/python/apache_beam/version.py git commit -m "Moving to ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master branch." Check out the release branch. @@ -224,8 +223,6 @@ The rest of this guide assumes that commands are run in the root of a repository ### Update the Python SDK version -In the master branch, update Python SDK [version](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/version.py) identifier to the next development version (e.g. `1.2.3.dev` to `1.3.0.dev`). - In the release branch, update the Python SDK version to the release version (e.g. `1.2.3.dev` to `1.2.3`). ### Update release specific configurations @@ -269,7 +266,8 @@ Set up a few environment variables to simplify the commands that follow. These i RC_NUM=1 -Use Gradle release plugin to build the release artifacts, as follows: +Use Gradle release plugin to build the release artifacts, and push code and +release tag to the origin repository (this would be the Apache Beam repo): ./gradlew release -Prelease.newVersion=${RELEASE}-SNAPSHOT \ -Prelease.releaseVersion=${RELEASE}-RC${RC_NUM} \ @@ -277,7 +275,7 @@ Use Gradle release plugin to build the release artifacts, as follows: Use Gradle publish plugin to stage these artifacts on the Apache Nexus repository, as follows: - ./gradlew publish -Prelease + ./gradlew publish -PisRelease Review all staged artifacts. They should contain all relevant parts for each module, including `pom.xml`, jar, test jar, javadoc, etc. Artifact names should follow [the existing format](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22) in which artifact name mirrors directory structure, e.g., `beam-sdks-java-io-kafka`. Carefully review any new artifacts. From 20a0a5aa79bc1fe6db73cd9b78a6e37ca581d4d9 Mon Sep 17 00:00:00 2001 From: Pablo Date: Fri, 18 May 2018 09:52:56 -0700 Subject: [PATCH 06/13] Address comments. --- src/contribute/release-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index e1580362c8a..c5648b2405e 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -158,7 +158,7 @@ Adjust any of the above properties to the improve clarity and presentation of th ### Verify that a Release Build Works -Run `./gradlew -PisRelease clean build` to ensure that the build processes that are specific to that profile are in good shape. +From a new workspace, run `git clean -fdx` to clean, and then `./gradlew -PisRelease build` to ensure that the build processes necessary for release are in good shape. ### Update and Verify Javadoc From cec43437ac7faeee86ff8bd738f3a145e597fb44 Mon Sep 17 00:00:00 2001 From: Pablo Date: Fri, 18 May 2018 10:02:35 -0700 Subject: [PATCH 07/13] Improving. --- src/contribute/release-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index c5648b2405e..1b9b7a549f9 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -158,7 +158,7 @@ Adjust any of the above properties to the improve clarity and presentation of th ### Verify that a Release Build Works -From a new workspace, run `git clean -fdx` to clean, and then `./gradlew -PisRelease build` to ensure that the build processes necessary for release are in good shape. +From a new workspace, run `git clean -fdx` to clean, and then `./gradlew -PisRelease build --no-parallel` to ensure that the build processes necessary for release are in good shape. ### Update and Verify Javadoc From 9eab89ca99ec4375e18b70a6b39598e417fb7ae5 Mon Sep 17 00:00:00 2001 From: Boyuan Zhang Date: Fri, 1 Jun 2018 17:13:29 -0700 Subject: [PATCH 08/13] Add more updates --- src/contribute/release-guide.md | 89 ++++++++++++++++++++++++++++++--- 1 file changed, 81 insertions(+), 8 deletions(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index 1b9b7a549f9..b1618fbdf94 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -63,6 +63,15 @@ To prepare for each release, you should audit the project status in the JIRA iss You need to have a GPG key to sign the release artifacts. Please be aware of the ASF-wide [release signing guidelines](https://www.apache.org/dev/release-signing.html). If you don’t have a GPG key associated with your Apache account, please create one according to the guidelines. +Get more entropy for creating a GPG key + + sudo apt-get install rng-tools + sudo rngd -r /dev/urandom + +Create a GPG key + + gpg --full-generate-key + Determine your Apache GPG Key and Key ID, as follows: gpg --list-keys @@ -128,6 +137,8 @@ Release manager needs to have an account with PyPI. If you need one, [register a When contributors resolve an issue in JIRA, they are tagging it with a release that will contain their changes. With the release currently underway, new issues should be resolved against a subsequent future release. Therefore, you should create a release item for this subsequent release, as follows: +Attention: Only committer has permission to perform this. + 1. In JIRA, navigate to the [`Beam > Administration > Versions`](https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions). 1. Add a new release: choose the next minor version number compared to the one currently underway, select today’s date as the `Start Date`, and choose `Add`. @@ -158,7 +169,52 @@ Adjust any of the above properties to the improve clarity and presentation of th ### Verify that a Release Build Works -From a new workspace, run `git clean -fdx` to clean, and then `./gradlew -PisRelease build --no-parallel` to ensure that the build processes necessary for release are in good shape. +Pre-installation for python build +* Install pip + + ``` + curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py + python get-pip.py + ``` +* Install virtualenv + + ``` + pip install --upgrade virtualenv + ``` +* Cython + + ``` + sudo pip install cython + sudo apt-get install gcc + sudo apt-get install python-dev + ``` +Make sure your ```time``` alias to ```/usr/bin/time```, if not: + +``` +sudo apt-get install time +alias time='/usr/bin/time' +``` + +Run gradle release build + +* Clean current workspace + + ``` + git clean -fdx + ./gradlew clean + ``` + +* Unlock the secret key + ``` + gpg --output ~/doc.sig --sign ~/.bashrc + ``` + +* Run build command + ``` + ./gradlew build -PisRelease --no-parallel --scan --stacktrace + ``` + +From a new workspace, run `git clean -fdx` and `./gradlew clean` to clean, and then `./gradlew -PisRelease build --no-parallel` to ensure that the build processes necessary for release are in good shape. ### Update and Verify Javadoc @@ -177,7 +233,9 @@ Javadoc to the Javadoc for other modules that Beam depends on. the version number has changed, download a new version of the corresponding `-docs/package-list` file. -### Create a release branch +### Create a release branch in apache/beam repository + +Attention: Only committer has permission to create release branch in apache/beam. Release candidates are built from a release branch. As a final step in preparation for the release, you should create the release branch, push it to the Apache code repository, and update version information on the original branch. @@ -223,22 +281,26 @@ The rest of this guide assumes that commands are run in the root of a repository ### Update the Python SDK version -In the release branch, update the Python SDK version to the release version (e.g. `1.2.3.dev` to `1.2.3`). +Update [sdks/python/apache_beam/version.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/version.py) in both master branch and release branch. + +* In the master branch, change the Python SDK version to the next dev version(e.g, `2.5.0-dev` to `2.6.0-dev`). + +* In the release branch, update the Python SDK version to the release version(e.g. `2.5.0-dev` to `2.5.0`). ### Update release specific configurations -1. Update archetypes: - [example](https://github.com/apache/beam/commit/d375cfa126fd7be9eeeec34f39c2b9b856f324bf) -2. Update runner specific configurations: +Update runner specific configurations: [example](https://github.com/apache/beam/commit/f572328ce23e70adee8001e3d10f1479bd9a380d) ### Start a snapshot build -Start a build of [the nightly snapshot](https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_NightlySnapshot/). +Start a build of [the nightly snapshot](https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_NightlySnapshot/) against master branch. Some processes, including our archetype tests, rely on having a live SNAPSHOT of the current version from the `master` branch. Once the release branch is cut, these SNAPSHOT versions are no longer found, so builds will be broken until a new snapshot is available. +Comment ```Run Gradle Publish``` in one pull request to trigger build. + ### Checklist to proceed to the next step 1. Release Manager’s GPG key is published to `dist.apache.org` @@ -259,12 +321,15 @@ so builds will be broken until a new snapshot is available. The core of the release process is the build-vote-fix cycle. Each cycle produces one release candidate. The Release Manager repeats this cycle until the community approves one release candidate, which is then finalized. -### Build and stage Java artifacts with Maven +### Build and stage Java artifacts with Gradle Set up a few environment variables to simplify the commands that follow. These identify the release candidate being built, and the branch where you will stage files. Start with `RC_NUM` equal to `1` and increment it for each candidate. RC_NUM=1 +Make sure your git config will maintain your account: + + git config credential.helper store Use Gradle release plugin to build the release artifacts, and push code and release tag to the origin repository (this would be the Apache Beam repo): @@ -283,6 +348,8 @@ Close the staging repository on Apache Nexus. When prompted for a description, e ### Stage source release on dist.apache.org +Attention: Only committer has permissions to perform following steps. + Copy the source release to the dev repository of `dist.apache.org`. 1. If you have not already, check out the Beam section of the `dev` repository on `dist.apache.org` via Subversion. In a fresh directory: @@ -313,6 +380,12 @@ Copy the source release to the dev repository of `dist.apache.org`. ### Build the Pydoc API reference +Make sure you have ```tox``` installed: + +``` +pip install tox + +``` Create the Python SDK documentation using sphinx by running a helper script. ``` cd sdks/python && tox -e docs From 9d0fd1c6967ee61209b2fadf86e99c6005ffd0ed Mon Sep 17 00:00:00 2001 From: Boyuan Zhang Date: Mon, 4 Jun 2018 13:04:28 -0700 Subject: [PATCH 09/13] More updates --- src/contribute/release-guide.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index b1618fbdf94..b18ef572e0a 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -214,8 +214,6 @@ Run gradle release build ./gradlew build -PisRelease --no-parallel --scan --stacktrace ``` -From a new workspace, run `git clean -fdx` and `./gradlew clean` to clean, and then `./gradlew -PisRelease build --no-parallel` to ensure that the build processes necessary for release are in good shape. - ### Update and Verify Javadoc The build with `-PisRelease` creates the combined Javadoc for the release in `sdks/java/javadoc`. @@ -279,18 +277,22 @@ Check out the release branch. The rest of this guide assumes that commands are run in the root of a repository on `${BRANCH_NAME}` with the above environment variables set. +### Update the beam version on master to the next dev version +* [beam/build_rules.gradle](https://github.com/apache/beam/blob/master/build_rules.gradle): update value of 'version'(e.g, '2.5.0' to '2.6.0') +* [beam/pom.xml](https://github.com/apache/beam/blob/master/pom.xml): update value of 'version' field(e.g, '2.5.0-SNAPSHOT' to '2.6.0-SNAPSHOT') + ### Update the Python SDK version Update [sdks/python/apache_beam/version.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/version.py) in both master branch and release branch. * In the master branch, change the Python SDK version to the next dev version(e.g, `2.5.0-dev` to `2.6.0-dev`). - * In the release branch, update the Python SDK version to the release version(e.g. `2.5.0-dev` to `2.5.0`). ### Update release specific configurations -Update runner specific configurations: - [example](https://github.com/apache/beam/commit/f572328ce23e70adee8001e3d10f1479bd9a380d) +Update Java runner specific configurations in release branch: +* [beam/runners/google-cloud-dataflow-java/build.gradle](https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/build.gradle): change value of 'dataflow.container_version' to 'beam-release_version_number'(e.g, 'beam-master-20180601' to 'beam-2.5.0') +* [beam/runners/google-cloud-dataflow-java/pom.xml](https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/pom.xml): change value of 'version' field (e.g, '2.5.0-SNAPSHOT' to '2.5.0') ### Start a snapshot build From d238c9ec3ba279188c33b098de0a3064fa8b4216 Mon Sep 17 00:00:00 2001 From: Boyuan Zhang Date: Tue, 5 Jun 2018 15:28:20 -0700 Subject: [PATCH 10/13] Addressed Alan's comments --- src/contribute/release-guide.md | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index b18ef572e0a..b564496e037 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -255,7 +255,7 @@ Version represents the release currently underway, while next version specifies BASE_RELEASE=2.5.0 RELEASE=2.5.1 - NEXT_VERSION_IN_BASE_BRANCH=2.5.0 + NEXT_VERSION_IN_BASE_BRANCH=2.6.0 git checkout tags/${BASE_RELEASE} Create a new branch, and update version files in the master branch. @@ -274,18 +274,13 @@ Create a new branch, and update version files in the master branch. Check out the release branch. git checkout ${BRANCH} - -The rest of this guide assumes that commands are run in the root of a repository on `${BRANCH_NAME}` with the above environment variables set. - -### Update the beam version on master to the next dev version -* [beam/build_rules.gradle](https://github.com/apache/beam/blob/master/build_rules.gradle): update value of 'version'(e.g, '2.5.0' to '2.6.0') -* [beam/pom.xml](https://github.com/apache/beam/blob/master/pom.xml): update value of 'version' field(e.g, '2.5.0-SNAPSHOT' to '2.6.0-SNAPSHOT') + +The rest of this guide assumes that commands are run in the root of a repository on `${BRANCH}` with the above environment variables set. ### Update the Python SDK version Update [sdks/python/apache_beam/version.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/version.py) in both master branch and release branch. -* In the master branch, change the Python SDK version to the next dev version(e.g, `2.5.0-dev` to `2.6.0-dev`). * In the release branch, update the Python SDK version to the release version(e.g. `2.5.0-dev` to `2.5.0`). ### Update release specific configurations From ab15c1bb19ac20c97652cca1c7fc085f6d1b6709 Mon Sep 17 00:00:00 2001 From: Boyuan Zhang Date: Wed, 6 Jun 2018 16:09:56 -0700 Subject: [PATCH 11/13] Add instructions about how to stage python binaries --- src/contribute/release-guide.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index b564496e037..35002e378e8 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -375,6 +375,28 @@ Copy the source release to the dev repository of `dist.apache.org`. 1. Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam). +### Stage python binaries on dist.apache.org + +Build python binaries in release branch in sdks/python dir. + + python setup.py sdist --format=zip + cd dist + mv apache-beam-${RELEASE}.zip apache-beam-${RELEASE}-python.zip + +Create hashes and sign the binaries + + gpg --armor --detach-sig apache-beam-${RELEASE}-python.zip + sha512sum apache-beam-${RELEASE}-python.zip > apache-beam-${RELEASE}-python.zip.sha512 + +Staging binaries + + svn co https://dist.apache.org/repos/dist/dev/beam + cd beam/${RELEASE} + svn add * + svn commit + +Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam). + ### Build the Pydoc API reference Make sure you have ```tox``` installed: From c6b6cc213504a8259f51201d2a0d014b42c6f294 Mon Sep 17 00:00:00 2001 From: Boyuan Zhang Date: Thu, 7 Jun 2018 09:14:25 -0700 Subject: [PATCH 12/13] copy renamed zi[ file to sdks/python/staging subdir --- src/contribute/release-guide.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index 35002e378e8..7dcb7c653f4 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -381,7 +381,8 @@ Build python binaries in release branch in sdks/python dir. python setup.py sdist --format=zip cd dist - mv apache-beam-${RELEASE}.zip apache-beam-${RELEASE}-python.zip + cp apache-beam-${RELEASE}.zip staging/apache-beam-${RELEASE}-python.zip + cd staging Create hashes and sign the binaries From 07ac72f4a5a8cc538aa09ad3ed7b62ea8904573b Mon Sep 17 00:00:00 2001 From: Boyuan Zhang Date: Thu, 7 Jun 2018 11:22:06 -0700 Subject: [PATCH 13/13] Update gradle publish with --no-parallel --no-daemon --- src/contribute/release-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md index 7dcb7c653f4..1608b05c598 100644 --- a/src/contribute/release-guide.md +++ b/src/contribute/release-guide.md @@ -337,7 +337,7 @@ release tag to the origin repository (this would be the Apache Beam repo): Use Gradle publish plugin to stage these artifacts on the Apache Nexus repository, as follows: - ./gradlew publish -PisRelease + ./gradlew publish -PisRelease --no-parallel --no-daemon Review all staged artifacts. They should contain all relevant parts for each module, including `pom.xml`, jar, test jar, javadoc, etc. Artifact names should follow [the existing format](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22) in which artifact name mirrors directory structure, e.g., `beam-sdks-java-io-kafka`. Carefully review any new artifacts.