[BEAM-3255] Updating release guide to use Gradle commands #424
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -63,6 +63,15 @@ To prepare for each release, you should audit the project status in the JIRA iss | |
|
|
||
| You need to have a GPG key to sign the release artifacts. Please be aware of the ASF-wide [release signing guidelines](https://www.apache.org/dev/release-signing.html). If you don’t have a GPG key associated with your Apache account, please create one according to the guidelines. | ||
|
|
||
| Get more entropy for creating a GPG key | ||
|
|
||
| sudo apt-get install rng-tools | ||
| sudo rngd -r /dev/urandom | ||
|
|
||
| Create a GPG key | ||
|
|
||
| gpg --full-generate-key | ||
|
|
||
| Determine your Apache GPG Key and Key ID, as follows: | ||
|
|
||
| gpg --list-keys | ||
|
|
@@ -128,6 +137,8 @@ Release manager needs to have an account with PyPI. If you need one, [register a | |
|
|
||
| When contributors resolve an issue in JIRA, they are tagging it with a release that will contain their changes. With the release currently underway, new issues should be resolved against a subsequent future release. Therefore, you should create a release item for this subsequent release, as follows: | ||
|
|
||
| Attention: Only committer has permission to perform this. | ||
|
|
||
| 1. In JIRA, navigate to the [`Beam > Administration > Versions`](https://issues.apache.org/jira/plugins/servlet/project-config/BEAM/versions). | ||
| 1. Add a new release: choose the next minor version number compared to the one currently underway, select today’s date as the `Start Date`, and choose `Add`. | ||
|
|
||
|
|
@@ -158,12 +169,54 @@ Adjust any of the above properties to the improve clarity and presentation of th | |
|
|
||
| ### Verify that a Release Build Works | ||
|
|
||
| Pre-installation for python build | ||
| * Install pip | ||
|
|
||
| ``` | ||
| curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py | ||
| python get-pip.py | ||
| ``` | ||
| * Install virtualenv | ||
|
|
||
| ``` | ||
| pip install --upgrade virtualenv | ||
| ``` | ||
| * Cython | ||
|
|
||
| ``` | ||
| sudo pip install cython | ||
| sudo apt-get install gcc | ||
| sudo apt-get install python-dev | ||
| ``` | ||
| Make sure your ```time``` alias to ```/usr/bin/time```, if not: | ||
|
|
||
| ``` | ||
| sudo apt-get install time | ||
| alias time='/usr/bin/time' | ||
| ``` | ||
|
|
||
| Run gradle release build | ||
|
|
||
| * Clean current workspace | ||
|
|
||
| ``` | ||
| git clean -fdx | ||
| ./gradlew clean | ||
| ``` | ||
|
|
||
| * Unlock the secret key | ||
| ``` | ||
| gpg --output ~/doc.sig --sign ~/.bashrc | ||
| ``` | ||
|
|
||
| * Run build command | ||
| ``` | ||
| ./gradlew build -PisRelease --no-parallel --scan --stacktrace | ||
| ``` | ||
|
|
||
| ### Update and Verify Javadoc | ||
|
|
||
| The build with `-PisRelease` creates the combined Javadoc for the release in `sdks/java/javadoc`. | ||
|
|
||
| The file `sdks/java/javadoc/ant.xml` file contains a list of modules to include | ||
| in and exclude, plus a list of offline URLs that populate links from Beam's | ||
|
|
@@ -178,64 +231,73 @@ Javadoc to the Javadoc for other modules that Beam depends on. | |
| the version number has changed, download a new version of the corresponding | ||
| `<module>-docs/package-list` file. | ||
|
|
||
| ### Create a release branch in apache/beam repository | ||
|
|
||
| Attention: Only committer has permission to create release branch in apache/beam. | ||
|
|
||
| Release candidates are built from a release branch. As a final step in preparation for the release, you should create the release branch, push it to the Apache code repository, and update version information on the original branch. | ||
|
|
||
| Check out the version of the codebase from which you start the release. For a new minor or major release, this may be `HEAD` of the `master` branch. To build a hotfix/incremental release, instead of the `master` branch, use the release tag of the release being patched. (Please make sure your cloned repository is up-to-date before starting.) | ||
|
|
||
| git checkout <master branch OR release tag> | ||
|
|
||
| **NOTE**: If you are doing an incremental/hotfix release (e.g. 2.5.1), please check out the previous release tag, rather than the master branch. | ||
|
|
||
| Set up a few environment variables to simplify Maven commands that follow. (We use `bash` Unix syntax in this guide.) | ||
|
|
||
| RELEASE=2.5.0 | ||
| NEXT_VERSION_IN_BASE_BRANCH=2.6.0 | ||
| BRANCH=release-${RELEASE} | ||
|
|
||
| Version represents the release currently underway, while next version specifies the anticipated next version to be released from that branch. Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4. | ||
|
|
||
| **NOTE**: Only if you are doing an incremental/hotfix release (e.g. 2.5.1), please check out the previous release tag, before running the following instructions: | ||
|
|
||
| BASE_RELEASE=2.5.0 | ||
| RELEASE=2.5.1 | ||
| NEXT_VERSION_IN_BASE_BRANCH=2.6.0 | ||
| git checkout tags/${BASE_RELEASE} | ||
|
|
||
| Create a new branch, and update version files in the master branch. | ||
|
|
||
| git branch ${BRANCH} | ||
|
There was a problem hiding this comment. Do we need to clarify that the branch is to made on the upstream apache/beam repo and pushed, and not on a fork, or is something done by the release plugin? There was a problem hiding this comment. It should be done in the upstream repo. I've mentioned 'Apache' there. LMK what you think. |
||
|
|
||
| # Now change the version in existing gradle files, and Python files | ||
| sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" build_rules.gradle | ||
| sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" gradle.properties | ||
| sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" sdks/python/apache_beam/version.py | ||
|
|
||
| # Save changes in master branch | ||
| git add gradle.properties build_rules.gradle sdks/python/apache_beam/version.py | ||
| git commit -m "Moving to ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master branch." | ||
|
|
||
| Check out the release branch. | ||
|
|
||
| git checkout ${BRANCH} | ||
|
|
||
| The rest of this guide assumes that commands are run in the root of a repository on `${BRANCH}` with the above environment variables set. | ||
|
|
||
| ### Update the Python SDK version | ||
|
|
||
| Update [sdks/python/apache_beam/version.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/version.py) in both master branch and release branch. | ||
|
|
||
| * In the release branch, update the Python SDK version to the release version(e.g. `2.5.0-dev` to `2.5.0`). | ||
|
|
||
| ### Update release specific configurations | ||
|
|
||
| Update Java runner specific configurations in release branch: | ||
| * [beam/runners/google-cloud-dataflow-java/build.gradle](https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/build.gradle): change value of 'dataflow.container_version' to 'beam-release_version_number'(e.g, 'beam-master-20180601' to 'beam-2.5.0') | ||
| * [beam/runners/google-cloud-dataflow-java/pom.xml](https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/pom.xml): change value of 'version' field (e.g, '2.5.0-SNAPSHOT' to '2.5.0') | ||
|
|
||
| ### Start a snapshot build | ||
|
|
||
| Start a build of [the nightly snapshot](https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_NightlySnapshot/) against master branch. | ||
| Some processes, including our archetype tests, rely on having a live SNAPSHOT of the current version | ||
| from the `master` branch. Once the release branch is cut, these SNAPSHOT versions are no longer found, | ||
| so builds will be broken until a new snapshot is available. | ||
|
|
||
| Comment ```Run Gradle Publish``` in one pull request to trigger build. | ||
|
|
||
| ### Checklist to proceed to the next step | ||
|
|
||
| 1. Release Manager’s GPG key is published to `dist.apache.org` | ||
|
|
@@ -256,31 +318,35 @@ so builds will be broken until a new snapshot is available. | |
|
|
||
| The core of the release process is the build-vote-fix cycle. Each cycle produces one release candidate. The Release Manager repeats this cycle until the community approves one release candidate, which is then finalized. | ||
|
|
||
| ### Build and stage Java artifacts with Gradle | ||
|
|
||
| Set up a few environment variables to simplify the commands that follow. These identify the release candidate being built, and the branch where you will stage files. Start with `RC_NUM` equal to `1` and increment it for each candidate. | ||
|
|
||
| RC_NUM=1 | ||
|
|
||
| Make sure your git config will maintain your account: | ||
|
|
||
| git config credential.helper store | ||
|
|
||
| Use Gradle release plugin to build the release artifacts, and push code and | ||
| release tag to the origin repository (this would be the Apache Beam repo): | ||
|
|
||
| ./gradlew release -Prelease.newVersion=${RELEASE}-SNAPSHOT \ | ||
| -Prelease.releaseVersion=${RELEASE}-RC${RC_NUM} \ | ||
| -Prelease.useAutomaticVersion=true --info --no-daemon | ||
|
|
||
| Use Gradle publish plugin to stage these artifacts on the Apache Nexus repository, as follows: | ||
|
|
||
| ./gradlew publish -PisRelease --no-parallel --no-daemon | ||
|
|
||
| Review all staged artifacts. They should contain all relevant parts for each module, including `pom.xml`, jar, test jar, javadoc, etc. Artifact names should follow [the existing format](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22) in which artifact name mirrors directory structure, e.g., `beam-sdks-java-io-kafka`. Carefully review any new artifacts. | ||
|
|
||
| Close the staging repository on Apache Nexus. When prompted for a description, enter “Apache Beam, version X, release candidate Y”. | ||
|
|
||
| ### Stage source release on dist.apache.org | ||
|
|
||
| Attention: Only committer has permissions to perform following steps. | ||
|
|
||
| Copy the source release to the dev repository of `dist.apache.org`. | ||
|
|
||
| 1. If you have not already, check out the Beam section of the `dev` repository on `dist.apache.org` via Subversion. In a fresh directory: | ||
|
|
@@ -289,30 +355,57 @@ Copy the source release to the dev repository of `dist.apache.org`. | |
|
|
||
| 1. Make a directory for the new release: | ||
|
|
||
| mkdir beam/${RELEASE} | ||
| cd beam/${RELEASE} | ||
|
|
||
| 1. Download source zip from GitHub: | ||
|
|
||
| wget https://github.com/apache/beam/archive/release-${RELEASE}.zip \ | ||
| -O apache-beam-${RELEASE}-source-release.zip | ||
|
|
||
| 1. Create hashes and sign the source distribution: | ||
|
|
||
| gpg --armor --detach-sig apache-beam-${RELEASE}-source-release.zip | ||
| sha512sum apache-beam-${RELEASE}-source-release.zip > apache-beam-${RELEASE}-source-release.zip.sha512 | ||
|
|
||
| 1. Add and commit all the files. | ||
|
|
||
| svn add beam/${RELEASE} | ||
| svn commit | ||
|
|
||
| 1. Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam). | ||
|
|
||
|
There was a problem hiding this comment. Need a step to build the python binaries and stage on dist.apache.org/dev/beam, like https://dist.apache.org/repos/dist/dev/beam/2.5.0/ |
||
| ### Stage python binaries on dist.apache.org | ||
|
There was a problem hiding this comment. Do we explain how to generate all python dependencies? (If not could we file a JIRA?) There was a problem hiding this comment. We don't explain that so far. Do we need this? There was a problem hiding this comment. I meant, all python "artifacts". Sorry for the confusion. I mean the tar ball, and platform dependent whl files. |
||
|
|
||
| Build python binaries in release branch in sdks/python dir. | ||
|
|
||
| python setup.py sdist --format=zip | ||
| cd dist | ||
| cp apache-beam-${RELEASE}.zip staging/apache-beam-${RELEASE}-python.zip | ||
| cd staging | ||
|
|
||
| Create hashes and sign the binaries | ||
|
|
||
| gpg --armor --detach-sig apache-beam-${RELEASE}-python.zip | ||
| sha512sum apache-beam-${RELEASE}-python.zip > apache-beam-${RELEASE}-python.zip.sha512 | ||
|
|
||
| Staging binaries | ||
|
|
||
| svn co https://dist.apache.org/repos/dist/dev/beam | ||
| cd beam/${RELEASE} | ||
| svn add * | ||
| svn commit | ||
|
|
||
| Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam). | ||
|
|
||
| ### Build the Pydoc API reference | ||
|
|
||
| Make sure you have ```tox``` installed: | ||
|
|
||
| ``` | ||
| pip install tox | ||
|
|
||
| ``` | ||
| Create the Python SDK documentation using sphinx by running a helper script. | ||
| ``` | ||
| cd sdks/python && tox -e docs | ||
|
|
@@ -334,14 +427,14 @@ candidate into the source tree of the website. | |
| Add the new Javadoc to [SDK API Reference page]({{ site.baseurl }}/documentation/sdks/javadoc/) page, as follows: | ||
|
|
||
| * Unpack the Maven artifact `org.apache.beam:beam-sdks-java-javadoc` into some temporary location. Call this `${JAVADOC_TMP}`. | ||
| * Copy the generated Javadoc into the website repository: `cp -r ${JAVADOC_TMP} src/documentation/sdks/javadoc/${RELEASE}`. | ||
| * Set up the necessary git commands to account for the new and deleted files from the javadoc. | ||
| * Update the Javadoc link on this page to point to the new version (in `src/documentation/sdks/javadoc/current.md`). | ||
|
|
||
| #### Create Pydoc | ||
| Add the new Pydoc to [SDK API Reference page]({{ site.baseurl }}/documentation/sdks/pydoc/) page, as follows: | ||
|
|
||
| * Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} src/documentation/sdks/pydoc/${RELEASE}`. | ||
| * Remove `.doctrees` directory. | ||
| * Update the Pydoc link on this page to point to the new version (in `src/documentation/sdks/pydoc/current.md`). | ||
|
|
||
|
There was a problem hiding this comment. Need to push the RC1 tag to github? There was a problem hiding this comment. I think this step will push RC1 tag to github: https://github.com/pabloem/beam-site/blob/372c93ecbafbf3a1440492df1e12050dfe939e91/src/contribute/release-guide.md#build-and-stage-java-artifacts-with-maven |
||
|
|
@@ -450,9 +543,9 @@ Use the Apache Nexus repository to release the staged binary artifacts to the Ma | |
| ### Deploy Python artifacts to PyPI | ||
|
|
||
| 1. Create a new release and upload the Python zip file for the new release using the [PyPI UI] (https://pypi.python.org/pypi/apache-beam) | ||
| 1. Alternatively, use the command line tool to upload the new release `twine upload apache-beam-${RELEASE}.zip` | ||
|
|
||
| Note: It is important to rename `apache-beam-${RELEASE}-python.zip` to `apache-beam-${RELEASE}.zip` before uploading, because PyPI expects a filename in the `<package-name>-<package-version>` format. | ||
|
|
||
| #### Deploy source release to dist.apache.org | ||
|
|
||
|
|
@@ -462,7 +555,7 @@ Copy the source release from the `dev` repository to the `release` repository at | |
|
|
||
| Create and push a new signed tag for the released version by copying the tag for the final release candidate, as follows: | ||
|
|
||
| VERSION_TAG="v${RELEASE}" | ||
| git tag -s "$VERSION_TAG" "$RC_TAG" | ||
| git push github "$VERSION_TAG" | ||
|
|
||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need
timefor release verification?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
python tests script uses
time. If/usr/bin/timeis not alias totime, tests will fail.