[BEAM-3255] Updating release guide to use Gradle commands #424

pabloem · 2018-04-19T23:23:16Z

No description provided.

alanmyrvold · 2018-04-20T13:06:28Z

src/contribute/release-guide.md

-        -DupdateWorkingCopyVersions=false \
-        -DupdateBranchVersions=true \
-        -DreleaseVersion="${VERSION}-SNAPSHOT"
+    git branch ${BRANCH}


Do we need to clarify that the branch is to made on the upstream apache/beam repo and pushed, and not on a fork, or is something done by the release plugin?

It should be done in the upstream repo. I've mentioned 'Apache' there. LMK what you think.

alanmyrvold · 2018-04-20T13:43:52Z

src/contribute/release-guide.md

+    git branch ${BRANCH}
+
+    # Now change the version in existing gradle files, and Python files
+    sed -i 's/'$RELEASE'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g' build_rules.gradle


The single quotes as written don't just match strings with single quotes.
Maybe
sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" build_rules.gradle

Nice! thanks.

swegner · 2018-04-20T16:09:09Z

src/contribute/release-guide.md


 Version represents the release currently underway, while next version specifies the anticipated next version to be released from that branch. Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4.

-Use Maven release plugin to create the release branch and update the current branch to use the new development version. This command applies for the new major or minor version. (Warning: this command automatically pushes changes to the code repository.)
+**NOTE**: Only if you are doing an incremental/hotfix releasei (e.g. 2.5.1), please check out the previous release tag, before running the following instructions:


typo: releasei -> release

Good catch. Thanks Scott.

alanmyrvold · 2018-04-20T18:11:18Z

src/contribute/release-guide.md

-    NEXT_VERSION="1.2.4"
-    BRANCH_NAME="release-${VERSION}"
-    DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT"
+    RELEASE_VERSION=2.5.0


Either this should be RELEASE=2.5.0 or occurrences of $RELEASE below changed to match

Ugh, sorry about that. Thanks.

alanmyrvold · 2018-04-20T18:23:05Z

src/contribute/release-guide.md

-        -DreleaseVersion=${VERSION} \
-        -Dtag=${RC_TAG} \
-        -DupdateWorkingCopyVersions=false
+    ./gradlew release -Prelease.newVersion=${RELEASE_VERSION}-SNAPSHOT \


Is release.newVersion 2.5.0-SNAPSHOT or 2.6.0-SNAPSHOT? ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT ?

pabloem · 2018-04-25T18:52:23Z

TODO: Add update of runners - and dataflow.container_version.

pabloem · 2018-04-25T19:39:09Z

TODO: Information on how to create the new key

swegner · 2018-05-01T21:39:04Z

I've proposed a set of changes for source distribution here: pabloem#2

pabloem · 2018-05-01T22:24:48Z

I've merged your changes, scott, and added a few to fix some todos that we discovered earlier. I'm still unclear if the instructions to update dataflow.container_version should be here, or in a Dataflow release guide.
@aaltay what should we do with dataflow.container_version?

swegner · 2018-05-02T22:55:00Z

@pabloem Can you elaborate on what you're referring to with dataflow.container_version? I don't see this in the existing release documentation or your current changes.

I want to get this unblocked so that we can follow this release guide for the 2.5.0 release.

I seem to recall that we need to update dataflow.container_version as some part of the release, but I don't think the requirements would change with the Gradle migration. Put otherwise: if documentation about dataflow.container_version is missing, then we should track that as a bug, but it shouldn't hold up this doc update.

pabloem · 2018-05-02T22:59:14Z

I think I agree with that. I've covered all the other TODOs that we ran into during our demo with Ahmet. So we can do the final review for this PR.

r: @aaltay PTAL

lukecwik · 2018-05-18T16:28:22Z

src/contribute/release-guide.md

@@ -158,12 +158,11 @@ Adjust any of the above properties to the improve clarity and presentation of th

 ### Verify that a Release Build Works

-Run `mvn -Prelease clean install` to ensure that the build processes that are specific to that
-profile are in good shape.
+Run `./gradlew -PisRelease clean build` to ensure that the build processes that are specific to that profile are in good shape.


From a clean workspace git clean -fdx, perform ./gradlew -PisRelease build

The gradle tasks you list don't run in the order that you specified, they just get added to the task tree and execute in dependency order (and if there is no dependency between them, then in arbitrary order).

I'm not sure whether we should instruct users to use --no-parallel or not here as well.

ah yes that's true

boyuanzz · 2018-06-02T00:16:37Z

Hey @pabloem @alanmyrvold , I added more updates in commit 9eab89c. Please take a look and review it. Thanks!

swegner · 2018-06-04T16:26:24Z

I didn't realize this PR is still open. The published release guide still references the old Maven commands. If this doc is close or good enough, can it be merged and continue to iterate in separate PR's?

pabloem · 2018-06-04T16:30:30Z

@jbonofre @aaltay WDYT? Should we merge this PR now?

boyuanzz · 2018-06-04T20:36:18Z

Hey @alanmyrvold, could you please review my recent updates: 9eab89c, 9d0fd1c ?

alanmyrvold · 2018-06-05T18:30:34Z

src/contribute/release-guide.md

+
+    BASE_RELEASE=2.5.0
+    RELEASE=2.5.1
+    NEXT_VERSION_IN_BASE_BRANCH=2.5.0


NEXT_VERSION_IN_BASE_BRANCH=2.6.0

alanmyrvold · 2018-06-05T18:31:39Z

src/contribute/release-guide.md


 The rest of this guide assumes that commands are run in the root of a repository on `${BRANCH_NAME}` with the above environment variables set.

+### Update the beam version on master to the next dev version


This step should be done on master and is the purpose of the sed scripts above?

alanmyrvold · 2018-06-05T18:32:34Z

src/contribute/release-guide.md


 Check out the release branch.

-    git checkout ${BRANCH_NAME}
+    git checkout ${BRANCH}


This step needs to be moved later, after changing versions on master?

We do all master update in previous sed scripts. The rest of instructions all happen in release branch.

boyuanzz · 2018-06-05T22:31:26Z

Hey @alanmyrvold , please review commit d238c9e, which addressed your comments above.

alanmyrvold · 2018-06-06T21:48:18Z

src/contribute/release-guide.md


 1. Add and commit all the files.

-        svn add beam/${VERSION}
+        svn add beam/${RELEASE}
        svn commit

 1. Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).



Need a step to build the python binaries and stage on dist.apache.org/dev/beam, like https://dist.apache.org/repos/dist/dev/beam/2.5.0/

alanmyrvold · 2018-06-06T21:52:48Z

src/contribute/release-guide.md

 * Set up the necessary git commands to account for the new and deleted files from the javadoc.
 * Update the Javadoc link on this page to point to the new version (in `src/documentation/sdks/javadoc/current.md`).

 #### Create Pydoc
 Add the new Pydoc to [SDK API Reference page]({{ site.baseurl }}/documentation/sdks/pydoc/) page, as follows:

-* Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} src/documentation/sdks/pydoc/${VERSION}`.
+* Copy the generated Pydoc into the website repository: `cp -r ${PYDOC_ROOT} src/documentation/sdks/pydoc/${RELEASE}`.
 * Remove `.doctrees` directory.
 * Update the Pydoc link on this page to point to the new version (in `src/documentation/sdks/pydoc/current.md`).



Need to push the RC1 tag to github?
And verify that the cythonized built python artifacts are on dist.apache.org along with the source?

I think this step will push RC1 tag to github: https://github.com/pabloem/beam-site/blob/372c93ecbafbf3a1440492df1e12050dfe939e91/src/contribute/release-guide.md#build-and-stage-java-artifacts-with-maven

boyuanzz · 2018-06-06T23:12:11Z

Hey @alanmyrvold @robertwb , please review this commit: ab15c1b . Thanks!

boyuanzz · 2018-06-06T23:21:48Z

Re: @robertwb, I think so. @alanmyrvold what do you think?

boyuanzz · 2018-06-07T16:15:51Z

@robertwb commit c6b6cc2 copies zip file into sdks/python/staging subdir

boyuanzz · 2018-06-08T18:35:45Z

retest this please

aaltay

@pabloem please address comments before merging.

aaltay · 2018-06-08T18:39:38Z

src/contribute/release-guide.md

+  sudo apt-get install gcc
+  sudo apt-get install python-dev
+  ```
+Make sure your ```time``` alias to ```/usr/bin/time```, if not:


Why do we need time for release verification?

python tests script uses time. If /usr/bin/time is not alias to time, tests will fail.

aaltay · 2018-06-08T18:41:11Z

src/contribute/release-guide.md

        svn commit

 1. Verify that files are [present](https://dist.apache.org/repos/dist/dev/beam).

+### Stage python binaries on dist.apache.org


Do we explain how to generate all python dependencies? (If not could we file a JIRA?)

We don't explain that so far. Do we need this?

I meant, all python "artifacts". Sorry for the confusion. I mean the tar ball, and platform dependent whl files.

boyuanzz · 2018-06-08T21:00:40Z

retest this please

boyuanzz · 2018-06-08T22:27:27Z

retest this please

boyuanzz · 2018-06-08T22:37:30Z

This PR is ready to merge. Issues above were discussed offline.

pabloem · 2018-06-08T23:03:33Z

@asfgit merge

Updating release guide to use Gradle commands.

55ccd27

alanmyrvold reviewed Apr 20, 2018

View reviewed changes

swegner reviewed Apr 20, 2018

View reviewed changes

Addressing comments. Improving commands.

ec3eb5a

alanmyrvold reviewed Apr 20, 2018

View reviewed changes

Addressing comments. Improving commands.

68ad3d6

alanmyrvold reviewed Apr 20, 2018

View reviewed changes

This comment has been minimized.

Sign in to view

swegner and others added 2 commits May 1, 2018 14:42

Update release guide for publishing source zip from GitHub

0ac546b

Addressing TODOs

372c93e

jbonofre self-requested a review May 3, 2018 18:38

lukecwik reviewed May 18, 2018

View reviewed changes

pabloem and others added 3 commits May 18, 2018 09:52

Address comments.

20a0a5a

Improving.

cec4343

Add more updates

9eab89c

More updates

9d0fd1c

alanmyrvold reviewed Jun 5, 2018

View reviewed changes

Addressed Alan's comments

d238c9e

alanmyrvold reviewed Jun 6, 2018

View reviewed changes

Add instructions about how to stage python binaries

ab15c1b

copy renamed zi[ file to sdks/python/staging subdir

c6b6cc2

Update gradle publish with --no-parallel --no-daemon

07ac72f

pabloem closed this Jun 8, 2018

pabloem reopened this Jun 8, 2018

aaltay approved these changes Jun 8, 2018

View reviewed changes

asfgit closed this in dac24e6 Jun 8, 2018

pabloem deleted the gradle-release-guide branch June 8, 2018 23:26

swegner pushed a commit to swegner/beam that referenced this pull request Jun 12, 2018

This closes apache/beam-site#424

5638167

charlesccychen pushed a commit to cosmoskitten/beam that referenced this pull request Jun 14, 2018

This closes apache/beam-site#424

1400d84


		The rest of this guide assumes that commands are run in the root of a repository on `${BRANCH_NAME}` with the above environment variables set.

		### Update the beam version on master to the next dev version

[BEAM-3255] Updating release guide to use Gradle commands #424

[BEAM-3255] Updating release guide to use Gradle commands #424

Conversation

pabloem commented Apr 19, 2018

alanmyrvold Apr 20, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pabloem commented Apr 25, 2018

This comment has been minimized.

This comment has been minimized.

pabloem commented Apr 25, 2018

swegner commented May 1, 2018

pabloem commented May 1, 2018 • edited

swegner commented May 2, 2018

pabloem commented May 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boyuanzz commented Jun 2, 2018

swegner commented Jun 4, 2018

pabloem commented Jun 4, 2018

boyuanzz commented Jun 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boyuanzz commented Jun 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boyuanzz commented Jun 6, 2018

boyuanzz commented Jun 6, 2018

boyuanzz commented Jun 7, 2018

boyuanzz commented Jun 8, 2018

aaltay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

boyuanzz commented Jun 8, 2018

boyuanzz commented Jun 8, 2018

boyuanzz commented Jun 8, 2018

pabloem commented Jun 8, 2018

alanmyrvold Apr 20, 2018 •

edited

pabloem commented May 1, 2018 •

edited