Prepare Spark release v3.0.0-preview-rc1 #26243

jiangxb1987 · 2019-10-24T11:35:07Z

What changes were proposed in this pull request?

To push the built jars to maven release repository, we need to remove the 'SNAPSHOT' tag from the version name.

Made the following changes in this PR:

Update all the 3.0.0-SNAPSHOT version name to 3.0.0-preview
Update the PySpark version from 3.0.0.dev0 to 3.0.0

Please note those changes were generated by the release script in the past, but this time since we manually add tags on master branch, we need to manually apply those changes too.

We shall revert the changes after 3.0.0-preview release passed.

Why are the changes needed?

To make the maven release repository to accept the built jars.

Does this PR introduce any user-facing change?

No

How was this patch tested?

N/A

jiangxb1987 · 2019-10-24T11:35:52Z

cc @srowen @cloud-fan @dongjoon-hyun @gatorsmile

srowen · 2019-10-24T12:40:53Z

Right, seems like we'd have to do this for release and then revert. OK

SparkQA · 2019-10-24T13:52:22Z

Test build #112604 has finished for PR 26243 at commit 3db7b71.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-24T14:25:06Z

Test build #112606 has finished for PR 26243 at commit ed65ac5.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-24T14:32:18Z

Test build #112607 has finished for PR 26243 at commit 8ac854b.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-10-24T15:53:57Z

Hi, @jiangxb1987 . We need to update R/pkg/DESCRIPTION, too.
cc @shivaram , @felixcheung , @HyukjinKwon

dongjoon-hyun · 2019-10-24T15:55:02Z

Also, this should be reverted this immediately after commit. Otherwise, our Jenkins will try to publish this to our snapshot repository.

dongjoon-hyun · 2019-10-24T16:01:23Z

Hi, @jiangxb1987 . I update the PR title for a commit log. Since we use Preparing Spark release v2.4.4-rc3, the following will be the better. (Also removed the [MINOR] prefix)

Prepare Spark release v3.0.0-preview-rc1

dongjoon-hyun · 2019-10-24T16:04:49Z

docs/_config.yml

@@ -14,7 +14,7 @@ include:

 # These allow the documentation to be updated with newer releases
 # of Spark, Scala, and Mesos.
-SPARK_VERSION: 3.0.0-SNAPSHOT
+SPARK_VERSION: 3.0.0-preview
 SPARK_VERSION_SHORT: 3.0.0


This also should be 3.0.0-preview. The whole string of 3.0.0-preview is the short version.

python/pyspark/version.py

dongjoon-hyun · 2019-10-24T16:09:22Z

Also, could you review this, @gatorsmile ?

SparkQA · 2019-10-24T17:04:51Z

Test build #112609 has finished for PR 26243 at commit 322fcab.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-10-24T17:10:30Z

The R failure is due to this.

Error in sparkR.session(master = sparkRTestMaster, enableHiveSupport = FALSE) : 
  (converted from warning) Version mismatch between Spark JVM and SparkR package. JVM version was 3.0.0-preview , while R package version was 3.0.0

SparkQA · 2019-10-24T17:52:58Z

Test build #112614 has finished for PR 26243 at commit ebc2345.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-24T19:03:32Z

Test build #112616 has started for PR 26243 at commit f3731db.

R/pkg/R/sparkR.R

R/pkg/DESCRIPTION

HyukjinKwon

BTW, shouldn't we cut the branch first and land this change to that branch?

srowen · 2019-10-25T00:31:45Z

Ideally yes we release Pyspark and SparkR. At least, Pyspark seems pretty important.

If it's a major problem that we're trying to do this in master, then I concede, let's make a branch. Does a name like "3.0.0.preview" cause fewer headaches? we do have to keep the branch if that's what we release, but, if we have to do it we have to.

If the idea is to go ahead and cut branch-3.0, I'd rather not just because of the headache of merging back every change from now until 3.0. But it's not out of the question.

HyukjinKwon · 2019-10-25T00:37:02Z

Ah, yea, I meant the branch like branch-3.0-preview or preview-3.0 for instance.

Does a name like "3.0.0.preview" cause fewer headaches?

If this works, yea it should be good. Let us know @jiangxb1987 :D.

shivaram · 2019-10-25T01:32:37Z

I don't think we should release the SparkR preview on CRAN. We can't overwrite versions and the default install from CRAN gives the latest version.

felixcheung · 2019-10-25T01:38:59Z

I am not sure CRAN accept preview in the version number also. And what Shivaram says

…

________________________________ From: Shivaram Venkataraman <notifications@github.com> Sent: Thursday, October 24, 2019 6:35:03 PM To: apache/spark <spark@noreply.github.com> Cc: Felix Cheung <felixcheung_m@hotmail.com>; Mention <mention@noreply.github.com> Subject: Re: [apache/spark] Prepare Spark release v3.0.0-preview-rc1 (#26243) I don't think we should release the SparkR preview on CRAN. We can't overwrite versions and the default install from CRAN gives the latest version. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#26243?email_source=notifications&email_token=ACENZ6YRTV45MTUH4PXGERDQQJEMPA5CNFSM4JESXMOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECG4ZKY#issuecomment-546163883>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACENZ63XMGEJSF6AC32PHZDQQJEMPANCNFSM4JESXMOA>.

dongjoon-hyun · 2019-10-25T02:07:18Z

Thank you all.

For SparkR, if we are going to skip CRAN, the problem is solved.
For PySpark, the branch was already created as branch-3.0-preview and deleted before due to the side effect (@HyukjinKwon . Please see the mailing list. We discussed the side-effect on dev mailing list).
If there is a way to avoid the side-effect on our merge script and release script. I'm also okay for re-creating.

Please check the feasibility before creating the branch. Maybe, it seems to depend on the branch naming.

HyukjinKwon · 2019-10-25T05:02:09Z

retest this please

HyukjinKwon · 2019-10-25T05:09:57Z

Thanks for pointer @dongjoon-hyun. Alright, then, we don't do PyPI too I guess @holdenk?

/tmp/tmp.RRgnNxjsCA/3.5/lib/python3.5/site-packages/setuptools/dist.py:398: UserWarning: Normalizing '3.0.0.preview' to '3.0.0rc0'
......
Requirement '/home/jenkins/workspace/SparkPullRequestBuilder@2/python/dist/pyspark-3.0.0.preview.tar.gz' looks like a filename, but the file does not exist
Processing ./python/dist/pyspark-3.0.0.preview.tar.gz
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/home/jenkins/workspace/SparkPullRequestBuilder@2/python/dist/pyspark-3.0.0.preview.tar.gz'

This looks just pip packaging test failure. setup.py is only for PIP packaging and we don't probably need it.

Can we just completely skip pip packaging or use some workarounds to disable the normalizing the name (pypa/setuptools#308) for this preview?

HyukjinKwon · 2019-10-25T05:20:38Z

If ^ is easy, and can be done in this PR, without a separate branch, 1. we can check if the tests pass 2. we release preview and 3. revert this commit from the master.

Sorry if this was discussed already somewhere but we're not going to vote on it and make multiple RCs right (since it's just a preview)?

It we're going to vote and make more RCs for preview, it might be best to have a separate branch. Seems like merging script issue can be easily fixed in this case. The latest branch is decided by sorting order so we can name it like branch-preview-3.0. JIRA fixed version we will have to fix this line

spark/dev/merge_spark_pr.py

Line 244 in 2ec3265

branch_ver = branch.replace("branch-", "")

SparkQA · 2019-10-25T07:05:02Z

Test build #112649 has finished for PR 26243 at commit f3731db.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

LGTM pending Jenkins.

SparkQA · 2019-10-28T07:05:02Z

Test build #112754 has finished for PR 26243 at commit 6459efe.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-10-28T08:38:07Z

retest this please

SparkQA · 2019-10-28T11:34:51Z

Test build #112761 has finished for PR 26243 at commit 6459efe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-10-28T11:48:04Z

I reverted 5fc363b which I merged by myself ... seems it causes the build failure on Windows (#26278). Let me rebase it here.

SparkQA · 2019-10-28T12:12:42Z

Test build #112770 has finished for PR 26243 at commit 6a59acd.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-10-28T13:30:46Z

retest this please

SparkQA · 2019-10-28T13:33:10Z

Test build #112772 has started for PR 26243 at commit 6a59acd.

dongjoon-hyun · 2019-10-28T16:29:28Z

Got it. Thank you for reverting and keeping this moving forward! @HyukjinKwon .

python/pyspark/version.py

SparkQA · 2019-10-29T00:30:58Z

Test build #112806 has finished for PR 26243 at commit ec571bb.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-10-29T00:36:16Z

retest this please

SparkQA · 2019-10-29T00:56:48Z

Test build #112808 has finished for PR 26243 at commit ec571bb.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-29T03:35:34Z

Test build #112809 has finished for PR 26243 at commit e1808a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-10-29T04:29:58Z

should be good to go.

cloud-fan · 2019-10-29T04:41:22Z

LGTM

jiangxb1987 · 2019-10-29T05:30:14Z

Thanks, merging to master!

jiangxb1987 · 2019-10-29T05:33:40Z

FYI, this PR has been reverted by b33a58c

dongjoon-hyun · 2019-10-29T17:24:36Z

Great! Thank you all!

dongjoon-hyun added the BUILD label Oct 24, 2019

dongjoon-hyun changed the title ~~[MINOR] Prepare Spark release 3.0.0-preview~~ Prepare Spark release v3.0.0-preview-rc1 Oct 24, 2019

dongjoon-hyun reviewed Oct 24, 2019

View reviewed changes

python/pyspark/version.py Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Oct 24, 2019

View reviewed changes

R/pkg/R/sparkR.R Show resolved Hide resolved

dongjoon-hyun reviewed Oct 24, 2019

View reviewed changes

R/pkg/DESCRIPTION Outdated Show resolved Hide resolved

dongjoon-hyun requested a review from felixcheung October 24, 2019 20:13

HyukjinKwon reviewed Oct 25, 2019

View reviewed changes

HyukjinKwon force-pushed the 3.0.0-preview-prepare branch from b10f174 to 6459efe Compare October 28, 2019 04:15

HyukjinKwon approved these changes Oct 28, 2019

View reviewed changes

HyukjinKwon force-pushed the 3.0.0-preview-prepare branch from 6459efe to 6a59acd Compare October 28, 2019 11:49

cloud-fan reviewed Oct 28, 2019

View reviewed changes

python/pyspark/version.py Outdated Show resolved Hide resolved

jiangxb1987 and others added 3 commits October 28, 2019 18:00

Prepare Spark release 3.0.0-preview

61750b8

Use JVM version check in SparkR

3149e7d

update python version

e1808a5

jiangxb1987 force-pushed the 3.0.0-preview-prepare branch from ec571bb to e1808a5 Compare October 29, 2019 01:00

jiangxb1987 closed this in 5eddbb5 Oct 29, 2019

jiangxb1987 mentioned this pull request Oct 30, 2019

[SPARK-29646][BUILD] Allow pyspark version name format ${versionNumber}-preview in release script #26306

Closed

Prepare Spark release v3.0.0-preview-rc1 #26243

Prepare Spark release v3.0.0-preview-rc1 #26243

Conversation

jiangxb1987 commented Oct 24, 2019 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

jiangxb1987 commented Oct 24, 2019

srowen commented Oct 24, 2019

SparkQA commented Oct 24, 2019

SparkQA commented Oct 24, 2019

SparkQA commented Oct 24, 2019

dongjoon-hyun commented Oct 24, 2019

dongjoon-hyun commented Oct 24, 2019

dongjoon-hyun commented Oct 24, 2019 • edited Loading

dongjoon-hyun Oct 24, 2019

Choose a reason for hiding this comment

dongjoon-hyun commented Oct 24, 2019

SparkQA commented Oct 24, 2019

dongjoon-hyun commented Oct 24, 2019

SparkQA commented Oct 24, 2019

SparkQA commented Oct 24, 2019

HyukjinKwon left a comment

Choose a reason for hiding this comment

srowen commented Oct 25, 2019

HyukjinKwon commented Oct 25, 2019

shivaram commented Oct 25, 2019

felixcheung commented Oct 25, 2019 via email

dongjoon-hyun commented Oct 25, 2019

HyukjinKwon commented Oct 25, 2019

HyukjinKwon commented Oct 25, 2019

HyukjinKwon commented Oct 25, 2019 • edited Loading

SparkQA commented Oct 25, 2019

HyukjinKwon left a comment

Choose a reason for hiding this comment

SparkQA commented Oct 28, 2019

HyukjinKwon commented Oct 28, 2019

SparkQA commented Oct 28, 2019

HyukjinKwon commented Oct 28, 2019

SparkQA commented Oct 28, 2019

HyukjinKwon commented Oct 28, 2019

SparkQA commented Oct 28, 2019

dongjoon-hyun commented Oct 28, 2019

SparkQA commented Oct 29, 2019

HyukjinKwon commented Oct 29, 2019

SparkQA commented Oct 29, 2019

SparkQA commented Oct 29, 2019

HyukjinKwon commented Oct 29, 2019

cloud-fan commented Oct 29, 2019

jiangxb1987 commented Oct 29, 2019

jiangxb1987 commented Oct 29, 2019

dongjoon-hyun commented Oct 29, 2019

jiangxb1987 commented Oct 24, 2019 •

edited

Loading

dongjoon-hyun commented Oct 24, 2019 •

edited

Loading

HyukjinKwon commented Oct 25, 2019 •

edited

Loading