Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Spark 2.4 #5756

Merged
merged 18 commits into from Apr 9, 2019
Merged

Update to Spark 2.4 #5756

merged 18 commits into from Apr 9, 2019

Conversation

chrisvittal
Copy link
Collaborator

@chrisvittal chrisvittal commented Apr 3, 2019

I have no idea if I've updated everything I need to update here. I haven't pushed the updated CI
build image or changed cloudtools (yet).

Update: pushed ci build image.

cc @cseed @danking

hail/build.gradle Outdated Show resolved Hide resolved
Copy link
Contributor

@tpoterba tpoterba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need changes in getting_started.rst and hail/hail-ci-deploy.sh

hail/build.gradle Show resolved Hide resolved
danking
danking previously requested changes Apr 3, 2019
Copy link
Collaborator

@danking danking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do a two-stage change to get this in. Arcturus added a configuration file to cloudtools that permits this.

https://github.com/Nealelab/cloudtools/pull/90/files#diff-8e7f8e85397f28e54901d3f7eb930902R137

Change CLOUDTOOLS_VERSION in hail-ci-deploy.sh to 4. This will prevent 3.x cloudtools from using any published hail jars. Then go update cloudtools at your leisure, bumping to 4.0.0. Once that's published, people will need to update cloudtools. They'll get a message to that effect.

hail/build.gradle Outdated Show resolved Hide resolved
@@ -17,8 +17,8 @@ Regardless of installation method, you will need:
<https://www.anaconda.com/download/>`_
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check getting_started_developing.rst too

@konradjk
Copy link
Collaborator

konradjk commented Apr 3, 2019

Yeah, is this going to deploy new jars only on 2.4 or also 2.2 for a while? This is good in the long run, but might be nice to keep 2.2 going in case there are any unintended consequences

@danking
Copy link
Collaborator

danking commented Apr 3, 2019

this will only do 2.4, but it's not much work to continue to provide both

@danking
Copy link
Collaborator

danking commented Apr 3, 2019

it looks like there's no source code changes

@chrisvittal chrisvittal dismissed danking’s stale review April 3, 2019 21:24

I think I got everything. We just need to update cloudtools now.

@tpoterba
Copy link
Contributor

tpoterba commented Apr 4, 2019

Dan, since you have a much better handle on how to get this in, can I give primary responsibility for this review to you?

I'm happy to accept a PR review in return.

@danking danking assigned danking and unassigned tpoterba Apr 5, 2019
@danking
Copy link
Collaborator

danking commented Apr 5, 2019

accepted

danking
danking previously requested changes Apr 5, 2019
if (sparkVersion == '2.2.0')
py4jVersion = '0.10.4'
else if (sparkVersion == '2.3.0')
py4jVersion = '0.10.6'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about 2.2.1 and 2.3.1 and friends? Should we truncate to the first three characters of sparkVersion for these checks?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the only ones that are different. Spark 2.2.1 is using 0.10.7, Spark 2.3.1 is using 0.10.7.

hail/python/hail/dev-environment-py3.7.yml Outdated Show resolved Hide resolved
hail/python/hail/docs/getting_started_developing.rst Outdated Show resolved Hide resolved
@danking
Copy link
Collaborator

danking commented Apr 5, 2019

ship it

@danking
Copy link
Collaborator

danking commented Apr 5, 2019

I added a cloudtools version pin in the cluster tests.

@chrisvittal chrisvittal force-pushed the spark-2.4 branch 3 times, most recently from 85857ae to ef3b289 Compare April 6, 2019 05:02
@chrisvittal
Copy link
Collaborator Author

The new cloudtools sha file needs to exist. I added it.

@danking
Copy link
Collaborator

danking commented Apr 6, 2019

Something is wrong with batch tests ...

@danking
Copy link
Collaborator

danking commented Apr 6, 2019

CI is quiet, so I'll retest now.

@danking
Copy link
Collaborator

danking commented Apr 6, 2019

I'll have to look into test_callback though.

@chrisvittal
Copy link
Collaborator Author

I believe that my work in #5813 will resolve the issues that we are seeing here. Please take a look.

For some reason, the name is now 1.4-debian9 rather than 1.4-deb9 🤷
@danking danking merged commit f37e25c into hail-is:master Apr 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants