Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Spark 2.4 #5756

Merged
merged 18 commits into from Apr 9, 2019

Conversation

Projects
None yet
4 participants
@chrisvittal
Copy link
Collaborator

commented Apr 3, 2019

I have no idea if I've updated everything I need to update here. I haven't pushed the updated CI
build image or changed cloudtools (yet).

Update: pushed ci build image.

cc @cseed @danking

Show resolved Hide resolved hail/build.gradle Outdated
@tpoterba
Copy link
Collaborator

left a comment

Need changes in getting_started.rst and hail/hail-ci-deploy.sh

Show resolved Hide resolved hail/build.gradle
@danking
Copy link
Collaborator

left a comment

We can do a two-stage change to get this in. Arcturus added a configuration file to cloudtools that permits this.

https://github.com/Nealelab/cloudtools/pull/90/files#diff-8e7f8e85397f28e54901d3f7eb930902R137

Change CLOUDTOOLS_VERSION in hail-ci-deploy.sh to 4. This will prevent 3.x cloudtools from using any published hail jars. Then go update cloudtools at your leisure, bumping to 4.0.0. Once that's published, people will need to update cloudtools. They'll get a message to that effect.

Show resolved Hide resolved hail/build.gradle Outdated
@@ -17,8 +17,8 @@ Regardless of installation method, you will need:
<https://www.anaconda.com/download/>`_

This comment has been minimized.

Copy link
@danking

danking Apr 3, 2019

Collaborator

check getting_started_developing.rst too

@konradjk

This comment has been minimized.

Copy link
Collaborator

commented Apr 3, 2019

Yeah, is this going to deploy new jars only on 2.4 or also 2.2 for a while? This is good in the long run, but might be nice to keep 2.2 going in case there are any unintended consequences

@danking

This comment has been minimized.

Copy link
Collaborator

commented Apr 3, 2019

this will only do 2.4, but it's not much work to continue to provide both

@danking

This comment has been minimized.

Copy link
Collaborator

commented Apr 3, 2019

it looks like there's no source code changes

@chrisvittal chrisvittal force-pushed the chrisvittal:spark-2.4 branch from fe75aa8 to fb28184 Apr 3, 2019

I think I got everything. We just need to update cloudtools now.

@tpoterba

This comment has been minimized.

Copy link
Collaborator

commented Apr 4, 2019

Dan, since you have a much better handle on how to get this in, can I give primary responsibility for this review to you?

I'm happy to accept a PR review in return.

@danking danking assigned danking and unassigned tpoterba Apr 5, 2019

@danking

This comment has been minimized.

Copy link
Collaborator

commented Apr 5, 2019

accepted

foo
if (sparkVersion == '2.2.0')
py4jVersion = '0.10.4'
else if (sparkVersion == '2.3.0')
py4jVersion = '0.10.6'

This comment has been minimized.

Copy link
@danking

danking Apr 5, 2019

Collaborator

what about 2.2.1 and 2.3.1 and friends? Should we truncate to the first three characters of sparkVersion for these checks?

This comment has been minimized.

Copy link
@chrisvittal

chrisvittal Apr 5, 2019

Author Collaborator

These are the only ones that are different. Spark 2.2.1 is using 0.10.7, Spark 2.3.1 is using 0.10.7.

Show resolved Hide resolved hail/python/hail/dev-environment-py3.7.yml Outdated
Show resolved Hide resolved hail/python/hail/docs/getting_started_developing.rst Outdated

Think I got everything

@chrisvittal chrisvittal force-pushed the chrisvittal:spark-2.4 branch from e838260 to 1703757 Apr 5, 2019

@danking

danking approved these changes Apr 5, 2019

@danking

This comment has been minimized.

Copy link
Collaborator

commented Apr 5, 2019

ship it

@danking

This comment has been minimized.

Copy link
Collaborator

commented Apr 5, 2019

I added a cloudtools version pin in the cluster tests.

@chrisvittal chrisvittal force-pushed the chrisvittal:spark-2.4 branch 3 times, most recently from 85857ae to ef3b289 Apr 6, 2019

@chrisvittal

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 6, 2019

The new cloudtools sha file needs to exist. I added it.

@chrisvittal chrisvittal force-pushed the chrisvittal:spark-2.4 branch from ef3b289 to ee8f9b4 Apr 6, 2019

@danking

This comment has been minimized.

Copy link
Collaborator

commented Apr 6, 2019

Something is wrong with batch tests ...

@danking

This comment has been minimized.

Copy link
Collaborator

commented Apr 6, 2019

CI is quiet, so I'll retest now.

@danking

This comment has been minimized.

Copy link
Collaborator

commented Apr 6, 2019

I'll have to look into test_callback though.

@chrisvittal

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 8, 2019

I believe that my work in #5813 will resolve the issues that we are seeing here. Please take a look.

chrisvittal added some commits Apr 9, 2019

Update image name
For some reason, the name is now 1.4-debian9 rather than 1.4-deb9 🤷

@chrisvittal chrisvittal force-pushed the chrisvittal:spark-2.4 branch from 7dac8bd to 72a9d9e Apr 9, 2019

@danking danking merged commit f37e25c into hail-is:master Apr 9, 2019

1 check passed

hail-ci-0-1 successful build
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.