Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Spark 2.4 #5756

Merged
merged 18 commits into from Apr 9, 2019
Merged

Update to Spark 2.4 #5756

merged 18 commits into from Apr 9, 2019

Conversation

@chrisvittal
Copy link
Collaborator

@chrisvittal chrisvittal commented Apr 3, 2019

I have no idea if I've updated everything I need to update here. I haven't pushed the updated CI
build image or changed cloudtools (yet).

Update: pushed ci build image.

cc @cseed @danking

hail/build.gradle Outdated Show resolved Hide resolved
Copy link
Collaborator

@tpoterba tpoterba left a comment

Need changes in getting_started.rst and hail/hail-ci-deploy.sh

hail/build.gradle Show resolved Hide resolved
Copy link
Collaborator

@danking danking left a comment

We can do a two-stage change to get this in. Arcturus added a configuration file to cloudtools that permits this.

https://github.com/Nealelab/cloudtools/pull/90/files#diff-8e7f8e85397f28e54901d3f7eb930902R137

Change CLOUDTOOLS_VERSION in hail-ci-deploy.sh to 4. This will prevent 3.x cloudtools from using any published hail jars. Then go update cloudtools at your leisure, bumping to 4.0.0. Once that's published, people will need to update cloudtools. They'll get a message to that effect.

hail/build.gradle Outdated Show resolved Hide resolved
@@ -17,8 +17,8 @@ Regardless of installation method, you will need:
<https://www.anaconda.com/download/>`_
Copy link
Collaborator

@danking danking Apr 3, 2019

check getting_started_developing.rst too

@konradjk
Copy link
Collaborator

@konradjk konradjk commented Apr 3, 2019

Yeah, is this going to deploy new jars only on 2.4 or also 2.2 for a while? This is good in the long run, but might be nice to keep 2.2 going in case there are any unintended consequences

@danking
Copy link
Collaborator

@danking danking commented Apr 3, 2019

this will only do 2.4, but it's not much work to continue to provide both

@danking
Copy link
Collaborator

@danking danking commented Apr 3, 2019

it looks like there's no source code changes

@chrisvittal chrisvittal dismissed danking’s stale review Apr 3, 2019

I think I got everything. We just need to update cloudtools now.

@tpoterba
Copy link
Collaborator

@tpoterba tpoterba commented Apr 4, 2019

Dan, since you have a much better handle on how to get this in, can I give primary responsibility for this review to you?

I'm happy to accept a PR review in return.

@danking danking assigned danking and unassigned tpoterba Apr 5, 2019
@danking
Copy link
Collaborator

@danking danking commented Apr 5, 2019

accepted

if (sparkVersion == '2.2.0')
py4jVersion = '0.10.4'
else if (sparkVersion == '2.3.0')
py4jVersion = '0.10.6'
Copy link
Collaborator

@danking danking Apr 5, 2019

what about 2.2.1 and 2.3.1 and friends? Should we truncate to the first three characters of sparkVersion for these checks?

Copy link
Collaborator Author

@chrisvittal chrisvittal Apr 5, 2019

These are the only ones that are different. Spark 2.2.1 is using 0.10.7, Spark 2.3.1 is using 0.10.7.

hail/python/hail/dev-environment-py3.7.yml Outdated Show resolved Hide resolved
hail/python/hail/docs/getting_started_developing.rst Outdated Show resolved Hide resolved
danking
danking approved these changes Apr 5, 2019
@danking
Copy link
Collaborator

@danking danking commented Apr 5, 2019

ship it

@danking danking closed this in #139 Apr 5, 2019
@danking danking reopened this Apr 5, 2019
@danking
Copy link
Collaborator

@danking danking commented Apr 5, 2019

I added a cloudtools version pin in the cluster tests.

@chrisvittal chrisvittal force-pushed the spark-2.4 branch 3 times, most recently from 85857ae to ef3b289 Apr 6, 2019
@chrisvittal
Copy link
Collaborator Author

@chrisvittal chrisvittal commented Apr 6, 2019

The new cloudtools sha file needs to exist. I added it.

@danking
Copy link
Collaborator

@danking danking commented Apr 6, 2019

Something is wrong with batch tests ...

@danking
Copy link
Collaborator

@danking danking commented Apr 6, 2019

CI is quiet, so I'll retest now.

@danking
Copy link
Collaborator

@danking danking commented Apr 6, 2019

I'll have to look into test_callback though.

@chrisvittal
Copy link
Collaborator Author

@chrisvittal chrisvittal commented Apr 8, 2019

I believe that my work in #5813 will resolve the issues that we are seeing here. Please take a look.

chrisvittal added 2 commits Apr 9, 2019
For some reason, the name is now 1.4-debian9 rather than 1.4-deb9 🤷
@danking danking merged commit f37e25c into hail-is:master Apr 9, 2019
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants