[query] Upgrade spark to 3.3.0 and dataproc to 2.1#12701
Merged
Conversation
daniel-goldstein
force-pushed
the
dataproc-2-1
branch
from
February 16, 2023 01:49
15d1aef to
d8c3a23
Compare
daniel-goldstein
force-pushed
the
dataproc-2-1
branch
from
February 16, 2023 19:34
5cb38e9 to
ba6e1d8
Compare
daniel-goldstein
marked this pull request as ready for review
February 16, 2023 19:41
danking
previously requested changes
Feb 16, 2023
| @@ -20,25 +20,22 @@ def init_parser(parser): | |||
|
|
|||
|
|
|||
| async def async_main(args): | |||
Contributor
There was a problem hiding this comment.
why make these changes? Do we report the username in the underlying exception?
Contributor
Author
There was a problem hiding this comment.
The linter complained about using too generic an error class, it wants us to subclass Exception. I removed the try except because I felt like it wasn't actually any more helpful than whatever stack trace was underneath, but I didn't think about the username not being in there. Can add that back in if you want.
| protobuf==3.20.2 | ||
| PyJWT | ||
| pyspark>=3.1.1,<3.2.0 | ||
| pyspark==3.3.0 |
Contributor
There was a problem hiding this comment.
Seems reasonable to do pyspark>=3.3,<3.4?
Contributor
Author
There was a problem hiding this comment.
I guess I was distrusting of pyspark and wanted the exact same version that there is in dataproc, but maybe that is unfounded.
danking
approved these changes
Feb 16, 2023
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CHANGELOG: Query on Spark now officially supports Spark 3.3.0 and Dataproc 2.1.x
Tested on dataproc via
make -C hail test-dataproc-37. Updating the dependencies introduced a few new linting checks that I fixed here. Updating pyspark necessitated a couple of changes, namely a different py4j jar and they removedSparkSession._wrapped(but maybe we didn't need that anyway? not sure). Most importantly, the newer spark version brings with it a newer jackson version which is sufficient for the azure-storage-blob dependency, meaning we don't need to build against two different spark versions for spark and batch.