Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix spark3 version #9

Merged
merged 1 commit into from
Jan 18, 2022
Merged

fix spark3 version #9

merged 1 commit into from
Jan 18, 2022

Conversation

rccern
Copy link
Contributor

@rccern rccern commented Jan 18, 2022

No description provided.

@rccern rccern merged commit 056f653 into master Jan 18, 2022
@rccern rccern deleted the spark32 branch January 18, 2022 09:58
@rahul26goyal
Copy link

@krishnan-r why was the version reverted?

@krishnan-r
Copy link
Contributor

Hi @rahul26goyal,

The sparklistener jar bundled with sparkmonitor for Spark 3.2 had a conflict with the version of json4s packaged in the jar and the one provided in the spark distribution.

To fix that we shaded json4s in the jar in #6, this allows supporting all the spark 3.x versions with a single JAR file.

When doing that we built the JAR using spark 3.2 as a dependency. However this broke earlier versions because the JAR built for 3.2 was not backward compatible, whereas the jar built for 3.0.1 works fine with 3.2 (if I remember correctly).

If this does not work with 3.2, we can perhaps have a separate JAR built and packaged with 3.2, similar to what we do between 2.x and 3.x. But this involves some more work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants