New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NoSuchMethodError due to kryo minor-version mismatch #955

Closed
ryan-williams opened this Issue Feb 22, 2016 · 4 comments

Comments

Projects
3 participants
@ryan-williams
Member

ryan-williams commented Feb 22, 2016

Not sure we want to or should do anything about this, but documenting it fwiw.

I have some code that depends on adam-core_2.10-0.18.2, and from that picks up a dependency on Kryo 2.24. Meanwhile, spark-core 1.6.0 depends on Chill 0.5.0 which depends on Kryo 2.21.

My code calls Input.total which is a long in Kryo 2.24 but an int in Kryo 2.21, so I get a NoSuchMethodError because my code was compiled against 2.24 but picks up Spark's 2.21 at runtime.

I'm going to look into whether this can be mitigated by one of these "user classpath first" options that I've seen discussed around Spark-land.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Feb 22, 2016

Member

At a quick glance it doesn't appear we declare a runtime or compile time dependency on kyro, even though we define a version for it in the <dependencyManagement> section. Maybe adding a runtime scope to your pom or to say adam-cli would override the Spark runtime dependency version?

Member

heuermh commented Feb 22, 2016

At a quick glance it doesn't appear we declare a runtime or compile time dependency on kyro, even though we define a version for it in the <dependencyManagement> section. Maybe adding a runtime scope to your pom or to say adam-cli would override the Spark runtime dependency version?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Feb 22, 2016

Member

Maybe adding a runtime scope to your pom or to say adam-cli would override the Spark runtime dependency version?

IIRC, it does, but you need to muck around with classpaths in Spark submit, or something of that sort. It's a bit ugly of a bug. I ran into this or something similar on a pre-1.6 custom build.

Member

fnothaft commented Feb 22, 2016

Maybe adding a runtime scope to your pom or to say adam-cli would override the Spark runtime dependency version?

IIRC, it does, but you need to muck around with classpaths in Spark submit, or something of that sort. It's a bit ugly of a bug. I ran into this or something similar on a pre-1.6 custom build.

@fnothaft fnothaft added this to the 0.20.0 milestone Jul 20, 2016

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Sep 8, 2016

Member

Is there anything actionable on this for the 0.20.0 release?

Member

heuermh commented Sep 8, 2016

Is there anything actionable on this for the 0.20.0 release?

@heuermh heuermh modified the milestones: 0.20.0, 0.22.0 Oct 13, 2016

@heuermh heuermh added this to Triage in Release 0.23.0 Mar 8, 2017

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft May 11, 2017

Member

I don't think there's anything actionable for this on 0.23.0. I'm going to close this as won't fix. @ryan-williams please reopen if you think I am in error.

Member

fnothaft commented May 11, 2017

I don't think there's anything actionable for this on 0.23.0. I'm going to close this as won't fix. @ryan-williams please reopen if you think I am in error.

@fnothaft fnothaft closed this May 11, 2017

@fnothaft fnothaft removed this from the 0.23.0 milestone May 11, 2017

@fnothaft fnothaft added the wontfix label May 11, 2017

@fnothaft fnothaft removed this from Triage in Release 0.23.0 May 11, 2017

@heuermh heuermh modified the milestone: 0.23.0 Jul 22, 2017

@heuermh heuermh added this to Completed in Release 0.23.0 Jan 4, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment