Bump Spark, Scala, Hadoop dependency versions #1221

Closed
heuermh opened this Issue Oct 23, 2016 · 4 comments

Comments

Projects
3 participants
@heuermh
Member

heuermh commented Oct 23, 2016

Apache Spark version 2.0.1 was released on 03 Oct 2016 and defaults to Scala version 2.11.8 and Hadoop 2.7 and later.

Apache Spark version 1.6.2 was released on 25 June 2016 and defaults to Scala version 2.10.5 and Hadoop 2.6. When bumping the Scala version to 2.10.5 downstream I see other libraries requiring 2.10.6.

@heuermh heuermh added the discussion label Oct 23, 2016

@DruidSmith

This comment has been minimized.

Show comment
Hide comment
@DruidSmith

DruidSmith Oct 25, 2016

Is ADAM working on Spark 2.0? Has anyone looked at running ADAM atop AWS EMR?

Is ADAM working on Spark 2.0? Has anyone looked at running ADAM atop AWS EMR?

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Oct 25, 2016

Member

Is ADAM working on Spark 2.0?

Yes, as of version 0.20.0, released last week.

Member

heuermh commented Oct 25, 2016

Is ADAM working on Spark 2.0?

Yes, as of version 0.20.0, released last week.

@heuermh

This comment has been minimized.

Show comment
Hide comment

@fnothaft fnothaft added this to the 0.21.0 milestone Nov 8, 2016

fnothaft added a commit to fnothaft/adam that referenced this issue Nov 8, 2016

fnothaft added a commit to fnothaft/adam that referenced this issue Nov 9, 2016

fnothaft added a commit to fnothaft/adam that referenced this issue Nov 9, 2016

@heuermh heuermh closed this in #1254 Nov 10, 2016

heuermh added a commit that referenced this issue Nov 10, 2016

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Nov 10, 2016

Member

Sigh! Reopening this issue. See comment; TL;DR is that #1254 spuriously passed but the move_to_spark2.sh script is broken now on ToT. If no one else is interested, I'll hack on this in the AM.

Member

fnothaft commented Nov 10, 2016

Sigh! Reopening this issue. See comment; TL;DR is that #1254 spuriously passed but the move_to_spark2.sh script is broken now on ToT. If no one else is interested, I'll hack on this in the AM.

@fnothaft fnothaft reopened this Nov 10, 2016

fnothaft added a commit to fnothaft/adam that referenced this issue Nov 10, 2016

fnothaft added a commit to fnothaft/adam that referenced this issue Nov 10, 2016

@heuermh heuermh closed this in 1b72f0b Nov 10, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment