Spark version compatibility and compilation error for Spark 2.0.1

[tkakantousis]

Hi all,

I'm trying compile latest master branch with spark 2.0.1 and hadoop 2.7.3 and I'm getting the following error:

[error] impossible to get artifacts when data has not been loaded. IvyNode = com.fasterxml.jackson.core#jackson-databind;2.5.4
java.lang.IllegalStateException: impossible to get artifacts when data has not been loaded. IvyNode = com.fasterxml.jackson.core#jackson-databind;2.5.4

Setting hadoop 2.4.0 and spark 1.6.1. compiles fine though. Could someone point me to documentation stating which spark and hadoop versions are supported? Should I manually set the dependencies and if so how?

Thanks!

[akshayrai]

@tkakantousis, the Spark fetcher is undergoing some major refactoring and we haven't tested it against all the Spark versions. In your case, can you try incrementing the jacksonVersion to 2.5.4 in the Dependencies.scala and then retry?

https://github.com/linkedin/dr-elephant/blob/master/project/Dependencies.scala#L28

[suiyuan2009]

I met the same problem, hadoop2.7.3 and spark 2.0.2, there is no 2.5.4...see mvn repo

[suiyuan2009]

I solve the problem by modify sbt.version to 0.13.9 in project/build.properties, no need to change jacksonVersion to 2.5.4.

[akshayrai]

@suiyuan2009, great! Thanks for posting your solution.

[superbobry]

A related question: is there a reason to use older sbt?

[xor007]

+1 hadoop 2.7.3/spark 2.1.0 (HDP 2.6) even with sbt 0.13.9 could not complete compilation for me