Upgrade the json4s version

[moranmathias]

The new version of json4s adds support to parse Map[String,Any] out the box. This is a very useful feature that i've been using in my custom version of spark.

[srowen]

Probably OK. Often we want a JIRA too to discuss the change but this is so small. Are you aware of any incompatibilities? I doubt it in a maintenance release, but worth researching in the commit log briefly.

[srowen]

ok to test

[pwendell]

@moranmathias Do you mind creating a JIRA for this? It's good to have an audit trail of any version upgrades, since sometimes they cause maintenance issues downstream

[SparkQA]

Test build #30939 has finished for PR 5691 at commit 1e79f5e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch adds the following new dependencies:
  • json4s-ast_2.10-3.2.11.jar
  • json4s-core_2.10-3.2.11.jar
  • json4s-jackson_2.10-3.2.11.jar
• This patch removes the following dependencies:
  • json4s-ast_2.10-3.2.10.jar
  • json4s-core_2.10-3.2.10.jar
  • json4s-jackson_2.10-3.2.10.jar

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

@moranmathias any updates? I think we just need some confidence that this is compatible to the best of our knowledge.

Do you need this in your code only? you shouldn't be depending on Spark for json4s, note. If so then mind closing this PR?

[wb14123]

Any update on this? I think this is a very useful feature. I searched on the web and there are many work around for this. It is good to see this to be merged.

[srowen]

@wb14123 I don't think the outstanding issues were resolved? you can see the discussion here.

[wb14123]

@srowen Sorry, I don't understand what do you mean. The discussion here seems that the developer don't follow up. But the issue is still here and I think it is important. Do I need to create a JIRA for this?

You said we shouldn't be depending on Spark for json4s. But I'm about to use json4s 3.2.11 in my spark application, I also write it as a dependency in my project, but while I submit the application as a JAR with spark-submit, the spark's json4s just override my own version of json4s.

[srowen]

@wb14123 This issue concerns updating Spark's json4s. If you want to pursue that, you can open a PR, but the question will be the same: does it cause any compatibility problem?

However your purpose is simply to use json4s in your app. It's not shaded in Spark. You need to use the 'userClassPathFirst' options in Spark to make your copy take precedence. This does not involve updating Spark.

[wb14123]

"userClassPathFirst" is marked "Experimental" in the doc for now. And it will use other library versions in my project like akka-slf4j which is incompatible with other akka libraries in Spark, and so on.

But maybe I should cleanup my project's dependency to find out which versions do I really need. Thanks for the explain.