[SPARK-25385][BUILD] Upgrade Hadoop 3.1 jackson version to 2.7.8

[wangyum]

What changes were proposed in this pull request?

Upgrade Hadoop 3.1 jackson version to 2.7.8 to fix JsonMappingException: Incompatible Jackson version: 2.7.8 as Hadoop 3.1 using 2.7.8: https://github.com/apache/hadoop/blob/release-3.1.0-RC1/hadoop-project/pom.xml#L72 .

How was this patch tested?

manual tests:

export SPARK_PREPEND_CLASSES=true
build/sbt clean package -Phadoop-3.1
spark-shell
scala> spark.range(10).write.mode("overwrite").parquet("/tmp/spark/parquet")

[SparkQA]

Test build #95846 has finished for PR 22372 at commit f68083a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[felixcheung]

LGTM, changes are only for 3.1 profile

[felixcheung]

do we have jenkins tests for 3.1 profile?

[wangyum]

We do not have jenkins tests for 3.1 profile:

spark/dev/run-tests.py

Lines 307 to 310 in 395860a

	sbt_maven_hadoop_profiles = {
	"hadoop2.6": ["-Phadoop-2.6"],
	"hadoop2.7": ["-Phadoop-2.7"],
	}

[HyukjinKwon]

@wangyum, are you doubly sure if SBT picks up the same Jackson? I roughly tried this a bit of while ago and found SBT doesn't pick up Maven's IIRC.

[HyukjinKwon]

Also, IIRC, #21596 change is needed for this Jackson upgrade.

[HyukjinKwon]

Also, I think we should fix #21588 first.

[HyukjinKwon]

Please test and check SBT's dependency tree as well.

[wangyum]

build/sbt dependency-tree -Phadoop-3.1

[wangyum]

We should clean first for package:

build/sbt clean package -Phadoop-3.1

and then check: assembly/target/scala-2.11/jars/:

[jerryshao]

Jackson version below 2.9.5 has CVE issues, I would suggest to upgrade to 2.9.6 as #21596 did.

[wangyum]

I did a simple test for 2.9.6. It works well. But that pr for 3.0. It means that a simple test on branch 2.4 will fail:

scala> spark.range(10).write.parquet("/tmp/spark/parquet")
com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.7.8
  at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
  at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
  at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:730)
  at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
  at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)

How about merge this pr to branch-2.4 only?

[jerryshao]

Do we officially support hadoop3 in branch 2.4? If branch 2.4 doesn't target to support Hadoop3 and this fix is only for Hadoop3, then I don't think it is meaningful to have this fix.

[jerryshao]

Btw, I don't think we can run current Spark with Hadoop 3.1 without any change.