[SPARK-35550][BUILD] Upgrade Jackson to 2.12.3 #32688

LuciferYang · 2021-05-28T02:36:17Z

What changes were proposed in this pull request?

This pr upgrade Jackson version to 2.12.3.
Jackson Release 2.12.3: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.12.3

Why are the changes needed?

Upgrade to a new version to bring potential bug fixes like FasterXML/jackson-modules-java8#207 and avro's master has been upgraded to Jackson to 2.12.3

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass the Jenkins or GitHub Action

SparkQA · 2021-05-28T03:25:32Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43557/

SparkQA · 2021-05-28T05:06:55Z

Test build #139039 has finished for PR 32688 at commit 72c425c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-05-28T05:32:11Z

@LuciferYang, the test failure should be now fixed in the latest master branch.

LuciferYang · 2021-05-28T05:41:50Z

thx @HyukjinKwon

SparkQA · 2021-05-28T06:30:51Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43567/

gengliangwang · 2021-05-28T07:58:28Z

Out of curiosity, what is the actual impact for Spark?

LuciferYang · 2021-05-28T08:08:46Z

Out of curiosity, what is the actual impact for Spark?

There may not be any real benefits at the moment, if we are going to upgrade avro version in the future, we will need upgrade Jackson to this version because the master of avro already used Jackson 2.12.3. Of course, we can wait until really need to upgrade.

gengliangwang · 2021-05-28T08:14:20Z

I see. I would suggest doing the upgrade until we have a better reason for this.

SparkQA · 2021-05-28T08:16:17Z

Test build #139047 has finished for PR 32688 at commit f5bafee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2021-05-29T14:12:06Z

I think it's OK to upgrade - we tend to want Jackson to stay pretty up to date anyway

gengliangwang · 2021-05-29T16:41:06Z

@srowen I see. I am OK to merge this as well.

HyukjinKwon · 2021-05-31T01:28:38Z

Merged to master.

LuciferYang · 2021-05-31T02:34:37Z

thx all ~

dchristle · 2021-06-04T00:27:10Z

Hi all @LuciferYang @HyukjinKwon @gengliangwang @srowen --

While attempting to run a job on k8s with a SNAPSHOT of today's master branch, I found there appears to be a conflict in our dependencies. The error occurs at the start of the job, and the stack trace is:

Exception in thread "main" java.lang.NoSuchMethodError: 'com.fasterxml.jackson.core.util.JacksonFeatureSet com.fasterxml.jackson.core.JsonParser.getReadCapabilities()'
	at com.fasterxml.jackson.databind.DeserializationContext.<init>(DeserializationContext.java:211)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.<init>(DefaultDeserializationContext.java:50)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext$Impl.<init>(DefaultDeserializationContext.java:391)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext$Impl.createInstance(DefaultDeserializationContext.java:413)
	at com.fasterxml.jackson.databind.ObjectMapper.createDeserializationContext(ObjectMapper.java:4656)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4585)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3593)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:247)
	at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:200)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:495)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:416)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:397)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:933)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:220)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:186)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:84)
...

The root cause may be that kubernetes-client has a dependency on Jackson 2.11.2: https://github.com/fabric8io/kubernetes-client/blob/d810aa1da99ff5cdb81cabea4f5a6dcf55ff0218/pom.xml#L87

When it initializes DeserializationContext, getReadCapabilities() is called here https://github.com/FasterXML/jackson-databind/blob/b5a60a816b9333edbba830bb252a1417dcd2686b/src/main/java/com/fasterxml/jackson/databind/DeserializationContext.java#L212

but that method has only existed since 2.12: https://github.com/FasterXML/jackson-core/blob/080eee156d82481625a38eda5a14ba4fbe2c4a46/src/main/java/com/fasterxml/jackson/core/JsonParser.java#L604

I will report back if downgrading the pom to 2.11.4 resolves the issue.

HyukjinKwon · 2021-06-04T00:37:13Z

This upgrades from 2.12.2 to 2.12.3 so the compatibility w/ 2.11 is unlikely from this PR. Does that happen after this PR?

HyukjinKwon · 2021-06-04T00:38:36Z

It's likely from #31878 if it's an issue. cc @wangyum and @xclyfe

HyukjinKwon · 2021-06-04T00:38:49Z

cc @dongjoon-hyun too FYI

dongjoon-hyun · 2021-06-04T01:38:28Z

Thank you for pinging me!

dongjoon-hyun · 2021-06-04T01:43:38Z

BTW, Apache Spark K8s IT has been healthy.
Maybe, it seems that we need to add some test coverage for the failed API.

dongjoon-hyun · 2021-06-04T01:45:13Z

@dchristle Could you elaborate about the following a little bit more?

The error occurs at the start of the job,

dchristle · 2021-06-04T01:49:03Z

@dongjoon-hyun Sure - the error triggers right after the job jar is added, i.e. there are no stages that start, and it's so early that the Spark UI is not accessible.

I tried rebuilding my Docker image from scratch and the error persists. I am using the GCS connector and Snowflake connectors for this job, which are technically only on 3.1. I am not sure how to debug whether there is something specific about my build (aka a problem locally) or if there truly is a dependency conflict in the master branch.

dongjoon-hyun · 2021-06-04T01:51:57Z

Thank you for the detail. Let me build the vanilla Spark distribution from the master branch manually and play with K8s a little. I'll post the result here.

dongjoon-hyun · 2021-06-04T01:55:38Z

BTW, this is the community Jenkins success result with this PR

[SPARK-35550][BUILD] Upgrade Jackson to 2.12.3

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20K8s%20Builds/job/spark-master-test-k8s/792/

dongjoon-hyun · 2021-06-04T02:29:00Z

I am able to build master branch and run TPCDS without any issue.

$ k get pod | head
NAME                  READY   STATUS    RESTARTS   AGE
tpcds-dhyun           1/1     Running   0          2m37s
tpcds-dhyun-exec-1    1/1     Running   0          2m3s
tpcds-dhyun-exec-10   1/1     Running   0          2m2s
tpcds-dhyun-exec-11   1/1     Running   0          2m1s
tpcds-dhyun-exec-12   1/1     Running   0          2m1s
tpcds-dhyun-exec-13   1/1     Running   0          2m1s
tpcds-dhyun-exec-14   1/1     Running   0          2m1s
tpcds-dhyun-exec-15   1/1     Running   0          2m1s
tpcds-dhyun-exec-16   1/1     Running   0          2m
...

$ k exec -it tpcds-dhyun -- /bin/bash
185@tpcds-dhyun:~/work-dir$ cd ../jars/
185@tpcds-dhyun:~/jars$ ls jackson*
jackson-annotations-2.12.3.jar	jackson-databind-2.12.3.jar	    jackson-datatype-jsr310-2.11.2.jar
jackson-core-2.12.3.jar		jackson-dataformat-cbor-2.12.3.jar  jackson-mapper-asl-1.9.13.jar
jackson-core-asl-1.9.13.jar	jackson-dataformat-yaml-2.12.3.jar  jackson-module-scala_2.12-2.12.3.jar
185@tpcds-dhyun:~/jars$ ls kube*
kubernetes-client-5.4.0.jar			  kubernetes-model-common-5.4.0.jar	   kubernetes-model-metrics-5.4.0.jar
kubernetes-model-admissionregistration-5.4.0.jar  kubernetes-model-coordination-5.4.0.jar  kubernetes-model-networking-5.4.0.jar
kubernetes-model-apiextensions-5.4.0.jar	  kubernetes-model-core-5.4.0.jar	   kubernetes-model-node-5.4.0.jar
kubernetes-model-apps-5.4.0.jar			  kubernetes-model-discovery-5.4.0.jar	   kubernetes-model-policy-5.4.0.jar
kubernetes-model-autoscaling-5.4.0.jar		  kubernetes-model-events-5.4.0.jar	   kubernetes-model-rbac-5.4.0.jar
kubernetes-model-batch-5.4.0.jar		  kubernetes-model-extensions-5.4.0.jar    kubernetes-model-scheduling-5.4.0.jar
kubernetes-model-certificates-5.4.0.jar		  kubernetes-model-flowcontrol-5.4.0.jar   kubernetes-model-storageclass-5.4.0.jar

dongjoon-hyun · 2021-06-04T02:30:00Z

Given the above, could you share your docker image's jar directory, @dchristle ? Do you have both Jackson 2.11.x and 2.12.3?

dongjoon-hyun · 2021-06-04T02:33:44Z

Or, is your Spark app using kubernetes-client library directly for some extension? It would be great if you can provide an reproducible example to us. We want to help you because K8s is GA in Apache Spark community. :)

dchristle · 2021-06-08T18:00:43Z

Or, is your Spark app using kubernetes-client library directly for some extension? It would be great if you can provide an reproducible example to us. We want to help you because K8s is GA in Apache Spark community. :)

That was the root cause. I refactored my workflow to compile Spark from the master branch inside my Docker image, so I could be sure the environment was clean. Replacing the GCS jar I was using with the -shaded variant seems to have fixed the problem, which suggests that jar was pulling in a slightly older incompatible Jackson version. Thank you for the help!

LuciferYang · 2021-06-09T10:38:26Z

@dongjoon-hyun @HyukjinKwon Thank you for your help, busy with release spark 3.1.x used in our company recently, haha

Upgrade Jackson to 2.12.3

72c425c

github-actions bot added the BUILD label May 28, 2021

Merge branch 'upmaster' into SPARK-35550

f5bafee

HyukjinKwon approved these changes May 31, 2021

View reviewed changes

HyukjinKwon closed this in ff27264 May 31, 2021

wangyum mentioned this pull request Jun 4, 2021

Upgrade Jackson to 2.12.3 fabric8io/kubernetes-client#3213

Closed

LuciferYang deleted the SPARK-35550 branch October 22, 2023 07:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-35550][BUILD] Upgrade Jackson to 2.12.3 #32688

[SPARK-35550][BUILD] Upgrade Jackson to 2.12.3 #32688

LuciferYang commented May 28, 2021 •

edited

Loading

SparkQA commented May 28, 2021

SparkQA commented May 28, 2021

HyukjinKwon commented May 28, 2021 •

edited

Loading

LuciferYang commented May 28, 2021

SparkQA commented May 28, 2021

gengliangwang commented May 28, 2021

LuciferYang commented May 28, 2021

gengliangwang commented May 28, 2021

SparkQA commented May 28, 2021

srowen commented May 29, 2021

gengliangwang commented May 29, 2021

HyukjinKwon commented May 31, 2021

LuciferYang commented May 31, 2021

dchristle commented Jun 4, 2021 •

edited

Loading

HyukjinKwon commented Jun 4, 2021

HyukjinKwon commented Jun 4, 2021

HyukjinKwon commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dchristle commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dchristle commented Jun 8, 2021

LuciferYang commented Jun 9, 2021 •

edited

Loading

[SPARK-35550][BUILD] Upgrade Jackson to 2.12.3 #32688

[SPARK-35550][BUILD] Upgrade Jackson to 2.12.3 #32688

Conversation

LuciferYang commented May 28, 2021 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented May 28, 2021

SparkQA commented May 28, 2021

HyukjinKwon commented May 28, 2021 • edited Loading

LuciferYang commented May 28, 2021

SparkQA commented May 28, 2021

gengliangwang commented May 28, 2021

LuciferYang commented May 28, 2021

gengliangwang commented May 28, 2021

SparkQA commented May 28, 2021

srowen commented May 29, 2021

gengliangwang commented May 29, 2021

HyukjinKwon commented May 31, 2021

LuciferYang commented May 31, 2021

dchristle commented Jun 4, 2021 • edited Loading

HyukjinKwon commented Jun 4, 2021

HyukjinKwon commented Jun 4, 2021

HyukjinKwon commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dchristle commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dongjoon-hyun commented Jun 4, 2021

dchristle commented Jun 8, 2021

LuciferYang commented Jun 9, 2021 • edited Loading

LuciferYang commented May 28, 2021 •

edited

Loading

HyukjinKwon commented May 28, 2021 •

edited

Loading

dchristle commented Jun 4, 2021 •

edited

Loading

LuciferYang commented Jun 9, 2021 •

edited

Loading