Skip to content

[SPARK-56318][BUILD][4.1] Upgrade jackson to 2.21.2#55135

Closed
manuzhang wants to merge 3 commits into
apache:branch-4.1from
manuzhang:upgrade-jackson-spark4.1
Closed

[SPARK-56318][BUILD][4.1] Upgrade jackson to 2.21.2#55135
manuzhang wants to merge 3 commits into
apache:branch-4.1from
manuzhang:upgrade-jackson-spark4.1

Conversation

@manuzhang
Copy link
Copy Markdown
Member

@manuzhang manuzhang commented Apr 1, 2026

What changes were proposed in this pull request?

Fixes vulnerability GHSA-72hv-8253-57qq

Why are the changes needed?

jackson-core 2.20.0 is affected.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests.

Was this patch authored or co-authored using generative AI tooling?

No.

@manuzhang
Copy link
Copy Markdown
Member Author

manuzhang commented Apr 2, 2026

@dongjoon-hyun @pan3793 Please help take a look, thanks!

@pan3793
Copy link
Copy Markdown
Member

pan3793 commented Apr 2, 2026

We generally don't upgrade minor versions of dependencies in the release branches. Is it possible to have a patched Jackson 2.20.x? Or is Jackson 2.21 fully compatible(e.g., no default behavior change, no removal of deprecated API) with 2.20?

cc @pjfanning, could you provide some info?

@manuzhang
Copy link
Copy Markdown
Member Author

manuzhang commented Apr 2, 2026

@pan3793 Looking at the release notes, there is only one patch between 2.20.0 and 2.21. Also, 2.21.1 is the patched version for affected versions >= 2.19.0, < 2.21.1.

@pan3793
Copy link
Copy Markdown
Member

pan3793 commented Apr 2, 2026

@manuzhang Jackson has a dozen code repos ... the CI failure is likely caused by jackson-module-scala pulling a new Scala version.

@manuzhang manuzhang force-pushed the upgrade-jackson-spark4.1 branch 2 times, most recently from 655b338 to 9bbb1db Compare April 2, 2026 10:14
@pjfanning
Copy link
Copy Markdown
Member

pjfanning commented Apr 2, 2026

Jackson 2.21 is LTS while 2.20 is not. 2.21.3 has been released.

@pan3793
Copy link
Copy Markdown
Member

pan3793 commented Apr 2, 2026 •
edited
Loading

Jackson 2.21 is LTS while 2.20 is not.

Alright, I used to think that Jackson's minor versions have the same support policy ...

cc @holdenk @dongjoon-hyun, do we want to accept such an upgrade for branch-4.1?

@manuzhang
Copy link
Copy Markdown
Member Author

manuzhang commented Apr 14, 2026

@pan3793 @holdenk @dongjoon-hyun gentle ping. What are your thoughts on this upgrade?

@holdenk
Copy link
Copy Markdown
Contributor

holdenk commented Apr 14, 2026

Seems like a reasonable upgrade but lets address the CI issue.

@manuzhang
Copy link
Copy Markdown
Member Author

manuzhang commented Apr 14, 2026

@holdenk Thanks for quick response. I don't think the CI failure is related.

/__w/spark/spark/docs/_plugins/build_api_docs.rb:177:in `build_r_docs': R doc generation failed (RuntimeError)
	from /__w/spark/spark/docs/_plugins/build_api_docs.rb:231:in `<top (required)>'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/external.rb:57:in `require'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/external.rb:57:in `block in require_with_graceful_fail'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/external.rb:55:in `each'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/external.rb:55:in `require_with_graceful_fail'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/plugin_manager.rb:96:in `block in require_plugin_files'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/plugin_manager.rb:94:in `each'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/plugin_manager.rb:94:in `require_plugin_files'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/plugin_manager.rb:21:in `conscientious_require'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/site.rb:131:in `setup'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/site.rb:36:in `initialize'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/commands/build.rb:30:in `new'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/commands/build.rb:30:in `process'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/command.rb:91:in `block in process_with_graceful_fail'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/command.rb:91:in `each'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/command.rb:91:in `process_with_graceful_fail'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `block in execute'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `each'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/mercenary-0.4.0/lib/mercenary/command.rb:221:in `execute'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/mercenary-0.4.0/lib/mercenary/program.rb:44:in `go'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/mercenary-0.4.0/lib/mercenary.rb:21:in `program'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/gems/jekyll-4.4.1/exe/jekyll:15:in `<top (required)>'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/bin/jekyll:25:in `load'
	from /__w/spark/spark/docs/.local_ruby_bundle/ruby/3.0.0/bin/jekyll:25:in `<main>'
Error: Process completed with exit code 1.

@manuzhang manuzhang force-pushed the upgrade-jackson-spark4.1 branch from 1b5bc71 to 51869cb Compare April 14, 2026 08:39
holdenk
holdenk reviewed Apr 15, 2026
View reviewed changes
Comment thread common/utils/pom.xml
Comment on lines +65 to +70
<exclusions>
<exclusion>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</exclusion>
</exclusions>
Copy link
Copy Markdown
Contributor

@holdenk holdenk Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes me anxious, if jackson is depending on a higher version of the scala-library we probably need to bump our scala version not exclude their scala library otherwise I suspect we would see weirdness in runtime with Jackson that will be difficult to track down. I'm open to the idea i'm wrong though.

Copy link
Copy Markdown
Member Author

@manuzhang manuzhang Apr 16, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just 2.13.18 and 2.13.17. We can remove the exclusion once they are aligned but I don't think it's blocking.

Copy link
Copy Markdown
Member

@pan3793 pan3793 Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be fine, as the commit history of jackson-module-scala does not indicate any incompatible issues between the scala 2.13.18 and 2.13.17

@pan3793
Copy link
Copy Markdown
Member

pan3793 commented Apr 16, 2026

I will merge this after CI is back to normal

@manuzhang
Copy link
Copy Markdown
Member Author

manuzhang commented Apr 21, 2026

@pan3793 The CI passes now. Do you have more comments?

@pan3793
Copy link
Copy Markdown
Member

pan3793 commented Apr 21, 2026

@manuzhang The branch-4.1 and branch-4.0 CI builds are broken, and the recent fixes on master haven't been fully backported.

pan3793 pushed a commit that referenced this pull request Apr 21, 2026
@manuzhang @pan3793
[SPARK-56318][BUILD][4.1] Upgrade jackson to 2.21.2
b0bd64e 
### What changes were proposed in this pull request?

Fixes vulnerability GHSA-72hv-8253-57qq

### Why are the changes needed?

jackson-core 2.20.0 is affected.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing tests.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #55135 from manuzhang/upgrade-jackson-spark4.1.

Authored-by: Manu Zhang <OwenZhang1990@gmail.com>
Signed-off-by: Cheng Pan <chengpan@apache.org>
@pan3793
Copy link
Copy Markdown
Member

pan3793 commented Apr 21, 2026

thanks, merged to branch-4.1 for 4.1.2

@pan3793 pan3793 closed this Apr 21, 2026
@manuzhang manuzhang deleted the upgrade-jackson-spark4.1 branch April 21, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@holdenk holdenk holdenk left review comments

@pan3793 pan3793 pan3793 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@manuzhang @pan3793 @pjfanning @holdenk