[SPARK-54597][BUILD] Upgrade lz4-java to 1.10.0

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to upgrade lz4-java to 1.10.0 and exclude the legacy groupID version.

Why are the changes needed?

Since lz4-java changed its repository, we had better depend on the live repository for future maintenance.

• https://github.com/lz4/lz4-java (Archived with the last release on 2021-06-19)
  • https://github.com/lz4/lz4-java/releases/tag/1.8.0 (2021-06-19)
• https://github.com/yawkat/lz4-java (New Fork)
  • https://github.com/yawkat/lz4-java/releases/tag/v1.9.0
    • Update lz4 to v1.10.0
  • https://github.com/yawkat/lz4-java/releases/tag/v1.10.0
    • Allow to create LZ4BlockInputStream with LZ4SafeDecompressor yawkat/lz4-java#3

Does this PR introduce any user-facing change?

No Spark behavior change.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

This is a dependency-only PR, cc @dbtsai , @HyukjinKwon , @LuciferYang , @yawkat , @SteNicholas .

To be clear, the security issue is not a scope of this PR.

[dongjoon-hyun]

Thank you, @HyukjinKwon . I'm going to add one more commit to ban this library explicitly.

[dongjoon-hyun]

All tests passed at the first commit and the subsequent two comments are only about excluding the transitive dependencies to make it sure.

• https://github.com/dongjoon-hyun/spark/actions/runs/19946487321

[HyukjinKwon]

LGTM

[SteNicholas]

@dongjoon-hyun, does it still need to switch fastDecompressor to safeDecompressor after upgrade?

[LuciferYang]

spark/LICENSE-binary

Line 391 in 212d0a7

org.lz4:lz4-java

spark/NOTICE-binary

Lines 205 to 208 in eaaa7ff

	* LICENSE:
	* license/LICENSE.lz4.txt (Apache License 2.0)
	* HOMEPAGE:
	* https://github.com/jpountz/lz4-java

Should the LICENSE-binary and NOTICE-binary be updated?

[dongjoon-hyun]

@dongjoon-hyun, does it still need to switch fastDecompressor to safeDecompressor after upgrade?

Exactly, that's @dbtsai 's contribution, @SteNicholas . This PR doesn't aim to do that. He will rebase his PR after merging this independently.

[dongjoon-hyun]

Thank you, @LuciferYang ! I'll update it.

[SteNicholas]

@dongjoon-hyun, I just confirm whether to switch fastDecompressor to safeDecompressor after upgrade to 1.10.0.

[dongjoon-hyun]

I just confirm whether to switch fastDecompressor to safeDecompressor after upgrade to 1.10.0.

@SteNicholas What I can say here is that it's beyond of this PR. Technically, we don't know what decision we are going to make eventually on the following yet because it's still Draft.

• [WIP][SPARK-54571][CORE] Use LZ4 safeDecompressor #53290

[dongjoon-hyun]

Please don't get me wrong. I'm trying to help that PR move forward by reducing the gap.

[LuciferYang]

LGTM ~

[dongjoon-hyun]

Thank you all! Merged to master for Apache Spark 4.2.0 (for now)