[FLINK-34955] Upgrade commons-compress to 1.26.0.

[slfan1989]

What is the purpose of the change

JIRA: [FLINK-34955] Upgrade commons-compress to 1.26.0.

commons-compress 1.24.0 has CVE issues, try to upgrade to 1.26.0, we can refer to the maven link

https://mvnrepository.com/artifact/org.apache.commons/commons-compress

Brief change log

(for example:)

• The TaskInfo is stored in the blob store on job creation time as a persistent artifact
• Deployments RPC transmits only the blob storage reference
• TaskManagers retrieve the TaskInfo from the blob cache

Verifying this change

Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (100MB)
• Extended integration test for recovery after master (JobManager) failure
• Added test that validates that TaskInfo is transferred only once across recoveries
• Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

CI report:

• 50992d4 UNKNOWN
• a361547 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[mbalassi]

Thanks, @slfan1989. The build looks healthy, please also update the notice files so that the relevant checks can pass:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58594&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb&l=44220

[slfan1989]

Thanks, @slfan1989. The build looks healthy, please also update the notice files so that the relevant checks can pass:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58594&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb&l=44220

@mbalassi Thank you very much for the review! I will continue to improve this pr.

[slfan1989]

In the compilation report, we can find the following error report. The report says this issue must be fixed, but I'm not sure if we should fix it in this PR because it doesn't seem to be related(Upgrade commons-compress to 1.26.0).

09:19:48,111 INFO  org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Problems were detected for a NOTICE file.
	flink-sql-avro-confluent-registry:
		 These issue are legally problematic and MUST be fixed: 
			Dependency commons-io:commons-io:2.11.0 is not listed.
			Dependency org.apache.commons:commons-lang3:3.12.0 is not listed.

[mbalassi]

I would assume these are pulled in as transitive dependencies via the upgrade, thanks for adding them.

[slfan1989]

@flinkbot run azure

[mbalassi]

@slfan1989 Thanks for addressing the Notice checks. We are almost there, the Python integration seems to have relied on a specific classname, that has probably changed. This causes a couple failures we are good otherwise.

Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.build.AbstractStreamBuilder
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58638&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=0f3adb59-eefa-51c6-2858-3654d9e0749d&l=9010

Let me know if you prefer getting some help with hunting this down.

[slfan1989]

@flinkbot run azure

[slfan1989]

@flinkbot run azure

[mbalassi]

LGTM, merging tomorrow if there are no objections.

[slfan1989]

LGTM, merging tomorrow if there are no objections.

Thank you very much for the review!

[Jiabao-Sun]

Hi @mbalassi, @slfan1989,

I think we needn't this dependency, the commons-codec's dependency is because commons-compress incorrectly depended on commons-codec's Charsets in version 1.26.0.

This issue has been fixed in version 1.26.1, so perhaps we should bump commons-compress version to 1.26.1.

see: https://issues.apache.org/jira/browse/COMPRESS-659

[Jiabao-Sun]

I have rechecked the dependency of commons-codec in commons-compress and it is no longer optional. Even if upgraded to 1.26.1, commons-codec will still be a transitive dependency.
Please ignore the previous noise, sorry for the disturbance.