[FLINK-25694][FileSystems][S3] Upgrade Presto to resolve GSON/Alluxio Vulnerability

[David-N-Perkins]

What is the purpose of the change

• Updated presto to the latest version due to GSON bug

Brief change log

• Updated prosto library to .272

Verifying this change

This change is already covered by existing tests.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
• The S3 file system connector: yes

Documentation

• Does this pull request introduce a new feature? no
• If yes, how is the feature documented? not applicable

[flinkbot]

CI report:

• 373b9be Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[MartijnVisser]

Thanks for the PR @David-N-Perkins but we need to also make changes to the NOTICE files, since this newer version has different version numbers

20:46:51,337 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency io.grpc:grpc-core:1.26.0 is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,338 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency io.netty:netty-codec-http2:4.1.51.Final is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,338 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency io.netty:netty-handler-proxy:4.1.51.Final is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,338 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency net.jpountz.lz4:lz4:1.3.0 is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,338 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency com.google.protobuf:protobuf-java:3.11.0 is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,338 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency io.netty:netty-handler:4.1.51.Final is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,338 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency io.grpc:grpc-stub:1.26.0 is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,338 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency org.bouncycastle:bcprov-jdk15on:1.54 is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,338 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency io.netty:netty-codec-http:4.1.51.Final is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,339 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency com.google.guava:guava:26.0-jre is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,339 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency com.jcraft:jzlib:1.1.3 is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,339 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency com.google.auth:google-auth-library-credentials:0.18.0 is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,339 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency io.netty:netty-codec:4.1.51.Final is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,339 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency io.netty:netty-transport-native-unix-common:4.1.51.Final is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,339 DEBUG org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency io.netty:netty-common:4.1.51.Final is mentioned in NOTICE file /__w/1/s/flink-python/src/main/resources/META-INF/NOTICE, but was not mentioned by the build output as a bundled dependency
20:46:51,342 ERROR org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Could not find dependency com.facebook.presto:presto-hive:0.272 in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE
20:46:51,342 ERROR org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Could not find dependency org.apache.hudi:hudi-presto-bundle:0.10.1 in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE
20:46:51,342 ERROR org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Could not find dependency com.facebook.presto:presto-hive-common:0.272 in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE
20:46:51,342 ERROR org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Could not find dependency com.facebook.presto:presto-hive-metastore:0.272 in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE
20:46:51,342 ERROR org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Could not find dependency org.alluxio:alluxio-shaded-client:2.7.3 in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE
20:46:51,342 ERROR org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Could not find dependency com.facebook.presto:presto-common:0.272 in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE
20:46:51,342 WARN  org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency com.facebook.presto:presto-common:0.257 is mentioned in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE, but is not expected there
20:46:51,342 WARN  org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency com.facebook.presto:presto-hive-metastore:0.257 is mentioned in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE, but is not expected there
20:46:51,342 WARN  org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency org.alluxio:alluxio-shaded-client:2.5.0-3 is mentioned in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE, but is not expected there
20:46:51,342 WARN  org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency com.facebook.presto:presto-hive:0.257 is mentioned in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE, but is not expected there
20:46:51,342 WARN  org.apache.flink.tools.ci.licensecheck.NoticeFileChecker     [] - Dependency com.facebook.presto:presto-hive-common:0.257 is mentioned in NOTICE file /__w/1/s/flink-filesystems/flink-s3-fs-presto/src/main/resources/META-INF/NOTICE, but is not expected there
20:46:51,466 INFO  org.apache.flink.tools.ci.licensecheck.JarFileChecker        [] - Checking directory /tmp/flink-validation-deployment with a total of 198 jar files.
20:48:28,002 WARN  org.apache.flink.tools.ci.licensecheck.LicenseChecker        [] - Found a total of 6 severe license issues
==============================================================================
License Check failed. See previous output for details.
==============================================================================

See https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=34528&view=logs&j=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5&t=54421a62-0c80-5aad-3319-094ff69180bb for all details

[David-N-Perkins]

I updated the NOTICE file.

[MartijnVisser]

@David-N-Perkins Thanks a lot for the fix! I've squashed the commits and rebased the PR. I've also modified the commit message to be in line with Flink code contribution guide, see https://flink.apache.org/contributing/contribute-code.html

Since this involves S3, I need to run some manually tests before I can merge the PR. I'll try to get that done today, finishtthe review and then I'll merge it (if everything is OK of course).

Thanks again for your help.

[MartijnVisser]

Verified that S3 is working as expected in https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=34682&view=results. Merging this now. Thanks again @David-N-Perkins !

[David-N-Perkins]

@MartijnVisser Does this need to get merged into any other support branches?
And is there a time frame on when this would get released? My company is tracking this vulnerability in our Flink deployments.

[MartijnVisser]

@David-N-Perkins I think we could consider backports to both release-1.15 and release-1.14, being the last 2 releases that are being supported. I'm not 100% sure if we could merge this before Flink 1.15 is released (since the release candidate has just been created and the release is really close), but let's first at least have those backports available :)