Flink version update #88

architgyl · 2023-04-29T00:28:14Z

Summary

Fix the Beam build issue to run against latest Flink version.

Testing

Tested for all versions of Flink and now Beam is able to fetch the latest Flink artifacts.

./gradlew :runners:flink:1.16:build
./gradlew :runners:flink:1.15:build
./gradlew :runners:flink:1.14:build
./gradlew :runners:flink:1.13:build
./gradlew :runners:flink:1.12:build

…rsionUpdate

venkata91 · 2023-04-29T00:31:15Z

Where are we setting the new flink version (11600.3)?

runners/flink/flink_runner.gradle

architgyl · 2023-04-29T02:23:46Z

Fixed the code.

xinyuiscool

Could you please verify the dependency by ./gradlew :runners:flink:1.16:dependenies? I want to make sure we are using our flink versions indeed. Thanks.

xinyuiscool · 2023-05-01T16:21:24Z

runners/flink/flink_runner.gradle

+  /* Flink 1.15 shades all remaining scala dependencies and therefor does not depend on a specific version of Scala anymore.
+   * Flink 1.16 onwards, flink version is 11600.* and does not depend on a specific version of Scala anymore.
+   */
+  if (flink_version.compareTo("1.15") >= 0 || flink_version.compareTo("11600.3") >= 0) {


I'm curious about this change: is it 11600.3 >= 1.15? If so, do we need the second part of the condition?

@xinyuiscool Would it be better to have an indirection say if the flink_version is 1.16 then use the corresponding linkedin_flink_version which is 11600.3 through a map or something. That way, when a new version is released in LinkedIn, we just need to change in the mapping table. Also this way even if LinkedIn Flink versioning scheme changes to Semantic versioning with its own major.minor.patch version, it won't be affected by the comparison operator.

architgyl · 2023-05-01T19:34:57Z

Could you please verify the dependency by ./gradlew :runners:flink:1.16:dependenies? I want to make sure we are using our flink versions indeed. Thanks.

Verified the changes locally.

`+--- org.apache.beam:beam-vendor-grpc-1_48_1:0.1 ()
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.10
+--- args4j:args4j:2.33
+--- org.apache.flink:flink-clients:11600.3
| +--- org.apache.flink:flink-core:11600.3
| | +--- org.apache.flink:flink-annotations:11600.3
| | | +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| | | +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| | | --- org.apache.flink:flink-shaded-force-shading:15.0
| | +--- org.apache.flink:flink-metrics-core:11600.3
| | | +--- org.apache.flink:flink-annotations:11600.3 ()
| | | --- org.apache.flink:flink-shaded-force-shading:15.0
| | +--- org.apache.flink:flink-shaded-asm-9:9.2-15.0
| | +--- org.apache.flink:flink-shaded-jackson:2.12.4-15.0
| | +--- org.apache.commons:commons-lang3:3.3.2 -> 3.9
| | +--- com.esotericsoftware.kryo:kryo:2.24.0 ()
| | +--- commons-collections:commons-collections:3.2.2
| | +--- org.apache.commons:commons-compress:1.21
| | +--- org.apache.flink:flink-shaded-guava:30.1.1-jre-15.0
| | +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| | +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| | --- org.apache.flink:flink-shaded-force-shading:15.0
| +--- org.apache.flink:flink-runtime:11600.3
| | +--- org.apache.flink:flink-core:11600.3 ()
| | +--- org.apache.flink:flink-rpc-core:11600.3
| | | +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| | | +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| | | --- org.apache.flink:flink-shaded-force-shading:15.0
| | +--- org.apache.flink:flink-rpc-akka-loader:11600.3
| | | +--- org.apache.flink:flink-core:11600.3 ()
| | | +--- org.apache.flink:flink-rpc-core:11600.3 ()
| | | +--- org.apache.flink:flink-shaded-force-shading:15.0
| | | +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| | | --- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| | +--- org.apache.flink:flink-java:11600.3
| | | +--- org.apache.flink:flink-core:11600.3 ()
| | | +--- org.apache.commons:commons-lang3:3.3.2 -> 3.9
| | | +--- org.apache.commons:commons-math3:3.6.1
| | | +--- com.twitter:chill-java:0.7.6
| | | | --- com.esotericsoftware.kryo:kryo:2.21 -> 2.24.0 ()
| | | +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| | | +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| | | --- org.apache.flink:flink-shaded-force-shading:15.0
| | +--- org.apache.flink:flink-queryable-state-client-java:11600.3
| | | +--- org.apache.flink:flink-shaded-netty:4.1.70.Final-15.0
| | | +--- org.apache.flink:flink-shaded-guava:30.1.1-jre-15.0
| | | --- org.apache.flink:flink-shaded-force-shading:15.0
| | +--- org.apache.flink:flink-hadoop-fs:11600.3
| | | +--- org.apache.flink:flink-core:11600.3 ()
| | | --- org.apache.flink:flink-shaded-force-shading:15.0
| | +--- commons-io:commons-io:2.11.0 -> 2.7
| | +--- org.apache.flink:flink-shaded-netty:4.1.70.Final-15.0
| | +--- org.apache.flink:flink-shaded-guava:30.1.1-jre-15.0
| | +--- org.apache.flink:flink-shaded-jackson:2.12.4-15.0
| | +--- org.apache.flink:flink-shaded-zookeeper-3:3.5.9-15.0
| | +--- org.apache.commons:commons-lang3:3.3.2 -> 3.9
| | +--- commons-cli:commons-cli:1.5.0
| | +--- org.javassist:javassist:3.24.0-GA
| | +--- org.xerial.snappy:snappy-java:1.1.8.3 -> 1.1.8.4
| | +--- org.lz4:lz4-java:1.8.0
| | +--- org.apache.flink:flink-shaded-force-shading:15.0
| | +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| | --- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| +--- org.apache.flink:flink-optimizer:11600.3
| | +--- org.apache.flink:flink-core:11600.3 ()
| | +--- org.apache.flink:flink-runtime:11600.3 ()
| | +--- org.apache.flink:flink-java:11600.3 ()
| | +--- org.apache.flink:flink-shaded-guava:30.1.1-jre-15.0
| | +--- org.apache.flink:flink-shaded-jackson:2.12.4-15.0
| | +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| | +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| | --- org.apache.flink:flink-shaded-force-shading:15.0
| +--- org.apache.flink:flink-java:11600.3 ()
| +--- commons-cli:commons-cli:1.5.0
| +--- org.apache.flink:flink-streaming-java:11600.3
| | +--- org.apache.flink:flink-core:11600.3 ()
| | +--- org.apache.flink:flink-file-sink-common:11600.3
| | | --- org.apache.flink:flink-shaded-force-shading:15.0
| | +--- org.apache.flink:flink-runtime:11600.3 ()
| | +--- org.apache.flink:flink-java:11600.3 ()
| | +--- org.apache.flink:flink-shaded-guava:30.1.1-jre-15.0
| | +--- org.apache.commons:commons-math3:3.6.1
| | +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| | +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| | --- org.apache.flink:flink-shaded-force-shading:15.0
| +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| --- org.apache.flink:flink-shaded-force-shading:15.0
+--- org.apache.flink:flink-streaming-java:11600.3 ()
+--- org.apache.flink:flink-core:11600.3 ()
+--- org.apache.flink:flink-metrics-core:11600.3 ()
+--- org.apache.flink:flink-java:11600.3 ()
+--- org.apache.flink:flink-runtime:11600.3 ()
+--- org.apache.flink:flink-optimizer:11600.3 ()
+--- project :model:fn-execution ()
+--- project :model:pipeline ()
+--- project :model:job-management ()
+--- project :sdks:java:fn-execution ()
+--- com.fasterxml.jackson.core:jackson-databind:2.14.1 ()
+--- org.apache.flink:flink-annotations:11600.3 ()
+--- org.apache.flink:flink-statebackend-rocksdb:11600.3
| +--- com.ververica:frocksdbjni:6.20.3-ververica-1.0
| +--- org.slf4j:slf4j-api:1.7.32 -> 1.7.30
| +--- com.google.code.findbugs:jsr305:1.3.9 -> 3.0.2
| --- org.apache.flink:flink-shaded-force-shading:15.0
--- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:2.14.1
+--- com.fasterxml.jackson.core:jackson-annotations:2.14.1 ()
+--- com.fasterxml.jackson.core:jackson-core:2.14.1 ()
+--- com.fasterxml.jackson.core:jackson-databind:2.14.1 ()
+--- jakarta.xml.bind:jakarta.xml.bind-api:2.3.3
| --- jakarta.activation:jakarta.activation-api:1.2.2
+--- jakarta.activation:jakarta.activation-api:1.2.2
--- com.fasterxml.jackson:jackson-bom:2.14.1 ()

(c) - dependency constraint
(*) - dependencies omitted (listed previously)

(n) - Not resolved (configuration is not meant to be resolved)

A web-based, searchable dependency report is available by adding the --scan option.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 24s
11 actionable tasks: 1 executed, 10 up-to-date`

xinyuiscool

LGTM!

architgyl added 3 commits April 28, 2023 17:12

Update Flink Version

e576bfa

Update Beam project to use latest Flink changes

ba507b3

Merge remote-tracking branch 'origin/FlinkVersionUpdate' into FlinkVe…

87179d2

…rsionUpdate

github-actions bot added flink runners labels Apr 29, 2023

venkata91 suggested changes Apr 29, 2023

View reviewed changes

runners/flink/flink_runner.gradle Outdated Show resolved Hide resolved

xinyuiscool reviewed Apr 29, 2023

View reviewed changes

runners/flink/flink_runner.gradle Outdated Show resolved Hide resolved

venkata91 suggested changes Apr 29, 2023

View reviewed changes

runners/flink/flink_runner.gradle Outdated Show resolved Hide resolved

Fixing version related comments

da4511e

xinyuiscool reviewed May 1, 2023

View reviewed changes

Address comments

32e5850

xinyuiscool approved these changes May 1, 2023

View reviewed changes

xinyuiscool merged commit 30488c6 into li_trunk May 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flink version update #88

Flink version update #88

architgyl commented Apr 29, 2023

venkata91 commented Apr 29, 2023

architgyl commented Apr 29, 2023

xinyuiscool left a comment

xinyuiscool May 1, 2023

venkata91 May 1, 2023 •

edited

Loading

architgyl commented May 1, 2023

xinyuiscool left a comment

Flink version update #88

Flink version update #88

Conversation

architgyl commented Apr 29, 2023

Summary

Testing

venkata91 commented Apr 29, 2023

architgyl commented Apr 29, 2023

xinyuiscool left a comment

Choose a reason for hiding this comment

xinyuiscool May 1, 2023

Choose a reason for hiding this comment

venkata91 May 1, 2023 • edited Loading

Choose a reason for hiding this comment

architgyl commented May 1, 2023

xinyuiscool left a comment

Choose a reason for hiding this comment

venkata91 May 1, 2023 •

edited

Loading