[KYUUBI #4367] Support Flink 1.17 by link3280 · Pull Request #4368 · apache/kyuubi

link3280 · 2023-02-19T01:38:45Z

Why are the changes needed?

Support Flink 1.17 and Flink SQL gateway.

Drop Flink 1.15
Migrate API to Flink SQL Gateway
Support Flink 1.17

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request

link3280 · 2023-02-19T01:41:31Z

cc @pan3793 @yanghua @SteNicholas @a49a @bowenliang123

bowenliang123 · 2023-02-19T01:46:11Z

Can you list here the breaking changes for the adaption for 1.17 and impacts for other versions?

link3280 · 2023-02-19T01:47:00Z

Note: The CI would NOT be successful because there are no flink 1.17 binaries on the apache site yet, plus I removed support for flink 1.16 or below temporarily. I will fix it when the flink 1.17 binaries are ready.

link3280 · 2023-02-19T01:50:56Z

Can you list out the main breaking changes and reasons of the adaption for 1.17?

Flink 1.17 refactors the SQL client module, adding a new module called Flink SQL gateway. Many classes Kyuubi relies on are refactored or relocated. For example, LocalExecutor and SessionContext.

WRT Flink 1.16 or below, surely I want to keep the support for at least 3 Flink minor versions, so I'm thinking of adding a new Flink engine module just for 1.17 or above if the community agrees.

bowenliang123 · 2023-02-19T02:16:28Z

Note: The CI would NOT be successful because there are no flink 1.17 binaries on the apache site yet, plus I removed support for flink 1.16 or below temporarily. I will fix it when the flink 1.17 binaries are ready.

Good to have a new profile and it job for flink 1.7. And we could test it in nighty workflows before it's released like we do to spark.

The versions of flink supported by kyuubi with this pr should be also clarified and discussed.

pan3793 · 2023-03-18T14:26:39Z

@link3280 this PR also includes some code refactoring, we can extract them in separating PR and put them into master first. And after #4517, we can drop support for Flink 1.14 in master, this could reduce lots of code differences.

### _Why are the changes needed?_ As discussed before, Kyuubi is going to support the latest 3 Flink versions, and to reduce the complexity of supporting Flink 1.17 #4368, we are going to remove support Flink 1.14 first. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4588 from pan3793/rm-flink-1.14. Closes #4387 97d2633 [Cheng Pan] Remove support for Flink 1.14 Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org>

codecov-commenter · 2023-04-08T17:37:19Z

Codecov Report

Merging #4368 (8eb4da6) into master (5faebb1) will increase coverage by 0.36%.
The diff coverage is 66.66%.

@@             Coverage Diff              @@
##             master    #4368      +/-   ##
============================================
+ Coverage     57.68%   58.04%   +0.36%     
  Complexity       13       13              
============================================
  Files           579      580       +1     
  Lines         32012    32235     +223     
  Branches       4275     4305      +30     
============================================
+ Hits          18465    18712     +247     
+ Misses        11784    11727      -57     
- Partials       1763     1796      +33

Impacted Files	Coverage Δ
...pache/kyuubi/engine/KyuubiApplicationManager.scala	`78.66% <0.00%> (-1.34%)`	⬇️
...ache/kyuubi/engine/flink/FlinkProcessBuilder.scala	`75.23% <75.00%> (-1.24%)`	⬇️

... and 29 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

...kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkEngineUtils.scala

...link-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/session/FlinkSessionImpl.scala

pan3793 · 2023-04-12T04:26:27Z

Thanks @link3280, this approach looks good to me, even we must drop Flink 1.15

…ions

...kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkEngineUtils.scala

...ink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/delegation/FlinkResultSet.scala

...ql-engine/src/main/scala/org/apache/kyuubi/engine/flink/delegation/FlinkSessionManager.scala

…uubi/engine/flink/FlinkEngineUtils.scala Co-authored-by: Cheng Pan <pan3793@gmail.com>

link3280 · 2023-04-17T02:19:04Z

Please kindly rerun the fail job @pan3793

pan3793 · 2023-04-17T12:12:24Z

Thanks for the great work, merged to master for 1.8

github-actions bot added kind:build module:flink labels Feb 19, 2023

pan3793 mentioned this pull request Mar 24, 2023

[KYUUBI #4387] Remove support for Flink 1.14 #4588

Closed

3 tasks

feat: Support Flink 1.17

49e0a94

link3280 force-pushed the flink-1.17 branch from 679203a to 49e0a94 Compare April 8, 2023 16:18

link3280 changed the title ~~[KYUUBI #4367] Support Flink 1.17~~ [WIP][KYUUBI #4367] Support Flink 1.17 Apr 10, 2023

[KYUUBI apache#4367] Fix Flink SessionManager compatibility issue

e18b3c2

github-actions bot added the kind:infra license, community building, project builds, asf infra related, etc. label Apr 10, 2023

link3280 added 2 commits April 10, 2023 23:10

[KYUUBI apache#4367] Remove support for Flink 1.15

ef6f4d4

[KYUUBI apache#4367] Flink FlinkSQLEngine capabilities with Flink 1.16

d289495

link3280 force-pushed the flink-1.17 branch from 00dd58b to d289495 Compare April 10, 2023 15:20

link3280 added 2 commits April 11, 2023 00:01

[KYUUBI apache#4367] Fix NPE in dependencies

6a08d0b

[KYUUBI apache#4367] Fix reflection failures

614ecc3

github-actions bot added the module:common label Apr 11, 2023

link3280 added 3 commits April 11, 2023 15:16

[KYUUBI apache#4367] Fix NoSuchFieldException

799c938

[KYUUBI apache#4367] Fix Flink conf dir not found

29728c0

[KYUUBI apache#4367] Improve code style

b685ff1

pan3793 reviewed Apr 12, 2023

View reviewed changes

...kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkEngineUtils.scala Outdated Show resolved Hide resolved

pan3793 reviewed Apr 12, 2023

View reviewed changes

...kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkEngineUtils.scala Outdated Show resolved Hide resolved

pan3793 reviewed Apr 12, 2023

View reviewed changes

...kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkEngineUtils.scala Outdated Show resolved Hide resolved

pan3793 reviewed Apr 12, 2023

View reviewed changes

...link-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/session/FlinkSessionImpl.scala Outdated Show resolved Hide resolved

link3280 added 8 commits April 15, 2023 02:57

[KYUUBI apache#4367] Fix IncompatibleClassChangeError

e1e5cd2

[KYUUBI apache#4367] Remove unnecessary reflection code

33eeb37

[KYUUBI apache#4367] Remove unnecessary variables

1c41e44

[KYUUBI apache#4367] Remove unnecessary classes

8530aec

[KYUUBI apache#4367] Improve code style for java.util.*

4ef8c52

[KYUUBI apache#4367] Improve code style for java.lang.Boolean

7e36e70

[KYUUBI apache#4367] Improve code style for error messages and iterat…

c2a23d5

…ions

[KYUUBI apache#4367] Improve code style for iterations

7b28eaf

github-actions bot removed the module:common label Apr 15, 2023

link3280 changed the title ~~[WIP][KYUUBI #4367] Support Flink 1.17~~ [KYUUBI #4367] Support Flink 1.17 Apr 15, 2023

link3280 added 2 commits April 15, 2023 10:29

[KYUUBI apache#4367] Fix spotless

ed95060

[KYUUBI apache#4367] Fix ambiguous reference

08f8999

link3280 requested a review from pan3793 April 15, 2023 06:33

pan3793 reviewed Apr 16, 2023

View reviewed changes

...kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkEngineUtils.scala Outdated Show resolved Hide resolved

pan3793 reviewed Apr 16, 2023

View reviewed changes

...kyuubi-flink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/FlinkEngineUtils.scala Outdated Show resolved Hide resolved

pan3793 reviewed Apr 16, 2023

View reviewed changes

...ink-sql-engine/src/main/scala/org/apache/kyuubi/engine/flink/delegation/FlinkResultSet.scala Outdated Show resolved Hide resolved

pan3793 reviewed Apr 16, 2023

View reviewed changes

...ql-engine/src/main/scala/org/apache/kyuubi/engine/flink/delegation/FlinkSessionManager.scala Outdated Show resolved Hide resolved

link3280 and others added 6 commits April 16, 2023 20:34

Update externals/kyuubi-flink-sql-engine/src/main/scala/org/apache/ky…

417d37b

…uubi/engine/flink/FlinkEngineUtils.scala Co-authored-by: Cheng Pan <pan3793@gmail.com>

[KYUUBI apache#4367] Improve java.long.Long usage

56567fc

[KYUUBI apache#4367] Improve code style

5c9d0aa

[KYUUBI apache#4367] Rename delegation package to shim

23d87ba

[KYUUBI apache#4367] Fix test failure

81a10f6

[KYUUBI apache#4367] Fix test failure

8eb4da6

pan3793 approved these changes Apr 17, 2023

View reviewed changes

pan3793 closed this in 553b2aa Apr 17, 2023

pan3793 assigned link3280 Apr 17, 2023

pan3793 added this to the v1.8.0 milestone Apr 17, 2023

Conversation

link3280 commented Feb 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

How was this patch tested?

Uh oh!

link3280 commented Feb 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bowenliang123 commented Feb 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

link3280 commented Feb 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

link3280 commented Feb 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bowenliang123 commented Feb 19, 2023

Uh oh!

pan3793 commented Mar 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Apr 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pan3793 commented Apr 12, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

link3280 commented Apr 17, 2023

Uh oh!

pan3793 commented Apr 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

link3280 commented Feb 19, 2023 •

edited

Loading

link3280 commented Feb 19, 2023 •

edited

Loading

bowenliang123 commented Feb 19, 2023 •

edited

Loading

link3280 commented Feb 19, 2023 •

edited

Loading

link3280 commented Feb 19, 2023 •

edited

Loading

pan3793 commented Mar 18, 2023 •

edited

Loading

codecov-commenter commented Apr 8, 2023 •

edited

Loading