[FLINK-25128][table] Reorganize table modules and introduce flink-table-planner-loader

[slinkydeveloper]

What is the purpose of the change

The goal of this PR is to allow arbitrary Scala versions in the user API, hiding the Scala version of the planner from the Scala version of the API. This required some changes in the module organization, as described in the changelog. I also included a README to give an overview to table api developers (not for end users) of the new package organization edf7350.

This PR fixes the following issues:

• https://issues.apache.org/jira/browse/FLINK-25128
• https://issues.apache.org/jira/browse/FLINK-25130
• https://issues.apache.org/jira/browse/FLINK-25131

I will open a followup PR to take care of the documentation.

Changelog

• PlannerBase is now using the class' classloader in order to load the ParserFactory. In case of planner_${scala.version}, this essentially makes no difference, while in case of planner-loader it will try to load first from its own classpath (which is ComponentClassLoader), and then from the parent classpath.
• Removed flink-table-uber module and replaced with flink-table-api-uber to ship only Java API related packages in a single uber jar.
• Now flink-table-runtime ships janino and code-splitter in its uber jar
• flink-table-planner pom has been reworked, removing dependencies shipped by flink-runtime and flink-dist like jackson and commons-lang3, and now it generates 2 uber jars: one is pretty much the same as we ship now on master, and the other, named loader-bundle, is a uber-jar which includes also scala, in order to be used by flink-table-planner-loader
• Introduce flink-table-planner-loader. This provides an implementation of the factories in org.apache.flink.table.delegation that uses an isolated classloader to load the implementations from flink-table-planner loader-bundle JAR.
• Rework SQL Client dependencies and remove scala suffix.
• Rework Flink distribution. Now we ship as separate jars flink-table-api-uber, flink-table-runtime, flink-table-planner-loader and flink-cep in /lib, while we continue to ship flink-table-planner_${scala.version} in /opt, in order to allow users to replace it with flink-table-planner-loader just in case they need it. This is the fundamental change for the user, and will documented in the followup doc PR.
• Use planner-loader wherever is possible in e2e tests and examples, and included a new test script that swaps the planner in order to check that both jars work fine.

There are a couple of details worth to mention in order to understand all the moving parts of this PR:

• Some planner dependencies are relocated, others are not, and note that every relocation performed by runtime needs to be performed by planner as well.
• The ComponentClassLoader used in PlannerModule allows only certain classes to be loaded from its parent classloader. Those are all the classes starting with "org.apache.flink" (which includes relocated dependencies) plus the classes starting with one of the prefixes in ownerClassPath

Verifying this change

A new e2e test has been included to check that both planner-loader and planner_${scala.version} works fine. All the previous e2e tests are now running with planner-loader, whenever is possible.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): yes
• The public API, i.e., is any changed class annotated with @Public(Evolving): mp

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? Will open a doc PR as followup

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 5481a24 (Thu Dec 16 14:43:28 UTC 2021)

Warnings:

• 14 pom.xml files were touched: Check for build and licensing issues.
• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[flinkbot]

CI report:

• 787be2a Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[twalthr]

Thanks for this great PR @slinkydeveloper. It looks we are approaching. Here is my first set of feedback.

[twalthr]

"depending" is ok but only as provided maybe you can rephrase this paragraph

[slinkydeveloper]

I rather prefer to keep it this way, just as a warning to everyone who even remotely thinks to add it as dependency :)

[twalthr]

But it needs to be added as a dependency. Otherwise you cannot run it in your IDE. Many people have problems with setting up Maven correctly. We should rephrase this to emphasize: provided is fine

[slinkydeveloper]

For testing you need to depend on table-planner-loader, not on table-planner. I added a sentence about testing.

[twalthr]

Awesome work @slinkydeveloper. This change will help users a lot.

[twalthr]

Isn't this only used for tests? What is it used for? JSON plan?

[slinkydeveloper]

Check ReflectionsUtil. It's used for the JSON plan but perhaps can be removed (not in this PR though)

[twalthr]

Why not provided anymore?

[slinkydeveloper]

Same as #18134 (comment)

[slinkydeveloper]

Actually, for this one I can remove it as we pull it from api-java-bridge

[twalthr]

LGTM given the build turns green 🥳