[FLINK-28203] Support Maven 3.3+

[zentol]

Based on #21346 (for test stability).

This PR sets up our build system to support Maven 3.3+ and our CI system to run with Maven 3.8.6.

In Maven 3.3 the dependency tree was made immutable at runtime, and thus can no longer be changed by the shade plugin. The plugin would usually remove a dependency from the tree when it is being bundled (known as dependency reduction).
While dependency reduction still works for the published poms (== what users consume) since it can still change the content of the final pom, while developing Flink it no longer works. This breaks plenty of things, since suddenly a whole bunch of dependencies are still visible to downstream modules that weren't before.
To workaround this we now mark all dependencies that we bundle as optional; this makes them non-transitive.
Behavior-wise a non-transitive dependency is identical to a removed dependency.

To also make this work in the IDE (which never interacts with jars within a project, in contrast to Maven) the optional flag is modeled as a property that defaults to true, but is set to false by a special profile when the module is imported into IntelliJ.

Naturally, requiring all dependencies that are bundled to be marked as optional is prone too errors. To that end the ShadeOptionalChecker is being introduced, which analyzes the bundled dependencies (based on the shade-plugin output) and the set of dependencies (based on the dependency plugin) to detect cases where a dependency is not marked as optional as it should.
The enforced rule is rather simple: Any dependency that is bundled, or any of it's parents, must show up as optional in the dependency tree.
The parent clause is required to cover cases where a module has 2 paths to a bundled dependency.
If a module depends on A1/A2, each depending on B, with A1 and B being bundled, then even if A1 is marked as optional B is still shown as a non-optional dependency (because the non-optional A2 still needs it!).

This has the caveat that going forward we may, at times, have to add/exclude dependencies purely for things to show up correctly in the dependency tree.

TODO: The ShadeOptionalChecker needs tests.

[flinkbot]

CI report:

• 2e1dbc0 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[zentol]

Some discourse on why the PR is the way it is:

Could this be limited to modules that actually need this?

Yes. We could determine which modules are relied upon by other modules and only enforce the optional flags for said module. This would however result in inconsistent module poms and additional work for the poor soul that added a dependency on another module.

Could this be achieved with exclusions?

Yes-ish. We could add exclusions for all bundled dependencies in the consuming modules, and optionally add another module in between to centralize these exclusions in a single place. (Similar to what the -loader modules do, at the cost of even more modules).
This however would require exclusions/bundling rules in 2 distinct files to be kept in sync. This is always problematic and can easily result in excess exclusions that may cause problems down the line.
It would additionally be more difficult to test this because there's no clear answer for a consuming module as to what should actually be excluded.
We'd implicitly limit the setup to the one described in Could this be limited to modules that actually need this?.
Making this work with IntelliJ would be quite noisy, as we'd have to setup the exclusions in a profile.

[alpinegizmo]

If a module depends on A1/A2, each depending on B, with A1 and B being bundled, then even if A1 is marked as optional B is still shown as a non-optional dependency (because the non-optional A2 still needs it!).

Are there are cases where this currently occurs?

[alpinegizmo]

I buy your reasoning as to why you've chosen to set it up this way, as opposed to the alternatives you described. This seems to create the most straightforward and most maintainable of the possible outcomes.

[alpinegizmo]

Some version of the explanation in this PR needs to put somewhere where flink developers can find it. Probably here: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/flinkdev/building/#dependency-shading

[zentol]

If a module depends on A1/A2, each depending on B, with A1 and B being bundled, then even if A1 is marked as optional B is still shown as a non-optional dependency (because the non-optional A2 still needs it!).

There are a few cases, yes. :(
Some examples:

flink-sql-parquet (because of a test dependency!):

[INFO] |  \- org.apache.avro:avro:jar:1.11.1:compile (optional) 
[INFO] |     +- com.fasterxml.jackson.core:jackson-core:jar:2.13.4:compile (optional) 
[INFO] |     +- com.fasterxml.jackson.core:jackson-databind:jar:2.13.4.2:compile (optional) 
[INFO] |     |  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.4:compile

flink-connector-kinesis:

[INFO] +- com.amazonaws:aws-java-sdk-kinesis:jar:1.12.276:compile (optional) 
[INFO] |  +- com.amazonaws:aws-java-sdk-core:jar:1.12.276:compile
[INFO] |  |  \- software.amazon.ion:ion-java:jar:1.0.2:compile

[zentol]

Some version of the explanation in this PR needs to put somewhere where flink developers can find it. Probably here: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/flinkdev/building/#dependency-shading

That page is all about building Flink from source and the peculiarities about building Flink on Maven 3.3+ (which this PR should actually update!). It's not really developer documentation on how to setup shading (or really changing Flink in any way) etc.

I'm inclined to add a page to the wiki / extend the Dependencies page.

[alpinegizmo]

Some version of the explanation in this PR needs to put somewhere where flink developers can find it. Probably here: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/flinkdev/building/#dependency-shading

That page is all about building Flink from source and the peculiarities about building Flink on Maven 3.3+ (which this PR should actually update!). It's not really developer documentation on how to setup shading (or really changing Flink in any way) etc.

I'm inclined to add a page to the wiki / extend the Dependencies page.

That page in the docs on building Flink is in a section entitled "Flink Development", and building Flink from source is the first step toward contributing to Flink. Perhaps developer documentation doesn't belong here, but there should at least be a noticeable pointer to the wiki. I say this as someone who was, until quite recently, unaware that there is valuable content for contributors (other than the FLIPs) in the wiki.

[zentol]

there should at least be a noticeable pointer to the wiki

Definitely. We haven't done a good job documenting the wiki as an actual source for developer docs.

[alpinegizmo]

Some typos.

[XComp]

Thanks, @zentol for this PR. I went over it. The biggest issue is the documentation, I guess. Some of the description that is added to this PR could be included in the code as well.

[XComp]

I went over the changes. One test needs to be revisited. PTAL

[XComp]

LGTM 👍

[XComp]

FYI: FLINK-32066 ...if you're waiting for another CI run to be picked up.

[zentol]

Green personal build: https://dev.azure.com/chesnay/flink/_build/results?buildId=3519&view=results

Merging.