[FLINK-16984][tests] Consolidate Log4j2-test.properties

[zentol]

Uses the same trick as #11633 to share a test resource across all modules, in this case log4j2-test.properties.

This PR removes all log4j2-test.properties files except in flink-test-utils-junit, where it is moved to into the main resource directory instead.

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 4dd50c1 (Sat Apr 04 16:42:43 UTC 2020)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[flinkbot]

CI report:

• 13fa6c2 Travis: SUCCESS Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run travis re-run the last Travis build
• @flinkbot run azure re-run the last Azure build

[rmetzger]

I wonder if it makes sense to keep the file around if there's some domain knowledge in those files. I guess for debugging or working on the Kafka tests it is helpful to suppress some log outputs.

[rmetzger]

In principle, I like the idea of removing a lot of duplicate files.
I see two problems:

• some modules seem to have some module specifics encoded in their logging config (I propose the keep the files for those modules)
• we need to make sure all the developers understand the new mechanism (I guess we should at least send a notice to the dev@ list?)

[zentol]

some modules seem to have some module specifics encoded in their logging config (I propose the keep the files for those modules)

I would only have special configs in leaf modules. For modules that are depended on you will usually want these logging configs to be propagated. Like for flink-runtime. I can't think of a good reason to not use the same logging in flink-tests.

I'm fine with keeping them in kafka and yarn-tests since they disable quite a lot (all of hadoop/zookeeper).

we need to make sure all the developers understand the new mechanism (I guess we should at least send a notice to the dev@ list?)

yes, that was my plan.

[tillrohrmann]

Thanks for creating this PR @zentol.

From a software engineering standpoint I agree with these changes. It makes sense to reduce duplication.

From a workflow perspective I think it worsens the situation a bit (or at least it disrupts the existing workflow). Currently when debugging a test from within the IDE, I jump to the test file, which brings me to the correct module, and then I change the log4j2-test.properties file in the resources folder of the test folder. This is quite easy and straight forward. With this change, now everyone has to remember that the logj42-test.properties comes from flink-test-utils-junit which is a higher intrinsic burden and not that obvious when you are new to the project.

Before doing any changes, I advise to start a discussion on dev about this topic outlining the benefits and drawbacks as well as the implications for the devs.