[FLINK-15785][travis][e2e] Rework E2E test activations

[zentol]

This PR reworks the activation logic for java end-to-end tests.

Introduces a new framework for specifying category inclusions/exclusions in flink-end-to-end-tests, that is capable of automatically aggregating properties across profiles.
Inclusions are defined by setting the e2e.include.<category-class>, conversely exclusions by setting e2e.exclude.<category-class>. The value of the property is a boolean that controls whether the inclusion/exclusions is enabled or not.
A groovy script searches for all properties matching the naming scheme, extracts the category class, aggregates inclusions/exclusions and sets the appropriate properties.

The script also prints the final inclusions/exclusions; I'd recommend trying it out and specifying different profile sets to see how it behaves.

Background: Maven does not really support combining categories. The surefire-plugin only accepts values like catA,catB, instead of a list where individual items have their own XML element. This implies that in order to combine category definitions one would have to concatenate the current value with some other value, i.e., do categories = categories + "hello". Maven doesn't support this however since it is a recursive definition of a property.

The following is a listing of every problem and how it was addressed:

1. travis activations are spread out over multiple files (.travis.yml, travis_watchdoh.sh)
   They are now consolidated in .travis.yml. While this does introduce some redundancy it is still preferable because .travis.yml is the most visible travis file, and it is easier to mvoe the pre-commit activation around.
2. setting multiple categories in .travis.yml is incredibly verbose
   Each category now has their dedicated profile in flink-end-to-end-tests that set the appropriate category. This is significantly more concise and provides an abstraction of the specific categories used. (In other words, we can modify things at will without breaking setups)
3. hadoop tests are not excluded by default
   Now disabled by default.
4. hadoop exclusions aren't setup (causing FLINK-15685)
   Are now setup.
5. automatic java11 exclusion is prone to not working as intended
   There is now a dedicated java11 profile in flink-end-to-end-tests to ensure the java11 exclusions is always applied.

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 0a4cdba (Tue Jan 28 12:54:58 UTC 2020)

Warnings:

• 1 pom.xml files were touched: Check for build and licensing issues.
• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

Details

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[flinkbot]

CI report:

• 0a4cdba Travis: FAILURE Azure: FAILURE
• 73b8f36 Travis: FAILURE Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run travis re-run the last Travis build
• @flinkbot run azure re-run the last Azure build

[tillrohrmann]

Thanks for creating this PR @zentol. The changes look good to me. I tried the PR out and it seems to generate the right sets of inclusions/exclusions. I had 2 minor comments. +1 for merging them after resolving the comments.

[tillrohrmann]

Do we want to keep these printlns?

[zentol]

I'm not sure. I like that you can tell at a glance whether something is off, which will be useful on Travis.

[zentol]

maybe keep them for a bit until the transition to azure is done and we are confident it works as it should.