[FLINK-12053][runtime] Calculate job vertex parallelism based on metrics

[eaglewatcherwb]

Change-Id: Ia593a04e73256d9a53c332fad964173e7770cb8a

What is the purpose of the change

Adaptively calculate job vertex parallelism based on metrics.

Brief change log

(for example:)

• Add enableAdaptiveParallelism in both job configuration and job manager configuration.
• Only enableAdaptiveParallelism of vertex with BLOCKING shuffle mode .
• Support decrease parallelism in the first step.
• Recursively adjust parallelism of the output vertices with POINTWISE shuffle pattern.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): no
• The public API, i.e., is any changed class annotated with @Public(Evolving): no
• The serializers: no
• The runtime per-record code paths (performance sensitive): no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
• The S3 file system connector: no

Documentation

• Does this pull request introduce a new feature? yes
• If yes, how is the feature documented? docs: https://docs.google.com/document/d/1ZxnoJ3SOxUk1PL2xC1t-kepq28Pg20IL6eVKUUOWSKY/edit?usp=sharing

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[tillrohrmann]

Thanks for opening this PR @eaglewatcherwb. I'm not sure whether we already reached agreement on the overall design for adaptive parallelism. I see some merge conflicts with the current effort to make the scheduler pluggable. I would suggest to wait until the new interfaces are in place before adding more things to the ExecutionGraph. The idea was to make the ExecutionGraph a leaner structure and let the specific Scheduler implementation handle things like adaptive parallelism. Therefore, I would suggest to close this PR.