[FLINK-29615] Retain active subtasks in TaskMetricStore when fetching metrics to accommodate dynamic scaling

[X-czh]

What is the purpose of the change

(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)

This pull request removes metrics of nonexistent subtasks when adaptive scheduler lowers job parallelism (aka, num of subtasks decreases).

Brief change log

Thanks to FLINK-29132 & FLINK-28588, Flink will now update current execution attempts when updating metrics. Since the active subtask info is included in the current execution attempt info, we are able to retain active subtasks using the current execution attempt info.

Verifying this change

Added UT to test the case where the number of subtasks decrease and tested on standalone clusters as the following:

1. Start a job with reactive mode enabled on a standalone cluster.
2. Remove TMs to decrease parallelism.
3. Checked that extra subtasks are removed in backpressure monitoring page.

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no) no
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no) no
• The serializers: (yes / no / don't know) no
• The runtime per-record code paths (performance sensitive): (yes / no / don't know) no
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) no
• The S3 file system connector: (yes / no / don't know) no

Documentation

• Does this pull request introduce a new feature? (yes / no) no
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

CI report:

• 2b41c49 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[X-czh]

@flinkbot run azure

[X-czh]

@flinkbot run azure

[xintongsong]

LGTM

[xintongsong]

@flinkbot run azure