[FLINK-17707][k8s] Support configuring replicas of JobManager deployment when HA enabled #15286

wangyang0918 · 2021-03-19T12:11:23Z

What is the purpose of the change

At the moment, in the native K8s setups, we hard code the replica of Deployment to 1. However, when users enable the ZooKeeper HighAvailabilityServices or Kubernetes HA service, they would like to configure the replica of JobManager deployment also for faster recovery.

In #15248, we have added the documentation for how to start standby JobManagers for standalone Flink cluster on K8s. This PR tries to make replica of JobManager deployment configurable for native mode.

Brief change log

Support configuring replicas of JobManager deployment when HA enabled

Verifying this change

Add dedicated tests for the changes

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

flinkbot · 2021-03-19T12:14:19Z

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 9742b4a (Thu Sep 23 17:22:53 UTC 2021)

✅no warnings

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

❓ 1. The [description] looks good.
❓ 2. There is [consensus] that the contribution should go into to Flink.
❓ 3. Needs [attention] from.
❓ 4. The change fits into the overall [architecture].
❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
@flinkbot approve all to approve all aspects
@flinkbot approve-until architecture to approve everything until architecture
@flinkbot attention @username1 [@username2 ..] to require somebody's attention
@flinkbot disapprove architecture to remove an approval you gave earlier

flinkbot · 2021-03-19T12:25:37Z

CI report:

9742b4a Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run travis re-run the last Travis build
@flinkbot run azure re-run the last Azure build

wangyang0918 · 2021-03-22T06:03:39Z

cc @tillrohrmann could you please have a look at your convenience?

wangyang0918 · 2021-03-24T10:26:58Z

Rebase latest master.

wangyang0918 · 2021-03-26T03:57:38Z

cc @xintongsong Could you please have a look?

xintongsong · 2021-03-26T04:43:55Z

Thanks for preparing the PR, @wangyang0918.

I'm afraid we are not ready for this feature ATM. As FLINK-21667 reveals, currently a standby RM can perform modifying actions on native Kubernetes deployment. I have not started working on FLINK-21667, because I'd rather not to make significant changes to the RM lifecycle management right before the 1.13 feature freeze.

I'd suggest to postpone this feature as well, marking it as blocked by FLINK-21667.

wangyang0918 · 2021-03-26T05:00:58Z

@xintongsong Thanks for your comments. Even though the standby only stops the terminated pod currently, but it is not a by-design behavior. The standby ResourceManagers should not perform the modifying actions. I agree with you that we could defer this PR after FLINK-21667.

wangyang0918 · 2021-06-04T00:17:46Z

Since the FLINK-21667 has been merged, I think this PR is ready for review. cc @xintongsong

xintongsong

Thanks @wangyang0918. Changes LGTM.
I have two additional questions.

Has the feature been verified on real clusters?
What documentation changes, in addition to the config option description, need to be made for this feature?

xintongsong · 2021-06-04T05:45:16Z

...n/java/org/apache/flink/kubernetes/kubeclient/parameters/KubernetesJobManagerParameters.java

+        checkArgument(
+                replicas > 0,
+                "'%s' should not be configured less than one.",
+                KubernetesConfigOptions.KUBERNETES_JOBMANAGER_REPLICAS.key());
+        if (replicas > 1 && !HighAvailabilityMode.isHighAvailabilityModeActivated(flinkConfig)) {
+            throw new IllegalArgumentException(
+                    "High availability should be enabled when starting standby JobManagers.");
+        }


IllegalConfigurationException is preferred rather than IllegalArgumentException.

…ent when HA enabled

wangyang0918 · 2021-06-04T10:07:42Z

@xintongsong Yes, this feature has been verified in a real K8s cluster. Actually, I think the Flink config option description is enough. But I will add a brief introduction in the native_kubernetes.md.

wangyang0918 · 2021-06-04T10:24:38Z

@xintongsong Thanks for your review. PR updated.

xintongsong

LGTM. Merging this.

rmetzger added review=description? component=Deployment/Kubernetes component=Runtime/Coordination labels Mar 19, 2021

wangyang0918 force-pushed the FLINK-17707-k8s-jm-replicas branch from 519d96d to 6a1950e Compare March 22, 2021 11:34

wangyang0918 force-pushed the FLINK-17707-k8s-jm-replicas branch from 6a1950e to 71743c8 Compare March 24, 2021 10:26

wangyang0918 marked this pull request as draft March 26, 2021 05:01

wangyang0918 force-pushed the FLINK-17707-k8s-jm-replicas branch from 71743c8 to 64a9b36 Compare June 3, 2021 11:58

wangyang0918 marked this pull request as ready for review June 3, 2021 12:02

wangyang0918 force-pushed the FLINK-17707-k8s-jm-replicas branch from 64a9b36 to 967f28c Compare June 4, 2021 00:16

xintongsong reviewed Jun 4, 2021

View reviewed changes

[FLINK-17707][k8s] Support configuring replicas of JobManager deploym…

9742b4a

…ent when HA enabled

wangyang0918 force-pushed the FLINK-17707-k8s-jm-replicas branch from 967f28c to 9742b4a Compare June 4, 2021 10:23

xintongsong approved these changes Jun 7, 2021

View reviewed changes

xintongsong closed this in 5fe3f5b Jun 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-17707][k8s] Support configuring replicas of JobManager deployment when HA enabled #15286

[FLINK-17707][k8s] Support configuring replicas of JobManager deployment when HA enabled #15286

wangyang0918 commented Mar 19, 2021 •

edited

flinkbot commented Mar 19, 2021 •

edited

flinkbot commented Mar 19, 2021 •

edited

wangyang0918 commented Mar 22, 2021

wangyang0918 commented Mar 24, 2021

wangyang0918 commented Mar 26, 2021

xintongsong commented Mar 26, 2021

wangyang0918 commented Mar 26, 2021 •

edited

wangyang0918 commented Jun 4, 2021

xintongsong left a comment

xintongsong Jun 4, 2021

wangyang0918 commented Jun 4, 2021

wangyang0918 commented Jun 4, 2021

xintongsong left a comment

[FLINK-17707][k8s] Support configuring replicas of JobManager deployment when HA enabled #15286

[FLINK-17707][k8s] Support configuring replicas of JobManager deployment when HA enabled #15286

Conversation

wangyang0918 commented Mar 19, 2021 • edited

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Mar 19, 2021 • edited

Automated Checks

Review Progress

flinkbot commented Mar 19, 2021 • edited

CI report:

wangyang0918 commented Mar 22, 2021

wangyang0918 commented Mar 24, 2021

wangyang0918 commented Mar 26, 2021

xintongsong commented Mar 26, 2021

wangyang0918 commented Mar 26, 2021 • edited

wangyang0918 commented Jun 4, 2021

xintongsong left a comment

Choose a reason for hiding this comment

xintongsong Jun 4, 2021

Choose a reason for hiding this comment

wangyang0918 commented Jun 4, 2021

wangyang0918 commented Jun 4, 2021

xintongsong left a comment

Choose a reason for hiding this comment

wangyang0918 commented Mar 19, 2021 •

edited

flinkbot commented Mar 19, 2021 •

edited

flinkbot commented Mar 19, 2021 •

edited

wangyang0918 commented Mar 26, 2021 •

edited