[FLINK-22006][k8s] Support to configure max concurrent requests for fabric8 Kubernetes client via JAVA opts or envs

[wangyang0918]

What is the purpose of the change

Flink now could only run 20 jobs or less in a K8s session(native and standalone) when K8s HA enabled.
The root cause is fabric8 Kubernetes client has configured the MaxRequests of OkHttpClient#Dispatcher[1] to 64[2], which means we could not create more than 64 watchers in the JobManager pod.

Normally, it could be configured in Flink via env or java opts. Unfortunately, the fabric8 Kubernetes client version 4.9.2 has a bug[3], which causes the max concurrent requests could not be set via system properties.

This PR is a temporary fix before we bumping the fabric8 Kubernetes version to 4.13.0+ or change the behavior how we watch the leader ConfigMaps.

After this commit, users could use any one in the following Flink config options to set the concurrent max requests, which allows to run more jobs in a session cluster. Please note that, each Flink job will consume 3 concurrent requests.

• containerized.master.env.KUBERNETES_MAX_CONCURRENT_REQUESTS: 200
• env.java.opts.jobmanager: "-Dkubernetes.max.concurrent.requests=200"

1. https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/utils/HttpClientUtils.java#L166
2. https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L135
3. Allow setting the max concurrent requests via sysprop / envvars fabric8io/kubernetes-client#2531

Brief change log

• Support to configure max concurrent requests for fabric8 Kubernetes client via JAVA opts or envs

Verifying this change

• Add two unit tests

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know)
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 2c5c9cd (Fri May 28 08:59:42 UTC 2021)

✅no warnings

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[wangyang0918]

cc @tillrohrmann This PR is ready for review. Please have a look.

[flinkbot]

CI report:

• 2c5c9cd Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run travis re-run the last Travis build
• @flinkbot run azure re-run the last Azure build

[tillrohrmann]

Thanks for creating this PR @wangyang0918. I think your changes should work. What I am missing is the documentation for this feature. In line with this thought, maybe it is clearer if we introduce a Flink configuration option to configure this value. If set, it would overwrite the default value/value configured through some other means. What do you think?

[tillrohrmann]

I am wondering whether we shouldn't rather make it configurable via a Flink configuration option? That way it would also be documented for our users and easier to use.

[wangyang0918]

Since fabric8 kubernetes-client provides many internal advanced configuration properties[1], I do not want to introduce such a special case for kubernetes.max.concurrent.requests. If you think it is a problem that we do not have the documentation, maybe we could add a section in native_kubernetes.md#Flink on Kubernetes Reference

[1]. https://github.com/fabric8io/kubernetes-client/blob/v4.9.2/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L65

[tillrohrmann]

Ok, then let's add a section about how to configure the Fabric8 kube client to native_kubernetes.md where we state how to set these experts options and also tell explicitly about KUBERNETES_MAX_CONCURRENT_REQUESTS if users want to submit a lot of jobs.

[wangyang0918]

@tillrohrmann I have pushed another commit for the documentation.

[tillrohrmann]

Thanks for updating this PR @wangyang0918. The changes look good to me. Merging this PR now.