[FLINK-24947] Support hostNetwork for native K8s integration on session mode

[spoon-lz]

What is the purpose of the change

Provide hostnetwork network mode for jobs running in native kubernetes mode to obtain better performance in unmanaged clusters(for session mode)

Brief change log

When a job is running in session mode, you can start the hostnetwork mode by configuring "kubernetes.hostnetwork.enabled=true". After it is turned on, the pod of JobManager and TaskManager will use the host network instead of the third-party CNI plug-in.

1. Use a random port when starting a session
2. Update the targetPort of the corresponding ports of InternalService and ExternalService to the real value whenever the session starts successfully

Verifying this change

The session and job are started when "kubernetes.hostnetwork.enabled=true" is configured. When the task is running normally, the JobManager is killed. After the JobManager is automatically started, the TaskManager will be correctly connected to the JobManager and can be connected to the session through flink-client

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
• The serializers: ( no)
• The runtime per-record code paths (performance sensitive): (no )
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes )
• The S3 file system connector: (no )

Documentation

• Does this pull request introduce a new feature? (yes )
• If yes, how is the feature documented? (docs)

[flinkbot]

CI report:

• 237027a Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[flinkbot]

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit 7fcbf27 (Wed Dec 15 12:16:33 UTC 2021)

Warnings:

• No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

• ❓ 1. The [description] looks good.
• ❓ 2. There is [consensus] that the contribution should go into to Flink.
• ❓ 3. Needs [attention] from.
• ❓ 4. The change fits into the overall [architecture].
• ❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
• @flinkbot approve all to approve all aspects
• @flinkbot approve-until architecture to approve everything until architecture
• @flinkbot attention @username1 [@username2 ..] to require somebody's attention
• @flinkbot disapprove architecture to remove an approval you gave earlier

[spoon-lz]

@flinkbot run azure

[spoon-lz]

@flinkbot run azure

[spoon-lz]

@flinkbot run azure

[wangyang0918]

@spoon-lz Thanks for creating this PR. I think it could work in your internal use case. But we need to refine the PR in a good shape before merging. I left some comments and please have a look.

[spoon-lz]

@wangyang0918 Thanks for helping review this PR, give me some time to modify them

[wangyang0918]

@spoon-lz I appreciate that you updated this PR. It has looked really better to me.

I have left some comments about the implementations, especially about how to get the dynamic ports in KubernetesResourceManagerDriver. Also the unit tests could be optimized.

[wangyang0918]

@spoon-lz Thanks for updating this PR again. It looks almost good to me and I just left some minor comments. Once they are addressed, I will try to merge this PR. Please make sure that this PR is also verified in minikube or a real K8s cluster.

Moreover, could you please squash the commits and refine the commit message. Maybe they could looks like following.

• [hotfix][runtime] Extract a common method to parse rest port from web interface URL
• [FLINK-24947][k8s] Support host network for native K8s integration

[wangyang0918]

@spoon-lz I have quickly verified this PR works well in a minikube. Great work.

[spoon-lz]

@wangyang0918 I think the code is sorted out already

[wangyang0918]

Thanks for updating this PR. I will address my last minor comments when merging.

[wangyang0918]

cc @spoon-lz If you have no other comments, I will merge this PR once the azure pipeline gives the pass.

[spoon-lz]

@wangyang0918 Thanks for your help, I think it's ok now

[wangyang0918]

@spoon-lz I believe that you forget the changes in ClusterEntrypoint.java when you squash and refine the commits.

[wangyang0918]

Force pushed because:

• fix the BlobServerOptions.PORT is not set in the ClusterEntrypoint.java
• rebase master

[spoon-lz]

@wangyang0918 Thanks for helping me modify the code

[wangyang0918]

Merging this PR.