[FLINK-20866][yarn] Set high-availability.cluster-id to application id if not configured #14577
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…d if not configured In order to easily connect to an HA enabled Yarn cluster, this commit sets the high-availability.cluster-id to the application id if not configured. This is symmetric to how we deploy HA enabled clusters and makes it unnecessary to explicitly set the high-availability.cluster-id if one tries to reconnect to the cluster.
Collaborator
|
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit 615f3b2 (Fri May 28 07:03:11 UTC 2021) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. DetailsThe Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
zentol
reviewed
Jan 7, 2021
| YarnConfigOptions.APPLICATION_ID, ConverterUtils.toString(clusterId)); | ||
|
|
||
| // set cluster-id to app id if not specified | ||
| if (!flinkConfiguration.contains(HighAvailabilityOptions.HA_CLUSTER_ID)) { |
Contributor
There was a problem hiding this comment.
Should we deduplicate/sync this block with the one in #startAppMaster?
Contributor
Author
There was a problem hiding this comment.
Ok, let me see what I can simplify here.
Contributor
Author
There was a problem hiding this comment.
Ok, I've removed YarnClusterDescriptor.zookeeperNamespace as it seems to be no longer used.
zentol
reviewed
Jan 7, 2021
| flinkConfiguration.set( | ||
| YarnConfigOptions.APPLICATION_ID, ConverterUtils.toString(clusterId)); | ||
|
|
||
| // set cluster-id to app id if not specified |
Contributor
There was a problem hiding this comment.
maybe rename the clusterId variable to appId
Contributor
Author
There was a problem hiding this comment.
Jup, good idea.
zentol
approved these changes
Jan 7, 2021
Contributor
zentol
left a comment
zentol
left a comment
There was a problem hiding this comment.
+1 functionality-wise, but ideally we do some small cleanups while we're at it.
Collaborator
…ation id if not configured
The YarnClusterDescriptor.zookeeperNamespace has been replaced by the configuration which is deployed together with the Yarn cluster. Hence, it is no longer needed.
Contributor
Author
|
Thanks for the review @zentol. I've addressed your comments and updated this PR. |
tillrohrmann
added a commit
to tillrohrmann/flink
that referenced
this pull request
Jan 8, 2021
…d if not configured In order to easily connect to an HA enabled Yarn cluster, this commit sets the high-availability.cluster-id to the application id if not configured. This is symmetric to how we deploy HA enabled clusters and makes it unnecessary to explicitly set the high-availability.cluster-id if one tries to reconnect to the cluster. This closes apache#14577.
tillrohrmann
added a commit
to tillrohrmann/flink
that referenced
this pull request
Jan 8, 2021
…d if not configured In order to easily connect to an HA enabled Yarn cluster, this commit sets the high-availability.cluster-id to the application id if not configured. This is symmetric to how we deploy HA enabled clusters and makes it unnecessary to explicitly set the high-availability.cluster-id if one tries to reconnect to the cluster. This closes apache#14577.
tillrohrmann
added a commit
that referenced
this pull request
Jan 8, 2021
…d if not configured In order to easily connect to an HA enabled Yarn cluster, this commit sets the high-availability.cluster-id to the application id if not configured. This is symmetric to how we deploy HA enabled clusters and makes it unnecessary to explicitly set the high-availability.cluster-id if one tries to reconnect to the cluster. This closes #14577.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
In order to easily connect to an HA enabled Yarn cluster, this commit sets the
high-availability.cluster-id to the application id if not configured. This is
symmetric to how we deploy HA enabled clusters and makes it unnecessary to
explicitly set the high-availability.cluster-id if one tries to reconnect
to the cluster.
Verifying this change
Added
YARNHighAvailabilityITCase.testClusterClientRetrievalDoes this pull request potentially affect one of the following parts:
@Public(Evolving): (yes / no)Documentation