SAMZA-2075: Add execEnvContainerId and execEnvAttemptId for diagnostics for Kubernetes job coordinator#1566
Merged
cameronlee314 merged 2 commits intoapache:masterfrom Dec 7, 2021
Conversation
…cs for Kubernetes job coordinator
Contributor
|
LGTM, but someone else who has more context on this should take a look and approve |
rmatharu-zz
reviewed
Dec 6, 2021
samza-core/src/main/java/org/apache/samza/metrics/reporter/MetricsHeader.java
Outdated
Show resolved
Hide resolved
rmatharu-zz
reviewed
Dec 6, 2021
samza-core/src/main/java/org/apache/samza/metrics/reporter/MetricsHeader.java
Outdated
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issues: In YARN today, a deployment attempt id which is shared between containers for a single Samza job can be extracted from the exec-env-container-id in the diagnostics information emitted by a job. However, in other execution environments (e.g. Kubernetes), there may not be an exec-env-container-id that can be parsed to get a deployment attempt id. It isn't ideal that this attempt id must be extracted from another field.
Changes:
samza-epoch-idfield toMetricsHeaderwhich contains the id for a deployment attempt and emit this field in the diagnostics messages.SAMZA_EPOCH_IDenvironment variable.Testing:
API changes (backwards compatible):
JSON representation of
MetricsHeader(used inMetricsSnapshot) has an additional optional field calledsamza-epoch-id, and this field is intended to contain the deployment attempt id which is shared across all containers (job coordinator and workers) of a Samza job. This field is filled in by reading theSAMZA_EPOCH_IDenvironment variable.