SAMZA-2604: Datamodel change to capture physical container id for AM HA#1445
Merged
mynameborat merged 5 commits intoapache:masterfrom Dec 1, 2020
Merged
Conversation
mynameborat
reviewed
Nov 30, 2020
Contributor
mynameborat
left a comment
mynameborat
left a comment
There was a problem hiding this comment.
I forgot about this PR and ended up reviewing these changes as part of the orchestration PR. Sorry about that. let me know if thats okay or I can port the comments over to reduce overhead.
lakshmi-manasa-g
changed the title
mynameborat
reviewed
Dec 1, 2020
Contributor
mynameborat
left a comment
mynameborat
left a comment
There was a problem hiding this comment.
Minor comment. Rest of it looks good to me.
samza-core/src/main/java/org/apache/samza/runtime/ContainerLaunchUtil.java
Outdated
Show resolved
Hide resolved
mynameborat
approved these changes
Dec 1, 2020
Contributor
|
Please resolve the conflict so that I can merge the PR. |
mynameborat
approved these changes
Dec 1, 2020
lakshmi-manasa-g
added a commit
to lakshmi-manasa-g/samza
that referenced
this pull request
Feb 9, 2021
…HA (apache#1445) Feature: Main feature is Cluster based Job coordinator (aka AM) high availability. The feature ensures that the new AM can establish connection with already running containers to avoid restarting all running containers when AM dies. This PR enables capturing of the physical execution environment container id (ex: yarn container id "container_123_123") mapping to Samza logical processor id (ex: "0"). In future PRs, this mapping will be used by the new AM. Changes: Introduce new Coordinator Stream Message and manager to read/write this message Container upon launch will write to c-stream, its logical and physical id Job Coordinator(AM) upon launch will read the mapping of all containers from c-stream. Tests: added unit test for new manager working on tests for other classes as they have no coverage for relevant code. API changes: New c-stream message Usage instructions: None Upgrade instructions: Backwards compatible. N/A
tranjith
pushed a commit
to tranjith/samza
that referenced
this pull request
Mar 23, 2021
…HA (apache#1445) Feature: Main feature is Cluster based Job coordinator (aka AM) high availability. The feature ensures that the new AM can establish connection with already running containers to avoid restarting all running containers when AM dies. This PR enables capturing of the physical execution environment container id (ex: yarn container id "container_123_123") mapping to Samza logical processor id (ex: "0"). In future PRs, this mapping will be used by the new AM. Changes: Introduce new Coordinator Stream Message and manager to read/write this message Container upon launch will write to c-stream, its logical and physical id Job Coordinator(AM) upon launch will read the mapping of all containers from c-stream. Tests: added unit test for new manager working on tests for other classes as they have no coverage for relevant code. API changes: New c-stream message Usage instructions: None Upgrade instructions: Backwards compatible. N/A
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature: Main feature is Cluster based Job coordinator (aka AM) high availability (HA) (TODO: sep/doc how?). The feature ensures that the new AM can establish connection with already running containers to avoid restarting all running containers when AM dies. This PR enables capturing of the physical execution environment container id (ex: yarn container id "container_123_123") mapping to Samza logical processor id (ex: "0"). In future PRs, this mapping will be used by the new AM.
Changes:
Tests:
API changes:
Usage instructions: None
Upgrade instructions: Backwards compatible. N/A