Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for handling multiple RMs sans Hadoop config #29

Merged
merged 1 commit into from
May 29, 2019

Conversation

kevin-bates
Copy link
Member

@kevin-bates kevin-bates commented Mar 21, 2019

Alternate address and port parameters can now be passed to the ResourceManager
constructor. This has been facilitated via a keyword args dict (kwargs) to
accommodate future parameters.
If both address and alt_address are provided,
address and port will be used to determine if the RM is active. If not, then
alt_address and alt_port will be used. If the latter is active, the address
and port member variables are reset to reflect the alternates.

Also added method get_active_host_port() to return the address, port tuple currently
in use for the ResourceManager instance. This is helpful to clients in cases where
the address was None or HA is configured.

Fixes: #9

EDIT: Removed use of kwargs per review comment.

@lresende
Copy link
Collaborator

While looking into this, a couple of things comes to mind:

  • In the case of reading the configuration from HADOOP_CONF we need to identify HA related information such as if standby node has been configured and set it properly on the yarn-client-api metadata.
  • Once we have the proper configuration and the client establishes the proper master at initialization time, we need to handle the case where the standby had been activated due to master failure and, during API invocation, start using the new master.

Having said that, this might be out of the scope of this initial PR and I would be ok to have this as a new open issue for a later time.

yarn_api_client/resource_manager.py Outdated Show resolved Hide resolved
Also added method `get_active_host_port()` to return the address,port
tuple currently in use for the ResourceManager instance.
@kevin-bates
Copy link
Member Author

@lresende - this has been rebased with master to include recent merge. (Thanks @dimon222)

Copy link
Collaborator

@lresende lresende left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lresende lresende merged commit 4c0612d into gateway-experiments:master May 29, 2019
@kevin-bates kevin-bates deleted the ha-rms branch September 5, 2019 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Yarn ha mode support is not very good
2 participants