Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[raycluster controller] Always honor maxReplicas #662

Conversation

DmitriGekhtman
Copy link
Collaborator

@DmitriGekhtman DmitriGekhtman commented Oct 28, 2022

Signed-off-by: Dmitri Gekhtman dmitri.m.gekhtman@gmail.com

Why are these changes needed?

If maxReplicas < Replicas, treat maxReplicas as the target replica count and log the discrepancy.
Partly addresses #560 in which some entity (possibly the Ray autoscaler) set replicas > maxReplicas.

See ray-project/ray#29770 for the corresponding safeguard in the autoscaler code.

Related issue number

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

Signed-off-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>
@DmitriGekhtman
Copy link
Collaborator Author

cc also @sihanwang41 @architkulkarni

Signed-off-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>
@DmitriGekhtman
Copy link
Collaborator Author

Will go ahead and add a test.

@peterghaddad
Copy link

Looks good to me!

Signed-off-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>
@DmitriGekhtman
Copy link
Collaborator Author

Will try it out manually as well.

Copy link
Member

@kevin85421 kevin85421 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank @DmitriGekhtman for this contribution!

@DmitriGekhtman
Copy link
Collaborator Author

DmitriGekhtman commented Oct 28, 2022

Confirming that I tested it and it works as expected. I think the behavior change should be fairly uncontroversial.

Copy link
Member

@kevin85421 kevin85421 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's wait for CI to finish.

@kevin85421 kevin85421 merged commit 5e862be into ray-project:master Oct 28, 2022
@DmitriGekhtman DmitriGekhtman deleted the dmitri/reduce-effective-max-workers branch October 29, 2022 00:14
lowang-bh pushed a commit to lowang-bh/kuberay that referenced this pull request Sep 24, 2023
* Honor max replicas

Signed-off-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>

* Tweak message.

Signed-off-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>

* Add to test logic.

Signed-off-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>

Signed-off-by: Dmitri Gekhtman <dmitri.m.gekhtman@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants