[Core] Fix placement group leaks #42942
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
can we have a unit test, e.g. start a pg and kill it right away, and see if there's any leak? |
added 2 commits
February 2, 2024 22:29
|
@rynewang I am figuring that out. This requires very subtle timing (indeed, I could repro when I created so many pgs only), so writing an unit test is not trivial. Writing cpp unit test is even harder because of how our code is structured. |
rkooo567
commented
Feb 6, 2024
| // Retry 10 * worker registeration timeout to avoid race condition. | ||
| // See https://github.com/ray-project/ray/pull/42942 | ||
| // for more details. | ||
| /*max_retry*/ RayConfig::instance().worker_register_timeout_seconds() * 10, |
There was a problem hiding this comment.
Q: should we do exponential backoff? Right now, we retry every 1 second.
rkooo567
changed the title
|
@jjyao it is ready to be reviewed |
|
Q: Right now ,we retry for 10 minutes with 1 second interval. Should we
|
When can it happen? |
When a worker is not started yet! |
jjyao
reviewed
Feb 7, 2024
| const rpc::CancelResourceReserveReply &reply) { | ||
| RAY_LOG(DEBUG) << "Finished cancelling the resource reserved for bundle: " | ||
| << bundle_spec->DebugString() << " at node " << node_id; | ||
| [this, bundle_spec, node_id, node, max_retry, current_retry_cnt]( |
There was a problem hiding this comment.
Where do you use the max retry
There was a problem hiding this comment.
oops.. let me fix this. I will also add an unit test
| @@ -59,8 +59,6 @@ void ClusterResourceManager::AddOrUpdateNode( | |||
|
|
|||
| void ClusterResourceManager::AddOrUpdateNode(scheduling::NodeID node_id, | |||
| const NodeResources &node_resources) { | |||
| RAY_LOG(DEBUG) << "Update node info, node_id: " << node_id.ToInt() | |||
There was a problem hiding this comment.
This seems useless and too verbose. I can bring it back if you think it is necessary
|
@jjyao it'd be great if you can approve the PR. I will add changes and merge it |
jjyao
approved these changes
Feb 7, 2024
ratnopamc
pushed a commit
to ratnopamc/ray
that referenced
this pull request
Feb 11, 2024
Root cause: When we schedule a task, we allocate resources and start a worker. If cancel bundle request is received before a worker is started, there's a leak because cancel bundle kills a worker and returns resources from "workers that are already started". It fixes the issue by retrying cancellation. This also means If a worker starts late (it has 60 seconds timeout), retry can fail as we have max number of retry. We retry for a very long time (10 * registration timeout), so it is unlikely to happen Alternatively, to improve the consistency, we can also do register removed pg and keep deleting resources (with a reconciler) until it is fully gone. register leased workers before workers are started. This can be a better solution, but the implication of this change is unknown. I chose the current solution as it is needed for a network issue as well. Signed-off-by: Ratnopam Chakrabarti <ratnopamc@yahoo.com>
rkooo567
added a commit
to rkooo567/ray
that referenced
this pull request
Feb 12, 2024
Root cause: When we schedule a task, we allocate resources and start a worker. If cancel bundle request is received before a worker is started, there's a leak because cancel bundle kills a worker and returns resources from "workers that are already started". It fixes the issue by retrying cancellation. This also means If a worker starts late (it has 60 seconds timeout), retry can fail as we have max number of retry. We retry for a very long time (10 * registration timeout), so it is unlikely to happen Alternatively, to improve the consistency, we can also do register removed pg and keep deleting resources (with a reconciler) until it is fully gone. register leased workers before workers are started. This can be a better solution, but the implication of this change is unknown. I chose the current solution as it is needed for a network issue as well.
aslonnie
pushed a commit
that referenced
this pull request
Feb 13, 2024
Root cause: When we schedule a task, we allocate resources and start a worker. If cancel bundle request is received before a worker is started, there's a leak because cancel bundle kills a worker and returns resources from "workers that are already started". It fixes the issue by retrying cancellation. This also means If a worker starts late (it has 60 seconds timeout), retry can fail as we have max number of retry. We retry for a very long time (10 * registration timeout), so it is unlikely to happen Alternatively, to improve the consistency, we can also do register removed pg and keep deleting resources (with a reconciler) until it is fully gone. register leased workers before workers are started. This can be a better solution, but the implication of this change is unknown. I chose the current solution as it is needed for a network issue as well.
tterrysun
pushed a commit
to tterrysun/ray
that referenced
this pull request
Feb 14, 2024
Root cause: When we schedule a task, we allocate resources and start a worker. If cancel bundle request is received before a worker is started, there's a leak because cancel bundle kills a worker and returns resources from "workers that are already started". It fixes the issue by retrying cancellation. This also means If a worker starts late (it has 60 seconds timeout), retry can fail as we have max number of retry. We retry for a very long time (10 * registration timeout), so it is unlikely to happen Alternatively, to improve the consistency, we can also do register removed pg and keep deleting resources (with a reconciler) until it is fully gone. register leased workers before workers are started. This can be a better solution, but the implication of this change is unknown. I chose the current solution as it is needed for a network issue as well. Signed-off-by: tterrysun <terry@anyscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Root cause: When we schedule a task, we allocate resources and start a worker. If cancel bundle request is received before a worker is started, there's a leak because cancel bundle kills a worker and returns resources from "workers that are already started".
It fixes the issue by retrying cancellation. This also means
Alternatively, to improve the consistency, we can also do
I chose the current solution as it is needed for a network issue as well.
Related issue number
Closes #26761
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.