[data] Fix bug in memory reservation #43511
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
raulchen
requested review from
ericl,
scv119,
c21,
amogkam,
scottjlee,
bveeramani,
stephanie-wang and
omatthew98
as code owners
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
c21
approved these changes
Feb 28, 2024
|
|
||
| E.g., | ||
| - "cur_map->downstream_map" will return [downstream_map]. | ||
| - "cur_map->limit1->limi2->downstream_map" will return [downstream_map]. |
Signed-off-by: Hao Chen <chenh1024@gmail.com>
bveeramani
approved these changes
Feb 28, 2024
Comment on lines
121
to
131
| @pytest.mark.parametrize( | ||
| "cluster_cpus, cluster_obj_store_mem_mb", | ||
| "cluster_cpus, cluster_obj_store_mem_mb, insert_limit_op", | ||
| [ | ||
| (3, 500), # CPU not enough | ||
| (4, 100), # Object store memory not enough | ||
| (3, 100), # Both not enough | ||
| (3, 500, False), # CPU not enough | ||
| (3, 500, True), # CPU not enough | ||
| (4, 100, False), # Object store memory not enough | ||
| (4, 100, True), # Object store memory not enough | ||
| (3, 100, False), # Both not enough | ||
| (3, 100, True), # Both not enough | ||
| ], | ||
| ) |
There was a problem hiding this comment.
Nit: To test all existing combinations with insert_limit_op, it might be cleaner to do another parametrize:
@pytest.mark.parametrize(
"cluster_cpus", "cluster_obj_store_mem_mb",
[
(3, 500),
(4, 100),
(3, 100),
]
)
@pytest.mark.parametrize("insert_limit_op", [False, True])
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
|
Can you help check that the failing tests on premerge are same as ones failing. in go/flaky. Thankks |
|
@can-anyscale checked. one is an existing flaky test, the other is because hugging face is down now. |
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Fix bugs in ReservationOpResourceAllocator (introduced by #43171)
update_resourcesalready handles this properly . But_should_unblock_streaming_output_backpressureand_op_outputs_reserved_remainingdidn't consider this.limitandstreaming_split, should setnum_cpus=0for their tasks._reserved_for_op_outputscurrently also includes op's internal output buffers. This is incorrect, because whenpreserve_order=True, task outputs will accumulate in op's internal output buffer, and use all the memory budget from_reserved_for_op_outputs. Then we still don't have memory budget to pull the blocks from the internal output buffer. Excluding the internal output buffer from_reserved_for_op_outputsfixes this issue.Also deflake
test_backpressure_from_outputandtest_e2e_autoscaling_up, as they depend on physical memory size of the node.Related issue number
closes #43493
closes #43490
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.