[k8s] Fix GPU count detection from autoscaler YAML #2636
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
landscapepainter
approved these changes
Nov 21, 2023
| rounded_count = math.ceil(res_count) | ||
| if resource_name == 'cpu': | ||
| # For CPU, we set minimum count to 1 because if CPU count is set to 0, | ||
| # (e.g., when request=0.5), ray will not be able to schedule any tasks. |
There was a problem hiding this comment.
From the comment, request=0.5, what is the request referring to? If it's referring to the request variable used in this function, rounded_count(CPU count) will be set to 1, not 0 when request=0.5. If it's not the variable, I think we should elaborate a bit more in the comment to show that it's not referring to the variable :)
There was a problem hiding this comment.
Good catch - clarified that it's referring to when skypilot task requests something like --cpus 0.5.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #2608.
Our resource detection from pod spec puts a minimum of 1 on CPU resource so that fractional CPU sized clusters can also be launched on k8s clusters. However, this was being incorrectly applied to GPU resources too, causing all clusters to be initialized with 1 GPU resource.
Tested (run the relevant ones):
bash format.shsky launch --cloud kubernetes -- ray statusnow does not show GPUs.pytest tests/test_smoke.py--kubernetes -k "not TestStorageWithCredentials"