-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[autoscaler][gcp] Retry GCP BrokenPipeError #16952
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM now! I really like this test! It is thorough and gives confidence that the wrapper works as intended!
Thanks! |
Windows failure is object spilling and thus unrelated. |
Why are these changes needed?
Closes #16072
Follows closely the strategy of kubeflow/pipelines#5250
On BrokenPipeError, retry and reconstruct google compute client.
Related issue number
Checks
scripts/format.sh
to lint the changes in this PR.Unit test for retry logic. Manual test with
ray up
to make sure nothing broke.