Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Autoscaler] Check if SSH is available every 5 sec, not 10 #14484

Merged
merged 3 commits into from
Mar 9, 2021
Merged

[Autoscaler] Check if SSH is available every 5 sec, not 10 #14484

merged 3 commits into from
Mar 9, 2021

Conversation

yiranwang52
Copy link
Contributor

@yiranwang52 yiranwang52 commented Mar 4, 2021 •
edited
Loading

Why are these changes needed?

Shorten the wait interval between SSH check, so the machine become ready sooner.

Related issue number

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

ijrsvt
ijrsvt reviewed Mar 4, 2021
View reviewed changes
python/ray/autoscaler/_private/command_runner.py Outdated
@@ -37,6 +37,7 @@
os.path.dirname(os.path.abspath(__file__)), "kubernetes/kubectl-rsync.sh")
MAX_HOME_RETRIES = 3
HOME_RETRY_DELAY_S = 5
SSH_RETRY_DELAY_S = 5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we define this in https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/_private/constants.py?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

ijrsvt
ijrsvt approved these changes Mar 4, 2021
View reviewed changes
Copy link
Contributor

@ijrsvt ijrsvt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@yiranwang52
Copy link
Contributor Author

@ijrsvt @AmeerHajAli can you help merge this?

@ijrsvt
Copy link
Contributor

ijrsvt commented Mar 4, 2021 •
edited
Loading

@yiranwang52 Can you fix lint first?

python/ray/autoscaler/_private/updater.py:13:80: E501 line too long (82 > 79 characters)

Running ./scripts/format.sh should also print this error!

AmeerHajAli
AmeerHajAli approved these changes Mar 4, 2021
View reviewed changes
Copy link
Contributor

@AmeerHajAli AmeerHajAli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Just to understand:

  1. this makes the ssh interval a parameter (default to 5)
  2. the NODE_START_WAIT_S is renamed to AUTOSCALER_NODE_START_WAIT_S.

@yiranwang52
Copy link
Contributor Author

LGTM.
Just to understand:

  1. this makes the ssh interval a parameter (default to 5)
  2. the NODE_START_WAIT_S is renamed to AUTOSCALER_NODE_START_WAIT_S.

Correct.

@ijrsvt ijrsvt added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Mar 4, 2021
@AmeerHajAli
Copy link
Contributor

@yiranwang52 , LINT:
python/ray/autoscaler/_private/updater.py:13:80: E501 line too long (82 > 79 characters)

@yiranwang52 yiranwang52 removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Mar 9, 2021
@ijrsvt ijrsvt changed the title [Autoscaler] Check if SSH is available every 5 sec instead of 10 sec [Autoscaler] Check if SSH is available every 5 sec, not 10 Mar 9, 2021
@ijrsvt ijrsvt merged commit a06dc39 into ray-project:master Mar 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AmeerHajAli AmeerHajAli AmeerHajAli approved these changes

@ijrsvt ijrsvt ijrsvt approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@yiranwang52 @ijrsvt @AmeerHajAli