Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tune] Cross-Node Recovery #3725

Merged
merged 10 commits into from
Jan 15, 2019
Merged

[tune] Cross-Node Recovery #3725

merged 10 commits into from
Jan 15, 2019

Conversation

richardliaw
Copy link
Contributor

@richardliaw richardliaw commented Jan 9, 2019 •
edited
Loading

Augments trial restore to also check if the runner is at the same
location. If not, the checkpoint files are pushed onto the new location.

TODO:

  • Test this on EC2
  • Autoscaler test (maybe add into stress tests?)
    • Option 1: Have a helper fn that blocks until the ray cluster is the expected size, then submit tune job, down scale cluster, and then make sure the exit code of the submit job is 0.

We don't have any tests for this right now...

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10691/
Test FAILed.

@richardliaw richardliaw mentioned this pull request Jan 9, 2019
Closed
3 tasks
@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10703/
Test PASSed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10830/
Test FAILed.

@richardliaw richardliaw changed the title [wip][tune] Cross-Node Recovery [tune] Cross-Node Recovery Jan 14, 2019
ericl
ericl reviewed Jan 14, 2019
View reviewed changes
python/ray/tune/logger.py
@@ -122,6 +123,12 @@ def flush(self):
self._log_syncer.sync_now(force=True)
self._log_syncer.wait()

def update_location(self, worker_ip):
"""Sends the current log directory to the remote node."""
if worker_ip != self._log_syncer.worker_ip:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this crash on non-autoscaler clusters?

Also, log a warning if sync is happening?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done; and no, won't crash (1sync_to_worker_if_possible1 catches it)

ericl
ericl reviewed Jan 14, 2019
View reviewed changes
python/ray/tune/logger.py Outdated
@@ -122,6 +123,12 @@ def flush(self):
self._log_syncer.sync_now(force=True)
self._log_syncer.wait()

def update_location(self, worker_ip):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update_location() sounds more harmless than this is, maybe sync_results_to_new_location()?

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10855/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10857/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10866/
Test PASSed.

@richardliaw
Copy link
Contributor Author

This is ready for another review.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10885/
Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10886/
Test PASSed.

@richardliaw richardliaw added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Jan 15, 2019
@richardliaw richardliaw merged commit 3918934 into ray-project:master Jan 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericl ericl ericl approved these changes

Assignees

@ericl ericl

Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@richardliaw @AmplabJenkins @ericl