Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial bare-metal implementation of elastic mode for fault tolerance and auto-scaling #1849

Merged
merged 116 commits into from May 15, 2020
Merged

Initial bare-metal implementation of elastic mode for fault tolerance and auto-scaling #1849

merged 116 commits into from May 15, 2020

Conversation

tgaddair
Copy link
Collaborator

@tgaddair tgaddair commented Apr 7, 2020 •
edited

Resolves #1907.

RFC.

@tgaddair
Initial commit of Elastic Horovod
1b87e1f 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed unit tests
2ad2107 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Removed elastic from run
b1e559f 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed Buildkite tests
3b3f960 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed blacklist check
c6cd306 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Added fault tolerance without scaling
3d3f864 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Refactored registration
8bc8f73 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed unit tests
6fa884d 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Added fault tolerance unit test
e40f034 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed file paths
e6a080c 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Added modules
c4a39e7 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed imports
0c12de9 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
More tests
4a4d128 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Reverting more tests
a0126ef 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed slots
eb6e0f5 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed keras state
a139280 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Removed requirement for host group
ac3963a 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed interactive tests
cda7f8d 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed keras tests
077c33d 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed imports
65e8e01 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Only test elastic
8269341 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed Spark tests
376b6ce 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed broadcast_object args
489d880 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed spark test
c7edff4 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Test remove test
0288423 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
TensorFlow 1.15
a4c80f3 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
TensorFlow CPU mode
37a3d05 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed unit tests
185d4b1 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Back to 1.14
3017726 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Renable all test
75c7cdd 
Signed-off-by: Travis Addair <taddair@uber.com>
tgaddair and others added 6 commits May 11, 2020 12:10
@tgaddair
Added more robust exception handling to controller
ab546b7 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed raw pointer accesses
a8176ad 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Added tests for killing process in addition to raising exceptions
d8150c8 
Signed-off-by: Travis Addair <taddair@uber.com>
@EnricoMi
Add elastic_timeout to ElasticSettings
d842e2a 
Renamed HOROVOD_ELASTIC_START_TIMEOUT to HOROVOD_ELASTIC_TIMEOUT
Renamed BaseSettings.timeout to BaseSettings.start_timeout

Signed-off-by: Enrico Minack <github@enrico.minack.dev>
@EnricoMi
Fix earlier commit
d628073 
Signed-off-by: Enrico Minack <github@enrico.minack.dev>
@EnricoMi
Guard against more psutil.NoSuchProcess
267fbb2 
Signed-off-by: Enrico Minack <github@enrico.minack.dev>
@EnricoMi
Copy link
Collaborator

Looks like commit 5f204ea broke tensorflow.

@tgaddair
Added tests
b8c7f62 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Force shutdown when initial host discovery fails and added test
a81e681 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Added test for min hosts
8804eab 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Changed min hosts check condition
220a868 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Added additional checks around Torch Horovod calls to raise HorovodIn…
617091c 
…ternalError

Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Renaming for consistency
4506a1d 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Do not call _get_host_assignments when we have insufficient slots
f4d7c0b 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Removed caching
959ef05 
Signed-off-by: Travis Addair <taddair@uber.com>
EnricoMi
EnricoMi reviewed May 13, 2020
View reviewed changes
horovod/run/elastic/driver.py
return False
return host in self._host_assignments and len(self._host_assignments[host]) > slot

@property
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why a @property, why not world_size, coordinator_info, results and finished as well? In any case, should not return the mutable dict self._host_assignments but a copy of it, deep copy not necessary here

tgaddair and others added 3 commits May 13, 2020 09:04
EnricoMi
EnricoMi approved these changes May 13, 2020
View reviewed changes
Copy link
Collaborator

@EnricoMi EnricoMi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All comments have been addressed, LGTM!

EnricoMi and others added 5 commits May 13, 2020 23:37
@EnricoMi
Removed obsolete test image
090b8b5 
Signed-off-by: Enrico Minack <github@enrico.minack.dev>
@EnricoMi
Fix gloo test excludes for python 2
d0fb6f4 
Signed-off-by: Enrico Minack <github@enrico.minack.dev>
@tgaddair
Upgrade to TensorFlow 2.2, fix skip tests for TensorFlow < 1.15
08f136e 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Updated Buildkite
dbe703f 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair
Fixed tests for TensorFlow 2.2
4fb7cb6 
Signed-off-by: Travis Addair <taddair@uber.com>
@tgaddair tgaddair merged commit bddcb9e into master May 15, 2020
@tgaddair tgaddair deleted the elastic branch May 16, 2020 17:34
@EnricoMi EnricoMi mentioned this pull request May 22, 2020
Merged
@tgaddair tgaddair mentioned this pull request Jun 22, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abditag2 abditag2 abditag2 left review comments

@nvcastet nvcastet nvcastet left review comments

@sboshin sboshin sboshin left review comments

@zw0610 zw0610 zw0610 left review comments

@EnricoMi EnricoMi EnricoMi approved these changes

@romerojosh romerojosh Awaiting requested review from romerojosh

Assignees
No one assigned
Labels
None yet
Milestone
0.20.0
Development

Successfully merging this pull request may close these issues.

Elastic mode to enable fault tolerance and auto-scaling
7 participants
@tgaddair @mshiryaev @EnricoMi @abditag2 @nvcastet @sboshin @zw0610