Fix multi-gpu metrics report #4187
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixed #2508. I'm not sure why GPU device does not allow reusing the same ClusterSpec but leaving it blank seems to work.
Tested with
GPU_NUM_DEVICES=8 python test_train_mp_mnist.py --metrics_debug --fake_data --num_epochs 1.Single-gpu metrics report: https://gist.github.com/ymwangg/609b79ed783a1797f87d55480d23bc8d
Multi-gpu metrics report: https://gist.github.com/ymwangg/8218ddf4ed8ce2d1ac35758effabdf6f
cc @YangFei1990