Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch Normalization when training with Multi GPU settings #15

Closed
shamanez opened this issue Oct 5, 2017 · 2 comments
Closed

Batch Normalization when training with Multi GPU settings #15

shamanez opened this issue Oct 5, 2017 · 2 comments

Comments

@shamanez
Copy link

shamanez commented Oct 5, 2017

Hi I have modified this code to run on multiple GPUs referencing Cifar 10 multi GPU example . But then I am not sure how to do batch normalization in distributed training . Specially

Calculating moving averages for use in the inference . How to do this ?
@yscacaca
Copy link
Owner

Is it possible to calculate the moving averages on CPU? Such as sess.run(mov_avg.assign(new_val))

@shamanez
Copy link
Author

possible. But then you have to get the statistics of one tower.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yscacaca @shamanez