-
Notifications
You must be signed in to change notification settings - Fork 74.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Synchronous Training using SyncReplicasOptimizer #8978
Comments
This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there. Thanks! |
hmm thanks for the reply. I just thought it could be a bug because I was following the instructions as stated on the website but can't get to synchronise the training. |
This is most likely something wrong with the code. Only the chief is supposed to update the variables including global step once the vars collected enough gradients. It is also expected that each worker will do this separately to keep the speed of async runs as much as possible... |
Thanks! I'm not sure how to keep the hidden state consistent across all machines since it's not a trainable parameter but I'll ask on StackOverflow. |
I'm trying to implement a synchronous distributed Recurrent Neural Network using TensorFlow on multiple servers. Here's the link to my code: https://github.com/tushar00jain/spark-ml/blob/master/rnn-sync.ipynb. I've also provided the relevant part below.
I want the computations within the same batch to happen in parallel but I think it's still computing separate RNNs on each worker server and updating the parameters on the parameter server separately. I know this because I am printing the _current_state variable after I run the graph for each batch. Also, the _total_loss for the same global step is different on each worker server.
I'm following the instructions provided at the following links: https://www.tensorflow.org/deploy/distributed#replicated_training https://www.tensorflow.org/api_docs/python/tf/train/SyncReplicasOptimizer
Is this a bug or is there something wrong with my code?
The text was updated successfully, but these errors were encountered: