-
Notifications
You must be signed in to change notification settings - Fork 195
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
why do we need x as argument of train() ? #16
Comments
great observation! the main reason for recomputing forward pass is that
there is a "lag" between prediction and training and the output of the
network (and as a result gradients) might be (probably are if there is more
than one agent) different in training. So, I don't think you can save any
computation. However, if the problem is communication-bound (i.e. CPU-GPU
connection is the bottleneck which is the case for Atari), then it is
possible to optimize the algorithm by keeping X inside the GPU and then
avoid sending it at training time.
…On Tue, Feb 28, 2017 at 4:54 AM, etienne87 ***@***.***> wrote:
In NetworkVP, ThreadTrainer, we see that we call train_model with x, r, a
(states, rewards, actions)
Wouldn't it be simpler to just to maintain an history of p,v inside for
each agent number inside NetworkVP & compute loss + backprop when rewards
come back in the train function with agent indices ?
Thus we would avoid recomputing forward (already been done in predict),
possibly even accelerating the whole process?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#16>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABEfT-U-L1klVw1ipFZA_iZsVtvpEL_sks5rg_zTgaJpZM4MOP4D>
.
|
@mbz Thanks a lot for taking the time to explain, makes more sense now! I understand more now the issue with "old gradients", since network has been updated in the mean time. Do you have any reference proving this is critical ? You are also right that most of the bottleneck is probably in the CPU-GPU connection for small models. If I understand correctly, we should still maintain a db indexed by ProcessAgent ids inside NetworkVP right & progressively concatenate x's at each predict call inside its own batch maintained on GPU? (there would be NAgent batches maintained inside NetworkVP) [EDIT] : I randomly stumbled upon this paper on drl for super smash brawl :
This looks pretty close to GA3C training configuration no? |
Yes, this seems to be very related to the lag that I was referring to. You can also refer to Policy Lag section of our ICLR paper for more details on it. Also yes, we need to keep the track of all the Xs and update their target values when we had them. Implementing this with TensorFlow might be a little tricky but I can help getting into more details if you are interested. |
the new version of your paper is great & more detailed ! Stabilizing (by increasing batch size, thus reducing delay) does not seem to improve convergence, (Figure 8), or is it just because model is less frequently updated? I am not sure to find in the paper if you tried to update the model with gradient which is not in sync? I mean without even recomputing forward (which makes things less delayed but delayed nonetheless If I understood correctly) I am interested in helping. I will try small changes & point them to you as it goes. |
Please have a look at the final version of the ICLR paper: here |
Thanks the precision @ifrosio . Just to add some results confirming what you say, I ran a small experiment in my branch "origin/gae", where I test 4 different configs on CartPole-v0 on 2 paramters : batchsize & advantage formula (R-V or Generalized Advantage Estimation). The results suggests Batch Size can have a huge impact on the stability of the algorithm. (Sorry for the time axis in hours, I should have kept the number of steps) All other hyperparams :
|
tried to apply loss in chainer variant . (By the way TF looks wayy faster (at least 3 times) than Chainer or Pytorch for the same problem). the "train_offpolicy" function did not succeed to make converge the basic CartPole problem. Closing issue as @mbz is right : the only thing possible is to keep x gpu tensors & concatenate them before training happen is the only credible acceleration. It also means LSTM will be at least 5x slower than FF one. |
In NetworkVP, ThreadTrainer, we see that we call train_model with x, r, a (states, rewards, actions)
Wouldn't it be simpler to just to maintain an history of p,v inside for each agent number inside NetworkVP & compute loss + backprop when rewards come back in the train function with agent indices ?
Thus we would avoid recomputing forward (already been done in predict), possibly even accelerating the whole process?
The text was updated successfully, but these errors were encountered: