Trust Region Updates

[random-user-x]

Hello @Kaixhin,

ACER/train.py

Line 97 in f22b07c

trust_loss += (param * z_star_p).sum()

I think that we should freeze the value of z_star_p by using z_star_p.detach().

In the second stage, we take advantage of back-propagation. Specifically, the updated gradient with
respect to φθ, that is z_∗, is back-propagated through the network to compute the derivatives with
respect to the parameters. 

Please let me know what do you think.

[random-user-x]

#13

[Kaixhin]

Ah yes the gradients are probably leaking through z_star_p, I think you are right on this one.