IN PPO, clipping the value loss with max is OK?

[CosmosShadow]

in file 'pposgd_simple.py' line 117,

vf_loss = .5 * U.mean(tf.maximum(vfloss1, vfloss2)) # we do the same clipping-based trust region for the value function

why not tf.minimum ?

[lezhang-thu]

I got the same doubt before. However when I saw the following:
pol_surr = - U.mean(tf.minimum(surr1, surr2)) # PPO's pessimistic surrogate (L^CLIP)
(baselines/baselines/pposgd/pposgd_simple.py)

Negative minimum is maximum, isn't it? So I think I can understand it now.

Supplement:
Sorry. I think there are bugs in my reasoning. Maybe it should be like this:
We need to maximize - vfloss, and hence it is equivalent to minimize vfloss.

Just consider maximize - vfloss. Who will be the final representative of vfloss?
There are just two candidates: vfloss1, vfloss2. Hence the - tf.maximum(vfloss1, vfloss2) will serve as a lower bound for - vfloss. I think it might have some intuition from the pessimistic bound for pol_surr.
Anyway, the final representative of vfloss is tf.maximum(vfloss1, vfloss2).

[CosmosShadow]

@lezhang-thu Thank you.
After rethinking, I got it.

when pi.vpred is between in oldpi.vpred and ret, and the distance from pi.vpred to oldpi.vpred is large than clip_param, the pi.vpred will be clipped. for example
oldpi.vpred < pi.vpred < ret, pi.vpred - oldpi.vpred > clip_param
then,
pi.vpred -> oldpi.vpred + clip_param

this is the result of tf.maximum(vfloss1, vfloss2)

[lezhang-thu]

@CosmosShadow I don't quite follow your idea.
Consider the example: pi.vpred < oldpi.vpred < ret, and pi.vpred - oldpi.vpred < -clip_param.
Then tf.maximum(vfloss1, vfloss2) would be tf.square(pi.vpred - ret), right?
Under this case, pi.vpred is not implicitly clipped.

Under your example, pi.vpred is implicitly clipped. That's true.

[CosmosShadow]

Yes, It's not clipped. then pi.vpred is far from target than oldpi.vpred, need to moving it to target.

Then 'pi.vpred' is closer to target than oldpi.vpred, it seems that the situation is being better. and In order to avoid drastic changes, clip the vpred.

[lezhang-thu]

@CosmosShadow Maybe it's a good idea to write in English, totally.
Your view seems to make sense. Maybe it gives some intuition to the question of why the code is written in this way.

ps,
After rethinking, I feel your view is good. Thanks!

[shtse8]

Sorry, I have the same question. After reading all comments, I still don't understand why we need to maximize the vfloss1 and vfloss2 for stability and there is no other resource on internet about this technique.

[lezhang-thu]

code from ppo2

        # Clip the value to reduce variability during Critic training
        # Get the predicted value
        vpred = train_model.vf
        vpredclipped = OLDVPRED + tf.clip_by_value(train_model.vf - OLDVPRED, - CLIPRANGE, CLIPRANGE)
        # Unclipped value
        vf_losses1 = tf.square(vpred - R)
        # Clipped value
        vf_losses2 = tf.square(vpredclipped - R)

        vf_loss = .5 * tf.reduce_mean(tf.maximum(vf_losses1, vf_losses2))

ppo alg. is sort of like trpo. both optimize the obj under the condition that the optimization is within a trust region.
i just want to describe one case so you can picture it.
consider the case when OLDVPRED < R.
now we have to update train_model.vf, i.e. vpred, so vpred is closer to R after the update.
case 1 is if vpred is within the trust region of OLDVPRED, then nothing needs to be done, tf.maximum(vf_losses1, vf_losses2) would degenerate to tf.square(vpred - R).
case 2 if vpred is outside trust region of OLDVPRED. then there would be case 2.1 and case 2.2.
case 2.1 is for the case when after the update vpred is closer to the trust region, and also closer to R.
case 2.1 is a perfect case, as we need to update vpred to optimize the objective of closing to R, also to be closer to trust region.
so for case 2.1, it happens when train_model.vf < OLDVPRED - CLIPRANGE. and now you'll see why tf.maximum(vf_losses1, vf_losses2) is needed here, as we want to keep tf.square(vpred - R).
case 2.2 is subtle. this is the case when after the update vpred is going away farther from the trust region, and also closer to R.
for case 2.2 we cannot update, as that would disobey the spirit of trust region, i.e., all the updates should be done within the trust region or for the "within the trust region" condition to be more possible.
so for case 2.2, it happens for example OLDVPRED + CLIPRANGE < train_model.vf < R. this time, tf.maximum then would choose vf_losses2. this is literally a constant, hence no grad, and no update ever happens.

hope this explanation helps. for anyone needed it.

[yueyang130]

@lezhang-thu very clear and logical explanation！thanks！

[Bonifatius94]

ppo alg. is sort of like trpo. both optimize the obj under the condition that the optimization is within a trust region.

Are you talking about maximizing utility $U(X)$ or minimizing loss $L(X) = - U(X)$? Mathematically $max \lbrace U(X) \rbrace = min \lbrace - U(X) \rbrace$ is very clear, but it's not so clear what objective you're referring to.