How to use custom loss ?

[duburcqa]

I would like to add the following extra term to the loss function,

where is the action sampled by the distribution, and can be computed by the actor.

What is the best way to do it using your framework ? The point is being able to take advantage of the analytical gradient computation.

The only way I can think of is to overwrite the whole learn method of the policy (i.e. PPO algorithm), but it feels inconvenient just to add an extra line of code...

Thank you in advance,

Best,
Alexis

[Trinkle23897]

That's a good question. Currently, you can either inherit the policy class (as you mentioned) or change the original framework's code to meet your expectations.

It can be discussed further. Some existing frameworks (like RLlib) modularized the loss function part. But in my opinion, this could be inconvenient for further development. Since the loss function is highly customizable, making the abstraction of the loss function will double the code complexity.

[duburcqa]

Ok, I got your point and I agree with you. But what about adding a loss_fn in the abstract base class for policies, that is basically doing nothing by default but can be overridden by the user? Because I really don't like the idea to have to overwrite 'learn' itself since it is a major source of error.

In this case, it is not a custom loss strictly speaking, but rather additional component to the original loss function (regularization), that may depend on the actor. So that it only consists in an extra function call before calling backward. I don't know if doing so is usual or not.

[duburcqa]

@Trinkle23897 Up !

[Trinkle23897]

@Trinkle23897 Up !

I have no time after #106 before this Friday...Many things to do

[duburcqa]

No problem ! I can do it ! But what do you think about the idea ?

[Trinkle23897]

I think that add loss_fn is okay, but what's its input?

[oldcricket]

@duburcqa It's a great idea to make it easier with a customized loss. I wondered if you have made any progress on that. Thanks!

[MischaPanch]

The loss is an integral part of the algorithm, so maybe inheriting and overriding is better than allowing users to pass custom losses. It's a central design question, I don't see it being necessary for the 1.0.0 release, but would keep the issue open