policy gradient learn function casting episode returns to int when using discrete actions

[Kenneth-Schroeder]

In the learn(...) function of the PGPolicy class, the to_torch_as(...) function is used on the return values of the input batch to convert them to a tensor of similar characteristics as the actions. When using discrete actions, this results in a problem as continuous rewards/returns are casted to integers.

tianshou/tianshou/policy/modelfree/pg.py

Line 134 in c25926d

ret = to_torch_as(minibatch.returns, result.act)

This can be fixed by checking self.action_type of the PGPolicy object and transforming the return values accordingly.

[Trinkle23897]

How about

ret = to_torch(minibatch.returns, result.act.device, torch.float)

[Kenneth-Schroeder]

works as well :)

[Trinkle23897]

Thanks for pointing it out! Are you willing to submit a pull request to fix this issue? (I believe it occurs many times)

[Kenneth-Schroeder]

Yes, I will take a look at it :)

[Kenneth-Schroeder]

Done for PGPolicy. The parameter order is actually ret = to_torch(minibatch.returns, torch.float, result.act.device) :)

[Kenneth-Schroeder]

I fixed code style, ran tests etc. and also checked all other occurrences of to_torch_as(...), but none other looked suspicious. PR should be ready for merge. @Trinkle23897