[bug-fix] Separate critic only for PPO

[ervteng]

Proposed change(s)

This might be a controversial change, but probably less so with hybrid-actions on the horizon.

In PPO-Torch, we use a separate actor and critic network for continuous actions, and a shared one for discrete. In TF, we always used a separate network. This causes some differences in performance between TF and Torch (particularly with the FoodCollector environment, where the combined network does not have enough capacity for both action and value).

In the principle of least surprise, I think we should change Torch to use a separate network always, at least until TF is deprecated.

Types of change(s)

• Bug fix
• New feature
• Code refactor
• Breaking change
• Documentation update
• Other (please describe)

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)
• Updated the migration guide (if applicable)

Other comments

[vincentpierre]

Do you plan on adding this to release 10 ?