RL: first round of some fixes and improvements

[azrael417]

This MR adds a few fixes and improvements to the RL code in the branch. Most notably:

• I somehow misunderstood the real use of target vs active networks. The latter are the important ones, the former are just for stabilization and should never be used externally, i.e. during inference. I fixed this throughout the code. This can be considered a critical fix and should improve skill drastically.
• In SAC, target_entropy was not correctly wired up in the interface and always ignored
• Fixed some warnings/errors.
• added comments for practitioners about useful parameter choice.
• added missing DDP all-reduces for loss reporting in some algorithms. This is not a problem for training, just a reporting problem.
• In DDPG and SAC, the polka update was not tuned to the optimizer updates when using gradient accumulation. This was fixed. For the other algorithms this was already correct.

[azrael417]

/build_and_test

[github-actions]

🚀 Build workflow triggered! View run

[github-actions]

✅ Build workflow passed! View run

[romerojosh]

LGTM!