Fixes actor_loss shape for SAC continuous

[dosssman]

Description

Address issues pointed out in #379

Types of changes

• Bug fix

Checklist:

• I've read the CONTRIBUTION guide (required).
• I have ensured pre-commit run --all-files passes (required).
• I have updated the tests accordingly (if applicable).
• I have updated the documentation and previewed the changes via mkdocs serve.
  • I have explained note-worthy implementation details.
  • I have explained the logged metrics.
  • I have added links to the original paper and related papers.

If you need to run benchmark experiments for a performance-impacting changes:

• I have contacted @vwxyzjn to obtain access to the openrlbenchmark W&B team.
• I have used the benchmark utility to submit the tracked experiments to the openrlbenchmark/cleanrl W&B project, optionally with --capture-video.
• I have performed RLops with python -m openrlbenchmark.rlops.
  • For new feature or bug fix:
    • I have used the RLops utility to understand the performance impact of the changes and confirmed there is no regression.
  • For new algorithm:
    • I have created a table comparing my results against those from reputable sources (i.e., the original paper or other reference implementation).
  • I have added the learning curves generated by the python -m openrlbenchmark.rlops utility to the documentation.
  • I have added links to the tracked experiments in W&B, generated by python -m openrlbenchmark.rlops ....your_args... --report, to the documentation.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
cleanrl	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 8, 2023 3:35am

[dosssman]

Didn't manage to get the rlops working yet, so regression report was done manually:

https://api.wandb.ai/links/openrlbenchmark/on2vqz6u

[timoklein]

https://api.wandb.ai/links/openrlbenchmark/on2vqz6u

So the version that has the bug fixed is actually worse? That's odd.

[dosssman]

Happened a few times before. Probably due to stochasticity that occurs during the sampling process, or due to the difference en environment / hardware.
Might add more runs to ascertain it, if you feel like it is necessary.
Performance regression is only on Walker2d it seems, the rest has very close performance to the rl-pilot baseline.

[timoklein]

Might add more runs to ascertain it, if you feel like it is necessary.

I can run a couple of experiments if you like but not before May 18th. But to me, it looks OK.

Probably due to stochasticity

I agree. We'd probably need 50+ runs to properly verify anything anyway, that's a little excessive :D

[dosssman]

All good on my side too.

[dosssman]

Fixes #379