Support experiment tracking with W&B

[vwxyzjn]

Description

closes #212

Motivation and Context

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

[araffin]

Thanks for the PR =)
In fact, that was something I wanted to add at some point. Could you also document the use in the readme and update the changelog?

[vwxyzjn]

Also a quick question, I would like to also log raw episodic return instead of the average one rollout/ep_rew_mean. What do you think is the best way to do it?

[araffin]

What do you think is the best way to do it?

There are two options:

• use a callback that uses the infos dict to retrieve episodic returns (similar to how we compute rollout/ep_rew_mean)
• read the csv files with a callback

in both cases, you need a callback and you need to be careful when using multiple envs.
For option 2, you could even flush the graph everytime if it's possible (would be the easiest but also more costly in computation time).

[vwxyzjn]

in both cases, you need a callback and you need to be careful when using multiple envs.

This is interesting. I guess for Mutil-agent envs, this can be quite tricky. how do you currently log it?

[araffin]

I guess for Mutil-agent envs, this can be quite tricky. how do you currently log it?

I tend to use an EvalCallback anyway, because the training performance is different from the true performance.
Otherwise, we just loop through the infos list (https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/base_class.py#L459) and update the buffer.
For the monitor files, they are handled by each separately and contains the timestamps which allow to order them afterward.

[vwxyzjn]

Added the documentation. I tried playing with the callbacks but couldn't quite find a way to pass callback other than overwriting the yaml files... Perhaps the raw stats stuff should be put into a separate PR?

[araffin]

Perhaps the raw stats stuff should be put into a separate PR?

yes, please

[araffin]

minor comments, otherwise LGTM =)