Prototype RLops Utility

[vwxyzjn]

Description

Related to #296. To get started, run the following command

python rlops.py --exp-name ddpg_continuous_action_jax \
    --wandb-project-name cleanrl \
    --wandb-entity openrlbenchmark \
    --tags  v1.0.0b2-9-g4605546 latest \
    --env-ids Hopper-v2 Walker2d-v2 HalfCheetah-v2 \
    --output-filename compare.png \
    --report

This PR generates this wandb report (which does not display properly due to wandb/wandb#4421; update: a temporary fix pip install git+https://github.com/wandb/wandb.git@6ccd140dabb0af529302b07f4d7ecc1235773900 works (https://github.com/wandb/wandb/tree/andrew/report-api-fixes)) and the following image

As part of this effort, I have labeled all of the existing runs in openrlbenchmark with rlops-pilot if the do not already have a label.

Types of changes

• Bug fix
• New feature
• New algorithm
• Documentation

Checklist:

• I've read the CONTRIBUTION guide (required).
• I have ensured pre-commit run --all-files passes (required).
• I have updated the documentation and previewed the changes via mkdocs serve.
• I have updated the tests accordingly (if applicable).

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
cleanrl	❌ Failed (Inspect)		Dec 15, 2022 at 3:07AM (UTC)

[kinalmehta]

Great work!!

• I believe currently, only comparisons within openrlbenchmark are supported.
  • If someone wants to use openrlbenchmark to compare with their own results(stored in their own wandb account and not openrlbenchmark), they'll have to manually modify this to include entity and project matching for both versions(openrlbenchmark and their own wandb account details).
• might have to add expt to the dependencies

Everything else LGTM

[vwxyzjn]

Support more advanced filter syntax

python -m cleanrl_utils.rlops --wandb-project-name cleanrl \
    --wandb-entity openrlbenchmark \
    --filters 'ddpg_continuous_action_jax?user=joaogui1&tag=rlops-pilot' 'ddpg_continuous_action_jax?user=joaogui1&tag=pr-298' \
        'ddpg_continuous_action_jax?user=costa-huang&tag=rlops-pilot' 'ddpg_continuous_action_jax?user=costa-huang&tag=pr-298' \
        'ddpg_continuous_action?user=costa-huang&tag=pr-299' 'ppo_continuous_action?user=costa-huang&tag=rlops-pilot'  \
    --env-ids Hopper-v2 Walker2d-v2 HalfCheetah-v2 \
    --output-filename compare.png --report

which produces the following image and a wandb report with the same line color, thanks to @andrewtruong's amazing PR pip install git+https://github.com/wandb/wandb.git@6ccd140dabb0af529302b07f4d7ecc1235773900

[vwxyzjn]

Closed by #368