feature(zlx): League training with slime volley env

[LuciusMos]

Related Issues

#17

TODO

• Add slimevolleygym env in dizoo
• PPO tuning in SlimeVolley-v0
• slimevolleygym env visualization
  (ding -m eval -c slime_volley_ppo_config.py -s 8 --load-path=./ckpt_best.pth.tar --replay-path=./video)
• self-play pipeline and tuning

[codecov-commenter]

Codecov Report

Merging #23 (9f64438) into main (c500a2e) will increase coverage by 0.01%.
The diff coverage is n/a.

❗ Current head 9f64438 differs from pull request most recent head 94287f6. Consider uploading reports for the commit 94287f6 to get more accurate results

@@            Coverage Diff             @@
##             main      #23      +/-   ##
==========================================
+ Coverage   89.03%   89.05%   +0.01%     
==========================================
  Files         356      356              
  Lines       25912    25912              
==========================================
+ Hits        23071    23076       +5     
+ Misses       2841     2836       -5     

Flag	Coverage Δ
unittests	89.05% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
ding/model/template/vac.py	98.55% <ø> (ø)
ding/envs/env_manager/subprocess_env_manager.py	83.41% <0.00%> (+0.48%)	⬆️
ding/worker/learner/comm/flask_fs_learner.py	91.87% <0.00%> (+1.87%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c500a2e...94287f6. Read the comment docs.

[zxzzz0]

@PaParaZz1 Could we fix the unit test failure?

[zxzzz0]

Could we enable league metric (TrueSkill) here as we did in the league demo. This will be useful to see the training progress.

[zxzzz0]

@PaParaZz1 Follow up on this. Is it possible that we can have another new config like xxx_league_PPO_using_pretrain_config.py, where use_pretrain is set to True rather than False here so that we don't train from scratch (Motivation is that training from scratch can be slow and we believe using pretrain model can benefit the training)

In this new config each time the model is mutated, it will be reset to the pretrain model. Assume you obtained this pretrain model offline by running PPO to train with the built-in bot dizoo/slime_volley/config/slime_volley_ppo_config.py for a very short period of time.

[PaParaZz1]

Here are DI-engine PPO (vs bot) results in SlimeVolley-v0 env (5 random seed):

[zxzzz0]

Here are DI-engine PPO (vs bot) results in SlimeVolley-v0 env (5 random seed):

@PaParaZz1 Well done! What a nice curve. This curve is very similar to the other people's PPO study as reported here.

Let's use this model as pretrain model and start the league train with TrueSkill.

[PaParaZz1]

Here are DI-engine PPO self-play training results (win rate with built-in AI) in SlimeVolley-v0 env (2 random seed), its performance has outperformed than that in original repo:

[PaParaZz1]

rule-based bot vs trained agent

slime_volley.mp4

[zxzzz0]

(1) CPU problem
We've tried this and found that only one core out of our 64 cores got 100% utilization. The remaining cores are nearly 2% utilization all the time. This situation remains the same even if we change collector_env_num from 8 to 64. Do you know how to fully utilize all cores? Should we change SyncSubprocessEnvManager into something else? @PaParaZz1
(2) GPU problem
We have one machine with 2 GPUs and 8 GiB memory each. Although we've changed the batch_size in learner to a 16x larger number. The memory usage for the first GPU is still only 1.6 GiB and the second GPU memory usage is always 0 GiB.

[PaParaZz1]

For GPU problem, I want to know what kind of multi-gpu implementation you use, torch.nn.DataParallel, torch.DistributedDataParallel or other methods? And you can open another issue to track these two problems.

[zxzzz0]

For GPU problem, we don't do any multi-gpu implementation and simply running slime_volley_league_ppo_config.py from master branch. But we want to fully utilize this single machine with multiple GPUs. I've tried adding one line to learner multi_gpu=True and wrap main with with DistContext():. However it raised errors as we reported here. It looks like it's doesn't support single machine with multiple GPUs.

[PaParaZz1]

OK, I will add torch.DataParallel in DI-engine before 10.25, for single machine with multiple GPUs. And you can pay attention to related PR.

[zxzzz0]

Thanks. In the related PR please also update slime_volley_ppo_config.py to use it.