Support to Multi-node Training

[BiEchi]

• I have marked all applicable categories:
  • exception-raising bug
  • RL algorithm bug
  • documentation request (i.e. "X is missing from the documentation.")
  • new feature request
• I have visited the source website
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and environment, where applicable:

  import tianshou, gymnasium as gym, torch, numpy, sys
  print(tianshou.__version__, gym.__version__, torch.__version__, numpy.__version__, sys.version, sys.platform)

Hello, we're using Tianshou for a large model, which requires multi-node training. I see the multi-GPU training tutorial here: https://tianshou.readthedocs.io/en/master/tutorials/cheatsheet.html#multi-gpu. However, I'm not sure whether this can be applied to multi-node training. Any previous attempts are greatly appreciated!

[MischaPanch]

In principle this should work since tianshou supports ray for parallelization, which in turn supports multiple nodes. @Trinkle23897 may know more.

I haven't used tianshou for LLMs yet (trlx seemed like a good fit for that) but would love to hear experiences and use cases. Could you provide some info on your use case and why you want to use tianshou, or is it non-disclosable for your case?

[BiEchi]

Thank you for the prompt response. Unfortuntely, we cannot provide additional details on the specific use cases as this is an ongoing research project. Thank you again for your help and your understanding!

[MischaPanch]

I see, thanks. Currently, multi-node parallelization and LLM support are not the main points on my radar, so I won't be able to help you with this at least until 2024. But other users/devs might have more experience with this