Running soft body envs on multiple GPUs

[cchang021]

Hi there! I'm trying to run soft body envs on multiple GPUs, but it's only okay to run on the first card (that card has monitor plugged in). When I'm running on other cards, I did the followings:

export CUDA_VISIBLE_DEVICES=1
python -m mani_skill2.examples.demo_random_action -e Pinch-v0

But I get the following error

RuntimeError: vk::PhysicalDevice::createDeviceUnique: ErrorInitializationFailed

[xuanlinli17]

We found that if you are running ManiSkill2 on a device where the DISPLAY variable is non-empty (e.g., if you are using (or vnc into) a device with a monitor connected), then ErrorInitializationFailed will occur on GPUs that are not rendering the screen, regardless of rigid & soft body environments. Instead, if you ssh into the machine or if you are using a headless machine, in which case the DISPLAY environment variable is empty, then you can run ManiSkill2 envs (regardless of rigid or soft body) on all GPUs.

The current temporary solution is to unset the DISPLAY variable by using the command unset DISPLAY, or remove the DISPLAY variable during program execution by using this command (which I prefer):

DISPLAY="" python -m mani_skill2.examples.demo_random_action -e PickCube-v0
CUDA_VISIBLE_DEVICES=1  DISPLAY="" python -m mani_skill2.examples.demo_random_action -e Pinch-v0

We'll figure out a permanent solution later.

[cchang021]

OK, I run the code on the other GPU successfully! Thanks!