Slow simulation (1000 iterations take several minutes without rendering)

[Benabdellah22]

I’ve been running the VectorizedMultiAgentSimulator and noticed significant performance issues when compared to other multi-agent environments like PettingZoo. Specifically, the simulator runs much slower even when rendering is turned off.

[matteobettini]

Hello

Running slower without rendering is technically impossible as rendering is an optional function and the omission of a function call cannot slow down a programme.

With regards to comnparison to other enviuronments, the goal of vmas is exactly to be orders of magnitude faster than any other non-vectorized environments (inlcuding anything in PettingZoo) by using gpu batching.

We have comparison scripts with OpenAI MPE (also in PettingZoo) which you can run from the folder https://github.com/proroklab/VectorizedMultiAgentSimulator/tree/main/mpe_comparison and show the scaling laws of vmas on gpu and cpu

I can personally say that I use vmas in my research and I am able to collect hundreds of thousands of frames per second on GPU.

If you want, you can send a minimally reproducible script for the benchmarks you are running and we could disect it together and figure out if there is something wrong

• Could it be that you are simulating just one environment?
• Or maybe you are comparing the time vmas takes in $n$ environments to the time PettingZoo takes for one?

[Benabdellah22]

Hi Matteo Bettini,
Thank you for your response.
I was testing using only one environment. Could that be causing the simulation to run slowly?

[matteobettini]

With one environment the benefits of vmas vectorization are not used and it should be similar to other simulators.

I would try with a few more and then divide the total time by the number of environments. Also, I suggest using a GPU if you have one.

That said, even in the case of one env, vmas should not be significantly slower than average simulators for the same task.

Can I ask what scenario are you running for 1000 steps? Some scenarios are more computationally intensive than others (could be due to number of agents, joints, shapes, and so on)

I suggest try with different scnearios and if there is one that seems unreasonably slow we can have a look.

For example you can compare all the MPE scenarios between vmas and PettingZoo

[Benabdellah22]

here is a short code, i'm testing delta time between one step,
i got 0.18ms,

import vmas
import time

env = vmas.make_env(scenario="joint_passage", num_envs=32, device="cpu", continuous_actions=True,
max_steps=None, seed=None, n_agents=2 )
obs = env.reset()
previous_time = time.time()
obs, rews, dones, info = env.step(env.get_random_actions())
current_time = time.time()
dt = current_time - previous_time
print(dt)

[matteobettini]

Yeah that seem aligned given that joint passage is one of the heaviest tasks (we have multiple collidable boxes and a joint constraint)

other scenarios should be much faste

I can anyway have a look at joint passage and see if we can optimize further

[Benabdellah22]

i understand, i will close this as resolved,