We should check this at least briefly, especially for the non-gym environments (pufferlib, DACBench, CARL).