Interactive rendering for SafetyGym

[joshwa71]

Hi there,

Thanks for the codebase, an awesome algorithm to play around with. I'm trying to render the env to sanity check the agent and am wondering how this is done. I've tried calling the render function in the from_gym.py script but this is not working. Simply running env.render in the step function also is not working.

Any help appreciated, thanks.

[jobesu14]

That might help.

#2 (comment)

[joshwa71]

I'm just trying to render a gym env not pygame, any idea in which file I need to call env.render() in order for this to work?

[jobesu14]

If I am not mistaken, you can still render the gym env as it is done here and get rid of all the pygame specific code...

The render() function of line 22 should be the one you are interested in (the one from your gym env).

Note: in the linked code, the extra callback to render the env is added to the embodied.run.eval_only script, but you can do the same for the embodied.run.train script.

[joshwa71]

I have instead added a call to the render() function in the from_gym.py step function, though it seems to only display the first frame and then goes black which is odd as the steps counter continues. Code below:

def step(self, action):
if action['reset'] or self._done:
self._done = False
obs = self._env.reset()

  return self._obs(obs, 0.0, is_first=True)
if self._act_dict:
  action = self._unflatten(action)
else:
  action = action[self._act_key]
self._env.render()
obs, reward, self._done, self._info = self._env.step(action)
return self._obs(
    obs, reward,
    is_last=bool(self._done),
    is_terminal=bool(self._info.get('is_terminal', self._done)))

Edit: I am using SafetyGym which uses Mujoco-py backend.

[danijar]

You can move the env out of a subprocess/thread with --envs.parallel none, maybe that fixes rendering in that environment?

In general, you can view videos in TensorBoard if your environment has an image observation key (or you can specify different keys via e.g. --run.log_video_keys top_down_image side_image.

For mujoco specifically, you may want to check/change the env variable MUJOCO_PY, which I set to egl in the FromDM wrapper to allow rendering on headless servers.

[vtopiatech]

Hey @danijar excellent work with DreamerV3! Do you know is there any simple way to "play together" with the AI e.g. in Crafter? Meaning if I press a key, it takes an action in the game (while the AI is playing)