Rendering a single frame and 'human' with MuJoCo

[tudorjnu]

Hello,

I have a problem with the new renderer when combined with MuJoCo.

I am creating a new environment that uses an image-based observation which works well with render_mode="single_rgb_array". However, I would like to be able to visualise the model with MuJoCo's interactive renderer which currently doesn't seem possible.

Any current workarounds or ideas on how this can be accomplished? Thank you!

Additionally, a possible bug. When rendering with a greater image size than 500, I get the following error:

RuntimeError: Offscreen rendering not supported

[pseudo-rnd-thoughts]

Would you be able to provide an example script to test with?
An example that doesn't work and an example of what you would like to run

[tudorjnu]

No problem!

Let's take Humanoid-v4 as an example. A basic gym loop would be the following:

import gym

env = gym.make('Humanoid-v4')
obs = env.reset()
for _ in range(1000):
    action = env.action_space.sample()
    obs, reward, done, info = env.step(action)
    env.render()

Now, let's assume that in the scene, there is a camera in the scene that should give the observation space for the agent. A problem will raise from this as I cannot get the single RGB array from the scene while still being able to watch the scene through MuJoCo's renderer.

A partial solution is to show the scene by showing the images, such as:

import gym
import cv2

env = gym.make('Humanoid-v4', render_mode="single_rgb_array")
obs = env.reset()
for _ in range(1000):
    action = env.action_space.sample()
    obs, reward, done, info = env.step(action)
    image = env.render()
    cv2.imshow("image", image)
    cv2.waitKey(1)

Is there any solution in which both are working?

Regarding the other issue. To replicate. just set the height and width greater than 500:

import gym
import cv2

env = gym.make('Humanoid-v4', render_mode="single_rgb_array",
               width=600, height=600)
obs = env.reset()
for _ in range(1000):
    action = env.action_space.sample()
    obs, reward, done, info = env.step(action)
    image = env.render()
    cv2.imshow("image", image)
    cv2.waitKey(1)

, which throws the following error:

RuntimeError: Offscreen rendering not supported

The latter seems to be an issue more related to MuJoCo itself.

[younik]

@tudorjnu If I understood correctly, you want to use human mode and single_rgb_array simultaneously.
We deprecated this possibility since changing the mode on-the-fly causes problems with other environments.

For your case, I suggest to initialize the env with single_rgb_array and then using this Wrapper https://github.com/openai/gym/blob/master/gym/wrappers/human_rendering.py

In this way, using the Wrapper you will see the human rendering, and when you need you can call gather an RGB frame from the original env.

[tudorjnu]

Yup, that is exactly what I was trying to do.

It seems that when I wrap the environment it renders a black screen only.

[younik]

Yup, that is exactly what I was trying to do.

It seems that when I wrap the environment it renders a black screen only.

You are right, this snippet is not working properly; @rodrigodelazcano @Markus28 do you have any idea why?

import gym
from gym.wrappers import HumanRendering

env = HumanRendering(gym.make('Humanoid-v4', render_mode="single_rgb_array"))
env.reset()

for _ in range(100):
    env.step(env.action_space.sample())

env.close()

[Markus28]

Yes, we are investigating it. It looks like something weird is happening in MuJoCo rendering. Opening a blank PyGame window (completely independently of what we do with the environment) will cause MuJoCo to render only black frames. Check out this snippet:

import numpy as np
import gym
import pygame


env = gym.make("Ant-v4", render_mode="single_rgb_array")
env.reset()
env.render()

########## Comment out to compare #############
pygame.init()
pygame.display.init()
window = pygame.display.set_mode((259, 342))
########## Comment out to compare #############

for _ in range(500):
    _, _, done, _ = env.step(env.action_space.sample())
    print(np.max(env.render()))
    if done:
        env.reset()

We tried using osmesa instead of glfw but that didn't fix the problem. I guess this problem might be difficult to debug.

[tudorjnu]

Ohh, that's a bummer. I will use the standard cv2.imshow for now.

In terms of the rendering, would it be possible in the case of MuJoCo, that the renderer from the wrapper would also be MuJoCo's renderer? This is because it offers some extra functionality such as showing contact forces, increasing the speed and so on.

[Markus28]

Nope I don’t think that will be possible. The new render API essentially forces us to choose one render mode during initialization, so we can’t really access human and rgb_array rendering of mujoco environments at the same time. The original idea behind the HumanRendering wrapper wasn’t actually to circumvent this, but to add human rendering to environments that don’t implement it natively, using their rgb_array renders.

[tudorjnu]

Ohh I see. So the image-based environments would lose their native rendering capabilities.
Truthfully, this didn't work in the previous gym iterations, but I was hoping it would work in this one.

I would leave the issue open for the other two problems, the wrapper not rendering and the size >500 making the environment crash for now.

[YouJiacheng]

Hi! For "So the image-based environments would lose their native rendering capabilities." I guess you just use this for visualization and debug purpose, thus the performance is not critical.
So you can create two exactly identical environments and step them identically. We can add a wrapper for this suggested by #3038 (comment).

[YouJiacheng]

gym/gym/envs/mujoco/mujoco_env.py

Lines 376 to 412 in 6e2e921

	def _render(
	self,
	mode: str = "human",
	width: int = DEFAULT_SIZE,
	height: int = DEFAULT_SIZE,
	camera_id: Optional[int] = None,
	camera_name: Optional[str] = None,
	):
	assert mode in self.metadata["render_modes"]
	if mode in {
	"rgb_array",
	"single_rgb_array",
	"depth_array",
	"single_depth_array",
	}:
	if camera_id is not None and camera_name is not None:
	raise ValueError(
	"Both `camera_id` and `camera_name` cannot be"
	" specified at the same time."
	)
	no_camera_specified = camera_name is None and camera_id is None
	if no_camera_specified:
	camera_name = "track"
	if camera_id is None:
	camera_id = mujoco.mj_name2id(
	self.model,
	mujoco.mjtObj.mjOBJ_CAMERA,
	camera_name,
	)
	self._get_viewer(mode).render(width, height, camera_id=camera_id)
	if mode in {"rgb_array", "single_rgb_array"}:
	data = self._get_viewer(mode).read_pixels(width, height, depth=False)

gym/gym/envs/mujoco/mujoco_env.py

Lines 429 to 448 in 6e2e921

	def _get_viewer(
	self, mode, width=DEFAULT_SIZE, height=DEFAULT_SIZE
	) -> Union["gym.envs.mujoco.Viewer", "gym.envs.mujoco.RenderContextOffscreen"]:
	self.viewer = self._viewers.get(mode)
	if self.viewer is None:
	if mode == "human":
	from gym.envs.mujoco import Viewer
	self.viewer = Viewer(self.model, self.data)
	elif mode in {
	"rgb_array",
	"depth_array",
	"single_rgb_array",
	"single_depth_array",
	}:
	from gym.envs.mujoco import RenderContextOffscreen
	self.viewer = RenderContextOffscreen(
	width, height, self.model, self.data
	)

That is - offscreen viewer(RenderContextOffscreen) need width and height, but _get_viewer use the DEFAULT_SIZE, while read_pixels use width and height passed in constructor(__init__)

gym/gym/envs/mujoco/mujoco_env.py

Lines 77 to 83 in 6e2e921

	render_frame = partial(
	self._render,
	width=width,
	height=height,
	camera_name=camera_name,
	camera_id=camera_id,
	)

[pseudo-rnd-thoughts]

Would you want to make a PR to solve the issue

[YouJiacheng]

Yes I can make a PR, but I'am not sure that is the root cause. I am trying to reproduce the problem locally, then I will make a PR.

[YouJiacheng]

Bad news: it seems that this is not the root cause.

[YouJiacheng]

Yes, we are investigating it. It looks like something weird is happening in MuJoCo rendering. Opening a blank PyGame window (completely independently of what we do with the environment) will cause MuJoCo to render only black frames. Check out this snippet:

import numpy as np
import gym
import pygame


env = gym.make("Ant-v4", render_mode="single_rgb_array")
env.reset()
env.render()

########## Comment out to compare #############
pygame.init()
pygame.display.init()
window = pygame.display.set_mode((259, 342))
########## Comment out to compare #############

for _ in range(500):
    _, _, done, _ = env.step(env.action_space.sample())
    print(np.max(env.render()))
    if done:
        env.reset()

We tried using osmesa instead of glfw but that didn't fix the problem. I guess this problem might be difficult to debug.

I guess this problem might be related to some global state of GL... (i.e. makecurrent)
Just a guess:
pygame makecurrent its own gl context in the same thread.
But this problem doesn't exist on Windows...

[pseudo-rnd-thoughts]

@YouJiacheng @tudorjnu Is this issue solved? If not, what is the remaining issue?

[tudorjnu]

@pseudo-rnd-thoughts Can confirm that now rendering works with different image sizes.

Furthermore, wrapping the "rgb_arrray" environment in HumanRendering is not working still.

To clarify, the remaining issue is the one mentioned by @YouJiacheng one comment above, in which the renderer produces only black frames.

[pseudo-rnd-thoughts]

Thanks @tudorjnu, @rodrigodelazcano (when you have time), could you see if this is an issue with our implementation or mujoco.