Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[render_gl] Remove X server requirement #21050

Open
pathammer opened this issue Feb 26, 2024 · 4 comments
Open

[render_gl] Remove X server requirement #21050

pathammer opened this issue Feb 26, 2024 · 4 comments
Assignees
Labels
component: geometry perception How geometry appears in color, depth, and label images (via the RenderEngine API) priority: medium type: feature request

Comments

@pathammer
Copy link
Contributor

pathammer commented Feb 26, 2024

Is your feature request related to a problem? Please describe.

According to @jwnimmer-tri, there's no reason for an X server to be a strict requirement for renderering. It makes it harder to deploy to headless environments for containerized environments.

Describe the solution you'd like

RenderEngineGL could use EGL to create it's OpenGL context.

VTK can automatically use EGL if built with that option. Edit: See #21700.

Describe alternatives you've considered

I considered using xvfb but that does non-HW-accelerated rendering and would be a bottleneck.

@jwnimmer-tri jwnimmer-tri added the component: geometry perception How geometry appears in color, depth, and label images (via the RenderEngine API) label Feb 26, 2024
@jwnimmer-tri
Copy link
Collaborator

FYI on a related topic...

As of Bazel 7.0, to access a real Xorg server during Bazel tests downstream projects must add this line in their .bazelrc:

build --sandbox_add_mount_pair=/tmp/.X11-unix

Otherwise, you'll see an failure such as DRAKE_THROW_UNLESS(display != nullptr); around internal_opengl_context.cc:85.

This is due to the following change in Bazel 7: --incompatible_sandbox_hermetic_tmp is enabled by default.

@jwnimmer-tri
Copy link
Collaborator

jwnimmer-tri commented Mar 30, 2024

I tried some prototyping today...

I used this WIP branch that switches VTK rendering to use EGL instead of GLX. I built a wheel file, installed it on Deepnote, and did apt install libegl-mesa0 instead of install_xvfb. I ran rendering_multibody_plant.ipynb and it worked fine. Thus, EGL seems like a viable option for CPU-based rendering without an X server.

As I understand it, there are ways to teach EGL to use the GPU, but I haven't looked into that specifically yet.

When using EGL, the camera option to show_window = True seemingly has no effect, at least not in my WIP branch. Possibly some device configuration is necessary, or maybe I need to be using Wayland instead of Xorg. Also, most of the VTK render engine tests pass, but two cases fail (for me).

Assuming that EGL can't be made strictly better than GLX, the next steps would be to see if we can compile both GLX and EGL support into Drake's VTK, and offer a RenderEngineVtkParams option for which one to use.

@jwnimmer-tri
Copy link
Collaborator

For our Kitware helpers -- let's say the first milestone on this issue is just RenderEngineVtk. We'll leave the RenderEngineGl for a second milestone.

@jwnimmer-tri
Copy link
Collaborator

Actually, I take that back. Let's use #21700 for the VTK option, and keep this ticket for the RenderEngineGl changes.

@jwnimmer-tri jwnimmer-tri changed the title Remove X server requirement from OpenGL and VTK renderers [render_gl] Remove X server requirement Jul 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: geometry perception How geometry appears in color, depth, and label images (via the RenderEngine API) priority: medium type: feature request
Projects
Status: Todo
Development

No branches or pull requests

5 participants