Error starting rviz2 from within container - libGL error: MESA-LOADER: failed to retrieve device information

[tleyden]

When I start an autoware container with:

$ rocker --nvidia --x11 --user --volume $HOME/Development/autoware --volume $HOME/Development/autoware_map -- ghcr.io/autowarefoundation/autoware-universe:latest-cuda

and try to start rviz2 (or autoware tries to launch rviz2), it fails with errors:

$ rviz2
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-tleyden'
libGL error: MESA-LOADER: failed to retrieve device information
libGL error: MESA-LOADER: failed to retrieve device information
[ERROR] [1666389050.804997231] [rviz2]: RenderingAPIException: OpenGL 1.5 is not supported in GLRenderSystem::initialiseContext at /tmp/binarydeb/ros-galactic-rviz-ogre-vendor-8.5.1/.obj-x86_64-linux-gnu/ogre-v1.12.1-prefix/src/ogre-v1.12.1/RenderSystems/GL/src/OgreGLRenderSystem.cpp (line 1201)
[ERROR] [1666389050.805238544] [rviz2]: rviz::RenderSystem: error creating render window: RenderingAPIException: OpenGL 1.5 is not supported in GLRenderSystem::initialiseContext at /tmp/binarydeb/ros-galactic-rviz-ogre-vendor-8.5.1/.obj-x86_64-linux-gnu/ogre-v1.12.1-prefix/src/ogre-v1.12.1/RenderSystems/GL/src/OgreGLRenderSystem.cpp (line 1201)
[ERROR] [1666389050.805275164] [rviz2]: InvalidParametersException: Window with name 'OgreWindow(0)' already exists in GLRenderSystem::_createRenderWindow at /tmp/binarydeb/ros-galactic-rviz-ogre-vendor-8.5.1/.obj-x86_64-linux-gnu/ogre-v1.12.1-prefix/src/ogre-v1.12.1/RenderSystems/GL/src/OgreGLRenderSystem.cpp (line 1061)

Workaround/fix

This stack overflow post suggested a workaround of adding:

--devices /dev/dri

to the rocker launch command, which fixed the issue for me.

$ rocker --nvidia --x11 --devices /dev/dri --user --volume $HOME/Development/autoware --volume $HOME/Development/autoware_map -- ghcr.io/autowarefoundation/autoware-universe:latest-cuda

Now rviz2 successfully launches from the container when run standalone or from autoware.

$ rviz2
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-tleyden'
[INFO] [1666394264.504178562] [rviz2]: Stereo is NOT SUPPORTED
[INFO] [1666394264.504282283] [rviz2]: OpenGl version: 4.6 (GLSL 4.6)
[INFO] [1666394264.527670450] [rviz2]: Stereo is NOT SUPPORTED

According to the rocker docs:

For Intel integrated graphics support you will need to mount the /dev/dri directory as follows: --devices /dev/dri

Are other docker users hitting the same issue or is there something broken with my setup that requires this flag?

[evshary]

I tested on two machines with GPU cards by my side. Both are Ubuntu 20.04 (x86)

1. Ubuntu 20.04 + GTX 1050
2. Ubuntu 20.04 + RTX 5000

I met the same issue on GTX 1050 but run successfully on RTX 5000.
I think the problem is that rviz2 does not use NVIDIA GPU but Intel integrated graphics.
You can force to use NVIDIA GPU by the following commands:

prime-select query
# It should show on-demand by default
sudo prime-select nvidia
# Force to use NVIDIA GPU

After reboot, your system will use NVIDIA GPU card by default and rviz2 can run successfully.

Another solution:
If you don't want to use GPU card and want to run with software computation, you can add LIBGL_ALWAYS_SOFTWARE=1.
But it causes bad performance and I don't think that is what you want.

rocker -e LIBGL_ALWAYS_SOFTWARE=1 --x11 --user --volume $HOME/autoware --volume $HOME/autoware_map -- ghcr.io/autowarefoundation/autoware-universe:latest-cuda

[tleyden]

If I run glxinfo on the host with these env variables as suggested in this nvidia forums post then it shows nvidia as the vendor:

$ __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo | grep vendor
server glx vendor string: NVIDIA Corporation
client glx vendor string: NVIDIA Corporation
OpenGL vendor string: NVIDIA Corporation

but for some reason that only works on the host.

In the container, it still shows OpenGL vendor string: Intel Open Source Technology Center even with the additional env variables.

[evshary]

Yes, it can work with __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia.
Actually, you can run RViz with these environmental variables on the host, and it'll use NVIDIA GPU. (I've tried that and use nvidia-smi to check)
That means at least you can run Autoware with NVIDIA GPU on the host.
But for the container, I'm not really sure why it can't.

Maybe you can try to enable gpu-manager first to see whether it can solve the issue or not.

sudo systemctl enable gpu-manager
sudo prime-select nvidia
# reboot

[tleyden]

That's good to know, I will try running autoware from source instead of docker and compare the behavior.

I did try to enable the gpu-manager on the host, re-run prime-select nvidia and reboot, but it didn't seem to change anything.

@evshary if you run __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo | grep vendor in a rocker/docker container, does it show nvidia or intel?

Also did you run rviz2 inside a container or directly on a host? If it was in a container, were you able to verify that it's using the gpu via nvidia-smi?

[evshary]

@tleyden

if you run __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo | grep vendor in a rocker/docker container, does it show nvidia or intel?

While prime-select on-demand, it shows Intel, the same as yours.
While prime-select nvidia, it shows something like this:

server glx vendor string: NVIDIA Corporation
client glx vendor string: Mesa Project and SGI
OpenGL vendor string: Mesa/X.org

Also did you run rviz2 inside a container or directly on a host? If it was in a container, were you able to verify that it's using the gpu via nvidia-smi?

That's a little tricky. nvidia-smi does not show the process in the containers by my side, but I can see that the memory and GPU usage changes while running RViz. So, I think it's using GPU.

[tleyden]

I was able to solve it by setting my host machine to default to the nvidia GPU driver.

I'm running on a system76 machine, and they have their own version of nvidia prime-select. I was able to run this on the host :

system76-power graphics nvidia
sudo reboot

which basically defaults every GUI app to run on the GPU. Now when I start an autoware container with this command:

rocker --nvidia --x11 --user --volume $HOME/Development/autoware --volume $HOME/Development/autoware_map -- ghcr.io/autowarefoundation/autoware-universe:latest-cuda

and run rviz2, it is able to run without errors:

rviz2
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-tleyden'
[INFO] [1667247760.746904715] [rviz2]: Stereo is NOT SUPPORTED
[INFO] [1667247760.747021907] [rviz2]: OpenGl version: 3.1 (GLSL 1.4)
[INFO] [1667247760.767387690] [rviz2]: Stereo is NOT SUPPORTED

It doesn't show up on nvidia-smi for me either, but I verified it was using the GPU by running several rviz2 instances and seeing the available GPU memory decrease each time.

I think this approach would work with nvidia prime-select as well, but I don't have a system to test it.

Thanks again @evshary for all the help!