Headless OpenGL rendering

[qianyizh]

This can be useful in scenarios where users want to render a (long) video.
It can also be useful when Open3D is deployed in a remote server and window creation is a problem.

Some information regarding GLFW: glfw/glfw#648

[syncle]

I am looking into this issue and read some documents.
According to this glfw 3.3 support headless rendering using OSMesa.
The glfw version included in Open3D seems to be 3.1. Can we upgrade glfw for this?

[qianyizh]

Yes.

[syncle]

As we discussed today, glfw 3.3 is not available yet. So we need to find the detour.
I tested following code with one of GCP virtual machine

glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE);

but got the error.

GLFW Error: X11: The DISPLAY environment variable is missing
Failed to initialize GLFW
Failed creating OpenGL window.

Basically glfwWindowHint(GLFW_VISIBLE, GLFW_FALSE); hides existing window. This method does not working in the environment GLFW cannot make a window.

[qianyizh]

Recently I have the needs for glfw rendering on headless server and have gain some success.
See this:
glfw/glfw#1004

In short, Alexey's email is absolute correct.

First, you need to setup Xorg correctly.
You can check /etc/X11/xorg.conf to see if it is configured headlessly (usually via sudo nvidia-xconfig -a --use-display-device=none).
Stop lightdm and restart Xorg to make sure the correct configuration is running.

Then, if you don't want to specify a GPU, you can just setup the environment variable DISPLAY to :0 and test the OpenGL setup.

export DISPLAY=:0
glxinfo | grep OpenGL

You should see something like:

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: TITAN Xp/PCIe/SSE2
OpenGL core profile version string: 4.5.0 NVIDIA 384.90
OpenGL core profile shading language version string: 4.50 NVIDIA
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.5.0 NVIDIA 384.90
OpenGL shading language version string: 4.50 NVIDIA
OpenGL context flags: (none)
OpenGL profile mask: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 384.90
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
OpenGL ES profile extensions:

If your server has multiple GPUs, you will be able to see multiple copies of the above information.

If you have passed this test, setup the DISPLAY environment variable and try some program with glfwInit(), say

export DISPLAY=:0
./some_program_that_has_glfwinit

There is a risk that you will have this error

glfw.GLFWError: (65544) b'X11: RandR gamma ramp support seems broken'

This is an error of version 3.2 and older. They don't support headless rendering. Since 3.3 (current master) it is supported.
Then everything should be fine.

In addition, if you want to specify GPU, you can use VirtualGL and TurboVNC to mock a display. This is done with the following steps.

First, install VirtualGL and TurboVNC, something like:

sudo dpkg -i virtualgl_xxx.deb
sudo dpkg -i turbovnc_xxx.deb

Then launch vnc server to run glfw.

// These two steps make sure that the right headless Xorg is configured
sudo service lightdm stop
sudo nohup Xorg :0 &

// Launch a vncserver to mock display on :1
// You can change it to :2, :3, ...
/opt/TurboVNC/bin/vncserver :1

// Launch program on :1
export DISPLAY=:1
vglrun -d :0.0 glxinfo
vglrun -d :0.0 <a script that calls glfw>

[qianyizh]

Another option is to mimic libigl's tcpviewer.
See this for reference:
https://github.com/libigl/libigl/blob/master/python/tcpviewer.py

Lower priority. Close for now.

[vkoltun]

This is pretty important. Do we have to close this issue? Can we release Open3D while this is open? I'm guessing we can. But I don't want this to fall through the cracks, I want this clearly on the list of issues that need to be addressed soon. (Perhaps not prior to release, but soon after.) Vladlen

…

On Mon, Dec 4, 2017 at 11:18 AM, Qianyi Zhou ***@***.***> wrote: Another option is to mimic libigl's tcpviewer. See this for reference: https://github.com/libigl/libigl/blob/master/python/tcpviewer.py Lower priority. Close for now. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJra2imGPNUwSeKsdH4b2VLZU35rtm_mks5s9EWcgaJpZM4NXPQW> .

[qianyizh]

Reopen per request.

[bhack]

Other than VNC there is also xpra https://xpra.org/trac/wiki/Usage/OpenGL

[rukie]

As another user, this would be an excellent feature! 👍

[syncle]

Working on this. I am using glfw3.3_dev to avoid hassles.

Edit: my working branch with glfw3.3_dev.

[takanokage]

Another approach is to use Docker to create a headless environment for Open3D.
I already have a docker image based on ubuntu 16.04 with xvfb and x11vnc that works. I've also tested one light window manager (ratpoison) but I'm not too happy with it and want to test others. Next I need to test building/running Open3D inside it.

If we need access to the host GPU (NVIDIA) there are already good solutions to get that: NVIDIA-docker and NVIDIA docker images with CUDA and cuDNN.

I've also been looking into using Alpine as the base if the image size is a concern. The Ubuntu image with xvfb and x1vnc gets to about 450MB while the Alpine image to just about 70MB.

[qianyizh]

The purpose is to make Open3D rendering part running on cloud (AWS/GCloud). @syncle can you help test the solution on GCloud?

[syncle]

Sure. I can use a fresh Ubuntu machine running on GCloud. @takanokage, can you elaborate your steps for me? That will be used for documents too.

[takanokage]

I will post more once I have it working.
I'm at the point where I can build Open3D inside the docker container. However the verification step after the build fails, for this reason I added issue 241.

[takanokage]

I got Open3D built and working inside a Docker container.
However I'm getting the same X11: RandR gamma ramp support seems broken error.
Next: I'll check the VirtualGL suggestion by Qianyi.

$ cd ~/Open3D/build/lib/Tutorial/Basic
$ python3 rgbd_redwood.py
press q

Read Redwood dataset
RGBDImage of size 
Color image : 640x480, with 1 channels.
Depth image : 640x480, with 1 channels.
Use numpy.asarray to access buffer data.
GLFW Error: X11: RandR gamma ramp support seems broken
GLFW Error: Linux: Failed to watch for joystick connections in /dev/input: No such file or directory
GLFW Error: Linux: Failed to open joystick device directory /dev/input: No such file or directory

[syncle]

From @qianyizh's post

There is a risk that you will have this error

glfw.GLFWError: (65544) b'X11: RandR gamma ramp support seems broken'

This is an error of version 3.2 and older. They don't support headless rendering. Since 3.3 (current master) it is supported. Then everything should be fine.

It might be good to try Open3D + GLFW3.3_dev solution via

git clone -b headless_rendering https://github.com/syncle/Open3D.git

[takanokage]

Jaesik, I just tried your branch with glfw 3.3 inside my docker container.
Somehow I'm still getting the same X11: RandR gamma ramp support seems broken error.
I seem to remember you mentioned I need to build with a particular flag, can you tell me what that was again? tx!

[syncle]

Yes. Try it with

cmake -DOpen3D_USE_NATIVE_DEPENDENCY_BUILD=OFF ../src

Edit:
Check out what cmake says. It should say like building glfw from source.

[takanokage]

Thanks!
So that did something:
GLFW Error: OSMesa: Library not found
Failed to create window
[DrawGeometries] Failed creating OpenGL window.

Did you install a special package for this?

[takanokage]

I have headless rendering working inside docker with the master branch of Open3D.
Next I'll be working on the documentation before submitting a pull request.

[takanokage]

Bind mounts (#254)
I propose we make the whole Open3D folder available both inside the docker container and on the host side. In this way we not only gain direct access to the images generated but we also enable developing Open3D on the host side and testing on both host side and docker side of things.

[takanokage]

Because we moved #254 under #17 I need to postpone the pull request until after I implement/test it.

[takanokage]

I got headless working together with bind mounting.
Bind mounting was a ton of work and should've been an independent pull request.

[takanokage]

Usage notes:

1. clone my branch
2. $ cd <Open3D path>/utilities/docker/ubuntu-xvfb/tools
3. $ ./build.sh to build the image
4. $ ./attach.sh to clone/build the master Open3D repo and start the Open3D container
5. $ ./headless_sample.sh to run the sample that renders some images and saves them to disk.
   The sample will render some images which can be accessed on the host at ~/Open3D_docker. These files don't go away if the container is stopped.

Notes:

• the sample will not return. Ctrl+c to exit. Need to update the sample to exit on it's own.
• the cloning is done for now to ~/Open3D_docker. We have options: find a way to let the user specify the destination or just reuse the current location of Open3D.
• TODO: uncomment entries in the dependencies section inside the Dockerfile.

[syncle]

Addressed in #273.