Implement CPU based offscreen rendering

[ScheiklP]

Pyglet supports offscreen rendering through EGL, but PyGame does not.
The combination of RenderFramework.PYGAME and RenderMode.HEADLESS thus currently not work.

For the future, we should fallback to LLVMpipe and OSMesa for this case.

import os
import numpy as np
from PIL import Image
from OpenGL.GL import *
from OpenGL.GLUT import *
from OpenGL.raw.osmesa import mesa as osmesa

# Use LLVMpipe for rendering
os.environ['LIBGL_ALWAYS_SOFTWARE'] = '1'
os.environ['GALLIUM_DRIVER'] = 'llvmpipe'

# create a context
ctx = osmesa.OSMesaCreateContextExt(osmesa.OSMESA_RGBA, 32, 0, 0, None)
if not ctx:
    print('Could not create OSMesa context')
    exit(1)

# create a framebuffer
width, height = 640, 480
buffer = np.zeros([height, width, 4], dtype=np.uint8)

# make the context current
if not osmesa.OSMesaMakeCurrent(ctx, buffer, GL_UNSIGNED_BYTE, width, height):
    print('Could not make the OSMesa context current')
    exit(1)

# setup viewport
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)
glMatrixMode(GL_PROJECTION)
glLoadIdentity()
glOrtho(-1, 1, -1, 1, -1, 1)
glMatrixMode(GL_MODELVIEW)
glLoadIdentity()

# render a red triangle
glBegin(GL_TRIANGLES)
glColor3f(1, 0, 0)  # Red color
glVertex2f(-0.5, -0.5)
glVertex2f(0.5, -0.5)
glVertex2f(0, 0.5)
glEnd()

# unbind the context before deletion
osmesa.OSMesaMakeCurrent(None, None, GL_UNSIGNED_BYTE, 0, 0)

# clean up
osmesa.OSMesaDestroyContext(ctx)

# flip the buffer vertically (origin is different in OSMesa and PIL)
buffer = np.flipud(buffer)

# convert the buffer to an image and save it
img = Image.fromarray(buffer, 'RGBA')
img.save('output.png')

[somefoo]

For systems with the Nvidia GPUs and their proprietary driver: make sure to set __GLX_VENDOR_LIBRARY_NAME=mesa

Example to test if LLVMpipe is available (output should be OpenGL renderer string: llvmpipe ...):
__GLX_VENDOR_LIBRARY_NAME=mesa LIBGL_ALWAYS_SOFTWARE=1 GALLIUM_DRIVER=llvmpipe glxinfo | grep "OpenGL renderer"

Command to run glxgears (apt install mesa-utils) with LLVMpipe:
__GLX_VENDOR_LIBRARY_NAME=mesa LIBGL_ALWAYS_SOFTWARE=1 GALLIUM_DRIVER=llvmpipe glxgears

[ScheiklP]

Since Gymnasium uses pygame, I took a look, and they use Surface instead of display, when the render_mode is set to rgb_array. Maybe that fixes the issue?

• https://github.com/Farama-Foundation/Gymnasium/blob/main/gymnasium/envs/toy_text/frozen_lake.py#L355
• https://github.com/Farama-Foundation/Gymnasium/blob/main/gymnasium/envs/toy_text/frozen_lake.py#L435

[ScheiklP]

Does not work.

[ERROR]   [SofaRuntime] ValueError: vector<bool>::_M_fill_insert
....
    self.sofa_simulation.initVisual(self._sofa_root_node)