Add Text rendering support to Visualizer

[qianyizh]

https://learnopengl.com/#!In-Practice/Text-Rendering
https://learnopengl.com/code_viewer.php?code=in-practice/text_rendering

This should go into a Misc category as one of ShaderWrapper's sub-classes.
Main roadblock:
FreeType is a complicated library to compile with.
This is also related to #1

[bigsheep2012]

Hello guys,
Many thanks for open3d 0.6.
Is there any way that I could render some text or numbers (e.g., nms scores) in the visualizer now?
Though I can try to figure out the scores' belongings in a mess scene, it would be nice if I can attach some number/text/captions to specific geometries.

Thanks.

[neoavalon]

https://learnopengl.com/#!In-Practice/Text-Rendering
https://learnopengl.com/code_viewer.php?code=in-practice/text_rendering

This should go into a Misc category as one of ShaderWrapper's sub-classes.
Main roadblock:
FreeType is a complicated library to compile with.
This is also related to #1

Thanks for all your work on open3d.

I would like to see this feature as well. It would be very useful when visualizing 3d box annotations on a point cloud to be able to print object class names and/or prediction scores. When rotating/moving the view it would be nice to have the text always face the camera as well.

As a workaround, is there anything reasonably simple I could do in the meanwhile? Like using pyglfw to grab a handle to the visualizer's window and inject text that way? If I sound crazy don't mind me, I'm completely new to open3d and opengl

Thanks

[dopeboy]

@neoavalon, you can try using a text-to-image technique like shown here and then use read_image to overlay the photo. Here's the code I wrote for my own use case:

WINDOW_WIDTH=1920 # change this if needed
WINDOW_HEIGHT=1080 # change this if needed
img = Image.new('RGB', (WINDOW_WIDTH, WINDOW_HEIGHT), color = (255,255,255))
fnt = ImageFont.truetype('/usr/share/fonts/truetype/ubuntu/Ubuntu-R.ttf', 64)
d = ImageDraw.Draw(img)
d.text((1300,100), "STATUS: GOOD", font=fnt, fill=(0,0,0)) # puts text in upper right
img.save('pil_text.png')

im = o3d.io.read_image("./pil_text.png")
vis.add_geometry(im)

And where ever you create your window, be sure to set the height and width of it using those WINDOW_ variables like so:

vis.create_window(width=WINDOW_WIDTH, height=WINDOW_HEIGHT)

[syncle]

Thanks @dopeboy for the solution that can be adopted readily. Closing for now. FYI, we are working on this issue. Here is a working branch

[heiwang1997]

Here is another interesting version to draw the text as 3D point cloud. Just in case others need this:

def text_3d(text, pos, direction=None, degree=0.0, font='DejaVu Sans Mono for Powerline.ttf', font_size=16):
    """
    Generate a 3D text point cloud used for visualization.
    :param text: content of the text
    :param pos: 3D xyz position of the text upper left corner
    :param direction: 3D normalized direction of where the text faces
    :param degree: in plane rotation of text
    :param font: Name of the font - change it according to your system
    :param font_size: size of the font
    :return: o3d.geoemtry.PointCloud object
    """
    if direction is None:
        direction = (0., 0., 1.)

    from PIL import Image, ImageFont, ImageDraw
    from pyquaternion import Quaternion

    font_obj = ImageFont.truetype(font, font_size)
    font_dim = font_obj.getsize(text)

    img = Image.new('RGB', font_dim, color=(255, 255, 255))
    draw = ImageDraw.Draw(img)
    draw.text((0, 0), text, font=font_obj, fill=(0, 0, 0))
    img = np.asarray(img)
    img_mask = img[:, :, 0] < 128
    indices = np.indices([*img.shape[0:2], 1])[:, img_mask, 0].reshape(3, -1).T

    pcd = o3d.geometry.PointCloud()
    pcd.colors = o3d.utility.Vector3dVector(img[img_mask, :].astype(float) / 255.0)
    pcd.points = o3d.utility.Vector3dVector(indices / 100.0)

    raxis = np.cross([0.0, 0.0, 1.0], direction)
    if np.linalg.norm(raxis) < 1e-6:
        raxis = (0.0, 0.0, 1.0)
    trans = (Quaternion(axis=raxis, radians=np.arccos(direction[2])) *
             Quaternion(axis=direction, degrees=degree)).transformation_matrix
    trans[0:3, 3] = np.asarray(pos)
    pcd.transform(trans)
    return pcd

[MaxChanger]

Thanks to @heiwang1997 for the effective solution to my needs, although there may be a more perfect solution provided by the official I didn't found.
There can be some improvements to the above scheme, such as controlling the font size and point cloud density. For ubuntu users I am not sure that it is necessary to specify the font to an absolute path, you could find some *.tif in /usr/share/fonts. For more flexible control, I made some minor modifications as follows:

def text_3d(text, pos, direction=None, degree=0.0, density=10, font='/usr/share/fonts/truetype/freefont/FreeMono.ttf', font_size=10):
    ......
    font_obj = ImageFont.truetype(font, font_size * density)
    ...
    pcd.points = o3d.utility.Vector3dVector(indices / 1000 / density)
    ...

chessboard_coord = o3d.geometry.TriangleMesh.create_coordinate_frame(
        size=0.02, origin=[0, 0, 0])
pcd_10 = text_3d('Test-10mm', pos=[0, 0, 0.01], font_size=10, density=10)
pcd_20 = text_3d('Test-20mm', pos=[0, 0, 0], font_size=20, density=2)
o3d.visualization.draw_geometries([pcd_10, pcd_20, chessboard_coord])

As shown in the picture, font_size is equivalent to controlling the width of the picture in mm,
font_size=10 means the image width is 10mm, at the same time, the density parameter can be used to control the number of the point cloud of text. (The length of the coordinate axis is 2cm)

[vhu474]

Thanks to @heiwang1997 for the effective solution to my needs, although there may be a more perfect solution provided by the official I didn't found. There can be some improvements to the above scheme, such as controlling the font size and point cloud density. For ubuntu users I am not sure that it is necessary to specify the font to an absolute path, you could find some *.tif in /usr/share/fonts. For more flexible control, I made some minor modifications as follows:

def text_3d(text, pos, direction=None, degree=0.0, density=10, font='/usr/share/fonts/truetype/freefont/FreeMono.ttf', font_size=10):
    ......
    font_obj = ImageFont.truetype(font, font_size * density)
    ...
    pcd.points = o3d.utility.Vector3dVector(indices / 1000 / density)
    ...

chessboard_coord = o3d.geometry.TriangleMesh.create_coordinate_frame(
        size=0.02, origin=[0, 0, 0])
pcd_10 = text_3d('Test-10mm', pos=[0, 0, 0.01], font_size=10, density=10)
pcd_20 = text_3d('Test-20mm', pos=[0, 0, 0], font_size=20, density=2)
o3d.visualization.draw_geometries([pcd_10, pcd_20, chessboard_coord])

As shown in the picture, font_size is equivalent to controlling the width of the picture in mm, font_size=10 means the image width is 10mm, at the same time, the density parameter can be used to control the number of the point cloud of text. (The length of the coordinate axis is 2cm)

Due to how the direction always assumes you are looking from the origin, I couldn't get the text to flip from looking behind the text. Add

img = ImageOps.flip(img)

to fix this

[jiapei-nexera]

How to output characters on top of the 3D viewer's canvas?

[lucasjinreal]

How can I visualzed 2 d text instead?