Camera view from intrinsic/extrinsic parameters

[pablospe]

How to set camera view using intrinsic/extrinsic parameters (used for Computer Vision people)?

intrinsic = [f 0 cx
             0 f cy
             0 0 1]

extrinsic = [R t]

where

f: focal lenght
cx, cy: principal point
R, t: rotation and translation mapping world points to camera coordinates

and width and height is problably needed too.

A 2nd question is: what is the coordinate system used in pyvista/VTK? +X left, +Y down, +Z forward?

    ^
   /
 z
/
--- x --->
|
y
|
v

[github-actions]

Hi and welcome! Thanks for posting your first issue in the PyVista project! Someone from @pyvista/developers will chime in before too long. If your question is support related, it may be automatically transferred to https://github.com/pyvista/pyvista-support

[pablospe]

The following code is an working example (based on make_vtk_camera.cpp).

fragment.ply can be downloaded here.

Code:

import pyvista as pv
import vtk
import math
import numpy as np
import matplotlib.pyplot as plt


def trans_to_matrix(trans):
    """ Convert a numpy.ndarray to a vtk.vtkMatrix4x4 """
    matrix = vtk.vtkMatrix4x4()
    for i in range(trans.shape[0]):
        for j in range(trans.shape[1]):
            matrix.SetElement(i, j, trans[i, j])
    return matrix

def main():
    w = 1024
    h = 768

    intrinsic = np.array([[665.10751011,   0.        , 511.5],
                          [  0.        , 665.10751011, 383.5],
                          [  0.        ,   0.        ,   1. ]])

    extrinsic = np.array([[ 0.95038793,  0.0954125 , -0.29607301, -1.84295291],
                          [-0.1222884 ,  0.98976322, -0.07358197, -1.2214318 ],
                          [ 0.28602154,  0.10613772,  0.95232687,  0.6428006 ],
                          [ 0.        ,  0.        ,  0.        ,  1.        ]])


    # renderer
    p = pv.Plotter(off_screen=True, window_size=[w,h])


    #
    # load mesh or point cloud
    #
    mesh = pv.read("fragment.ply")
    p.add_mesh(mesh, rgb=True)


    #
    # intrinsics
    #

    cx = intrinsic[0,2]
    cy = intrinsic[1,2]
    f = intrinsic[0,0]

    # convert the principal point to window center (normalized coordinate system) and set it
    wcx = -2*(cx - float(w)/2) / w
    wcy =  2*(cy - float(h)/2) / h
    p.camera.SetWindowCenter(wcx, wcy)

    # convert the focal length to view angle and set it
    view_angle = 180 / math.pi * (2.0 * math.atan2(h/2.0, f))
    p.camera.SetViewAngle(view_angle)


    #
    # extrinsics
    #

    # apply the transform to scene objects
    p.camera.SetModelTransformMatrix(trans_to_matrix(extrinsic))

    # the camera can stay at the origin because we are transforming the scene objects
    p.camera.SetPosition(0, 0, 0)

    # look in the +Z direction of the camera coordinate system
    p.camera.SetFocalPoint(0, 0, 1)

    # the camera Y axis points down
    p.camera.SetViewUp(0,-1,0)


    #
    # near/far plane
    #

    # ensure the relevant range of depths are rendered
    # depth_min = 0.1
    # depth_max = 100
    # p.camera.SetClippingRange(depth_min, depth_max)
    # # depth_min, depth_max = p.camera.GetClippingRange()
    p.renderer.ResetCameraClippingRange()

    # p.show()
    # p.render()
    p.store_image = True  # last_image and last_image_depth
    p.close()


    # get screen image
    img = p.last_image

    # get depth
    # img = p.get_image_depth(fill_value=np.nan, reset_camera_clipping_range=False)
    img = p.last_image_depth

    plt.figure()
    plt.imshow(img)
    plt.colorbar(label='Distance to Camera')
    plt.title('Depth image')
    plt.xlabel('X Pixel')
    plt.ylabel('Y Pixel')
    plt.show()


if __name__ == "__main__":
    main()

[pablospe]

I have some questions:

1. Is this the correct way of doing offline rendering? Even though I set up off_screen=True, when I do p.close() a windows appears and disappears instantly (at least in Ubuntu, it doesn't happen on Windows). I would like to do headless rendering, without opening a window.

2. What is the correct way to save/load a depth map image like this?

I think a version of this could be a library example. I believe CV guys would need this if they want to use the library

Thanks!