How to calculate the clicked pixel in the function ViserServer.on_scene_click

[zjcs]

I want to get the clicked pixel when I click the screen. Actually, I find that the normalize pixel (xy_norm) is diffrent to the camera fov setting: (1) xy_norm[0] is not equal to tan_fov[0] when I click the right side; (2) while xy_norm[1] is similar to the tan_fov[1] when I click the bottom side. I think the situation (1) is wrong, anyone can help me?

    @self.server.on_scene_click
    def _(point):
            for client in self.server.get_clients().values():
                camera = client.camera
                
                ray_d = tf.SO3(camera.wxyz).inverse().as_matrix() @ point.ray_direction
                # dxyz, the ray in the camera, under the OpenGL coordination: x->right, y->up, z->far
                xy_norm = torch.Tensor(ray_d[:2])/ray_d[2]
                
                print("camera wxyz    :", camera.wxyz)
                print("camera position:", camera.position)
                print("ray_origin     :", point.ray_origin)
                print("ray_direction  :", point.ray_direction)
                print("ray_d          :", ray_d)
                print("xy_norm        :", xy_norm)
                
                print("aspect    :", camera.aspect)
                fov = torch.Tensor([camera.fov*camera.aspect, camera.fov])
                print("fov rad   :", fov)
                print("fov degree:", np.rad2deg(fov.numpy()))
                print("tan fov   :", torch.tan(fov/2))

output of situation(1)

camera wxyz : [-0.50603067 0.52730842 0.49247169 -0.47259965] [26/12 17:43:21]
camera position: [ 5.17964479 -0.35429741 0.21389997] [26/12 17:43:21]
ray_origin : (5.179644789687916, -0.35429740985762054, 0.2138999720126518) [26/12 17:43:21]
ray_direction : (-0.7240712823396855, 0.6897037779662408, 0.00542924949302979) [26/12 17:43:21]
ray_d : [ 0.63868355 -0.03709912 0.76857464] [26/12 17:43:21]
xy_norm : tensor([ 0.8310, -0.0483]) [26/12 17:43:21]
aspect : 1.8641304347826086 [26/12 17:43:21]
fov rad : tensor([1.5708, 0.8426]) [26/12 17:43:21]
fov degree: [90. 48.27988] [26/12 17:43:21]
tan fov : tensor([1.0000, 0.4482]) [26/12 17:43:22]

output of situation (2)

camera wxyz : [-0.50603067 0.52730842 0.49247169 -0.47259965] [26/12 17:53:23]
camera position: [ 5.17964479 -0.35429741 0.21389997] [26/12 17:53:23]
ray_origin : (5.179644789687916, -0.35429740985762054, 0.2138999720126518) [26/12 17:53:23]
ray_direction : (-0.8963325810726669, 0.016132824571412917, -0.44308874515040325) [26/12 17:53:23]
ray_d : [-0.04507268 0.40585627 0.91282481] [26/12 17:53:23]
xy_norm : tensor([-0.0494, 0.4446]) [26/12 17:53:23]
aspect : 1.8641304347826086 [26/12 17:53:23]
fov rad : tensor([1.5708, 0.8426]) [26/12 17:53:23]
fov degree: [90. 48.27988] [26/12 17:53:23]
tan fov : tensor([1.0000, 0.4482]) [26/12 17:53:24]

[zjcs]

I find out the bug: camera aspect should scale the tan_fov_2 instead of fov degree, then xy_norm is equal to tan_fov_2

tan_fov_2 = torch.Tensor([np.tan(camera.fov/2)*camera.aspect, np.tan(camera.fov/2)])

[brentyi]

Glad you figured out the bug!

For getting the pixel coordinate, I did notice that the GaussianEditor folks forked viser and added the 2D position directly to the click event: heheyas@4c4d8be

This seems cleaner than jumping through the unprojection hoops; would take a PR if you have time. 🙂

[zjcs]

I'm sorry, due to the company's privacy policy, committing code to GitHub is prohibited. I have informed the author of GaussianEditor about this implementation #21 .