Files:

data_collection.py: teleoperates the robot according to keyboard commands (arrow keys, z and c keys, m key to open/close gripper, o and p to rotate gripper). Escape can be used to render the view and save the pose data. The location of the nerf config file should be set at the top of this file teleop.py: contains helper functions for operating the robot render_viewpoint.py: given a nerf file and a novel view matrix, render the view inside the nerf from the camera position. calibrate.py: given the colmap image, the colmap to NeRF matrix and scale, and the matrix from the colmap transforms.json, output the world to nerf matrix. This matrix can then be saved at the top of data_collection.py run_arm.py: given a grasp pose, go to that position, make the grasp,

Information to render a feature field nerf is given in the F3RM repository, which also must be installed: https://github.com/f3rm/f3rm

Please also install pynput

How to get calibration data and run that file

Pick an image from NeRF training dataset that has a clear AprilTag view. From the COLMAP transforms.json, get camera parameters and corresponding transform pose of the camera. We input these values into the calibration file. We run PnP on the detected AprilTag corners and camera transform to get the T_cam_world. We input this value into the data_collection.py file.

From the dataparser_output.json file generated by NeRFStudio, we take the applied_tranform matrix and the scaling value as well for the data_collection.py file. We take the world coordinates of the desired new viewpoint, and create a lookat matrix for it.

We then transform the lookat matrix to COLMAP space by multiplying T_cam_world to give T_colmap_world. We then take this, and flip the axes so it corresponds to the NeRF coordinate convention, and multiply the translation vector with the scaling factor. We then multiply it all with applied_tranform matrix to get the new viewpoint to be generated.