Robotic Pose Estimation using Segmented 3-Dimensional images
For full documentation of usage/setup please follow the steps outlined in GUIDE.md.
For prediction-specific documentation please see PREDICTION.md.
This tool uses Artificial Intelligence (AI), specifically image segmentation algorithms provided by PixelLib, to predict robot joint angles.
This is accomplished by segmenting the robot and each of its joints from an RGBD (3-D) image. With a known camera pose relative to the robot, these segmented sections of the image can be compared to a 3-D rendering of the robot in any possible pose.
An error function can be defined, quantitatively comparing these images based on visual and depth-based similarity. For any given input, this characterizes loss as a function of the rendered pose of the robot.
Traversing this loss to find minima therefore enables the depicted robot pose to be estimated.
The aforementioned 3D Rendering is done via Pyrender, with forward kinematics handled by Klamp't.
This requires Tensorflow for both segmentation and pose estimation.
CUDA and cuDNN should first be installed according to the Tensorflow Installation Instructions.
The recommended versions are:
It is recommended to not install Visual Studio integration with CUDA (do a custom install and deselect VS integration).
It is recommended to simply install requirements with pip:
pip install --upgrade -r requirements.txtKeras
Keras sometimes includes .decode('utf8') in its code. This is unnecessary, and causes issues when loading and saving hd5f files.
Notably, every instance of .decode('utf8') in "tensorflow_core\python\keras\saving\hdf5_format.py" can be removed.
This will often cause issues when loading a model for segmentation.
Numpy
This is avoided if using requirements.txt to install
Numpy 1.19.5 may be automatically installed with Tensorflow. This version of Numpy presents memory issues on some machines when utilizing the multiprocessing module, as this repository does.
Numpy 1.19.2 should work with this repository.
Many static constants are specified in robotpose/constants.py. Many of these can be easily modified to change program behavior.
Some constants only change visual appearance of some functions whereas others can change how the entire program behaves; take caution whenever changing more technical constants.
Released under an Apache 2.0 License. See LICENSE for details.