Synthetic data generation of surgical instruments' 6D pose datasets​

Package to generate 6D pose datasets of surgical instruments. The package is based on the Surgical Robotic challenge simulation environment and the BOP toolkit.

Main functionalities:

1. Replaying of pre-recorded trajectories (See simple_replay.py)
2. Recording of 6D pose datasets in the BOP format (See collect_data.py).
3. Reader class for datasets generated in BOP format (See BopDatasetReader).

Getting started

The following code base will require Ubuntu 20.04, ROS noetic, AMBF, and the surgical robotics challenge simulation environment. After obtaining all dependencies, first install ambf6dpose package with:

pip install -e .

and then install additional dependencies with:

pip install -r requirements/requirements.txt
sudo apt install ros-noetic-ros-numpy

Data generation

To generated data, you will first need to open the surgical robotics challenge scene and then replay motions with the simple_replay.py script. While the motions are being replayed, you can collect data with the collect_data.py script. For more information about to each script refer see below.

Replay instrumention motion

python scripts/simple_replay.py single-replay --help

Usage: simple_replay.py single-replay [OPTIONS]

Options:
  --bag_path PATH            Path to bag file
  --percent_to_replay FLOAT  Path to bag file
  --ecm_pos FLOAT LIST       ECM joint position, specifid as a string of 4
                             float separated with a space,     e.g., '1.0 1.0
                             1.0 1.0'. If not provide current camera pose will
                             be used.
  -r                         Record images
  -o, --output_p DIRECTORY   Only required if record flag (-r) is set.
  --help                     Show this message and exit.

Sample trajectories are available upon request.

Collection of image and pose data

python scripts/collect_data.py --help

Usage: collect_data.py [OPTIONS]

  6D pose data collection script. Instructions: (1) Run ambf simulation (2)
  run recorded motions (3) run collection script.

Options:
  --path TEXT          Path to save dataset  [required]
  --sample_time FLOAT  Sample every n seconds
  --help               Show this message and exit.

Troubleshooting:

Ros sync client did not received any data/timeout exceptions

Ros topic for images are hardcoded on the Rostopics.py. If you are using different topic names, the ROS sync client will not generate any data to be saved. In particular, check if the toolpitchlink state for PSM1 and PSM2 are being published. These are not published by default on the simulation environment

/ambf/env/psm1/toolpitchlink/Command
/ambf/env/psm2/toolpitchlink/Command

To start publishing change the toolpitchlink BODY passive flag to false in the PSM ADF files. You will have to do this for both PSMs.

Incompatibility issues with numpy greater than 1.20

If you find incompatibility issues with your numpy version, you will probably need to modify some source files of ros_numpy to remove the numpy deprecated attributes. Replace np.float for float at line 224 of point_cloud2.py.

File "/opt/ros/noetic/lib/python3/dist-packages/ros_numpy/point_cloud2.py", li ne 224 , in def get_xyz_points(cloud_array, remove_nans=True, dtype=np.float):
File "/home/jin/.local/lib/python3.8/site-packages/numpy/init.py", line 30 5 , in _getattr

raise AttributeError(former_attrs[attr])
AttributeError: module 'numpy' has no attribute 'float'.
'np.float" was a deprecated alias for the builtin 'float'. To avoid this error 
In existing code, use 'float' by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use 'np.float64" here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidan ce see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

Citation

If you find this work useful, please cite it as: