CLVR Jaco Play Dataset

Shivin Dass¹, Jullian Yapeter¹, Jesse Zhang¹, Jiahui Zhang¹, Karl Pertsch¹, Stefanos Nikolaidis¹, Joseph J. Lim^1,2

¹University of Southern California ²KAIST

The CLVR Jaco Play Dataset consists of 1085 teleoperated robot episodes, including language annotations. Please find details for loading and using the data below and in the provided Colab.

For further questions, please contact {sdass, pertsch}@usc.edu.

Downloading the Data

Please install gdown (pip install gdown) and run the provided download script.

bash download_jaco_data.sh ./data

Alternatively, you can manually download the data from this link (~16GB). We found manual download to be faster in some cases.

Using the Data

Please check out our Colab for an example how to unpack and use the data!

The dataset was collected at a frequency of 10Hz. It has the following structure:

• Observations are split into 5 attributes,
  • front_cam_ob : observations from 3rd person cam
  • mount_cam_ob : observations from mounted camera
  • ee_cartesian_pos_ob : end effector cartesian position. ee_cartesian_pos_ob[0:3] corresponds to position and ee_cartesian_pos_ob[3:7] corresponds to orientation in quarternian format
  • ee_cartesian_vel_ob : end effector cartesian velocity. ee_cartesian_pos_ob[0:3] corresponds to change in position and ee_cartesian_pos_ob[3:6] corresponds to change in orientation in roll, pitch yaw format
  • joint_pos_ob : joint positions of the jaco arm (we only use the last 2 elements of this that correspond to the gripper joints)
• actions : first 3 elements are cartesian deltas and 4th element is a label from {0, 1, 2} meaning {open gripper, don't move gripper, close gripper}
• terminals : 1 at the end of each skill
• prompts : Natural language description of the goal
• reward : 1 at the end of each skill

Citation

If you find this work useful in your research, please consider citing:

@software{dass2023jacoplay,
  author = {Dass, Shivin and Yapeter, Jullian and Zhang, Jesse and Zhang, Jiahui
            and Pertsch, Karl and Nikolaidis, Stefanos and Lim, Joseph J.},
  title = {CLVR Jaco Play Dataset},
  url = {https://github.com/clvrai/clvr_jaco_play_dataset},
  version = {1.0.0},
  year = {2023}
}

License

The data is available under the Creative Commons Attribution 4.0 International License.