WiDEVIEW: An UltraWideBand and Vision Dataset for Deciphering Pedestrian-Vehicle Interactions

    

Jia Huang¹, Alvika Gautam¹, Junghun Choi² and Srikanth Saripalli¹
1. Texas A&M University;  2. Hyundai Motor Company

Overview

Recording Platform

• Polaris GEM e6

Sensor Setup

• One 32 Channels Lidar: Velodyne Ultra Puck
• Three point grey RGB Cameras: TBA
• Inertial Navigation System (GPS/IMU): Vectornav VN-300 Dual Antenna GNSS/INS
• Ultra-Wideband sensors(UWB): Decawave (now Qorvo) modules (TREK 1000 RTLS evaluation kit with DW 1000 UWB chip configured as 4 Anchors and 6 Tags)

Folder structure

UWB_Dataset
├── UWB-gcart-camera-calibration    --directory for the three front cameras calibration by using Kalibr
├── benchmarks                      --directory for object tracking benchmark by using Yolov5 and DeepSORT
├── decawave_ros                    --directory for TREK1000 UWB ros package containing ros node and messages
├── lidar2camera                    --directory for lidar to middle camera calibration by using SensorsCalibration toolbox
├── sensor_syn.py                   --script for sensor data approximate synchronization for future sensor fusion

Annotated Data:

The dataset contains 30,936 frames which come from the front three cameras with 10,312 frames each. Every 5-th frame was annotated manually and carefully in Computer Vision Annotation Tool (CVAT) for the front middle camera frames while the ones in between are interpolated automatically without adjustment. There are a total of 8,368 bounding boxes manually annotated for pedestrians and cyclists along with six tag IDs for those who carried tags, and 33,239 bounding boxes are interpolated in CVAT. These annotations are exported in MOT 1.1 format for the following multi-object tracking task evaluation for images. The annotations for the left and right cameras frames are in progress and will be updated on the Github gradually.

Annotations and Annotation tasks (include original videos) Download:

Annotations include raw images in the img1 folder, annotation tasks include the raw videos named output.mp4 used for annotation in CVAT.

Annotation folder structure

rosbag_filename(subfolder_middle_annotation).zip
├── gt
      ├── gt.txt           
      └── labels.txt 
├── img1
      ├── 000001.PNG          
      ├── 000002.PNG     
      └── ......

Example of rosbag_filename(subfolder_middle_annotation).zip : 2023-02-04-16-04-35_0(1_middle_annotation)

Annotation task folder structure

rosbag_filename(subfolder_middle_task).zip
├── annotations.json
├── task.json
├── data
      ├── index.json          
      ├── manifest.jsonl     
      └── output.mp4

Example of rosbag_filename(subfolder_middle_task).zip : 2023-02-04-16-04-35_0(1_middle_task)

Sequence 1_1: 2023-02-04-16-04-35_0.bag: (Annotation) (Task)

Sequence 1_2: 2023-02-04-16-04-35_0.bag: (Annotation) (Task)

Sequence 2: 2023-02-10-15-23-44_0.bag: (Annotation) (Task)

Sequence 3_1: 2023-02-10-15-26-14_1.bag:(Annotation) (Task)

Sequence 3_2: 2023-02-10-15-26-14_1.bag:(Annotation) (Task)

Sequence 4: 2023-02-10-15-28-44_2.bag: (Annotation) (Task)

Sequence 5_1: 2023-02-10-15-35-56_0.bag: (Annotation) (Task)

Sequence 5_2: 2023-02-10-15-35-56_0.bag: (Annotation) (Task)

Sequence 6: 2023-02-10-15-38-30_0.bag: (Annotation) (Task)

Sequence 7: 2023-02-12-13-59-09_0.bag: (Annotation) (Task)

Sequence 8: 2023-02-12-14-13-00_0.bag: (Annotation) (Task)

Sequence 9: 2023-02-12-14-14-40_0.bag: (Annotation) (Task)

Sequence 10:2023-02-12-14-20-14_0.bag: (Annotation) (Task)

Sequence 11:2023-02-12-14-24-09_0.bag: (Annotation) (Task)

Sequence 12:2023-02-12-14-25-15_0.bag: (Annotation) (Task)

Sequence 13:2023-02-12-14-31-29_0.bag: (Annotation) (Task)

Sequence 14:2023-02-17-15-40-11_0.bag: (Annotation) (Task)

Sequence 15:2023-02-17-15-42-27_0.bag: (Annotation) (Task)

Sequence 16:2023-02-17-15-44-23_0.bag: (Annotation) (Task)

Sequence 17:2023-02-17-16-00-23_0.bag: (Annotation) (Task)

Sequence 18_1:2023-02-17-16-53-27_0.bag: (Annotation) (Task)

Sequence 18_2:2023-02-17-16-53-27_0.bag: (Annotation) (Task)

Sequence 19:2023-02-17-16-54-38_0.bag: (Annotation) (Task)

Sequence 20:2023-02-17-17-01-15_0.bag: (Annotation) (Task)

Sequence 21:2023-02-17-17-03-49_0.bag: (Annotation) (Task)

Calibration:

Multi-cameras Intrinsic and Extrinsic Calibration

The sensor suite cameras (left, middle and right on the autonomous shuttle roof rack) have a 30$^{\circ}$ overlapping field of view between adjacent cameras. The intrinsic and extrinsic camera parameters are estimated by the multiple camera calibration tool in the Kalibr toolbox. A 6 x 6 aprilgrid target with spacing of 0.03m is used. We utilize a pinhole projection model for our cameras, where a three-dimensional scene is projected onto an image plane through a perspective transform. The calibration details can be found in the folder UWB-gcart-camera-calibration.

Lidar to Middle Camera Calibration

To prepare data for LiDAR and middle camera calibration, the data across both sensors is first matched by finding the pairs with the closest time stamps between two sensors. Then the point cloud and image pairs are calibrated in SensorsCalibration toolbox to facilitate the sensor fusion process. The calibration details can be found in the folder lidar2camera.

Benchmarks

2D Object Detection and Tracking

Case	HOTA	MOTA	MOTP	IDF1	FP	FN	ID Sw.	Recall	Precision	Dets	GT Dets	IDs	GT IDs
1	41.577	42.046	72.496	54.958	9144	11006	729	69.451	73.236	34165	36027	491	231
2	45.873	44.204	78.173	57.293	1607	2038	398	71.874	76.42	6815	7246	425	231

Benchmark Reproduction

To reproduce the results, please refer to here

ROS Bag Raw Data

Data included in raw ROS bagfiles:

Topic Name	Message Tpye	Message Descriptison
/Decawave	decawave_ros/uwb_distance	Distances between Anchors and Tags
/left/image_raw	sensor_msgs/Image	Images from the left camera
/middle/image_raw	sensor_msgs/Image	Images from the middele camera
/right/image_raw	sensor_msgs/Image	Images from the right camera
/vectornav/fix	sensor_msgs/NavSatFix
/vectornav/gps	vn300/gps	GPS data from VectorNav-VN300
/vectornav/imu	vn300/sensors	Imu data from VectorNav-VN300
/vectornav/ins	vn300/ins	INS data from VectorNav-VN300
/velodyne_points	sensor_msgs/PointCloud2	PointCloud produced by the Velodyne Lidar

ROS Bag Download

The ROS bag information can be found on the google drive link. The following are the google drive links for the ROS Bag files.

Sequence 1: 2023-02-04-16-04-35_0.bag: (19.91GB)

Sequence 2: 2023-02-10-15-23-44_0.bag: (19.94GB)

Sequence 3: 2023-02-10-15-26-14_1.bag: (20.07GB)

Sequence 4: 2023-02-10-15-28-44_2.bag: (18.42GB)

Sequence 5: 2023-02-10-15-35-56_0.bag: (7.29GB)

Sequence 6: 2023-02-10-15-38-30_0.bag: (8.04GB)

Sequence 7: 2023-02-12-13-59-09_0.bag: (6.66GB)

Sequence 8: 2023-02-12-14-13-00_0.bag: (5.97GB)

Sequence 9: 2023-02-12-14-14-40_0.bag: (15.71GB)

Sequence 10:2023-02-12-14-20-14_0.bag: (5.69GB)

Sequence 11:2023-02-12-14-24-09_0.bag: (9.16GB)

Sequence 12:2023-02-12-14-25-15_0.bag: (8.66GB)

Sequence 13:2023-02-12-14-31-29_0.bag: (5.59GB)

Sequence 14:2023-02-17-15-40-11_0.bag: (4.26GB)

Sequence 15:2023-02-17-15-42-27_0.bag: (7.53GB)

Sequence 16:2023-02-17-15-44-23_0.bag: (16.47GB)

Sequence 17:2023-02-17-16-00-23_0.bag: (7.04GB)

Sequence 18:2023-02-17-16-53-27_0.bag: (3.93GB)

Sequence 19:2023-02-17-16-54-38_0.bag: (6.57GB)

Sequence 20:2023-02-17-17-01-15_0.bag: (6.57GB)

Sequence 21:2023-02-17-17-03-49_0.bag: (4.43GB)

Working in progress

• Add camera and UWB calibration results
• Annotate the left and right camera frames
• Annotate the lidar point clouds

Citation

@misc{huang2023wideview,
      title={WiDEVIEW: An UltraWideBand and Vision Dataset for Deciphering Pedestrian-Vehicle Interactions}, 
      author={Jia Huang and Alvika Gautam and Junghun Choi and Srikanth Saripalli},
      year={2023},
      eprint={2309.16057},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

License

All datasets and code on this page are copyright by us and published under the GNU General Public License v2.0.