3. Sensor Details

Sensor Timestamps

All timestamps associated with data streams (e.g., sensor readings, egomotion) are provided relative to a common start timestamp for each clip. Consequently, each 20-second clip nominally spans a time range from 0 to 20,000,000 microseconds.

Asynchronous Sensor Data

The sensors are asynchronous, i.e., they come in at different timestamps, as shown in this example:

• Even within the same sensor type (e.g., two different cameras), there can be small differences in the timestamp of arrival of a sensor reading
• Even within a single sensor, consecutive readings do not necessarily come in at exactly the number of timestamps specified by the sensor frequency—there can be small offsets

Two Types of Timestamps

LiDAR and radar data have two kinds of timestamps:

1. Scan Timestamp (Reference Timestamp):

   • Represents the start time of a complete sensor scan/sweep
   • For LiDAR: The timestamp when the 360° rotation began
   • For Radar: The timestamp when the scan cycle started
   • Stored in parquet files as reference_timestamp (LiDAR) or timestamp (Radar)

2. Return Timestamp (Per-Point/Per-Detection):

   • Individual timestamp for each point/detection within a scan
   • Represents the exact moment when that specific measurement was captured
   • For LiDAR: Each point has its own timestamp (encoded in point cloud)
   • For Radar: Each detection has its own sensor_timestamp

Asynchronicity and Negative Timestamps

Due to the asynchronous nature of the various data acquisition systems, the presence of negative timestamps (i.e., timestamps preceding the t=0 mark) is a common and expected artifact:

• Data points with slightly negative timestamps represent events that occurred just before the official start of the clip's 20-second nominal period
• This is normal and should be handled gracefully in your data processing pipeline

Egomotion Timestamp Structure

The egomotion data stream features a specific and intentional structure to provide richer context:

Historical Data: The stream includes a history of egomotion data leading up to the t=0 mark (negative timestamps)

Extended Future Data: The stream extends significantly beyond the nominal 20-second clip duration, typically providing up to two minutes of future data

This extended temporal window is provided to support tasks that require:

• Modeling historical context (e.g., "Where was the vehicle 5 seconds ago?")
• Predicting future routing intent (e.g., "Where is the vehicle heading in the next minute?")

For more details, please refer to Egomotion and Motion Compensation.

---

Camera Data

Hardware Configuration

7 Cameras providing 360° RGB coverage around the vehicle:

Camera Name	Location
camera_front_wide_120fov	Front windshield/roof
camera_front_tele_30fov	Front windshield/roof
camera_cross_left_120fov	Left side (cross-traffic)
camera_cross_right_120fov	Right side (cross-traffic)
camera_rear_left_70fov	Rear left
camera_rear_right_70fov	Rear right
camera_rear_tele_30fov	Rear

Data Format

File Structure (HuggingFace):

camera/{camera_name}/{camera_name}.chunk_XXXX.zip
└── {clip_id}.{camera_name}.mp4                      # RGB Video (20 seconds)
    {clip_id}.{camera_name}.timestamps.parquet       # Per-frame timestamps
    {clip_id}.{camera_name}.blurred_boxes.parquet    # Privacy blur metadata

RGB Video Format:

• MP4 video files containing RGB color images (~14.7 MB per 20-second clip)
• Frame rate: ~30 FPS
• Total frames per clip: ~605 frames
• Color space: RGB (3 channels: Red, Green, Blue)

Timestamp Schema:

{
    'frame_index': int64,          # Frame number (0, 1, 2, ...)
    'timestamp': int64,            # Timestamp in microseconds (relative to clip start)
}

Camera Model: f-theta vs Pinhole

f-theta Camera Model:

• Fisheye-like distortion model designed for wide field-of-view capture
• Enables wider FOV compared to standard pinhole cameras
• Native camera model used for data capture in this dataset

Rectification to Pinhole:

• f-theta images can be rectified to standard pinhole projection
• Trade-offs: FOV reduction and some quality degradation after rectification
• Useful when standard pinhole model is required for downstream algorithms

---

Lidar Data

Hardware

• Type: Mechanical rotating LiDAR with 128 laser channels
• Mounting: Roof-mounted on the vehicle
• Spin Duration: 100ms per complete rotation
• Timestamp Tolerance: ±5ms synchronization tolerance

Data Format

File Structure (HuggingFace):

lidar/lidar_top_360fov/lidar_top_360fov.chunk_XXXX.zip
└── {clip_id}.lidar_top_360fov.parquet  (200 spins per file)

Parquet Schema:

{
    'reference_timestamp': int64,           # Reference timestamp (microseconds)
    'draco_encoded_pointcloud': binary,     # DracoPy-encoded point cloud
}

Decoded Point Cloud Fields:

After decoding with DracoPy, each point has the following fields:

{
    'x': float32,           # X coordinate in meters (forward)
    'y': float32,           # Y coordinate in meters (left)
    'z': float32,           # Z coordinate in meters (up)
    'timestamp': int64,     # Absolute timestamp in microseconds
    'intensity': uint8      # Reflectivity value (0-255)
}

Intensity Values:

• Represents the strength of the returned laser signal (0-255)
• Higher values indicate stronger reflections (e.g., retroreflectors, metallic surfaces)
• Lower values indicate weaker reflections (e.g., dark/matte surfaces, oblique angles)
• Zero intensity points are normal and expected for absorptive materials or extreme angles

Timestamp Encoding:

Timestamps are encoded in the DracoPy color channels (Red, Green, Blue):

• Red channel: Intensity value
• Green + Blue channels: Encoded relative timestamp within the spin
• 16-bit timestamp encoded as: (Blue << 8) | Green
• Normalized to range: [-105,000, +105,000] microseconds relative to reference_timestamp
• Absolute timestamp = reference_timestamp + decoded_relative_timestamp

Compression and Quantization:

• Point clouds are compressed using DracoPy with 15-bit quantization
• Quantization errors (P95/P99 percentiles):
  • XYZ positions: ~5.6 / 6.1 millimeters
  • Intensity: negligible (~0.00 / 0.01)
  • Timestamps: ~2 / 2 microseconds

---

Radar Data

Hardware Configuration

9 Radar Sensors providing 360° coverage around the vehicle:

Sensor Name	Location
radar_front_center	Front bumper center
radar_corner_front_left	Front left corner
radar_corner_front_right	Front right corner
radar_side_left	Left side
radar_side_right	Right side
radar_corner_rear_left	Rear left corner
radar_corner_rear_right	Rear right corner
radar_rear_left	Rear left
radar_rear_right	Rear right

Data Format

File Structure (HuggingFace):

radar/{sensor_name}/{sensor_name}.chunk_XXXX.zip
└── {clip_id}.{sensor_name}.parquet  (~181k detections per file)

Parquet Schema:

Each row represents a single radar detection/return. All 15 fields are stored per detection:

Scan-Level Metadata (same for all detections in a scan):

{
    'scan_index': int64,           # Scan/frame number
    'timestamp': int64,            # Reference timestamp (microseconds)
    'sensor_timestamp': int64,     # Sensor-specific timestamp (microseconds)
    'num_returns': int64,          # Number of returns for this scan
    'doppler_ambiguity': float,    # Velocity ambiguity factor
    'sensor_id': uint8,            # Sensor identifier
    'max_returns': int64,          # Maximum returns per scan
    'radar_model': uint8,          # Radar hardware model ID
}

Per-Detection Measurements (unique for each radar return):

{
    'detection_index': int64,      # Detection sequence number within scan
    'azimuth': float,              # Horizontal angle (radians)
    'elevation': float,            # Vertical angle (radians)
    'distance': float,             # Range to target (meters)
    'radial_velocity': float,      # Doppler velocity (m/s)
    'rcs': float,                  # Radar Cross Section (dBsm)
    'snr': float,                  # Signal-to-Noise Ratio (dB)
    'exist_probb': uint8           # Existence probability (0-100)
}

Key Radar Features:

1. Velocity Measurement

   • Unlike lidar, radar provides radial velocity via Doppler effect
   • Positive velocity: Target moving away from sensor
   • Negative velocity: Target approaching sensor
   • Units: meters/second

2. Radar Cross Section (RCS):

   • Measures how much radar energy is reflected back
   • High RCS: Large metallic objects (vehicles, buildings)
   • Low RCS: Small objects, non-metallic materials
   • Units: dBsm (decibels relative to square meter)

3. Signal Quality Metrics

   • Signal-to-Noise Ratio: Higher = more reliable detection
   • Existence Probability: Confidence that detection is real