Perception

The perception system processes camera inputs to understand the driving environment. It consists of three main components that work together to enable autonomous driving capabilities.

Lane Detection

Input: Camera frames from front-facing camera

Process:

• Image preprocessing and resizing for model input
• GPU-accelerated neural network inference
• Deep learning model inference for initial lane segmentation
• Inverse Perspective Mapping (IPM) to transform to top-down view
• Connected Components analysis to identify distinct lane clusters
• Lane validation and matching algorithm:
  • Validates clusters against history and distance constraints
  • If two valid lanes detected: maintains both, updates lane width history
  • If only one lane detected: creates synthetic second lane using width history
  • If no valid matches: defaults to clusters closest to frame center
• Midcurve generation to create trajectory line

Output:

• Lane mask showing detected and/or synthetic boundaries
• Midpoint error (lateral offset) for PID steering control
• Trajectory line (midcurve) for navigation
• Visualization frames for debugging/monitoring

Key Features:

• Model-based detection for robustness
• Lane history tracking for temporal consistency
• Synthetic lane generation for single-lane scenarios
• Adaptive to varying road conditions through historical constraints

Object Detection

Input: Camera frames from front-facing camera

Process:

• Image preprocessing and resizing for model input
• GPU-accelerated neural network inference
• Detection and classification of objects (vehicles, pedestrians, signs, lights)
• Road/non-road segmentation for drivable area identification
• Distance estimation based on object position in frame
• Relative velocity approximation from sequential frames

Output:

• Bounding boxes with class labels for detected objects
• Road/non-road mask for trajectory validation
• Object positions and sizes
• Distance estimations for ACC and safety functions
• Emergency brake signal when obstacles intersect trajectory

Key Features:

• Real-time detection using GPU acceleration
• Supports ACC by detecting and measuring distance to leading vehicles
• Triggers emergency braking when obstacles detected in path
• Identifies regions of interest for traffic sign/light classification
• Provides road surface estimation to prevent off-road routing

Traffic Sign/Light Classification

Input: Cropped regions of interest from object detection

Process:

• Secondary classification on sign/light regions
• Multi-class recognition for traffic signs (stop, yield, speed limits)
• State detection for traffic lights (red, yellow, green)
• Text recognition for speed limit values

Output:

• Sign/light classifications with confidence scores
• Speed limit values (numeric)
• Traffic light states
• Control signals (stop, yield, speed adjustment)

Key Features:

• Enables appropriate responses to traffic rules
• Adjusts vehicle behavior based on road signage
• Supports safe navigation through intersections

Integration with TrajectoryDefinition

The TrajectoryDefinition component integrates outputs from lane detection and object detection to create a safe driving path:

• Combines lane boundaries with road surface detection
• Validates trajectory against road/non-road masks
• SAE_3: Uses nominal trajectory based on lanes only
• SAE_4: May re-route trajectory to avoid obstacles while staying on road
• Provides emergency signals when no valid path exists

The perception pipeline operates in real-time with frame synchronization to ensure consistent decision-making across all components.