VisionPilot 1.0

Vision Pilot 1.0 is an open source L2 ADAS system which enables warning and actively safety features as well as hands free autopilot using a single 2MP camera. Vision Pilot serves as an open and customizable software substrate which can help automotive OEMs and Tier-1 suppliers save time, money, and resources whilst acquiring technology ownership and sovereignty. This release enables the following production-ready features:

ACC - autonomous cruise control
FCW - forward collision warning
AEB - autonomous emergency braking
LKAS - lane keep assist
LDW - lane departure warning
ISA - intelligent speed assist
Autopilot - single-lane hands-free highway autopilot

It leverages three open-source AI models developed by the Autoware Foundation:

AutoSpeed - Closest in-path object detection
AutoSteer - Ego path future waypoint detection
AutoDrive - End to End distance/in-path object presence detection and road curvature estimation

The Vision Pilot 1.0 release also features cross modal AI fusion alongside trajectory planning for a safe and introspectable driving policy.

The system supports deployment without middleware and provides interfaces to integrate with ROS2-based systems.

Vision Pilot 1.0 can be deployed on low-cost, mass-market automotive chipsets in the 3-5 INT8 TOPs range to achieve a complete pipeline speed of over 10 FPS.

Contributions

@m-zain-khawaja along with the team @atanasko, @pranavdoma, and @TranHuuNhatHuy

VisionPilot 0.9

Key Features Added in v0.9

• Longitudinal Perception & Tracking

  • Integrated AutoSpeed detector and ObjectFinder multi-object tracker.
  • Robust CIPO selection (Level 1 & Level 2), Kalman-based distance and relative-speed, cut‑in detection, Kalman reset handling.

• Speed Planning (RSS-inspired)

  • New SpeedPlanner computes safe following distance d_min and ideal_speed_ms.
  • Encodes FCW/AEB zones, exposes fcw_active and aeb_active.

• Longitudinal Controller

  • New longitudinal PID controller tracks ideal_speed_ms and outputs control_effort_ms2 (accel/decel).

• Parallel Lateral + Longitudinal Pipelines

  • Shared capture buffer at configurable pipeline.target_fps.
  • All longitudinal tuning (AutoSpeed thresholds, PID gains, ego-speed fallback) moved into visionpilot.conf.

• Shared-Memory IPC

  • POSIX shm segment /visionpilot_state exposing a single VisionPilotState struct with:
    • Lateral outputs, longitudinal CIPO/safety metrics, control effort, and CAN/ego state.
  • tools/shm_reader utility to inspect outputs frame‑by‑frame.

System Architecture

Vision Pilot 0.9 processes images from a single front-facing camera to enable both ADAS features and highway autopilot within a single driving lane. Compared to Vision Pilot 0.9, this version of system incorporates both lateral perception and planning modules alongside longitudintal perception and planning modules in two parallel streams, through the additional integration of the AutoSpeed neural network for closest-in-path-object detection. This enables features such as Autonomous Cruise Control, Forward Collision Warning, and Automatic Emergency braking. In order to estimate the distance of the closest-in-path-object, a homography transform is utilized which maps image pixels to road coordinates, providing an estimate of real-world distances in metres. A Kalman filter is used to track the distance of the closest-in-path-object and estimate its velocity. To maintain a safe following distance to the lead vehicle, the system complies with Mobileye's Responsibility Sensitive Safety framework.

Contributions

@m-zain-khawaja along with the team @pranavreddy23, @atanasko , and @TranHuuNhatHuy

VisionPilot 0.5

What's new?

Version 0.5 is based on EgoLanes and AutoSteer models for autonomous steering, and it's tested with constant velocity on a real vehicle.

Description

Vision Pilot 0.5 is an autonomous lateral steering system which processes images from a single front-facing camera for autonomous steering control features such as lane centering and lane-keep assist. Vision Pilot 0.5 has been tested with an automotive OEM on their private test track and the outputs of the system were compared with ground-truth data from human drivers. The system was able to achieve over 90% accuracy compared to human driven steering at highway driving speeds in high curvature turns (20+ degree steering angle) and high bank angles.

Vision Pilot 0.5 uses two neural networks, EgoLanes for segmenting and classifying lane lines, and AutoSteer for estimating the steering angle to follow the road. In parallel, a traditional tracking pipeline is utilized to measure the cross-track error and yaw error and a feedback plus feedforward controller is utilized to remain centered in-lane. The system can run headless or a visualzation can be shown which overlays the detected lanes and predicted steering angle on the input image. Although EgoLanes runs on each image individually, the AutoSteer network requires the current and previous images as input to capture spatio-temporal features. A moving average smoothing is applied to the output of AutoSteer to ensure that steering signal commands do not suffer from noise.

System Architecture

Contributions

@m-zain-khawaja along with the team @atanasko, @pranavreddy23, @TranHuuNhatHuy