Challenges in Hybrid E2E + Rule-Based Architecture for Real-Time Autonomous Driving

[YSH-research]

1. Introduction and Research Motivation

I am a master's student who deeply aligns with the vision of Autoware and Tier IV.

Our lab's key milestone this year is implementing robust static obstacle avoidance and dynamic lane changing in a real vehicle environment. I have previous experience deploying and operating Autoware on actual vehicles, and I am currently focused on building a system to develop and validate a VAD (Vectorized Autonomous Driving) model in CARLA for eventual real-vehicle deployment.

To this end, I designed a Hybrid Architecture combining Autoware's rule-based system with an E2E model. However, I have encountered serious challenges regarding real-time performance and synchronization. I am seeking the community's insights to help complete this system and contribute to real-vehicle applications.

---

2. Current Research Progress

I have implemented the following components to bridge the gap between Autoware and the E2E model:

1. Curvature-based HLC (High Level Command) Publisher

Purpose: Enable the VAD model to issue appropriate driving commands based on the overall path geometry.

Implementation: Created a custom package that subscribes to Autoware's Global Trajectory. It calculates path curvature in real-time and dynamically publishes HLCs (e.g., straight, left turn, right turn).

Video : Only HLC Publisher

2. Rule-Based & E2E Fusion Driving Attempt (Bench2Drive Benchmarking)

Purpose: Inspired by the Bench2Drive architecture, aimed to enhance decision-making in complex scenarios.

Implementation: Rather than simply switching control authority, I attempted a pipeline that compares trajectories generated by Autoware (Rule-Based) and the VAD model (E2E), selecting and generating the optimal path for the current situation.

3. CARLA Town 04 Map Topology Modification for Lane Changing

Purpose: Enable lane changing functionality that was impossible with the provided Autoware map.

Implementation: Identified topology errors caused by overlapping linestrings in the provided CARLA Town 04 Lanelet2 map. Manually cleaned up the overlapping lines using Vector Map Builder, allowing the planner to generate proper lane change paths.

4. Data Bottleneck Analysis (Current Limitations)

Purpose: Analyze system load and latency during hybrid operation.

Status: Failed to achieve real-time performance due to severe data bottlenecks.

Temporary Workaround: Currently, the system only operates stably when CARLA simulation speed is reduced to 0.5x. This confirms that the current architecture has excessive overhead for real-vehicle application, indicating an urgent need for optimization.

---

3. Problems and Limitations Encountered

Despite these efforts, I am reaching the conclusion that simple module integration is unsuitable for real-time applications due to the following issues:

A. External HLC's Lack of Static Obstacle Awareness

The current external HLC generator relies only on static map (Lanelet2) information. Therefore, it cannot reflect temporary static obstacles detected by Perception (parked vehicles, construction cones, etc.).

Problem: Even when my lane is blocked by an obstacle, the HLC maintains a 'straight' command because the global map shows it as a 'straight' section. This prevents the system from triggering necessary lane changes or avoidance maneuvers.

Failure Case - Obstacle Avoidance
Failure Case - Obstacle Avoidance 02

B. Latency and Real-time Performance Degradation

Data Overhead: Significant ROS 2 communication delay when transmitting high-resolution images from CARLA to the VAD node.

Problem: Despite using use_sim_time for clock synchronization, the actual wall-clock processing time for image transmission and inference is too long. This causes the system to miss control cycle deadlines.

C. Control Instability due to Execution Delay

Problem: While a slight drop in control frequency (Hz) is manageable, normal control becomes impossible due to end-to-end execution delay.

Result: By the time the VAD-generated trajectory reaches the controller, it represents an outdated vehicle state relative to the vehicle's current condition. As the controller attempts to execute a path calculated for a past situation, the vehicle jerks or fails to properly follow the path.

Turning Failure due to Control Latency
Turning Failure due to Control Latency 02

---

### 4. Discussion and Hypotheses (Questions)

Based on my experiments, I have drawn some tentative conclusions and would like to hear your thoughts:

1. Hypothesis on Computing Architecture: I have concluded that running everything on a single machine is infeasible. To use the full Autoware stack with a heavy E2E model, I believe computing resources must be physically separated into these three components:

(1) E2E Inference (dedicated GPU/machine)

(2) Rule-Based Planning & Control (main Autoware PC)

(3) Sensing/Perception Head

Question: Do you agree that a distributed computing setup is essential for this hybrid E2E approach?

2. Hypothesis on External HLC Utility: The current external HLC module relying solely on static maps seems ineffective for real-vehicle driving (unless in predetermined cases like Bench2Drive). If HLC isn't tightly integrated with real-time Perception information, it appears to offer little help for dynamic path planning.

Question: Is there a design pattern that allows external HLC to be aware of Perception information without creating circular dependencies?

3. Feasibility of Pure E2E Avoidance Driving: Honestly, due to the aforementioned latency and synchronization issues, implementing pure E2E-based obstacle avoidance and path following within the Autoware ecosystem feels nearly impossible at present.

Question: Has anyone in the community successfully performed similar work? I feel like I've hit a dead end and worry that I might be attempting something structurally impossible with the current ROS 2 / Autoware architecture.

Any shared experiences or advice would be tremendously helpful.

Thank you.

[Max-Bin]

@YSH-research

First of all, thank you for your outstanding work and this detailed report. We deeply appreciate your contributions to the community, and we would warmly welcome any Pull Requests regarding the improvements you've made.

Regarding the challenges you raised, I have some suggestions and thoughts:

1. Optimization: Disabling LiDAR Simulation

Regarding the latency and system load, have you tried disabling the LiDAR simulation output in Autoware? Since VAD model is vision-centric, the LiDAR data might be unnecessary for your specific E2E pipeline, yet it places a significant burden on the processor and the bridge. You can try disabling it in the configuration file below:
autoware_universe/simulator/autoware_carla_interface/config/sensor_mapping.yaml
2. Hardware Specs & Operational Speed

Could you please share the specifications of the hardware you are currently using? Additionally, at what vehicle speed does the latency/instability typically start to occur?

3. Profiling Data We are continuing to look into other possible optimization points.

It would be extremely grateful if you could provide a breakdown of the specific execution time (latency) for each component. Knowing exactly where the bottleneck lies will help us provide more targeted advice.

4. Discussion on HLC and Obstacle Avoidance

4.1 Regarding the issue where the external HLC fails to account for static obstacles: In theory, within the Bench2Drive implementation, lateral obstacle avoidance should be handled during the model's forward pass without requiring a change in the High-Level Command (i.e., the model should nudge around the obstacle even while the command remains "Forward").

4.2 I suspect the failure might be attributed to the model's performance or the quality of the Bench2Drive dataset itself. I have visualized some samples from the dataset and noticed inconsistent quality, which might be confusing the model.

4.3 This is a fascinating and significant challenge often seen in model-based planners. Investigating how to refine the generated trajectory or implementing a rule-based fallback mechanism for safety would be a very valuable direction for future research.

Thank you again for your efforts, and we look forward to further discussion!