Perceptinet

An ego vehicle can miss a road user hidden by an occluding truck or other traffic. Perceptinet explores how an elevated roadside unit (RSU) can provide a complementary view, transmit observations over a simulated V2I link, and support earlier risk assessment.

The project models the complete perception → communication → fusion → risk → control workflow, rather than stopping at object-detection overlays.

• Multi-class, dual-camera perception — YOLO26n inference on onboard and RSU RGB feeds, targeting pedestrians, bicycles, cars, motorcycles, buses, and trucks.
• Simulation-assisted spatial localization — CARLA camera geometry, projected actor matching, and world-coordinate observations.
• Cooperative sensor fusion — spatial association, temporal tracks, and short-term dropout handling across the two viewpoints.
• Configurable V2I fault injection — timestamped packets, simulation-time delivery queues, latency/jitter, packet loss, stale-data rejection, and temporary blackouts.
• Risk-aware speed control — path-relative distance, closing speed, time-to-collision (TTC), hazard levels, and a deterministic braking controller.
• Live observability — interactive Driver Mode / Pro Mode dashboards, per-cycle telemetry, JSON/CSV exports, and comparative reports.

Two recordings from the CARLA experiments show the same occluded pedestrian crossing under different communication conditions. Each recording includes the onboard and RSU feeds plus the live dashboard.

Scenario 1 — normal V2I	Scenario 2 — degraded V2I
Clear-weather RSU warning, braking and controller STOPPED state	Heavy rain, injected communication blackout and observed kinematic footprint contact
Watch the full recording (MP4)	Watch the full recording (MP4)

Recorded evidence: The GIFs are short previews; click either image or its MP4 link to see the complete recording. These are simulated CARLA runs, not a production vehicle test. Scenario 2's CONTACT (SIM) means geometric overlap in the kinematic model—not a validated physical collision.

flowchart TB
    W["CARLA urban simulation"] --> O["Onboard RGB camera"]
    W --> R["Roadside RGB camera · RSU"]

    O --> P1["YOLO26n detection"]
    R --> P2["YOLO26n detection"]

    P1 --> L1["World-coordinate localization"]
    P2 --> L2["World-coordinate localization"]

    L2 --> V["Simulated V2I channel<br/>latency · loss · outage"]
    L1 --> F["Spatial + temporal sensor fusion"]
    V --> F

    F --> K["Risk Engine<br/>distance · closing speed · TTC"]
    K --> B["Braking Controller<br/>risk-based target speed"]
    B --> E["CARLA ego vehicle<br/>kinematic motion"]

    F --> T["Telemetry + metrics"]
    K --> T
    B --> T
    T --> D["Live Dashboard<br/>Driver Mode · Pro Mode"]

Two controlled intersection/occlusion experiments share the same pedestrian crossing configuration, perception model, scene geometry, and control logic. The treatment changes the weather visualization and V2I reliability.

What happens? A pedestrian emerges from behind an occluding vehicle. The RSU can provide a different view from the onboard camera. Perceptinet fuses available observations and computes a TTC-based risk state; the ego controller responds to the resulting warning level.

The crossing begins when the pedestrian is about 11.5 m longitudinally ahead of the ego vehicle, at a commanded 0.95 m/s walking speed. Scenario outcomes are measured, not hard-coded: the degraded-link case is not guaranteed to collide, and the baseline is not automatically marked safe.

Experimental integrity: CARLA rain changes the simulated scene's appearance. The communication degradation is injected separately in Python; the project does not claim that CARLA weather physically creates RF interference. The baseline V2I model annotates latency, while Scenario 2 uses a queue that withholds packets until their simulated delivery time.

For reproducible parameters and interpretation, see Scenario comparison and Engineering notes.

• Windows desktop session, Python 3.12, and a running/launchable CARLA 0.9.16 installation.
• Python dependencies from requirements.txt. Use a CARLA Python package/wheel compatible with your local simulator and Python environment.
• YOLO26n model weights accessible to Ultralytics. The teacher-demo script looks for C:\CARLA\yolo26n.pt and otherwise attempts to load yolo26n.pt by name.
• A graphics-capable machine for CARLA and inference. The current teacher-demo configuration uses YOLO device 0 (typically the first CUDA GPU); adjust YOLO_DEVICE in the demo script for your setup.
• CARLA RPC reachable at 127.0.0.1:2000. The launcher checks the actual CARLA world/map RPC, not just whether the port is open.

git clone https://github.com/vinceyuchen/Perceptinet.git
cd Perceptinet
py -3.12 -m pip install -r requirements.txt

# Verify launcher paths and selected scenario without starting CARLA
py -3.12 .\Perceptinet_Teacher_Demo.py --dry-run

Start CARLA before the demo, or provide its executable with --carla-exe / the PERCEPTINET_CARLA_EXE environment variable. The launcher also checks common Windows CARLA installation paths.

Double-click a launcher, or run one cycle from PowerShell:

py -3.12 .\Perceptinet_Teacher_Demo.py --scenario clear --cycles 1
py -3.12 .\Perceptinet_Teacher_Demo.py --scenario rain_v2i --cycles 1
py -3.12 .\Perceptinet_Teacher_Demo.py --scenario alternate --cycles 2

The live demo opens the CARLA spectator view, onboard/RSU detection feeds, and the Perceptinet Dashboard. Switch dashboard views with the Driver / Pro buttons (or D / P); press Q in the camera window or launcher console, or close the dashboard, to stop. The launcher requests scene cleanup.

Each completed teacher-demo cycle writes separate artifacts under results/teacher_demo/runs/cycle_*/, including:

• scenario_result.json — completion reason, braking/detection timestamps, walker motion, closest centre-to-centre separation, and observed contact indicators.
• metrics_summary.json and metrics_events.csv — latency summaries, detection timing, braking events, and other diagnostic metrics.
• occluded_pedestrian_world_v1.csv — matched detection and world-coordinate observations.

To generate a comparison report from your own recorded runs:

py -3.12 -m pip install reportlab
py -3.12 .\scripts\generate_teacher_demo_report.py

Reports are written to results/teacher_demo/report/ as PDF, CSV, and JSON. Missing run data are left pending rather than replaced with fabricated measurements. Runtime artifacts are ignored by Git.

The Core Python checks workflow can be run automatically on relevant code changes or manually from GitHub Actions. It executes eight source-level regressions on Python 3.12 without launching CARLA:

Run the same checks locally from the repository root:

py -3.12 .\scripts\test_risk_engine.py
py -3.12 .\scripts\test_braking_controller.py
py -3.12 .\scripts\test_v2i_channel.py
py -3.12 .\scripts\test_metrics_collector.py
py -3.12 .\scripts\test_fusion_engine.py
py -3.12 .\scripts\test_temporal_fusion.py
py -3.12 .\scripts\test_teacher_demo_paths.py
py -3.12 .\scripts\test_teacher_demo_report.py

Important: These are targeted unit/regression checks, not evidence of a successful full CARLA run, verified physical braking or production safety. Live scenario testing requires CARLA, cameras, compatible model weights and the appropriate inference device. The current workflow run status must be checked on GitHub Actions; merely adding the workflow does not establish a passing result.

Perceptinet/
├── perception/                 # YOLO26n inference and world-coordinate projection
├── v2i/                        # V2I metadata and impaired packet delivery
├── fusion/                     # Spatial association and temporal fusion
├── risk/                       # Path-relative and TTC risk assessment
├── control/                    # Deterministic braking controller
├── simulation/                 # CARLA scene, road context and walker logic
├── dashboard/                  # Driver Mode and Pro Mode UI
├── metrics/                    # Latency, detection and braking measurements
├── validation/                 # Evaluation and validation helpers
├── scripts/                    # Experiments, tests and report generation
├── Perceptinet_Teacher_Demo.py  # Live demo orchestrator
└── START_*_DEMO.bat             # Windows one-click launchers

Perceptinet is a simulation research prototype, not a production autonomous-driving or safety-certified system.

• In the current teacher demo, ego motion is kinematic (CARLA vehicle physics disabled); braking is represented by commanded speed and transforms, not a physically validated vehicle dynamics model.
• Spatial association can use CARLA actor identity and simulator geometry. This is simulation-assisted localization, not an independently validated real-world 3D detection system.
• A kinematic_contact event means 2D actor footprint overlap, not a proven physics-simulated crash. CARLA collision-sensor events are recorded separately when available.
• Weather and V2I impairments are separately configured; the radio model is not a calibrated wireless or meteorological propagation model.
• Demo parameters, TTC thresholds, and deceleration rates are for controlled evaluation, not certified road-safety performance.

Improve degraded-link resilience (adaptive freshness checks, fallback strategies, and fault-aware fusion), extend multi-road-user evaluation, and validate braking/contact behavior with a physics-enabled ego model.