Design: CARLA × Autoware integration architecture for closed-loop evaluation

[hakuturu583]

CARLA × Autoware integration design

Sharing a proposal of integration architecture for using CARLA as a closed-loop evaluation / digital-twin backend for Autoware. The design decomposes the integration into three layers around CARLA as the simulation core. (Legend: solid arrows = runtime communication, dashed arrows = file/data handoff.)

1. Sensor Simulation Layer

Generates photorealistic sensor input from the CARLA scene and feeds it to the Autoware under test.

• Renderer obtains the 3D scene state from CARLA and generates sensor inputs (camera/LiDAR).
• 3D assets are fed in as files (see usdz note below).

2. Simulation Core

• CARLA is the center: it owns the physics engine and master clock, driving time synchronization and binding the sensor layer and traffic layer together.
• autoware_carla_interface mediates between Autoware and CARLA: it forwards sensor data (DDS) into Autoware and returns control commands (DDS) back, plus initialization / pose sync.

3. Traffic Environment Layer

Reproduces realistic surrounding traffic.

• Scenario (Python) input scripts describe the scenario.
• Scenario Engine takes the Python input and mediates it into CARLA.
• Traffic Engine is pluggable — e.g. SUMO or a Self-Play policy — so traffic-flow algorithms can be swapped to produce natural traffic.

Note 1 — Scenarios as a Python package

Scenarios are authored and managed as a Python package rather than ad-hoc scripts. Benefits:

• Version management — scenarios ship as a versioned Python package, so they can be pinned, reused, and tracked like any dependency.
• API stability via pinned CARLA Python client — by fixing the version of the CARLA Python client, scenarios do not break as long as CARLA's own communication API is unchanged. The scenario package is decoupled from CARLA-core churn.
• Agent-authorable — scenarios can be written by coding agents (LLMs), since they are plain, structured Python.
• Existing base library — a base library already exists: tier4/autoware_lanelet2_to_opendrive — autoware_carla_scenario.

Note 2 — Input asset format: usdz

Scene assets enter the pipeline in a USD-based usdz archive (a ZIP package). Representative contents:

• default.usda — USD root stage (World / Z-up ENU)
• metadata.yaml — uuid / scene_id / version
• scene.json — scene index (coordinate frames, ECEF, map/CARLA references)
• tileset.json — 3D Tiles (EXT_3dgs_spz)
• chunks/*.spz — 3DGS body (Gaussians; position/rotation/scale/color(SH)/opacity, Niantic SPZ compression)
• chunks/*.json.gz — LiDAR sidecar (depth/intensity)
• entities/<id>/*.spz — dynamic objects as individual 3DGS
• rig_trajectories.json / sequence_tracks.json — sensor-rig poses / dynamic-actor trajectories
• ppisp.json — camera appearance correction

Common coordinate system: right-handed Z-up ENU, geo-referenced via an ECEF anchor. (Ref: github.com/autowarefoundation/3dgs_io, scene_usdz.py.)

Why this design — AlpaSim alignment. The usdz layout is modeled after AlpaSim, so that we keep compatibility with AlpaSim as far as possible going forward. We are also keeping the ongoing AlpaSim closed-loop integration in mind — it is actively being explored in autowarefoundation/auto_e2e#140, where an early working prototype of an AutoE2E policy running a closed-loop session in AlpaSim has already been demonstrated (renderer choice and the full RL loop are still under active discussion).

Gaussian data format — open standard. The exact data format for the Gaussians packed inside usdz was not clearly defined. To resolve this, we propose packing the 3D GS data based on the emerging open standard from Khronos/OGC that adds 3D Gaussian Splats to the glTF asset standard (announcement), and carrying that standardized 3D GS representation inside the usdz package.

[hakuturu583]

Reference: measured latency of the CARLA client API path (why the Renderer publishes sensor topics directly over DDS)

To back the design decision of having the Renderer publish sensor topics directly over DDS — rather than pulling sensor data through the CARLA client API and re-publishing it — we benchmarked the sensor-data latency of the CARLA 0.10.0 client API itself.

Setup: CARLA 0.10.0 (UE 5.5, official carlasim/carla:0.10.0 Docker image, Town10HD_Opt), RTX 4090, localhost, synchronous mode fixed at 20 Hz. One vehicle carrying one LiDAR (32 ch, 600k pts/s) and one RGB camera (800×600). Latency = world.tick() call → sensor callback fired in the Python client (mean over n=100, receive rate 100/100).

Configuration	End-to-end (tick start → client receive)	Transport only (tick end → receive)
LiDAR alone	16.9 ms (p95 19.9)	7.3 ms
Camera alone	91.1 ms (p95 109.9)	59.1 ms
LiDAR (camera attached too)	51.6 ms (p95 64.0)	15.4 ms
Camera (LiDAR attached too)	104.3 ms (p95 125.0)	68.1 ms

For scale: a bare RPC round-trip is ~0.03 ms and world.tick() without sensors is ~9 ms, so the cost is not the RPC layer but the sensor-data pipeline behind the client API (render-target readback → serialization of raw BGRA / point cloud → TCP stream → client-side deserialization). Attaching multiple sensors makes it worse: the camera and LiDAR streams contend, degrading LiDAR delivery from ~17 ms to ~52 ms.

Two takeaways relevant to this architecture:

1. Any path that routes sensor data through the CARLA client API unavoidably pays this latency. Even on localhost with a high-end GPU, a single 800×600 camera costs ~90–105 ms from simulation step to client receive — before a bridge re-publishes it into DDS. Publishing sensor topics directly from the renderer avoids this hop entirely.
2. CARLA 0.10 ships native DDS (ROS 2) server support (no CARLA-ROS bridge needed), so CARLA-side sensor streams can also be published as DDS topics without the client-API hop. From Autoware's point of view, sensor data is then handled through a single, unified API — DDS topics — regardless of whether it originates from CARLA's native ROS 2 interface or from the external Renderer proposed here.

Benchmark script (RPC RTT / world.tick() cost / per-sensor tick→callback latency, sync & async modes): https://gist.github.com/hakuturu583/f144445e4c406f839ba92ce2a044c664

[mitsudome-r]

related discussion: https://github.com/orgs/autowarefoundation/discussions/7133

[technolojin]

How simulating sensor kit information to be managed?
In current AWSIM, the sensor information lives in two places, AWSIM and autoware_launch repository.
updating AWSIM to update/modify sensor configuration is in high learning curve.

Is possible to integrate simulating sensor kit information to one place?

[hakuturu583]

I think it would be best to store sensor parameters (such as LiDAR resolution) in autoware_carla_itneraface using a key-value store where the sensor name serves as the key, and then have the launch process specify that key to retrieve the parameters.
While detailed sensor parameters may inevitably be simulator-specific, the management of what is placed where should be handled by the launch process.

[hakuturu583]

@mitsudome-r As far as I understand, this discussion was addressed by the Software WG, and approval was granted on the understanding that the details of the file format would be worked out as development progresses.
Do you share this understanding, Mitsudome-san?

[hakuturu583]

autoware-carla-scenario integration wiill be integrated into Autoware soon!

autowarefoundation/autoware_universe#13319

[hakuturu583]

Proposal: donate the toolchain repository to AWF — and split it in two while doing so

As a follow-up to this design, I would like to donate tier4/autoware_lanelet2_to_opendrive to the Autoware Foundation, so that the pieces this architecture depends on live under AWF governance and can be maintained by the community.

What the repository contains today

It is currently a single uv workspace holding two packages with very different jobs:

Package	Role	Where it sits in the design above
autoware_lanelet2_to_opendrive	convert CLI: Lanelet2 .osm → ASAM OpenDRIVE .xodr, with Lanelet2 preprocessing, ASAM QC validation and a CARLA-specific output overlay	Map/asset preparation — the file/data handoff into CARLA (dashed arrow)
autoware_carla_scenario	Hydra-driven scenario runner: loads the converted map into CARLA, drives the Autoware ego, evaluates pass/fail conditions, records video/JSON results, ships a FastAPI viewer	Traffic Environment Layer — the "Scenario (Python)" + "Scenario Engine" boxes, and the base library referenced in Note 1

Why they should be separated before/at contribution

Keeping both in one repository was convenient while bootstrapping, but it is confusing from the outside — the repository name advertises a map converter, while half of it is a closed-loop scenario framework. Concretely:

1. Different audiences. Map conversion is useful to anyone who needs an OpenDRIVE road network from a Lanelet2 map, with no interest in CARLA or in running Autoware. Scenario authoring is only relevant to people doing closed-loop evaluation.
2. Different dependency footprints. The converter depends on the Lanelet2 Python API and the ASAM QC framework and needs no CARLA client at all; the scenario package pins a specific CARLA Python client, which is exactly the API-stability mechanism described in Note 1. Merging the two forces every user to resolve the union of both dependency sets.
3. Different release cadences and versioning. Note 1 argues scenarios should ship as a versioned Python package that can be pinned and reused like any dependency. That only works cleanly if the scenario package has its own version number and release history, decoupled from converter changes.
4. Different maintainership. Map/HD-map expertise and simulation/scenario expertise are not the same reviewer pool; separate repositories let AWF assign code owners that actually match.

Proposed contribution shape

Donate as two repositories under autowarefoundation:

• autoware_lanelet2_to_opendrive — the Lanelet2 → OpenDRIVE converter only (CLI, preprocessing, QC validation, CARLA output overlay, its Docker image and CI).
• autoware_carla_scenario — the scenario library/runner only (scenario definitions, Hydra configs, evaluation + recording, viewer, CARLA integration tests), consuming the converter's output as a file artifact, not as a source-level dependency.

Notes on the mechanics:

• The split should be done with git history preserved for each package (e.g. git filter-repo), not as a fresh squashed import.
• Repository-level machinery (Dockerfile, docker-compose.yml profiles, pre-commit, CI workflows, docs) needs to be duplicated and trimmed per repository; the current uv workspace root goes away and each package becomes a standalone project.
• The cross-package touchpoint is small and already file-based (converted .xodr + map assets), so no API coupling has to be invented for the split.
• The Autoware-side integration of autoware_carla_scenario is already in flight in autowarefoundation/autoware_universe#13319; the donated repository would be what that integration pins against.

Feedback on the proposed boundary (and on which AWF WG should own each repository) is very welcome before I start the split.

[mitsudome-r]

https://pypi.org/org/autoware-foundation/
If you could share us your PyPI account, we can add you to our organization

[hakuturu583]

Released! 👍 https://pypi.org/project/autoware-carla-scenario/

[hakuturu583]

Proposal (1/3): donate tier4/qc-opendrive to AWF — the checker that verifies the conversion result

Following the donation proposal above, there are two more repositories on the OpenDRIVE side of this architecture, and both are tier4 forks today. I would like to donate them to the Autoware Foundation as well, so that the donated autoware_lanelet2_to_opendrive does not pin anything owned by tier4. This comment covers the first one, the next one covers qc-baselib-py, and a third comment covers an unrelated proposal on the CARLA client side (typesafe_carla / carla_canvas).

What it is

tier4/qc-opendrive is a fork of asam-ev/qc-opendrive (MPL-2.0), the ASAM Quality Checker bundle for OpenDRIVE. It is how we verify the output of the Lanelet2 → OpenDRIVE conversion: the converter's analyze / qc-validate path runs this bundle over the generated .xodr and turns the findings into a report, so a converted map is judged against the ASAM spec rather than by "CARLA loaded it without complaining". Without it, the dashed map/asset handoff arrow in the design has no acceptance criterion.

It is already a hard dependency of the repository proposed for donation above:

# tier4/autoware_lanelet2_to_opendrive — pyproject.toml
[tool.uv.sources]
asam-qc-opendrive = { git = "https://github.com/tier4/qc-opendrive.git", rev = "f8b186ba..." }

As long as that fork stays under tier4/, the donated converter cannot be built from AWF-owned sources alone. That is the main reason to move it.

Why a fork exists at all

main of the fork is currently identical to upstream — every change lives on a branch with an open PR, so nothing has been quietly diverged:

Change	Why we need it
Handle an omitted <link> in the contact-point check (#5)	<link> is minOccurs="0" in the schema (OpenDRIVE §10.3, "Isolated roads may omit this element"). The checker calls road.find("link").find(...) unconditionally and aborts with AttributeError on a spec-legal map — which our converter emits.
Use road_belongs_to_junction() instead of @junction == 1 (#6)	@junction is -1 outside junctions and the junction id inside one, so comparing it to 1 only skips roads that happen to belong to junction 1. Every other junction road is falsely reported.
Memoise getpath() (#8)	All 42 issue-location call sites go through lxml's xmlGetNodePath, which walks the preceding siblings — i.e. the roads — at each level. Any checker reporting per road is therefore quadratic in map size, which is what we hit on city-scale maps.
Relax the numpy upper bound (#9)	numpy = "^1.26.0" excludes numpy 2.x, while the converter locks numpy 2.2.6. Same intent as upstream asam-ev/qc-opendrive#138. This is the commit currently pinned by rev.
Drop an unused numpy.ma.core import (#7)	Trivial, but it is the base of the stacked PRs above.

In other words, the fork is not a product fork — it is the set of upstream bugs that a Lanelet2-derived OpenDRIVE map happens to trigger.

Upstream-first, and the fork is meant to shrink

The goal is not to maintain a permanent downstream copy of a well-run ASAM project. The policy I would propose for the AWF-owned fork:

1. Everything goes to asam-ev first; the fork only carries what is still in review there.
2. main of the fork stays a fast-forward of asam-ev/main — patches live in branches that get dropped as upstream merges them.
3. The converter pins the fork by rev only while a fix is pending, and moves back to the upstream PyPI release (asam-qc-opendrive>=1.0.0) as soon as it can.

If upstream takes all of the patches, the fork ends up holding nothing and can simply be archived. That is the success case.

Proposed shape

• Create autowarefoundation/qc-opendrive as a GitHub fork of asam-ev/qc-opendrive (not a fresh import), so the fork relationship — and therefore one-click PRs to upstream — is preserved. License stays MPL-2.0; no relicensing is involved or needed.
• Push the five branches above to it and reopen the PRs there, keeping the review history readable.
• Switch autoware_lanelet2_to_opendrive to pin the AWF fork at the same revision, so the donated converter has no tier4 remote left.
• Ownership: the same WG that ends up owning autoware_lanelet2_to_opendrive, since the people who review a map-conversion change are the ones who care whether the checker accepts its output.

Feedback welcome — in particular on whether AWF prefers hosting such upstream-tracking forks at all, versus vendoring the patches in the converter until ASAM merges them.

[hakuturu583]

Proposal (2/3): donate tier4/qc-baselib-py to AWF — the library the QC report itself is built with

Second half of the previous comment. tier4/qc-baselib-py is a fork of asam-ev/qc-baselib-py (MPL-2.0), the base library that every ASAM Quality Checker bundle — including qc-opendrive from the previous comment — is written against: it owns the checker/issue model, the Result object, register_issue() / load_from_file(), and the writing of the schema-conformant report file.

The two repositories therefore travel together: the converter's validation step runs qc-opendrive, and qc-opendrive builds its report through qc-baselib-py. A defect here does not make the checker crash — it makes the report wrong or impossibly slow, which is worse, because the report is exactly the artifact we want to trust when accepting a converted map.

Status of the fork

Unlike qc-opendrive, this fork's main is 4 commits ahead of upstream (two PRs already merged into the fork), with three more in review:

Change	What goes wrong without it
Reserve the issue ids already used by a loaded report (#10, merged)	load_from_file() leaves the id counter at 0, so an issue registered on a loaded report gets an id the report already uses. _get_issue() then returns whichever duplicate it finds first, and add_*_location() / add_domain_specific_info() silently mutate the wrong issue. No exception — the written report is just wrong.
Look issues up through an index kept by their checker (#11, merged)	_get_issue() scans every issue of the checker, so adding a location to each issue is O(N²). On a city-scale map that is the difference between a report and a hang.
Keep a rejected issue out of the report (#7)	register_issue() appends the issue before validating, so an issue the validators reject stays on the checker: it is written to the report, and every later registration on that checker fails with the same exception.
Keep a rejected status off the checker (#8)	Same shape in set_checker_status() — pydantic validates after assignment, so e.g. a SKIPPED status on a checker that holds issues sticks, and the report is written with status="skipped" next to <Issue> elements, which the schema forbids.
Validate only the issue being registered (#9)	register_issue() re-validates every issue registered so far on each call — O(N²) again. The PR adds a guard test pinning the validators, so the optimisation cannot silently stop being equivalent.

Two of these are correctness bugs that produce a silently wrong report, and three are the reason a QC run over a large converted map is slow enough to discourage running it at all. Both categories matter for using QC as an acceptance gate in CI rather than as something a human runs occasionally.

Current dependency situation

autoware_lanelet2_to_opendrive resolves asam-qc-baselib 1.1.0 from PyPI today and pins only the qc-opendrive fork by rev. Pinning the fixes above means pointing at the fork as well — which is precisely why it should not be a tier4 repository by then.

Proposed shape

Identical to the previous comment, and for the same reasons:

• Create autowarefoundation/qc-baselib-py as a GitHub fork of asam-ev/qc-baselib-py, MPL-2.0 unchanged, so upstream PRs stay one click away.
• Upstream-first: main tracks asam-ev/main, the fork only carries what is pending review upstream, and patches are dropped as they are merged there. The fork shrinking to nothing is the intended end state.
• Same owning WG as qc-opendrive and autoware_lanelet2_to_opendrive — the three are one toolchain and splitting their review across WGs would only add latency.

If AWF would rather not host upstream-tracking forks at all, the alternative is to vendor these patches inside the converter until ASAM merges them; I think a fork is the cleaner and more honest option, but I am happy to follow whatever the foundation prefers.

[hakuturu583]

Proposal (3/3): support typesafe_carla + carla_canvas in autoware_carla_interface

Separate from the two donation proposals above, this one is about the CARLA client side of the Simulation Core box — how autoware_carla_interface talks to CARLA, and where the Renderer of the Sensor Simulation Layer plugs in.

The two libraries

Library	What it is
typesafe_carla (MIT, PyPI typesafe-carla 0.8.5)	A CARLA-API-compatible client that is statically type-checked. Code is written against the same API shape as the carla package (Client / World / Actor / Sensor / command.* …), but it is compiled by Codon and calls LibCarla through a narrow C ABI — CPython and the carla package are not in the path. Passing a Transform where a VehicleControl is expected is a compile error, not a failure 40 s into a scenario run. The surface is generated from binding specs (252 C ABI functions) validated against LibCarla with libclang, so the coverage is mechanical rather than hand-maintained.
carla_canvas (MIT, PyPI carla-canvas 0.2.1)	The abstraction of the rendering layer implemented inside typesafe_carla. typesafe_carla has a sensor-source ABI: a plugin can supply the measurements and the ROS 2 output of the sensors it claims, instead of the CARLA server. carla_canvas is that plugin — it hands the claimed sensors to a renderer service over gRPC. Which sensors are swapped and which renderer is used is a configuration file (CARLA_CANVAS_CONFIG); the program is unchanged, listen() / poll() / wait_for_data() / enable_for_ros() keep working, measurements arrive as LibCarla's own Image / LidarMeasurement, and the ROS 2 topics keep CARLA's names and types. See the repositories for the details.

carla_canvas is exactly the Sensor Simulation Layer boundary drawn in the design above, made concrete: a 3DGS renderer, or a world model such as cosmos3-canvas (NVIDIA Cosmos3-Nano-Sim-Auto generating the ego's cameras and LiDAR closed-loop), becomes a renderer behind the CARLA API — with no change to autoware_carla_interface and no change to the topics Autoware subscribes to.

Why support it in Autoware

1. It is an installable dependency. Both libraries are released on PyPI, so a rosdep key resolves them and autoware_carla_interface can finally declare its CARLA client in package.xml. Today that package declares no CARLA client at all, and the README asks the user to install a third-party wheel or to add an egg file to PYTHONPATH by hand — a step that cannot be reproduced by rosdep install and that every CI or container image has to special-case.
2. Type checking where it pays off. The bridge and the scenarios are the parts where an API mistake surfaces late, in the middle of a long simulation run, and often as a confusing CARLA-side error. Catching it at build time is worth more here than in ordinary application code.
3. Renderer swapping becomes configuration, not a fork. The alternative to carla_canvas is patching the sensor path of the bridge per renderer, which is what we would otherwise end up doing for every new sensor-simulation backend.

Constraint: UE5 only, so UE4 needs a fallback

typesafe_carla targets LibCarla of the UE5 line (0.10.0, ue5-dev, ue58-dev). The combination documented for autoware_carla_interface today is still CARLA 0.9.15 (UE4), and that must keep working for everyone using it.

So the integration shape I would propose is:

• Put a thin client abstraction in autoware_carla_interface with two backends: typesafe_carla (+ carla_canvas when a canvas configuration is given) for UE5-line servers, and the existing carla Python package for the UE4 line.
• Select the backend explicitly (launch parameter, defaulting to what the connected server reports), and fall back to the existing path when typesafe_carla is not installed or the server is 0.9.x. No current user loses their setup, and the UE4 code path stays exactly as it is.
• Only once that fallback exists does the new client get wired in as the default for the UE5 line.

On upstreaming to CARLA

The long-term goal is not to maintain a parallel client forever. CARLA itself is currently in feature freeze for the UE5.8 release, so proposing this upstream right now is not realistic — nothing would move until the freeze lifts. Once it does, I would like to spend the time to propose integrating these capabilities into CARLA upstream: a typed client surface, and the renderer/sensor-source plugin boundary that carla_canvas relies on. Until then the two libraries live outside CARLA, which is also the honest place for them while the ABI is still moving.

Feedback welcome on where the client abstraction should sit (inside autoware_carla_interface, or as its own package that both the bridge and autoware_carla_scenario use), and on whether AWF would want these two repositories under the foundation as well, or prefer to consume them from PyPI as ordinary upstream dependencies.