Request for Comments: Modular AirStack — module repos, slot contracts, marketplace, and distributed CI/docs #379
andrewjong
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
RFC: Modular AirStack — module repos, slot contracts, a module marketplace, and distributed CI/docs
This RFC is the output of a design session on making AirStack modular rather than monolithic. It is written to be self-contained: a reader (human or AI agent) should be able to understand the full design and implement it from this document alone. Repo facts referenced below are accurate as of
develop@55d9b887(VERSION0.19.0-alpha.9).1. Problem statement
Projects branched from AirStack develop features we want to offer to trunk users as optional capabilities, but today the only integration path is merging into the monolith or living forever in a fork. Concrete motivating cases:
The specific pain points this design addresses:
robot/ros_ws/src/local/local_bringup/launch/containslocal.launch.xml,local_droan_cpu.launch.xml, andlocal_macvo_obstacle_avoidance.launch.xml— near-copies differing by which module fills a role, with alternate wirings living as commented-out lines.Dockerfile.robot, GCS, Isaac). This breaks when deps are cross-repo, and the permutation space of module combinations is too large to prebuild.mikeversioning); module docs will be distributed across repos but should still be served coherently on the public site.Design principles (decided)
2. Module anatomy: a thin repo with a manifest
A module repo contains its ROS packages / Isaac extension / assets plus a
module.yamlat the root:The three motivating examples map as: OptiTrack →
ros_package,slot: state_estimator,provides: [odometry]; custom planner →slot: global_planner; DFM2 →isaac_extension(Isaac supports extension search paths, so it's the lowest-coupling pilot).3. Sync mechanism and launch integration
Sync: vcstool + a
modules/overlay (not submodules)modules.reposfile (vcstool format).airstack module add <name-or-url>clones into a gitignoredmodules/directory at repo root and updatesmodules.repos.type/targetssay: ROS packages symlinked intorobot/ros_ws/src/modules/(colcon picks them up automatically), Isaac extensions onto the extension search path, compose fragments merged viaCOMPOSE_FILE.airstack module add|remove|list|sync|doctor|create|search|extract, plusairstack init --release <X>,module create --in-tree, andmodule doctor --drift(see §11, researcher workflow).Slots + one stack config: kill the variant launch files
The layers already implicitly define roles. Make them explicit slots (
state_estimator,local_planner,local_world_model,global_planner,global_world_model,controller,behavior, …). Each slot has a contract: the standard topics (already documented indocs/robot/autonomy/integration_checklist.md) become required launch args. Every module — including trunk's own — ships its own launch file honoring that arg interface.Replace hand-edited bringup XML with a per-deployment
stack.yaml:A generic Python bringup reads
stack.yaml, includes each selected module's launch file, and passes the contract topics. Enabling/disabling a module = one line. The threelocal_*.launch.xmlvariants collapse into three presetstack.yamlfiles. (Precedent for YAML-driven collection already exists in trunk:tests/colcon_unit_test_packages.yaml.)Contracts are more than topic names. The slot contract spec must also capture QoS profiles (a best-effort subscriber against a reliable-only publisher silently receives nothing) and frame/units conventions (TF tree shape, ENU vs NED, frame naming). Cheap to declare, miserable to debug in the field.
airstack module doctorchecks these.4. Modules that don't fit a slot (e.g., an end-to-end VLA planner)
A rigid slot taxonomy breaks on modules that swallow several slots — e.g., an end-to-end vision-language-action planner spanning perception → global planning → local planning. Three refinements:
(a) Slots are contracts at narrow waists, not boxes. The stack's real invariants are a handful of interchange points — rungs on a ladder: sensor bus → state estimate (
/odometry) → global plan → trajectory interface (trajectory_segment_to_add/trajectory_override) → task action servers (tasks/navigate) → low-level interface commands. A conventional module taps in at one rung and out at the next; an end-to-end module taps in high and out low.(b) A module can declare what it replaces:
The resolver vacates replaced slots (no double-publishing), verifies everything consumed is still provided by something, and warns about orphaned downstream consumers (e.g., a GCS map view fed by the vacated world model).
(c) Three module tiers of decreasing verifiability, all first-class:
stack_module— declares consumes/produces/replaces; doctor verifies wiring and conflicts, not internals.Non-negotiable safety floor across all tiers: command authority flows through the trajectory controller / interface layer, where arming, safety monitoring, and takeover live. A module emitting
trajectory_overrideinherits the whole safety apparatus for free (timeout behavior, land-on-state-estimate-loss, GCS override) — a selling point for authors, not a constraint. A module that genuinely must bypass it (direct attitude/rate commands) must declareproduces: [attitude_setpoint]and the doctor flags it loudly as safety-critical.Cross-module compatibility uses the same machinery:
provides/requirescapability names + slot uniqueness gets ~90% of the value cheaply.doctorreports: two modules claiming one slot, arequiresnobody provides, message-type mismatches vsairstack_msgs. Do not build a full ontology — names + message types, validated at sync time and optionally at launch.5. Version compatibility and test separation
The substrate already exists
Trunk's
docker-build.ymlpushes cosign-signed images tagged by.envVERSIONon merge tomain/develop. Published images are the compatibility contract.Module CI: a reusable workflow trunk owns
Trunk publishes a
workflow_call-able workflow (.github/workflows/module-system-tests.yml). A module repo's entire CI:The reusable workflow: (1) checks out trunk at
airstack_ref; (2) checks out the module and runsairstack module add ./— the same overlay path a developer uses, so the module'sslot:declaration wires it into the test bring-up'sstack.yaml(their planner runs instead of the default); (3) runs the existing suite unchanged viaairstack test -m "<marks>".test_build_packagesbuilds their package because it's in the workspace;waypoint_flightexercises their planner because it fills the slot. The module repo never copies a single test.System tests double as slot conformance tests.
tests/waypoint_checker.pyjudging the odometry track (in-order corridor arrival, goal tolerance, per-waypoint timeout) doesn't care which planner produced the motion — passing it is the behavioral definition of "a working global planner." Formalize: each slot contract names the marks a filler must pass (global_planner→waypoint_flight,autonomy;state_estimator→liveliness,sensors,takeoff_hover_land). The registry compat badge for (module version × AirStack version) is granted only when that mark set passes.GPU runner access (system tests need GPU + sim license)
Trunk's GPU jobs run on OpenStack ephemeral runners via
.github/orchestrator/(per-job VM, JIT token, destroyed on completion). Two paths, both used:repository_dispatchwith{module_repo, module_ref, airstack_ref}; trunk runs the suite on its own runners and posts a check-run/status back to the module's commit via a GitHub App. Secrets, licenses, runner control never leave trunk; bench time can be gated/rate-limited.Cost ladder (encode in the reusable workflow defaults)
unit+build_packagesinside the published image — minutes, no GPU, no orchestrator.liveliness(+sensorsfor perception-adjacent modules);msairsimis the cheaper bring-up.develop, so breakage is discovered by trunk the day it lands — not by module authors months later.Metrics: let module runs'
metrics.jsondiff against trunk's baseline for the same AirStack version via the existingtests/parse_metrics.py. Badges can then carry e.g. "cross-track RMSE vs default planner," making the marketplace informative rather than a link list.6. Docker images: base + composed module layers
Never publish permutations. Trunk keeps publishing exactly one signed base image per host type per version. Composed images are built where they're used, from cached layers.
Three dependency tiers (declared in
module.yaml)rosdep/apt/pip lists (default). Deps in
package.xml(rosdep) +deps: {apt, pip}in the manifest. No Dockerfile in the module repo.Dockerfile fragment for real install steps — written against a build arg, never a fixed base:
Prebuilt overlay image for monster deps (e.g., a VLA's 20 GB CUDA/PyTorch). The module's own CI publishes
ghcr.io/<org>/<module>-overlay:<airstack_version>builtFROM ${BASE_IMAGE}; the manifest points at it so developers pull instead of build.Composition chain
airstack module syncsorts enabled modules deterministically (by name), groups by target host, and builds:bwsinside the container, unchanged).modules.lock(resolved chain + image digests).airstack upknows precisely when a rebuild is needed and says so ("optitrack deps changed, rebuilding 2/4 layers"). The lockfile also gives reproducibility: compat badges record the digest chain they were earned with; a bug report of "modules A+B+C on 0.19.2" is one file.Two failure modes to design for
numpy<2, another needs>=2):doctorrunspip check/ apt dry-run against the composed layer set at compose time and names the two modules fighting — not a colcon failure forty minutes in.Dockerfile.modulein chain order. Modules keep their fragment buildable and treat the published overlay as a cache, not the source of truth.Side effect: trunk's Dockerfiles thin out over time — deps merged into
Dockerfile.robotbecause some configuration needs them (MACVO networks, mocap SDKs) migrate into the modules that own them. The monolith unwinds itself module by module.7. Marketplace / registry
An
airstack-modules-indexrepo (rosdistro'sdistribution.yamlpattern): one YAML entry per module — name, repo URL, description, maintainer, CI-generated compat matrix.airstack module search|list|addreads it; MkDocs renders it into a browsable catalog with compat badges. Getting listed = PR to the index, which doubles as the quality gate (manifest valid, CI green, README template sections present, license checked).Release sets: the answer to N×N module compatibility
You can't test every module pairing. Periodically publish a curated release set — "AirStack 0.20 + these module versions, tested together" — one lockfile in the registry, blessed by a single CI run with all of them enabled. Individual badges mean "works with trunk"; release sets mean "works together." Most users pull a release set; à-la-carte is for developers. Release sets also drive versioned docs (§9).
8. Governance (the part most likely to decide success)
airstack_msgsruthlessly, as its own versioned package. ROS 2 type hashes mean a module built against a different msg definition doesn't degrade gracefully — it silently fails to connect.doctorcompares msg-package versions across the composed workspace.9. Docs
Hybrid, with embedding automated. Pure link-outs to repo READMEs fail users: no unified search, no consistent theme, and no versioning (a GitHub README always shows
main— wrong for someone on 0.19). Trunk already versions docs withmike; module docs ride that machinery.docs/), declared viadocs:in the manifest and validated against the README template at registration. The trunk docs deploy workflow gains a pre-build step: for each registered module, shallow-clone at a pinned ref, drop content undermodules/<name>/, splice a nav entry. (Plugins exist —mkdocs-multirepo-plugin— but a fetch loop driven by the registry index is simpler and easier to debug. This extends trunk's existingsame-dirpattern of including package READMEs from outsidedocs/.)develop/latest site fetches module default branches.repository_dispatch) + existing push triggers + nightly backstop. "Within a day, exact at releases" is the right target.10. Smaller items (one line each, all decided)
assets: [{url, sha256, dest}];module syncfetches to a cache. No Git LFS blobs in module repos.airstack module creategenerates the template repo (manifest, CI stub, launch, README); update.agents/skills/so AI-agent contributions build modules the new way instead of reintroducing the monolith pattern.11. The researcher workflow: research in a fork, graduate to a module
A principle worth stating explicitly: research happens in a fork; modularity is a graduation step, not an entry fee. Nobody exploring an algorithm wants to maintain a manifest, CI, and contract compliance while their idea changes daily — if the process demands that, researchers will ignore it and we get whole-repo forks again (the DFM2 story). The design's job is to make the fork cheap to extract from later, which means the critical moment isn't extraction — it's day one.
Stage 0 — start from a release set.
airstack init --release 0.20clones trunk at the blessed tag with the release-set lockfile;airstack module add <name>pulls any modules the experiments need. The researcher pins here and stays pinned through the research — stability through the paper deadline, no chasing trunk.Stage 1 — research in-tree, but inside a boundary. The expensive part of extraction later is never moving files — it's the archaeology: edits sprinkled across trunk launch files, hardcoded topics, deps snuck into
Dockerfile.robot. So impose exactly one cheap discipline at the start:airstack module create --in-tree my_plannerscaffolds a directory underrobot/ros_ws/src/modules/my_planner/— a normal colcon package with a stubmodule.yamland its own launch file — living in the researcher's fork, committed to their repo, with zero CI or registry obligations. They hack freely inside it. All ceremony is deferred; only the directory boundary exists from day one, and that boundary is what makes extraction mechanical instead of archaeological.To keep the boundary honest without policing:
airstack module doctor --driftdiffs the fork against the pinned release and classifies changes — contained in the module dir (fine) vs. trunk file edits (extraction debt, each listed). Researchers can run it never, occasionally, or in their fork's CI; either way, at extraction time the drift report is a precise, machine-readable work list rather than a surprise.The boundary is a default, not a cage. Some feature work genuinely is repo-wide, and the drift report informs — it never blocks: no CI failure, no refusal to sync, no gate anywhere. A trunk edit in the drift report has three healthy destinies rather than being forbidden: (1) upstream it — bug fixes and genuine improvements belong to everyone; PR to trunk and it leaves the report entirely; (2) carry it as extraction debt — perfectly fine to defer; at graduation it becomes a
stack.yamloverride, launch arg, or Dockerfile fragment; (3) recognize it as a missing rung — sometimes a trunk edit is evidence the contracts lack a tap point the researcher needed, which is exactly what the contract RFC process (§8) is for; drift reports are how those gaps get discovered.Stage 2 — graduate. Once the algorithm is ironed out:
airstack module extract my_plannermoves the directory into a fresh repo from the template (manifest completed, CI stub pinned to their release version, README skeleton) and emits the drift report as the remaining task list — convert this hardcoded topic to a contract arg, move these threeDockerfile.robotlines intodeps:/Dockerfile.module, turn that trunk config edit into astack.yamloverride. Then the standard loop: conformance marks against the pinned version (should pass immediately — same code), then against current trunk (the deferred compat work happens once, at the end — matching researcher incentives), then register in the index.Agent-assisted extraction. Extraction is an ideal coding-agent task for a specific reason: it has an objective success criterion. Package it as an
.agents/skills/extract-moduleskill whose context is themodule.yamlschema, the contract/rung specs, the template repo, and the drift report as input. The agent does the mechanical conversion and usesdoctor+ the slot's conformance marks (e.g.,waypoint_flightpassing with the researcher's planner in the slot) as its verification loop — it isn't hoping the refactor worked; it flies the drone until it does. A second, smaller skill matters just as much: guidance for agents helping during Stage-1 research, so an agent asked to "add a parameter to my planner" keeps changes inside the module directory instead of reintroducing trunk edits. Cheap prevention that makes Stage 2 nearly free.Stage 3 — maintain or park. Nightly canary + compat bumps while the author is around; the unmaintained shelf when they graduate. The graduating-student story now has a good ending: an afternoon of agent-assisted extraction turns a thesis fork into a registered module, instead of the work dying with the fork.
12. Phased implementation plan
module.yamlschema (+ JSON Schema validation); reusablemodule-system-tests.ymlworkflow; extract DFM2 disturbances as pilot (sim-side, lowest coupling: Isaac extension path, no slot). Proves "own repo, own CI, own tests."modules/overlay dir +modules.repos;airstack module add|remove|sync|doctorsubcommands in the CLI; Docker composition chain +modules.lock; extract OptiTrack and the custom planner. Researcher-workflow tooling (§11):module create --in-tree,doctor --drift,module extract, and theextract-moduleagent skill.stack.yaml-driven bringup; collapse thelocal_*.launch.xmlvariants into presets. (Big refactor — fixes launch clutter for trunk modules too.)airstack-modules-indexregistry repo; docs catalog + module-docs embedding; nightly canary; org runner group + dispatch test bench; release sets; governance policies (maintainer lifecycle, deprecation RFCs).Beyond these phases, cross-embodiment generalization (platform modules, trait-based compatibility) is future work tracked in RFC #380 — sequenced only after phases 1–4 are proven.
Ranked success factors: (1) the CI-against-published-images loop, (2) contract governance/deprecation policy, (3) maintainer lifecycle policy. Everything else is buildable later; those three determine whether the ecosystem stays trustworthy.
Appendix: trunk facts an implementer needs
.envVERSION="0.19.0-alpha.9"tags docker images;check-version-increment.ymlgates PRs on semver bump..github/workflows/docker-build.ymlbuilds, pushes, cosign-signs all compose images on merge tomain/developwhenVERSIONchanges.tests/system/(pytest marks intests/pytest.ini:unit,build_docker,build_packages,integration,liveliness,sensors,takeoff_hover_land,autonomy,waypoint_flight); harness intests/harness/+tests/conftest.py; unit-test package list intests/colcon_unit_test_packages.yaml(precedent for YAML-driven collection); metrics viatests/parse_metrics.py(diff-vs-baseline, exits 1 on regression); waypoint pass/fail via standalonetests/waypoint_checker.py.runs-on: [self-hosted, airstack-ephemeral]→ OpenStack ephemeral VMs via.github/orchestrator/(spawn loop, JIT tokens, cloud-init, reap loop). Fork PRs blocked from runners.robot/ros_ws/src/autonomy_bringup/launch/robot.launch.xml, role-dispatched byAUTONOMY_ROLE(full|onboard|offboard); per-layer bringups likerobot/ros_ws/src/local/local_bringup/launch/(where the variant explosion lives).deploy.replicas: ${NUM_ROBOTS};ROBOT_NAME/ROS_DOMAIN_IDresolved at container start byrobot/docker/.bashrc+robot/docker/robot_name_map/resolve_robot_name.py./{robot}/odometry,/{robot}/global_plan,/{robot}/trajectory_controller/{trajectory_override,trajectory_segment_to_add,look_ahead,tracking_point,set_trajectory_mode},/{robot}/tasks/{navigate,takeoff,land,fixed_trajectory},/{robot}/interface/*— seedocs/robot/autonomy/integration_checklist.md.mikeversioning,same-dirplugin for out-of-tree READMEs, deploy workflows per branch (deploy_docs_from_{main,develop,release}.yaml).castacksorg (disturbances under itssimulation/), OptiTrack integration (natnet_ros2exists in trunk today), custom global planner.This RFC came out of a design brainstorm session (2026-08-04). Comments and pushback welcome — especially from anyone who'd author a module or maintain the registry.
All reactions