Proposal: Dora AI Fabric — A Native AI/ML Execution Layer for Dora

[SaitejaKommi]

Proposal: Dora AI Fabric — A Native AI/ML Execution Layer for Dora

Status: Draft
Author: @SaitejaKommi
Category: Ideas
Focus: AI/ML · Dataflow Infrastructure · Observability · Reproducibility

---

Abstract

Dora already provides a strong foundation for building distributed, low-latency dataflows for robotics and real-time applications.

As AI systems become increasingly composed of multiple models and processing stages, there is an opportunity to make Dora a stronger infrastructure layer for modern AI/ML pipelines.

A typical AI application may look like:

Camera
   ↓
Preprocessing
   ↓
Vision Model
   ↓
Embedding Model
   ↓
RAG / Vector Search
   ↓
LLM / VLM
   ↓
Decision / Action

Today, each component can already be implemented as a Dora node. However, AI/ML developers still need to reason about:

AI/ML Concern | Example -- | -- Tensor representation | Shape, dtype, layout Model compatibility | Input/output requirements Model metadata | Model identity and version Performance | Inference and end-to-end latency Profiling | Bottlenecks and throughput Reproducibility | Configuration and model versions Recording / replay | Reproducing failures Interoperability | AI/robotics tooling

AI metadata should preferably be descriptive rather than transformational.

For example, describing:

float32[768] CUDA

should not require copying the tensor merely to validate its metadata.

---

Backward Compatibility

Existing Dora dataflows should continue working unchanged.

AI metadata should be optional.

flowchart LR
    A[Existing Node] --> B[Dora Runtime]
    B --> C[Works Exactly as Before]

AI-aware nodes could additionally provide:

AI Node
   │
   ├── Data contract
   ├── Model metadata
   └── Observability metadata

This allows the ecosystem to adopt the capabilities incrementally.

---

Non-Goals

This proposal does not aim to:

• Create a new deep-learning framework

• Replace PyTorch

• Replace TensorRT

• Replace ONNX Runtime

• Replace Hugging Face

• Implement model training

• Create a vector database

• Replace Dora's transport layer

• Immediately replace .drec

• Require every Dora node to become AI-aware

• Introduce GPU-specific abstractions into the core runtime without evidence

---

Success Criteria

A first prototype should demonstrate a real AI pipeline, rather than only adding abstractions.

For example:

Camera
   ↓
Vision Model
   ↓
Embedding
   ↓
RAG
   ↓
LLM

The prototype should demonstrate:

• Contracts preventing invalid connections

• Measurable end-to-end latency

• Model metadata

• Recording / replay

• At least one external visualization or interoperability workflow

The prototype should also include benchmarks showing the overhead introduced by each capability.

---

Why This Could Be Valuable for Dora

Most AI frameworks primarily focus on model execution.

Dora has an opportunity to focus on something different:

The infrastructure surrounding AI models.

A production AI system is rarely just a model.

It is usually:

Sensors
  ↓
Preprocessing
  ↓
Model
  ↓
Postprocessing
  ↓
Retrieval
  ↓
Reasoning
  ↓
Decision
  ↓
Action

The reliability and latency of the entire pipeline matter just as much as the performance of an individual model.

Dora already provides the dataflow foundation for connecting these components.

The proposed AI Fabric would make that foundation more useful to AI/ML developers without changing Dora's core identity.

---

Why I Am Interested in This

My primary background is in Machine Learning, Generative AI, and software engineering, rather than robotics.

While contributing to Dora's Rust codebase, I became interested in the boundary between:

AI Models
     +
Dataflow Infrastructure
     +
Real-Time Systems

I think this is an interesting area where my AI/ML experience can contribute while continuing to learn Dora's systems and robotics architecture.

I would rather validate the direction with the Dora maintainers and community before implementing a large feature.

---

Open Questions

I would especially appreciate feedback from the Dora maintainers and community on:

1. Should AI/ML contracts build on Dora's existing semantic output/type mechanisms?

2. Which AI/ML types would provide the most value initially?

3. Would AI pipeline profiling be useful as a first-class Dora capability?

4. Would an offline .drec → .mcap exporter be a useful first interoperability project?

5. Are there existing roadmap items or abstractions that this proposal should build upon?

6. Which real AI/robotics workloads would be the best reference implementation?

7. Of the four areas — contracts, observability, reproducibility, and interoperability — which one would provide the highest value to Dora users today?

---

Conclusion

I am proposing this as a direction for discussion rather than a fixed API design.

The core idea is:

Dora should not become an AI framework. Dora could become excellent infrastructure for composing, observing, and reproducing AI systems.

I would start with a small, measurable prototype — potentially AI data contracts + pipeline observability — and expand only if the results demonstrate real value.

I would appreciate feedback from the maintainers and community on whether this direction fits Dora's roadmap and, more importantly:

Which part would be most valuable to prototype first?

@heyong4725 @phil-opp

[heyong4725]

Thanks for writing this up, @SaitejaKommi, and apologies for the slow reply. The framing in your conclusion is one I agree with: dora should not become an AI framework, and the infrastructure around models is where a dataflow runtime earns its keep. Where I land differently is on how much of this is new. Three of the four pillars already have a home in dora, so I'd rather point you at those, and be clear about what is and is not on the roadmap, than stand up an "AI Fabric" layer beside them.

First, a mechanical problem: the post is a pasted HTML fragment and the middle is missing. It jumps from the "AI/ML Concern" table straight to "AI metadata should preferably be descriptive". The sections that presumably describe the four pillars (contracts, observability, reproducibility, interoperability) are not there, so nobody can review the actual design. Could you repost those in plain markdown?

Taking your open questions in turn against main:

1. Should contracts build on the existing type mechanism? Yes, and there is more of it than the proposal assumes. docs/types.md defines optional output_types / input_types on ports, checked by dora validate --strict-types and by dora build at build time with zero runtime cost. That is exactly your "descriptive, not transformational" property. Types are URNs with parameters, e.g. std/media/v1/AudioFrame[sample_type=f32,channels=2], with wildcard matching when one side is unparameterized. The registry lives in libraries/core/types/std/ (core, math, media, vision, control).

2. Which AI/ML types first? There is no std/tensor namespace yet. A std/tensor/v1/Tensor[dtype=f32,rank=1] (or a shape= parameter, if the matching rules can carry it) plus Embedding and Logits aliases would cover your float32[768] CUDA example and is a small, self-contained PR: a YAML file in libraries/core/types/std/, validation tests, a docs/types.md section. Keep device placement (CUDA) out of the type: it is a transport property, and the tensor-pool extension (libraries/extensions/tensor-pool/) already owns pinned-host / CUDA zero-copy handoff, explicitly outside the 1.0 stability surface because CI has no GPU runners. Anything GPU-aware should land there, not in core.

3. Pipeline profiling as first-class? Most of it exists. The daemon has an OpenTelemetry/OTLP exporter behind the telemetry feature (Jaeger, Tempo, Zipkin), dora trace shows coordinator spans with no external infrastructure, and libraries/extensions/telemetry/metrics exports host metrics. What is missing is a guide: "instrument a vision → embedding → LLM pipeline end to end, read per-node and per-hop latency in Jaeger." That is a docs PR plus possibly a docs/debugging.md section, not a subsystem. If you find a real hole while writing it (e.g. a span the node API should emit and does not), that becomes a well-motivated, small code PR.

4. Offline .drec → .mcap exporter? It is the one item in the proposal with no existing counterpart: there is no mcap anywhere in the repo. To be clear about where it stands, it is not on the maintainers' roadmap and we are not committing to build or own it. It is also the kind of thing that fits a community contribution well: offline, no core change, testable without hardware, and useful only to the people who actually run Foxglove or ROS tooling against dora recordings. If that need is real for you, a standalone exporter built against the .drec layout documented in libraries/recording/src/lib.rs (header, RecordEntry sequence, footer) is the right shape. Note that crate is explicitly outside the 1.0 stability guarantee, so the exporter should expect to track it. Arrow IPC as the mcap message encoding would keep it lossless. Whether it later merits a place in the repo depends on demand, not on the proposal.

5. Existing roadmap to build on? Model identity and version belong in the node hub manifest (docs/plan-node-hub.md), which is already typed and versioned and reuses the type URN registry as its contract language. A separate "model metadata" layer would duplicate it.

6 / 7. Reference workload and highest-value pillar? Contracts are the highest-value piece dora can take upstream today: the tensor types from question 2 are small, measurable, and slot into a mechanism that already exists. The mcap exporter is the highest-value piece for interop, but per question 4 that one is community-driven and demand-gated. For a reference pipeline, the examples/ tree already has camera → detection → visualization flows in Python; extending one with an embedding + LLM stage would be a realistic end-to-end demo for the profiling guide.

What I'd decline: an "AI Fabric" as a named layer or crate. The title says "native execution layer" while the non-goals correctly say it executes nothing; what remains is metadata + tooling, and dora's pattern for that is small opt-in extensions with measured overhead. The node-hub proposal only moved once it became a concrete spec with a checklist; the same will be true here. If the mcap exporter and tensor types land and turn out to want a shared surface, that is the moment to name it.

Happy to review a tensor-types PR or the profiling guide. If you build an mcap exporter and find users for it, bring the numbers and we can talk about where it lives.

[SaitejaKommi]

Thanks for the detailed feedback @heyong4725! This is really helpful.

Four Pillars

1. AI/ML Contracts

Extend Dora's existing type system with semantic types for common AI/ML data rather than introducing a separate contract system.

For example:

• std/tensor/v1/Tensor[dtype=f32,rank=1]
• std/tensor/v1/Embedding
• std/tensor/v1/Logits

These types would allow AI/ML nodes to declare what they consume and produce, enabling dora validate --strict-types and dora build to catch incompatible pipelines before execution.

The type should describe the data itself (dtype, rank/shape, semantic meaning), while device placement such as CUDA remains a transport/runtime concern handled by existing mechanisms such as tensor-pool.

2. AI/ML Observability

Use Dora's existing OpenTelemetry and tracing infrastructure to make AI pipelines easier to profile end-to-end.

For a pipeline such as:

Camera → Vision Model → Embedding Model → RAG → LLM

the goal is to make it straightforward to identify:

• per-node inference latency
• dataflow/hop latency
• bottlenecks between model stages
• end-to-end pipeline latency

This would primarily be a profiling/debugging guide using Dora's existing telemetry capabilities, with code changes only where an actual observability gap is identified.

3. Reproducibility

Treat AI/ML pipeline execution as something that should be inspectable and reproducible.

A recorded pipeline should preserve enough information to understand what data flowed through the system and how the pipeline was structured, while model identity/version information should build on Dora's existing node-hub manifest mechanism rather than introducing a separate model metadata layer.

The goal is to make it easier to reproduce and debug AI pipeline behavior without adding AI-specific execution semantics to Dora core.

4. Interoperability

Provide a practical bridge between Dora recordings and existing robotics/AI tooling.

One concrete direction is a standalone .drec → .mcap exporter that:

• reads Dora's existing recording format
• preserves recorded data without requiring changes to the runtime
• encodes messages using Arrow IPC where appropriate
• produces MCAP files usable by existing ROS/Foxglove tooling

This would remain an offline/community contribution rather than becoming a new core Dora subsystem.