Proposal: Dora Hub — A Multi-Registry Node Ecosystem for Dora

[LeonRust]

Proposal: Dora Hub — A Multi-Registry Node Ecosystem for Dora

Status: Draft
Author: @LeonRust
Date: 2026-06-09
Discussion: https://github.com/orgs/dora-rs/discussions/2095

---

Abstract

This proposal introduces Dora Hub, a multi-registry package ecosystem for discovering,
distributing, and verifying Dora nodes. It enables
developers to publish reusable operator nodes with typed data contracts, and dataflow authors
to discover and install them with a single command. The design separates the Hub (registry)
from the Runtime (execution), keeping each layer focused and maintainable.

---

Motivation

Current state

• Nodes are written as local files referenced by path in dataflow.yml
• No standard way to share nodes across projects or teams
• No mechanism for versioning, dependency resolution, or schema validation of third-party nodes
• Every team reinvents camera drivers, ML inference wrappers, and protocol adapters

Why now

The Dora ecosystem has stabilized around the 1.0 runtime. The missing piece is a
community distribution channel — a central registry where nodes can be published, discovered, and installed with a single command.

---

Goals

1. Publish: developers can publish typed, versioned, CI-verified nodes
2. Discover: dataflow authors can search and browse available nodes
3. Install: one command to install a node and all its runtime dependencies
4. Validate: Arrow schema contracts are checked at install time, preventing runtime surprises
5. Enterprise-ready: organizations can run private registries behind their firewall

---

Design

Architecture

Web Platform (Dashboard + Node Pages)
         │
    Registry API (REST + search + auth)
         │
    ┌────┴────┐
PostgreSQL     S3
 (metadata)    (artifacts)
         │
    CI System (GitHub Actions shared workflow)

Hub is a registry only. It does not participate in dataflow execution — that remains the
responsibility of the Dora Runtime.

CLI Surface (11 commands)

dora hub login              # Paste API token
dora hub logout             # Clear local token

dora hub search <q>         # Search the registry
dora hub install <n>@<v>    # Install a node
dora hub uninstall <n>      # Remove a node
dora hub info <n>           # View metadata & schema

dora hub publish            # Publish from local node.json
dora hub yank <n>@<v>       # Yank a version

dora hub outdated           # Check for updates
dora hub update             # Apply compatible updates
dora hub list               # List installed nodes

node.json — the core abstraction

Every published node carries a node.json manifest that declares:

{
  "id": "camera",
  "version": "1.2.5",
  "runtime": "python",
  "entrypoint": "camera.py",

  "inputs":  [{"id": "tick", "type": "uint64"}],
  "outputs": [{"id": "image", "type": "uint8", "shape": [1080, 1920, 3], "layout": "HWC"}],

  "config": {
    "resolution": {"type": "string", "default": "720p", "enum": ["480p", "720p", "1080p"]}
  },

  "requirements": {
    "hardware": {"camera": {"interface": "usb"}},
    "dependencies": {"python": ["opencv-python>=4.5"]}
  },

  "category": "sensor",
  "keywords": ["camera", "opencv", "vision"]
}

This manifest is the source of truth for the entire lifecycle: search indexing, schema
validation, CI verification, and dependency resolution.

Data contracts

• Type: Arrow types only (uint8, float32, int64, …)
• Shape: independent field (Arrow has no native multi-dimensional shape)
• Layout: optional (HWC, CHW); mismatch generates a warning, not an error
• Validation: dora validate checks that connected ports have compatible schemas before execution

Multi-registry model

hub: camera@^1.2                       # Official registry (hub.dora-rs.ai)
hub: hub.my-company.com/lidar@1.0      # Self-hosted registry (FQDN required)
hub: hub.other-org.com/filter@2.0      # Any third-party registry (FQDN required)

No prefix defaults to the official registry. All non-official registries must use
a fully qualified domain name. Self-hosted registries are backed by PostgreSQL + S3 with OIDC-based SSO.

---

Publishing Workflow

$ dora hub test                     # 4-step local verification
  1. Schema validation
  2. Build (cargo / pip / cmake)
  3. Smoke test (launch + mock input)
  4. Output schema check

$ git tag v1.2.5 && git push        # Triggers CI webhook
→ Hub CI validates → passes → published
→ Hub CI validates → fails → rejected with details

CI runs on Ubuntu 22.04, macOS 14, and Windows 2022. Timeout: 30 min (configurable).

C/C++ nodes: three-tier distribution

Tier	Strategy	When
L1	Precompiled binaries	Official critical nodes (ROS2 bridge etc.)
L2	Source + Docker CI build	Community standard path
L3	Source + local compilation	Escape hatch

System dependencies are declared per-distro and detected at install time with fail-fast
diagnostics (no auto-install of system packages).

---

Authentication & Authorization

• GitHub OAuth for Web login
• API tokens (Publish / Read-only / Full) managed via Web Dashboard
• Namespace = GitHub username or org name, auto-bound on first OAuth login
• CLI: dora hub login pastes a token; no browser-based OAuth flow in the terminal

---

Community Governance

• Free publishing: any GitHub user can register a namespace and publish nodes
• Version policy: semver, yanked versions block new installs but existing ones continue
• Deprecation: message + replaced_by fields guide users to successors
• Quality signals: CI status, download count, example presence; no subjective scoring
• Abuse prevention: 10-version/hour cap for new accounts; namespace squatting reclaimable after 6 months

---

Non-Goals

The following are explicitly out of scope for the Hub:

• Dataflow orchestration: dora run remains the domain of Dora Runtime
• Visual editor: no drag-and-drop dataflow builder
• In-browser testing: nodes are tested locally via dora hub test
• Built-in discussions: use GitHub Issues for community feedback
• Proprietary licensing enforcement: MIT/Apache-2.0 preferred, but no DRM

---

Implementation Plan

Phase	Scope	Estimate
MVP	CLI (search, install, publish, info, login), PostgreSQL registry, GitHub OAuth, basic Web Dashboard, CI verification pipeline	3 months
Phase 2	Enterprise private registries, L2 Docker CI builds for C/C++, outdated/update commands, offline mode	2 months
Phase 3	L1 cross-compiled binaries, pull-through mirror, node promotion workflow, Meilisearch backend	2 months

---

Prior Art

This proposal draws from established package registries:

• crates.io: namespace model, token-based CLI auth, yank semantics, rate limiting
• Docker Hub: multi-registry FQDN resolution, pull-through mirror
• npm: deprecated + replaced_by fields, outdated/update commands
• PyPI: free publishing without pre-review, semver conventions

---

Risks & Mitigations

Risk	Mitigation
C/C++ build matrix explosion	Three-tier strategy; L3 source compilation as default, L1/L2 only for critical/official nodes
Python dependency conflicts across nodes	Runtime-layer intersection solver, with per-node-venv escape hatch
Namespace squatting	6-month inactivity → dispute mechanism
Centralization risk	Multi-registry from day one; enterprises run their own

[heyong4725]

Thanks @LeonRust for this proposal — the level of detail (31 explicit decisions, clean Hub/Runtime separation) is a great foundation for discussion. A node ecosystem is one of the most important long-term goals for dora, which is exactly why I'd like to pressure-test the architecture before any implementation starts. I reviewed the full design against the current codebase and the existing node distribution channels. Detailed spec-level feedback is in #2097; this comment focuses on the architecture.

What I think is right

• The Hub/Runtime separation (registry stays out of orchestration)
• Typed data contracts in a node manifest — this is the single highest-value idea in the proposal. Today dataflow.yml outputs are untyped strings, and schema mismatches surface only at runtime
• Standard registry semantics borrowed correctly: semver, yank-blocks-new-installs-only, deprecated + replaced_by, fail-fast system deps with no auto-sudo, explicit transform nodes
• Multi-registry with FQDN resolution for enterprise deployments

Core question: what is the package format?

As written, the Hub is a metadata registry with pointers, not a package system:

• Rust/Python nodes: the "package" is a git ref. Install clones source and builds locally, or runs pip install for deps and copies the entry file. Copying a single entry file breaks for any multi-file node — and nearly all of the ~60 existing node-hub nodes are proper multi-file Python packages.
• C/C++ L1: the artifact is a single raw binary at an external URL (e.g. GitHub Releases) with a publisher-supplied sha256 in node.json.

This means there is no defined archive layout, and no immutability: git tags are mutable, GitHub Release assets can be re-uploaded, and the publisher controls both the file and its hash. Yank semantics, offline mirrors, and supply-chain integrity all assume the registry has custody (or at least an immutable pin) of the bytes — as designed, it has neither. The concrete question I think the design needs to answer first: what bytes does dora hub publish upload, and what bytes does dora hub install download?

Reconciling with the existing ecosystem

The proposal's "Current state" section says nodes are only local path references, but dora already has a working distribution channel: dora-hub contains ~60 nodes published individually to PyPI as wheels, consumed today via:

- id: dora-whisper
  build: pip install dora-distil-whisper
  path: dora-distil-whisper

The descriptor also already supports git: + branch/tag/rev sources with build: commands. Any Hub design needs a migration/coexistence story for these nodes — and "pip install deps + copy entry file" is strictly weaker than the wheel-based distribution they use today (no entry points, no manylinux binary wheels, no sdist, no cross-node dependency resolution).

Operational scope

The design commits the project to operating a crates.io-class service: PostgreSQL, S3, OAuth + token lifecycle, a web platform, plus a CI farm that runs docker buildx on untrusted community code for up to 30 minutes across three OSes — which is a crypto-mining and supply-chain-attack magnet, and a 24/7 security/moderation/legal responsibility. crates.io has paid full-time staff behind it. Who operates and funds hub.dora-rs.ai is not addressed, and I think the 3-month MVP estimate is off by a large factor once auth, abuse handling, and CI hardening are included.

Alternative: same goals, same UX, ~10% of the infrastructure

I'd propose keeping the proposal's CLI surface and manifest, but changing what's underneath: artifact bytes go to each ecosystem's native registry; dora maintains only the manifest + an immutable pin.

Node type	Artifact format & host
Python	wheel on PyPI (status quo for the ~60 existing nodes — zero migration)
Rust	crates.io for source installs; prebuilt binaries via OCI/GitHub Releases (cargo-binstall pattern)
C/C++, mixed, weight-heavy	OCI artifact on ghcr.io (ORAS)

dora hub publish becomes an orchestrator, not an uploader:

1. Validate node.json (schema, contracts, smoke test)
2. Publish to the native backend (twine / cargo publish / oras push)
   — skippable if the publisher's own release CI already did it
3. Capture the immutable identifier (wheel sha256 / crate version / OCI digest)
4. Submit { node.json + pin } to the dora index

dora hub install camera@^1.2 resolves against the index, dispatches to uv/cargo/oras under the hood, verifies the hash, writes the lockfile. Uniform UX does not require a uniform format — users never see which backend served the bytes.

The index itself can start as a git repo (dora-rs/node-index) mapping name@version → { manifest, backend pin } — the same model as cargo's index, rosdistro, Homebrew taps, vcpkg, and winget. Publishing = a PR (or a small GitHub Action). This delegates auth, review, hosting, CDN, and abuse handling to GitHub and the native registries. The index stores the manifest itself (not a pointer to it), so search, dora hub info, and pre-install schema validation never need to download artifacts.

The OCI choice is worth highlighting because it dissolves several sections of the proposal at once:

• Content-addressed digests give immutability and fix the publisher-controlled-hash problem
• OCI manifest lists natively express the per-target artifacts table
• The enterprise story (§4.3 private registry + OIDC, §4.4 pull-through mirror) is a shipped feature of Harbor / Artifactory / ECR / ACR, which robotics companies already run for containers
• The proposal's own syntax hub.my-company.com/lidar@1.0 is already an OCI reference, one character away from hub.my-company.com/lidar:1.0
• Precedent: Homebrew bottles, Helm charts, and WASM modules all migrated to exactly this

The honest cost of the hybrid: publish-side failure modes become per-backend (PyPI auth vs. crates.io rate limits vs. ghcr tokens), so CLI error handling must be good per backend. That is real work — but it is CLI work, not service operation, and it is the same tradeoff Homebrew and conda-forge accepted deliberately.

Suggested phasing (inverts the risk)

1. Manifest + validation (pure client work, no servers): standardize node.json with typed contracts, ship dora validate and descriptor-level schema checking. Valuable even for today's path:/git: nodes, and it forces the contract-model questions (see Feature —— Dora Hub Design #2097) to be answered first.
2. Git-backed index + CLI (search/install/info/publish over the index): full publish/discover/install loop with zero hosted services. Private registry = a private index repo; offline = clone the index.
3. Hosted layer only if scale demands it: search API, dashboard, download stats built on top of the same index data; the web UI starts as a static site generated from the index. Nothing from phases 1–2 is thrown away.

The proposal as written builds infrastructure first and defers the runtime-integration and contract semantics ("Runtime layer") that actually determine whether the ecosystem works. I'd do the semantics first and keep the infrastructure boring as long as possible.

Prior art worth adding

The cited registries (crates.io, npm, Docker Hub, PyPI) are good, but the closest analogs solved these exact problems in this exact domain: rosdep (per-distro system dependency keys — §8.3), rosdistro (a git-hosted YAML index as the entire registry), bloom, and conda-forge / RoboStack (community feedstock model = the proposed L2 tier, already built and operated). These are alternative architectures to study, not just references.

Happy to discuss any of this — and to help prototype the manifest + index approach if there's interest.

[heyong4725]

The implementation spec is now up for review: PR #2154 (docs/plan-node-hub.md), with #2097 restructured as the umbrella issue carrying the phase checklist.

The spec turns this discussion into an implementation contract: 11 use cases as acceptance tests, the manifest/index formats, the dora hub CLI surface, descriptor + multi-daemon integration grounded in the existing codebase (type-URN contracts, build lockfile, git-source machinery, managed uv envs), a security model (machine-enforced append-only index, namespace binding, full closure locking), the node-hub migration plan, and a 4-phase roadmap of small PRs.

Appendix A maps every decision from the original proposal to what was adopted (most of it) or changed (with reasons). Review window: 14 days — feedback on the PR or here. All roadmap items are seeking owners; @LeonRust, the index bootstrap (P2.3), category curation, and the Phase-4 static site (your §XIII web platform) have your name on them first if you want them.

[heyong4725]

Implementation update — the git-backed v1 is feature-complete

Following this proposal, the first version of Dora Hub shipped as a git-backed, cargo-style catalog (the simpler path — the hosted Registry API + Web Platform proposed here map to a future Phase 4). It is now done end-to-end and the catalog is populated.

Landed (Phases 1 → 3.4):

• Node manifest (dora-node.yml) + contract injection + dora hub init
• dora-hub-client + hub: resolution → pinned git/binary node, lockfiles (v2/v3), discovery, --hub-override, P2.8 binary source
• dora hub publish / yank / outdated / update
• node-hub migration (62 manifests) + catalog bootstrap + seed
• Weekly reachability + integrity audits; §7.4 namespace-identity binding; best-effort validate

State:

• Catalog: all 62 official dora-rs nodes published under node-index/; OWNERS = the dora-rs core team
• Index CI + auto-merge bot live (dora-index-ci: validate / append-only / namespace / decide / identity + the two periodic audits, and index-auto-merge.yml)
• Docs: a 6-chapter Hub guide (Overview → Using Nodes → Manifest → Reproducible Builds → Publishing → Indexes), both repos' READMEs, and the marketing site

Full PR-by-PR breakdown in the umbrella issue: #2097.

Remaining:

1. Activation — two repo settings on dora-rs/dora-hub (Allow auto-merge + branch protection requiring index-checks), per node-index/BOOTSTRAP.md. After that, a conformant owner publish auto-merges.
2. Phase 4 — the hosted Registry API / Web Platform / PostgreSQL + S3 / search + download stats from this proposal is not built; v1 used the git-backed catalog instead.

Thanks @LeonRust for the proposal that kicked this off. The two open decisions (go-live timing + whether to pursue the hosted platform as Phase 4) are laid out in #2097 — happy to scope Phase 4 from this proposal if we want to pursue it.