Initial xArm6 WorldBelief perception stack - #2665
Conversation
Greptile SummaryThis PR adds a WorldBelief perception stack for xArm6 object identity. The main changes are:
Confidence Score: 5/5This looks safe to merge.
Important Files Changed
Reviews (10): Last reviewed commit: "Merge branch 'main' into jhengyi/percept..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Replace the live-YOLO WorldBelief pipeline with query-on-demand perception. scan folds the COMPLETE recorded interval since the last fold into a rule-based identity core with measured constants. DINO crop galleries carry identity across sessions; recall searches whole-frame CLIP over a shared history index and detector-verifies each proposed moment before answering (CLIP proposes, the detector picks).
xArm6 WorldBelief — pulled perception with cross-session memoryPivot to pulled perception: nothing runs until asked; when asked, the answer covers the complete recorded interval since the last question — no observation holes. flowchart LR
CAM["camera just records<br/>color + depth + pose"] --> REC[("session<br/>recording")]
REC -- "scan (on demand)" --> SS["SceneScanner<br/>frames → 3D sightings"]
SS --> WB["WorldBelief<br/>sightings → stable identities"]
WB --> OUT["reply: name · id · xyz<br/>· trust · basis"]
WB <--> MEM[("long-term memory<br/>survives restarts")]
MEM -- "recall (on demand)" --> RC[""where did I last see X?""]
How a scan actually worksThe camera only records. RealSense color + depth frames and the camera's world pose stream into the session's recording Frame selection — fast-forward while nothing moves, slow down when something does. The recorded camera poses set the pace: ~1 frame/s while still, 7.5/s while moving. And if the set of detected objects changes between two kept frames (a hand swapped something while the camera was still), the scanner bisects the gap and looks between them. Net cost: 11–18 % of frames ever see the detector, with no blind spots. The worldbelief identity coreThe detector has no memory — it never says "that same mug". flowchart TB
S["detection arrives:<br/>label + 3D center + DINO embedding"] --> Q1{"Q1 · inside an active track's<br/>match radius?<br/>min(0.18 + 0.15·dt, 0.40) m"}
Q1 -- "yes — label agrees (or cos ≥ 0.8 override),<br/>no appearance contradiction (cos ≥ 0.5)" --> T1["update that track<br/><i>basis: tracked</i>"]
Q1 -- no --> Q2{"Q2 · inside an established track's<br/>0.13 m noise envelope?"}
Q2 -- yes --> T2["claimed by that track —<br/>mint suppressed (envelope default)"]
Q2 -- no --> Q3{"Q3 · gallery match to an absent or<br/>stored identity?<br/>cos ≥ 0.5, margin ≥ 0.05, same label"}
Q3 -- yes --> T3["identity revived<br/><i>basis: reacquired / restored</i>"]
Q3 -- no --> Q4["Q4 · mint a new track —<br/>established after 3 frames ∧ 1.5 s"]
P["established track goes unseen"] -- "depth ray passes THROUGH<br/>its center — 2 votes" --> GONE["state: absent<br/>(gallery kept)"]
GONE -. "revivable via Q3" .-> Q3
Q1 — Is it inside an active track's match radius? Every track predicts the object near its last observed position. The radius starts at 0.18 m and grows with time-unobserved ( Q2 — Is it inside an established track's 0.13 m noise envelope? Then that track claims it, by default. Two rigid objects can't co-locate: a detection landing within the envelope of an established object is that object unless stronger evidence says otherwise, and minting there is suppressed (the mint-guard). This envelope-as-null-hypothesis rule kills most identity-theft bugs. Q3 — Does its embedding match an absent or stored identity's gallery? No active track claims it — so compare the detection's DINO embedding against the galleries of every absent identity, from this session or any earlier one. The best match must cos ≥ 0.5, beat by a margin ≥ 0.05 (a near-tie between two candidates means the evidence can't tell them apart — refuse to guess), carry the same label, and resemble the absent identity more than any co-located neighbour. Success revives the identity: Q4 — Mint a new track. Only when Q1–Q3 all fail is a new Absence is evidence-only. A track is never dropped for being unseen — it goes Galleries are the long-term memory. Each established sighting adds sufficiently-novel views to the track's gallery in
Where each piece lives in the code:
How to Testdimos --xarm6-ip <ROBOT_IP> run xarm6-worldbelief
# in a second terminal:
dimos mcp call scan -a prompt='["mug","coke can"]'
dimos mcp call recall -a text="mug"Move an object by hand between scans → same id, |
|
@jhengyilin @mustafab0 (@leshy pls audit) I think it's fine to make world-belief a dedicated module as it makes lifecycle and api boundary clean, concurrent reads with single write is not a problem for sqlite as read operates on the snapshot. My only concern about having to specify db path for mutliple modules can be relieved in the future by providing some pre-launch config injection (I'm thinking of using some timestamp based default location and make sure two modules sync on this path) |
Address review feedback. WorldBeliefRecorder is now a dumb writer; the new WorldBeliefModule owns the belief, the scan/CLIP models, and the scan/recall skills. SceneScanner is stateless: callers pass the belief in. Hand-built matrices replaced with Transform/Pose/ PointStamped; tuning constants live in WorldBeliefConfig; _Track is a grouped slots dataclass. Gallery lookup now selects the newest view by append time (the per-track counter resets on restore).
xArm6 WorldBelief — pulled perception with cross-session memoryPivot to pulled perception flowchart TB
CAM["camera<br/>color + depth + pose"] --> REC["WorldBeliefRecorder<br/>"]
REC -- writes --> RDB[("session recording<br/>per-session .db")]
RDB -- "scan / recall<br/>(on demand)" --> WB["WorldBeliefModule<br/>@scan · @recall<br/>SceneScanner + WorldBelief"]
MEM[("long-term memory<br/>worldbelief_history.db<br/>survives restarts")] <--> WB
WB --> OUT["reply: name · id · xyz<br/>· trust · basis"]
WB -- "objects · detections_3d · pointcloud" --> CONS["PickAndPlace / Rerun"]
Two modulesThe stack is split as 2:
How a scan actually works — follow one callThe camera only records. RealSense color + depth frames and the camera's world pose stream into the session's recording Frame selection — fast-forward while nothing moves, slow down when something does. The recorded camera poses set the pace: ~1 frame/s while still, 7.5/s while moving. And if the set of detected objects changes between two kept frames (a hand swapped something while the camera was still), the scanner bisects the gap and looks between them. Net cost: 11–18 % of frames ever see the detector, with no blind spots. Ownership. The worldbelief identity coreThe detector has no memory — it never says "that same mug". flowchart TB
S["detection arrives:<br/>label + 3D center + DINO embedding"] --> Q1{"Q1 · inside an active track's<br/>match radius?<br/>min(0.18 + 0.15·dt, 0.40) m"}
Q1 -- "yes — label agrees (or cos ≥ 0.8 override),<br/>no appearance contradiction (cos ≥ 0.5)" --> T1["update that track<br/><i>basis: tracked</i>"]
Q1 -- no --> Q2{"Q2 · inside an established track's<br/>0.13 m noise envelope?"}
Q2 -- yes --> T2["claimed by that track —<br/>mint suppressed (envelope default)"]
Q2 -- no --> Q3{"Q3 · gallery match to an absent or<br/>stored identity?<br/>cos ≥ 0.5, margin ≥ 0.05, same label"}
Q3 -- yes --> T3["identity revived<br/><i>basis: reacquired / restored</i>"]
Q3 -- no --> Q4["Q4 · mint a new track —<br/>established after 3 frames ∧ 1.5 s"]
P["established track goes unseen"] -- "depth ray passes THROUGH<br/>its center — 2 votes" --> GONE["state: absent<br/>(gallery kept)"]
GONE -. "revivable via Q3" .-> Q3
Q1 — Is it inside an active track's match radius? Every track predicts the object near its last observed position. The radius starts at 0.18 m and grows with time-unobserved ( Q2 — Is it inside an established track's 0.13 m noise envelope? Then that track claims it, by default. Two rigid objects can't co-locate: a detection landing within the envelope of an established object is that object unless stronger evidence says otherwise, and minting there is suppressed (the mint-guard). This envelope-as-null-hypothesis rule kills most identity-theft bugs. Q3 — Does its embedding match an absent or stored identity's gallery? No active track claims it — so compare the detection's DINO embedding against the galleries of every absent identity, from this session or any earlier one. The best match must cos ≥ 0.5, beat by a margin ≥ 0.05 (a near-tie between two candidates means the evidence can't tell them apart — refuse to guess), carry the same label, and resemble the absent identity more than any co-located neighbour. Success revives the identity: Q4 — Mint a new track. Only when Q1–Q3 all fail is a new Absence is evidence-only. A track is never dropped for being unseen — it goes Galleries are the long-term memory. Each established sighting adds sufficiently-novel views to the track's gallery in
Where each piece lives in the code:
How to Testdimos --xarm6-ip <ROBOT_IP> run xarm6-worldbelief
# in a second terminal:
dimos mcp call scan -a prompt='["mug","coke can"]'
dimos mcp call recall -a text="mug"Move an object by hand between scans → same id, |
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #2665 +/- ##
==========================================
- Coverage 72.42% 71.82% -0.60%
==========================================
Files 1002 1009 +7
Lines 89281 90154 +873
Branches 8129 8248 +119
==========================================
+ Hits 64664 64756 +92
- Misses 22427 23286 +859
+ Partials 2190 2112 -78
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 39 files with indirect coverage changes 🚀 New features to boost your workflow:
|
TomCC7
left a comment
TomCC7
left a comment
There was a problem hiding this comment.
shape is good, left some comments 👍
- finish Recorder/query ownership and serialize shared model state - use global DINO re-ID with ambiguity abstention and cross-session galleries - publish 3D snapshots through the xArm Rerun validation blueprint - keep PickAndPlace, navigation, and shared infrastructure out of scope
WorldBelief — pulled perception with cross-session memoryThis supersedes the earlier spatial-radius/Q1–Q4 overview. The final implementation uses flowchart TB
CAM["RealSense<br/>color + depth + camera pose"] --> REC["WorldBeliefRecorder<br/>record plane"]
REC --> SESSION[("timestamped session DB<br/>raw evidence")]
SESSION -- "scan / recall<br/>on demand" --> WB["WorldBeliefModule<br/>query plane"]
HISTORY[("worldbelief_history.db<br/>DINO galleries + CLIP index")] <--> WB
WB --> REPLY["scan result<br/>name · full ID · xyz · trust · basis"]
WB --> OBJECTS["objects snapshot"]
WB --> VIS["detections_3d · pointcloud"]
CAM --> RERUN["Rerun validation"]
VIS --> RERUN
OBJECTS -. "future integration PR" .-> DOWNSTREAM["PickAndPlace / navigation"]
Two modules
The Recorder performs no detection. One
|
| Basis | Meaning |
|---|---|
new |
newly created candidate |
tracked |
matched in the next folded frame |
reacquired |
matched to an in-memory identity after a gap or absence |
restored |
matched to an identity persisted by an earlier process/session |
recall
recall(text) incrementally indexes immutable session frames with CLIP into the same history
database. CLIP ranks candidate moments across sessions; it is not trusted as localization.
Top moments are reopened from their source recordings and checked with a small YOLO-E + DINO
scan. The response includes the recording, timestamp, camera pose, uncalibrated CLIP similarity,
and detector-confirmed where_object. If no candidate is confirmed, where_object is None.
Configuration
Deployment policy stays in the xArm blueprint; the generic perception modules contain no xArm
paths or vocabulary. The main model and sampling choices are blueprint fields:
WorldBeliefModule.blueprint(
scan_prompts=[],
stationary_hz=4.0,
yoloe_model_name="yoloe-11l-seg.pt",
dino_model_name="facebook/dinov2-base",
clip_model_name="openai/clip-vit-base-patch32",
)Persistent gallery and CLIP streams are isolated by checkpoint ID, so changing DINO or CLIP
does not reuse incompatible embeddings.
How to test
# terminal 1
uv run dimos --xarm6-ip <ROBOT_IP> run xarm6-worldbelief
# terminal 2: an explicit prompt is required
uv run dimos mcp call scan -a prompt='["coke can", "mug", "yellow can"]'
uv run dimos mcp call recall -a text="mug"Expected checks:
- repeated scans keep a distinctive object's full ID;
- a moved or previously absent object reports
basis: reacquired; - after restart, a gallery match reports the same full ID with
basis: restored; - two novel camera viewpoints promote an established identity from
tentativetoconfirmed; - Rerun shows live RGB/depth plus published 3D detections and point cloud.
…ter worldbelief decision log
Head branch was pushed to by a user without write access
xArm6 WorldBelief — pulled perception with cross-session memoryThis supersedes the earlier spatial-radius/Q1–Q4 overview. The final implementation uses flowchart TB
CAM["RealSense<br/>color + depth + camera pose"] --> REC["WorldBeliefRecorder<br/>record plane"]
REC --> SESSION[("timestamped session DB<br/>raw evidence")]
SESSION -- "scan / recall<br/>on demand" --> WB["WorldBeliefModule<br/>query plane"]
HISTORY[("worldbelief_history.db<br/>DINO galleries + CLIP index")] <--> WB
WB --> REPLY["scan result<br/>established + unresolved observations"]
WB --> OBJECTS["established objects snapshot"]
WB --> VIS["established detections_3d · pointcloud"]
CAM --> RERUN["Rerun validation"]
VIS --> RERUN
OBJECTS -. "future integration PR" .-> DOWNSTREAM["PickAndPlace / navigation"]
Two modules
The Recorder performs no detection.
|
| Basis | Meaning |
|---|---|
new |
newly created candidate |
tracked |
matched in the next folded frame |
reacquired |
matched to an in-memory identity after a gap or absence |
restored |
matched to an identity persisted by an earlier process/session |
unresolved |
current physical observation without a defensible durable identity |
recall
recall(text) incrementally indexes immutable session frames with CLIP into the same history
database. Frames are processed chronologically and sampled into stable time buckets; the source
watermark advances only after an indexing top-up completes.
CLIP ranks candidate moments across sessions but is not trusted as localization. Distinct top
moments are reopened from their source recordings and checked with YOLO-E + DINO over a small
temporal window. The response includes the recording, timestamp, camera position, uncalibrated
CLIP similarity, and detector-confirmed where_object. If no candidate is confirmed, recall
still returns the best CLIP-ranked moment with where_object: None. If the index has no matching
moment, recall returns None.
Configuration
Deployment policy stays in the xArm blueprint; the generic perception modules contain no xArm
paths or vocabulary. The main model and sampling choices are blueprint fields:
WorldBeliefModule.blueprint(
scan_prompts=[],
stationary_hz=4.0,
yoloe_model_name="yoloe-11l-seg.pt",
dino_model_name="facebook/dinov2-base",
clip_model_name="openai/clip-vit-base-patch32",
)Persistent gallery and CLIP streams are isolated by checkpoint ID, so changing DINO or CLIP
does not reuse incompatible embeddings.
How to test
# terminal 1
uv run dimos --xarm6-ip <ROBOT_IP> run xarm6-worldbelief
# terminal 2: an explicit prompt is required
uv run dimos mcp call scan -a prompt='["coke can", "mug", "yellow can"]'
uv run dimos mcp call recall -a text="mug"Expected checks:
- repeated scans keep a distinctive object's full ID;
- unresolved observations remain visible in scan results but stay out of trusted publications;
- a moved or previously absent object reports
basis: reacquired; - after restart, a gallery match reports the same full ID with
basis: restored; - two novel camera viewpoints promote an established identity from
tentativetoconfirmed; - unconfirmed recall still returns the remembered moment with
where_object: None; - Rerun shows live RGB/depth plus published 3D detections and point cloud.
ce2c71a
* fix: better save path for nav recordings (dimensionalOS#2796) * feat(transport): rust zenoh support and api refactor (dimensionalOS#2753) * feat(arduino): add ArduinoModule with full arduino sim (and real arduino) support (dimensionalOS#1879) * Swastika/docs chore/cleanup and fix links (dimensionalOS#2832) * Velocity API (dimensionalOS#2859) * holonomic trajectory controller (dimensionalOS#2697) Co-authored-by: danvi <bogdan@dimensionalos.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: leshy <lesh@sysphere.org> * fix(config): merge nested module overrides (dimensionalOS#2829) * fix(mls-planner): macOS fix, link pyo3 cdylib (dimensionalOS#2836) * revert(voxels): drop unused time_window mode from VoxelGrid (dimensionalOS#2845) * Revert "feat(arduino): add ArduinoModule with full arduino sim (and r… (dimensionalOS#2868) * feat: external blueprint registration (dimensionalOS#2517) * Fix test_basic_deployment in CI (dimensionalOS#2767) * Run CI on merge queue (dimensionalOS#2936) * fix: disallow unused _ variables (dimensionalOS#2928) * Add Galaxea A1Z arm in simulation (dimensionalOS#2947) Co-authored-by: cc <cc7@duck.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: cc <55869557+TomCC7@users.noreply.github.com> * Initial xArm6 WorldBelief perception stack (dimensionalOS#2665) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat(g1-groot): navigation on the real G1 (Point-LIO + raytracing costmap) (dimensionalOS#2861) Co-authored-by: Jeff Hykin <jeff.hykin@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Swastika/docs/navigation relocalization (dimensionalOS#2764) Co-authored-by: Cursor <cursoragent@cursor.com> * docs(teleop): rename Hosted Teleop page to Remote Teleop (dimensionalOS#2967) * feat(hosted-teleop): Go2 teleoperation over Cloudflare broker (dimensionalOS#2798) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: spomichter <pomichterstash@gmail.com> Co-authored-by: Sam Bull <git@sambull.org> Co-authored-by: Paul Nechifor <paul@nechifor.net> * DimSim scene editing/authoring in JS + moving inside dimos (dimensionalOS#2187) Co-authored-by: Paul Nechifor <paul@nechifor.net> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(unitree): marshal stop_movement zero-twist onto the loop thread (dimensionalOS#2977) * chore: bump version to 0.0.14b1 (beta pre-release) (dimensionalOS#2971) * add latency graph and description to hosted teleop page (dimensionalOS#2982) * docs(hosted-teleop): operator how-to + config/architecture refresh (dimensionalOS#2945) Co-authored-by: stash <pomichterstash@gmail.com> * serve LFS media via dimos-docs-assets so it renders on mintlify (dimensionalOS#2973) * style(docs): fix trailing whitespace breaking lint on main (dimensionalOS#2991) * add dimTELE banner (dimensionalOS#2984) Co-authored-by: stash <pomichterstash@gmail.com> * feat: blueprint namespaces (dimensionalOS#2725) * fix: delete dead dimsim code (dimensionalOS#2980) Co-authored-by: leshy <lesh@sysphere.org> * fix: run all tests locally (dimensionalOS#2995) * add gitHub community files for contributors and AI-assisted development (dimensionalOS#2347) * fix: large file (dimensionalOS#3003) * Fix/ivan/3d planner voxel colors (dimensionalOS#2983) * feat: ban `__new__` (dimensionalOS#3009) * fix: remove removed cibuildwheel enable group breaking release builds (dimensionalOS#3007) * [Backport release/0.0.14b1] fix: remove removed cibuildwheel enable group breaking release builds (dimensionalOS#3013) Co-authored-by: stash <pomichterstash@gmail.com> * build(deps): bump the actions group across 1 directory with 6 updates (dimensionalOS#3056) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(manipulation): stabilize agentic xArm simulation (dimensionalOS#2999) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Paul Nechifor <paul@nechifor.net> * fewer fields (dimensionalOS#3052) * Build all GC Roots into Cachix (dimensionalOS#2866) * Control coordinator refactor to add declarative task (dimensionalOS#2959) Co-authored-by: Paul Nechifor <paul@nechifor.net> Co-authored-by: leshy <lesh@sysphere.org> * feat(hosted-teleop): Arm hosted teleoperation over the Cloudflare broker (dimensionalOS#3004) * feat(control): removes frozen set of input streams - now Any Input stream can be routed to the control coordinator (dimensionalOS#3110) * chore: add OpenYAM URDF support (dimensionalOS#3049) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat: add Viser robot mesh display options (dimensionalOS#3112) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * add links to cards (dimensionalOS#3065) * [Tests] Add unit tests for SequentialIds (dimensionalOS#3060) * feat(web): PR 1 - Deno (dimensionalOS#3042) * Ivan/feat/go2zenoh (dimensionalOS#3141) * feat: Piper teleop polishing (dimensionalOS#3101) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat(hosted-teleop): auto-select operator view via robot_type broker config (dimensionalOS#3144) * chore: remove get_project_root (dimensionalOS#3131) * Add backport label to Dependabot PRs and check docker weekly (dimensionalOS#3067) * fix memory issues DIM-1326 (dimensionalOS#3149) Co-authored-by: danvi <bogdan@dimensionalos.com> * docs(drone): fix dead pre-built RosettaDrone APK link (dimensionalOS#2546) Co-authored-by: Paul Nechifor <paul@nechifor.net> * feat: manipulation planning group (dimensionalOS#2645) Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Go2 holonomic Pose tracker trajectory controller (dimensionalOS#2948) Co-authored-by: Ivan Nikolic <lesh@sysphere.org> * feat(web): PR 2 - Relay (dimensionalOS#3043) * ai disclose rule update (dimensionalOS#3028) * Replace httpx with aiohttp (dimensionalOS#2974) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Paul Nechifor <paul@nechifor.net> * fix(memory2): recorder ingress reception_ts + poseless streams (dimensionalOS#2927) * ci: smoke-build a wheel on PRs touching release build config (dimensionalOS#3008) * Fix typos and grammar (dimensionalOS#2726) Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: Paul Nechifor <paul@nechifor.net> * feat: visualize manipulation obstacles (dimensionalOS#3108) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat(web): PR 3 - Python client (dimensionalOS#3044) * feat(web): PR 4 - E2E (dimensionalOS#3045) * Swastika/chore/readme update (dimensionalOS#3027) * Persist dependencies on disk (dimensionalOS#3167) * ci: label first-time contributor PRs (dimensionalOS#3171) * Add @Dreamsorcerer as global codeowner (dimensionalOS#3165) * feat(m20): integrate official MuJoCo model * docs(m20): add official MuJoCo integration guide * docs(m20): add English guide and fidelity review * docs(m20): make English guide primary * docs(m20): document lateral and yaw limitation * docs(m20): add MuJoCo test matrix * docs(m20): consolidate test matrix into guides * docs(m20): clarify guide reading order * fix(memory2): preserve recorder ingress timing * fix(memory2): avoid undefined frame in pose warning * fix(config): merge nested module overrides (dimensionalOS#2829) (cherry picked from commit 66ce423) * style: format Zenoh service --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Andrew Lauer <69774903+aclauer@users.noreply.github.com> Co-authored-by: Jeff Hykin <jeff.hykin@gmail.com> Co-authored-by: Swastika <44317853+swstica@users.noreply.github.com> Co-authored-by: Dan Vi <bogwi@tutamail.com> Co-authored-by: danvi <bogdan@dimensionalos.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: leshy <lesh@sysphere.org> Co-authored-by: cc <55869557+TomCC7@users.noreply.github.com> Co-authored-by: Paul Nechifor <paul@nechifor.net> Co-authored-by: Sam Bull <git@sambull.org> Co-authored-by: Krishna_Hundekari <113392023+KrishnaH96@users.noreply.github.com> Co-authored-by: cc <cc7@duck.com> Co-authored-by: Jheng-Yi Lin <75511000+jhengyilin@users.noreply.github.com> Co-authored-by: Pim Van den Bosch <49974392+Nabla7@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: stash <pomichterstash@gmail.com> Co-authored-by: ruthwikdasyam <63036454+ruthwikdasyam@users.noreply.github.com> Co-authored-by: Viswajit <39920874+Viswa4599@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dimos-release-bot[bot] <4143859+dimos-release-bot[bot]@users.noreply.github.com> Co-authored-by: Mustafa Bhadsorawala <39084056+mustafab0@users.noreply.github.com> Co-authored-by: 起岚 <155608604+xiaoyaoqilan@users.noreply.github.com> Co-authored-by: Daniel Eneh <72442267+Danny024@users.noreply.github.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
* fix: better save path for nav recordings (dimensionalOS#2796) * feat(transport): rust zenoh support and api refactor (dimensionalOS#2753) * feat(arduino): add ArduinoModule with full arduino sim (and real arduino) support (dimensionalOS#1879) * Swastika/docs chore/cleanup and fix links (dimensionalOS#2832) * Velocity API (dimensionalOS#2859) * holonomic trajectory controller (dimensionalOS#2697) Co-authored-by: danvi <bogdan@dimensionalos.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: leshy <lesh@sysphere.org> * fix(config): merge nested module overrides (dimensionalOS#2829) * fix(mls-planner): macOS fix, link pyo3 cdylib (dimensionalOS#2836) * revert(voxels): drop unused time_window mode from VoxelGrid (dimensionalOS#2845) * Revert "feat(arduino): add ArduinoModule with full arduino sim (and r… (dimensionalOS#2868) * feat: external blueprint registration (dimensionalOS#2517) * Fix test_basic_deployment in CI (dimensionalOS#2767) * Run CI on merge queue (dimensionalOS#2936) * fix: disallow unused _ variables (dimensionalOS#2928) * Add Galaxea A1Z arm in simulation (dimensionalOS#2947) Co-authored-by: cc <cc7@duck.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: cc <55869557+TomCC7@users.noreply.github.com> * Initial xArm6 WorldBelief perception stack (dimensionalOS#2665) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat(g1-groot): navigation on the real G1 (Point-LIO + raytracing costmap) (dimensionalOS#2861) Co-authored-by: Jeff Hykin <jeff.hykin@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Swastika/docs/navigation relocalization (dimensionalOS#2764) Co-authored-by: Cursor <cursoragent@cursor.com> * docs(teleop): rename Hosted Teleop page to Remote Teleop (dimensionalOS#2967) * feat(hosted-teleop): Go2 teleoperation over Cloudflare broker (dimensionalOS#2798) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: spomichter <pomichterstash@gmail.com> Co-authored-by: Sam Bull <git@sambull.org> Co-authored-by: Paul Nechifor <paul@nechifor.net> * DimSim scene editing/authoring in JS + moving inside dimos (dimensionalOS#2187) Co-authored-by: Paul Nechifor <paul@nechifor.net> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(unitree): marshal stop_movement zero-twist onto the loop thread (dimensionalOS#2977) * chore: bump version to 0.0.14b1 (beta pre-release) (dimensionalOS#2971) * add latency graph and description to hosted teleop page (dimensionalOS#2982) * docs(hosted-teleop): operator how-to + config/architecture refresh (dimensionalOS#2945) Co-authored-by: stash <pomichterstash@gmail.com> * serve LFS media via dimos-docs-assets so it renders on mintlify (dimensionalOS#2973) * style(docs): fix trailing whitespace breaking lint on main (dimensionalOS#2991) * add dimTELE banner (dimensionalOS#2984) Co-authored-by: stash <pomichterstash@gmail.com> * feat: blueprint namespaces (dimensionalOS#2725) * fix: delete dead dimsim code (dimensionalOS#2980) Co-authored-by: leshy <lesh@sysphere.org> * fix: run all tests locally (dimensionalOS#2995) * add gitHub community files for contributors and AI-assisted development (dimensionalOS#2347) * fix: large file (dimensionalOS#3003) * Fix/ivan/3d planner voxel colors (dimensionalOS#2983) * feat: ban `__new__` (dimensionalOS#3009) * fix: remove removed cibuildwheel enable group breaking release builds (dimensionalOS#3007) * [Backport release/0.0.14b1] fix: remove removed cibuildwheel enable group breaking release builds (dimensionalOS#3013) Co-authored-by: stash <pomichterstash@gmail.com> * build(deps): bump the actions group across 1 directory with 6 updates (dimensionalOS#3056) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(manipulation): stabilize agentic xArm simulation (dimensionalOS#2999) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Paul Nechifor <paul@nechifor.net> * fewer fields (dimensionalOS#3052) * Build all GC Roots into Cachix (dimensionalOS#2866) * Control coordinator refactor to add declarative task (dimensionalOS#2959) Co-authored-by: Paul Nechifor <paul@nechifor.net> Co-authored-by: leshy <lesh@sysphere.org> * feat(hosted-teleop): Arm hosted teleoperation over the Cloudflare broker (dimensionalOS#3004) * feat(control): removes frozen set of input streams - now Any Input stream can be routed to the control coordinator (dimensionalOS#3110) * chore: add OpenYAM URDF support (dimensionalOS#3049) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat: add Viser robot mesh display options (dimensionalOS#3112) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * add links to cards (dimensionalOS#3065) * [Tests] Add unit tests for SequentialIds (dimensionalOS#3060) * feat(web): PR 1 - Deno (dimensionalOS#3042) * Ivan/feat/go2zenoh (dimensionalOS#3141) * feat: Piper teleop polishing (dimensionalOS#3101) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat(hosted-teleop): auto-select operator view via robot_type broker config (dimensionalOS#3144) * chore: remove get_project_root (dimensionalOS#3131) * Add backport label to Dependabot PRs and check docker weekly (dimensionalOS#3067) * fix memory issues DIM-1326 (dimensionalOS#3149) Co-authored-by: danvi <bogdan@dimensionalos.com> * docs(drone): fix dead pre-built RosettaDrone APK link (dimensionalOS#2546) Co-authored-by: Paul Nechifor <paul@nechifor.net> * feat: manipulation planning group (dimensionalOS#2645) Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * Go2 holonomic Pose tracker trajectory controller (dimensionalOS#2948) Co-authored-by: Ivan Nikolic <lesh@sysphere.org> * feat(web): PR 2 - Relay (dimensionalOS#3043) * ai disclose rule update (dimensionalOS#3028) * Replace httpx with aiohttp (dimensionalOS#2974) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Paul Nechifor <paul@nechifor.net> * fix(memory2): recorder ingress reception_ts + poseless streams (dimensionalOS#2927) * ci: smoke-build a wheel on PRs touching release build config (dimensionalOS#3008) * Fix typos and grammar (dimensionalOS#2726) Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: Paul Nechifor <paul@nechifor.net> * feat: visualize manipulation obstacles (dimensionalOS#3108) Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> * feat(web): PR 3 - Python client (dimensionalOS#3044) * feat(web): PR 4 - E2E (dimensionalOS#3045) * Swastika/chore/readme update (dimensionalOS#3027) * Persist dependencies on disk (dimensionalOS#3167) * ci: label first-time contributor PRs (dimensionalOS#3171) * Add @Dreamsorcerer as global codeowner (dimensionalOS#3165) * feat: add RoboPlan multi-robot planning (dimensionalOS#3155) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Andrew Lauer <69774903+aclauer@users.noreply.github.com> Co-authored-by: Jeff Hykin <jeff.hykin@gmail.com> Co-authored-by: Swastika <44317853+swstica@users.noreply.github.com> Co-authored-by: Dan Vi <bogwi@tutamail.com> Co-authored-by: danvi <bogdan@dimensionalos.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: leshy <lesh@sysphere.org> Co-authored-by: cc <55869557+TomCC7@users.noreply.github.com> Co-authored-by: Paul Nechifor <paul@nechifor.net> Co-authored-by: Sam Bull <git@sambull.org> Co-authored-by: Krishna_Hundekari <113392023+KrishnaH96@users.noreply.github.com> Co-authored-by: cc <cc7@duck.com> Co-authored-by: Jheng-Yi Lin <75511000+jhengyilin@users.noreply.github.com> Co-authored-by: Pim Van den Bosch <49974392+Nabla7@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: stash <pomichterstash@gmail.com> Co-authored-by: ruthwikdasyam <63036454+ruthwikdasyam@users.noreply.github.com> Co-authored-by: Viswajit <39920874+Viswa4599@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dimos-release-bot[bot] <4143859+dimos-release-bot[bot]@users.noreply.github.com> Co-authored-by: Mustafa Bhadsorawala <39084056+mustafab0@users.noreply.github.com> Co-authored-by: 起岚 <155608604+xiaoyaoqilan@users.noreply.github.com> Co-authored-by: Daniel Eneh <72442267+Danny024@users.noreply.github.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com>
This reverts commit ce2c71a, except for dimos/models/embedding/dino.py, which is kept: DINOModel is a standalone EmbeddingModel whose only dependencies (dimos.models.base, dimos.models.embedding.base, dimos.msgs.sensor_msgs.Image) predate the reverted commit. Conflict in dimos/robot/all_blueprints.py resolved by keeping the wrist-camera entry added after the merge; only the two world-belief registry entries are removed.
3 participants
WorldBelief live object identity for xArm6 manipulation
flowchart TB camera([RealSense RGB-D camera]) detector["YOLO-E prompt detector<br/><i>prompt mode, tracker disabled in xArm blueprint</i>"] osr["ObjectSceneRegistration<br/><i>RGB-D objects + CLIP/DINO crop embeddings</i>"] wb["WorldBelief<br/><i>stable IDs, support windows, re-acquisition</i>"] rerun["Rerun<br/><i>annotated image + 3D workspace</i>"] manip["PickAndPlaceModule<br/><i>reads present objects</i>"] recorder["XArm6WorldBeliefRecorder<br/><i>records replay-critical streams</i>"] subgraph m2 ["memory2"] direction TB subgraph session ["per-run recording DB"] direction LR color[(color_image)] depth[(depth_image)] info[(camera_info)] det2d[(detections_2d)] det3d[(detections_3d)] audit[(worldbelief_audit)] end subgraph history ["worldbelief_history.db"] direction LR object_events[(object evidence)] semantic_vec[(semantic vectors)] visual_vec[(visual vectors)] end end camera --> detector detector --> osr camera --> osr osr -->|"frame objects"| wb osr -->|"annotated image + current-frame pointcloud"| rerun wb -->|"present objects"| manip wb -->|"detections_3d + audit"| rerun wb ==> object_events wb ==> semantic_vec wb ==> visual_vec object_events -. "rehydrate compact identity state on restart" .-> wb camera ==> recorder osr ==> recorder wb ==> recorder recorder ==> color recorder ==> depth recorder ==> info recorder ==> det2d recorder ==> det3d recorder ==> audit classDef stream fill:#fef3c7,stroke:#d97706,stroke-width:2px classDef module fill:#dbeafe,stroke:#2563eb,stroke-width:2px classDef memory fill:#dcfce7,stroke:#16a34a,stroke-width:2px classDef external fill:#f3f4f6,stroke:#6b7280,stroke-width:1px class color,depth,info,det2d,det3d,audit,object_events,semantic_vec,visual_vec stream class detector,osr,wb,rerun,manip,recorder module class m2,session,history memory class camera externalWhat this unlocks
This PR adds a live object identity layer for the xArm6 perception/manipulation stack. The detector still sees frame-local masks; WorldBelief turns those observations into stable workspace objects that pick/place and visualization can trust.
WorldBeliefassociates current 3D observations against maintained identity state using geometry, labels, CLIP, DINO, support windows, and re-acquisition policy.present_objectsto theobjectsport andDetection3DArray.How it works - walkthrough
Objectobservations.presentonly after enough recent support.present_objectsare published to theobjectsport anddetections_3d. Pick/place sees stable workspace objects rather than raw detector churn.Runtime object model
The PR deliberately separates four related concepts:
objects,detections_3d, pick/place, 3D boxes.This split is why the Rerun pointcloud can stay visually clean while manipulation still receives stable present objects.
Why this design
present_objectsanddetections_3dremain the manipulation-facing outputs.Main files
dimos/robot/manipulators/xarm/blueprints/worldbelief.py,dimos/robot/all_blueprints.py,dimos/manipulation/blueprints.pydimos/perception/object_scene_registration.pydimos/perception/detection/world_belief.py,identity_association.py,identity_features.pydimos/perception/detection/world_belief_history.pydimos/perception/detection/type/detection3d/object.pydimos/models/embedding/dino.py,clip.py,mobileclip.py,dimos/perception/detection/detectors/yoloe.pydimos/robot/manipulators/xarm/worldbelief_recorder.py,dimos/memory2/module.pyKnown limits / follow-ups