aruco emit detection3D

[bogwi]

This PR builds upon #2242 and closes awaited #1654

Make marker detection emit vision_msgs/Detection3DArray so downstream detection consumers - filtering - debug overlays - tracking - spatial memory - and Rerun visualization can treat markers like other 3D detections.

Architecture

 LIVE (#1654)
  ==========
  Camera.color_image ──┐
  Camera.camera_info ──┼──► MarkerDetectionStreamModule.start()
  TF (world<-optical frame) ──┘  │  optical = camera_optical_frame_id(image, info)
                                 ▼
                      _append_image_with_pose
                      skip if: no CameraInfo | no TF
                      out: stream.append(Image, pose=7-tuple)  # no frame name in tuple
                                 │
                                 ▼
                      QualityWindow(img.sharpness, 0.5s)
                      -> sharpest Image / window
                                 │
                                 ▼
                      [optional] SpeedLimit(mps, dps)
                                 │
                                 ▼
                      DetectMarkers -> detect_markers_in_image
                      (camera_optical_frame_id + pose -> world_T_optical)
                      in:  Observation[Image] + pose + CameraInfo
                      out: 0..N Observation[Detection3DMarker]  (track_id=-1 unless smoothing)
                            OR sentinel None (empty frame)
                                 │
                                 ▼
                      MarkersPerFrame
                      in:  fan-out markers + tags
                      out: Observation[Detection3DArray]
                            (empty detections_length OK)
                                 │
                ┌────────────────┴────────────────┐
                ▼                                 ▼
      LCM Detection3DArray              MarkerTfModule
      (wire: bbox center/orient)        world->markers->marker_{id}
                │
                ▼
      rerun bridge: msg.to_rerun() -> rr.Boxes3D


  OFFLINE (PR #2242 + reused by live module)
  ==========================================
  SqliteStore.color_image (obs.pose from recording)
      -> QualityWindow -> SpeedLimit? -> DetectMarkers
      -> list[Observation[Detection3DMarker]]  (no MarkersPerFrame in map CLI)
      -> direct rr.log / PGO-corrected poses

[codecov]

Codecov Report

❌ Patch coverage is 93.35233% with 70 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/perception/fiducial/marker_pose.py	71.83%	16 Missing and 4 partials ⚠️
...ception/fiducial/marker_detection_stream_module.py	78.48%	17 Missing ⚠️
dimos/perception/fiducial/marker_transformer.py	87.27%	6 Missing and 8 partials ⚠️
dimos/perception/fiducial/marker_detect.py	80.48%	4 Missing and 4 partials ⚠️
dimos/msgs/vision_msgs/Detection3DArray.py	77.41%	5 Missing and 2 partials ⚠️
...os/perception/detection/type/detection3d/marker.py	86.66%	1 Missing and 1 partial ⚠️
dimos/perception/fiducial/marker_tf_module.py	93.10%	0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR refactors live ArUco/AprilTag detection to emit Detection3DArray messages, enabling downstream consumers (tracking, filtering, Rerun, TF) to treat fiducial markers like any other 3D detection. Marker detection logic is extracted from MarkerTfModule into dedicated modules (marker_pose.py, marker_detect.py, marker_transformer.py, marker_detection_stream_module.py), and MarkerTfModule is narrowed to TF-only publication from an incoming Detection3DArray port.

• MarkerDetectionStreamModule is a new StreamModule that gates images on CameraInfo + TF availability, quality-windows them, runs DetectMarkers, fans out per-marker observations through MarkersPerFrame, and publishes one Detection3DArray per source frame via LCMTransport.
• MarkersPerFrame collapses per-marker fan-out into a single Detection3DArray per source frame using a count-based fast-path flush; Detection3DArray.to_rerun() lets the Rerun bridge log oriented boxes directly from the wire type.
• MarkerTfModule is simplified to subscribe to detections: In[Detection3DArray] and mirror bbox.center poses into the TF tree, fully decoupled from camera and detection logic.

Confidence Score: 5/5

Safe to merge; the refactoring is well-structured and all call sites are consistently updated.

The core detection and streaming logic is correctly implemented. MarkersPerFrame flush semantics handle all expected frame orderings. The two findings are both speculative edge cases that do not affect the current deployed configurations.

No files require special attention; the two suggestions in marker_detection_stream_module.py and desk_marker_tf.py are minor hardening improvements.

Important Files Changed

Filename	Overview
dimos/perception/fiducial/marker_detection_stream_module.py	New live-pipeline StreamModule; gates on CameraInfo+TF, builds DetectMarkers+MarkersPerFrame pipeline. Minor: camera_info is snapshotted at pipeline creation rather than passed as a callable, so post-start config updates are silently ignored by DetectMarkers.
dimos/perception/fiducial/marker_transformer.py	DetectMarkers and MarkersPerFrame transformers; flush logic (count fast-path + timestamp fallback) is correct for sequential frame delivery.
dimos/perception/fiducial/marker_tf_module.py	Successfully narrowed to TF-only publication; subscribes to Detection3DArray and mirrors bbox.center poses. All detection/camera logic removed cleanly.
dimos/perception/fiducial/marker_detect.py	New stateless per-frame detection helper; correctly handles size-mismatch guard, fisheye/radtan branching, and builds Detection3DMarker list.
dimos/msgs/vision_msgs/Detection3DArray.py	Adds to_rerun() and _label_for_detection(); slices detections_length correctly and handles empty arrays safely.
dimos/perception/detection/type/detection3d/marker.py	Detection3DMarker dataclass; post_init sets name from marker_label; to_detection3d_msg correctly overrides class_id and id.
dimos/robot/unitree/go2/blueprints/smart/unitree_go2.py	unitree_go2_markers blueprint updated to wire MarkerDetectionStreamModule → MarkerTfModule with LCMTransport. All call sites updated correctly.
dimos/perception/fiducial/marker_pose.py	Extracted shared pose helpers; fisheye path undistorts before solvePnP correctly.
dimos/perception/detection/type/detection3d/imageDetections3D.py	New helper converting Detection3DBBox list to Detection3DArray ROS message; clean and straightforward.
dimos/perception/fiducial/blueprints/desk_marker_tf.py	Blueprint updated to include MarkerDetectionStreamModule. create_desk_camera_info() is called at module import time rather than lazily.

Sequence Diagram

sequenceDiagram
    participant Cam as CameraModule
    participant MDSM as MarkerDetectionStreamModule
    participant DM as DetectMarkers
    participant MPF as MarkersPerFrame
    participant LCM as LCMTransport
    participant MTF as MarkerTfModule
    participant RR as Rerun

    Cam->>MDSM: color_image (Image)
    MDSM->>MDSM: _append_image_with_pose (TF lookup, gate on CameraInfo)
    MDSM->>DM: Stream[Observation[Image]] with pose tuple
    DM->>DM: QualityWindow / SpeedLimit
    DM->>DM: detect_markers_in_image
    DM-->>MPF: Observation[Detection3DMarker] x N
    DM-->>MPF: None sentinel (empty frame)
    MPF->>MPF: flush on count or ts-change
    MPF->>LCM: Detection3DArray (one per frame)
    LCM-->>MTF: Detection3DArray (in-process)
    LCM-->>RR: Detection3DArray (via LCM wire)
    MTF->>MTF: "_process_detections -> tf.publish"
    RR->>RR: "msg.to_rerun() -> rr.Boxes3D"

Loading

_{Reviews (9): Last reviewed commit: "fix test pydantic None" | Re-trigger Greptile}

[leshy]

super mega clean, thanks