WIP: Map comprehension experiments

[leshy]

run foxglove-bridge

export ALIBABA_API_KEY=...
pytest -sv dimos/agents2/skills/interpret_map/eval/test_ivan_eval.py

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[greptile-apps]

Greptile Summary

This PR adds point detection capabilities to the VLM (Vision-Language Model) system, enabling VLMs to identify specific point locations in images rather than just bounding boxes.

Key changes:

• Added Detection2DPoint class for representing 2D point detections with circle visualization support
• Extended VlModel base class with query_points() method for point-based queries
• Added native point detection support to MoondreamVlModel using Moondream's point() API
• Extended ImageAnnotations and ImageDetections to support circle annotations
• Added test infrastructure for map comprehension experiments

Critical issue:

• Detection2DPoint is missing the required abstract method to_ros_detection2d(), which will cause TypeError at instantiation time

Confidence Score: 2/5

• PR contains a critical abstract method implementation error that will cause runtime failures when using Detection2DPoint
• The missing to_ros_detection2d() abstract method in Detection2DPoint will cause a TypeError when attempting to instantiate the class. This is a blocking issue that needs to be fixed before merging.
• dimos/perception/detection/type/detection2d/point.py - missing abstract method implementation; dimos/agents2/skills/interpret_map/eval/test_ivan_eval.py - undefined variable bug

Important Files Changed

Filename	Overview
dimos/perception/detection/type/detection2d/point.py	New Detection2DPoint class for 2D point detections. Missing required abstract method to_ros_detection2d() which will cause instantiation to fail at runtime.
dimos/agents2/skills/interpret_map/eval/test_ivan_eval.py	New test file for map comprehension experiments. Contains undefined variable target (should be state.target) and several unused imports.
dimos/models/vl/base.py	Added query_points() method to VlModel base class and vlm_point_to_detection2d_point() helper function for converting VLM point results to Detection2DPoint.
dimos/models/vl/moondream.py	Added query_points() method using Moondream's native point detection API. Converts normalized coordinates to pixel coordinates correctly.
dimos/perception/detection/type/detection2d/base.py	Made to_text_annotation() and to_points_annotation() non-abstract with empty defaults. Added to_circle_annotation() method for circle visualizations.

Sequence Diagram

sequenceDiagram
    participant User
    participant VlModel
    participant MoondreamVlModel
    participant Detection2DPoint
    participant ImageDetections2D
    participant ImageAnnotations

    User->>VlModel: query_points(image, query)
    VlModel->>VlModel: _prepare_image(image)
    VlModel->>VlModel: query_json(scaled_image, full_query)
    
    loop For each point tuple
        VlModel->>Detection2DPoint: vlm_point_to_detection2d_point()
        Detection2DPoint-->>VlModel: Detection2DPoint instance
        VlModel->>ImageDetections2D: append detection
    end
    
    VlModel-->>User: ImageDetections2D

    User->>ImageDetections2D: to_foxglove_annotations()
    ImageDetections2D->>Detection2DPoint: to_circle_annotation()
    ImageDetections2D->>Detection2DPoint: to_text_annotation()
    ImageDetections2D-->>User: ImageAnnotations

Loading

[greptile-apps]

_{11 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: Missing abstract method to_ros_detection2d() required by base class Detection2D. This will cause TypeError: Can't instantiate abstract class Detection2DPoint with abstract method to_ros_detection2d at runtime when creating instances.

See Detection2DBBox.to_ros_detection2d() at dimos/perception/detection/type/detection2d/bbox.py:395 for reference implementation.

[greptile-apps]

syntax: target is undefined. Should be state.target.

Suggested change

	pose.publish(target)
	pose.publish(state.target)

[greptile-apps]

Too many files changed for review.

[greptile-apps]

Too many files changed for review.

[greptile-apps]

Too many files changed for review.

[greptile-apps]

Too many files changed for review.

[greptile-apps]

Too many files changed for review.

[greptile-apps]

Too many files changed for review.