class VLMAgent(AgentSpec, Module) for streamed VLM queries over Transport

[spomichter]

To test:
dimos --replay run unitree-go2-vlm-stream-test

[greptile-apps]

Greptile Summary

This PR implements a Vision-Language Model (VLM) agent with stream-based and RPC interfaces for processing image queries on the Unitree Go2 robot. The implementation adds a new VLMAgent class that subscribes to image streams and query messages, processes them through language models, and publishes responses.

Key changes:

• New VLMAgent class that extends AgentSpec with vision capabilities through color_image, query, and answer streams
• Supports multiple LLM providers (Ollama, HuggingFace, and others via LangChain)
• Implements both stream-based (_on_query) and RPC (query_image) interfaces for flexibility
• Added VlmStreamTester module to validate the agent with replayed images, testing both stream and RPC paths
• Extended base Agent and AgentSpec classes with query_image abstract method for consistency
• New blueprint vlm_stream_test registered for testing on Go2 replay data

Minor issue:

• Type annotation error in vlm_agent.py:123 where *msgs parameter is incorrectly annotated as list[...] instead of individual message types

Confidence Score: 4/5

• This PR is safe to merge with one minor type annotation fix needed.
• The implementation is well-structured with proper error handling, thread management, and clean separation of concerns. The VLMAgent correctly implements the AgentSpec interface and follows existing patterns in the codebase. The tester module includes appropriate timeout handling and logging. The only issue is a type annotation error that won't cause runtime problems but should be fixed for type checking correctness.
• dimos/agents/modules/vlm_agent.py needs the type annotation fix on line 123.

Important Files Changed

Filename	Overview
dimos/agents/modules/vlm_agent.py	New VLMAgent implementation with stream-based and RPC interfaces for vision-language queries. Type signature issue in append_history parameter.
dimos/agents/modules/vlm_stream_tester.py	Testing module for VLMAgent with stream and RPC query smoke tests. Clean implementation with proper thread management.
dimos/agents/spec.py	Added abstract query_image method to AgentSpec interface.

Sequence Diagram

sequenceDiagram
    participant Tester as VlmStreamTester
    participant VLM as VLMAgent
    participant LLM as Language Model
    participant ImageStream as Image Stream

    ImageStream->>VLM: color_image (subscribe)
    VLM->>VLM: _on_image(image)
    VLM->>VLM: Store latest_image
    
    ImageStream->>Tester: color_image (subscribe)
    Tester->>Tester: _on_image(image)
    Tester->>Tester: Update latest_image & timestamp
    
    Note over Tester: Worker thread runs queries
    
    Tester->>VLM: query.publish(HumanMessage)
    VLM->>VLM: _on_query(msg)
    VLM->>VLM: _extract_text(msg)
    VLM->>VLM: _invoke_image(image, query)
    VLM->>LLM: invoke(messages)
    LLM-->>VLM: AIMessage response
    VLM->>VLM: append_history(msg, response)
    VLM->>Tester: answer.publish(AIMessage)
    Tester->>Tester: _on_answer(msg)
    Tester->>Tester: Log answer
    
    Note over Tester: Also tests RPC path
    
    Tester->>VLM: RPC query_image(image, query)
    VLM->>VLM: _invoke_image(image, query)
    VLM->>LLM: invoke(messages)
    LLM-->>VLM: AIMessage response
    VLM-->>Tester: response.content
    Tester->>Tester: Log RPC answer

Loading

[greptile-apps]

_{7 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

syntax: Parameter type annotation incorrect - *msgs expects individual messages, not a list. The annotation should be *msgs: AIMessage | HumanMessage instead of *msgs: list[AIMessage | HumanMessage]

Suggested change

	def append_history(self, *msgs: list[AIMessage | HumanMessage]) -> None:
	def append_history(self, *msgs: AIMessage | HumanMessage) -> None:

[spomichter]

This can be deleted - not used

[spomichter]

This will also be deleted

[paul-nechifor]

If you want to query the latest image, why not publish to VLMAgent.query to avoid sending the latest image over RPC since VLMAgent already has the latest image?

[spomichter]

Good catch

[spomichter]

Ah sorry no this class was vibecoded - its showing an example of a one-off RPC call AND a stream query just as an example. Want to show how you can use VLMAgent both via a stream of humanMessage queries AND as a normal RPC call where you pass (query, image). FYI @ClaireBookworm

[paul-nechifor]

Oh, so VLMAgent isn't a replacement for Agent? In that case would we be running two agent loops?

[spomichter]

VLMAgent in this case is meant to be run one-off yeah. But i guess we could alos change VLMAgent to inherit from Agent instead of AgentSpec, so then i would run a agent loop / skill coordinator

[spomichter]

Yeah so we need to do this because agent runs its own agent loop but listens to /skill. We haven't solved multi-agent yet. So if VLMAgent also inherited from Agent it would get bogged down by tool responses in parallel.

[spomichter]

Soon we will fix this so each agent also has its own skill coordinator topics and then we can run 5 agents that don't clash