bug: we should fetch stream `stats` updates from RemoteConversation

[xingyaoww]

Currently, when we run uv run examples/02_remote_agent_server/01_convo_with_local_agent_server.py, we will see that the visualizer will print boxes showing cost $0.00 until the conversation.run returns.

The main reason is beacuse in openhands-sdk/openhands/sdk/conversation/impl/remote_conversation.py, we implement:

    @property
    def stats(self) -> ConversationStats:
        """Get conversation stats (fetched from remote)."""
        info = self._get_conversation_info()
        stats_data = info.get("stats", {})
        return ConversationStats.model_validate(stats_data)

And _get_conversation_info will always re-use the cached value instead of fetching the up-to-date value from remote.

This works for most state values (since they will stay unchanged), but not for "stats," since it will be updated on every agent action.

We should figure out a way to stream these "stats" changes as part of the conversation state update via WebSockets. We should also check if there's other fields in ConversationState that should be updated but we haven't do it

[xingyaoww]

@OpenHands please review the codebase and understand the reason for this bug. Do not modify code or create pull request, just analyze the reason and provide path to the corresponding files

[openhands-ai]

I'm on it! xingyaoww can track my progress at all-hands.dev

[openhands-ai]

Bug Analysis Summary for Issue #1087

I've completed a thorough analysis of the bug where the visualizer displays $0.00 cost during conversation.run() until completion, instead of showing real-time updates.

Root Cause

The bug has two interconnected issues:

1. Stats updates are not broadcasted during agent execution:

   • When LLM metrics are updated, the ConversationStats.usage_to_metrics dictionary is mutated directly (in register_llm())
   • This mutation doesn't trigger ConversationState.__setattr__() because the stats object itself is not reassigned
   • Therefore, no ConversationStateUpdateEvent is emitted when stats change during agent execution
   • The server only publishes full state updates at specific times (initialization, after run completes, after pause) but NOT during each agent step

2. RemoteState caching prevents fresh data reads:

   • Once RemoteState._cached_state is set, _get_conversation_info() always returns the cached value
   • The cache is only updated via ConversationStateUpdateEvent received through WebSocket
   • Since no stats update events are sent during execution, the cache contains stale data
   • The visualizer reads conversation_stats → state.stats → cached (stale) stats

Key Files Involved

Core Issue Files:

1. /workspace/project/software-agent-sdk/openhands-sdk/openhands/sdk/conversation/impl/remote_conversation.py

   • Lines 263-276: RemoteState._get_conversation_info() - implements caching mechanism
   • Lines 278-291: RemoteState.update_state_from_event() - handles state updates (works correctly)
   • Lines 382-386: RemoteState.stats property - reads from cached state

2. /workspace/project/software-agent-sdk/openhands-agent-server/openhands/agent_server/event_service.py

   • Lines 262-266: Sets up state change callback and publishes initial state
   • Lines 268-275: run() - only publishes state AFTER run completes
   • Lines 283-288: pause() - only publishes state after pause
   • Lines 343-356: _publish_state_update() - publishes full state snapshot
   • Missing: No periodic state updates during agent execution steps

3. /workspace/project/software-agent-sdk/openhands-sdk/openhands/sdk/conversation/conversation_stats.py

   • Lines 86-101: register_llm() - mutates usage_to_metrics dictionary directly without triggering state change events

4. /workspace/project/software-agent-sdk/openhands-sdk/openhands/sdk/conversation/state.py

   • Lines 220-261: ConversationState.__setattr__() - emits ConversationStateUpdateEvent on field changes
   • Lines 91-94: stats field definition
   • The problem: nested mutations don't trigger __setattr__

Visualizer Files (Stats Consumers):

5. /workspace/project/software-agent-sdk/openhands-sdk/openhands/sdk/conversation/visualizer/base.py

   • Lines 72-74: conversation_stats property reads from _state.stats

6. /workspace/project/software-agent-sdk/openhands-sdk/openhands/sdk/conversation/visualizer/default.py

   • Lines 270-282: _format_metrics_subtitle() displays cost/tokens from stats

Infrastructure (Already Exists):

7. /workspace/project/software-agent-sdk/openhands-sdk/openhands/sdk/event/conversation_state.py
   • Lines 18-96: ConversationStateUpdateEvent - supports individual field updates via key="stats"

Other Fields That Should Be Checked

According to the issue, we should verify other ConversationState fields that might need real-time updates:

• ✅ stats - Main issue, should update after each agent action
• ⚠️ execution_status - Should update during conversation state transitions (RUNNING, PAUSED, WAITING_FOR_CONFIRMATION, etc.) - may already work if set via state.execution_status = ...
• ⚠️ activated_knowledge_skills - Might change but likely rare
• ✓ Other fields (agent, workspace, persistence_dir, max_iterations, stuck_detection, confirmation_policy, secret_registry) are static configuration that don't change during execution

Test Coverage

The test file /workspace/project/software-agent-sdk/tests/sdk/conversation/test_remote_conversation_state_updates.py already includes:

• Lines 384-434: test_stats_update_via_state_event() - demonstrates the expected behavior where stats should be updated via ConversationStateUpdateEvent with key="stats"

This test shows the infrastructure is in place; it just needs to be utilized during agent execution.

---

Note: As requested, I have not modified any code or created a pull request. This is purely an analysis of the bug's root cause and the files involved.

_{View full conversation}

[xingyaoww]

@OpenHands review the analysis above and can you help me put together a fix for these

[openhands-ai]

I'm on it! xingyaoww can track my progress at all-hands.dev

[openhands-ai]

Summary

I've successfully fixed issue #1087 where RemoteConversation stats were not updating in real-time during agent execution. The fix has been implemented, tested, and submitted as PR #1142: #1142

✅ Checklist - All Requirements Met:

1. Root cause identified: Stats updates were mutating the usage_to_metrics dictionary directly without triggering state change notifications
2. Fix implemented: Added callback mechanism to detect and broadcast stats changes
3. Tests added: New tests verify stats changes trigger callbacks and state updates
4. All tests passing: Both new and existing tests pass successfully
5. Pre-commit checks passing: Code formatting, linting, and type checks all pass
6. PR created: Draft PR with comprehensive documentation

📝 Changes Made (106 lines):

Source Files:

• conversation_stats.py (+20 lines): Added _on_stats_change callback field, set_on_stats_change() method, and modified register_llm() to detect new registrations
• state.py (+22 lines): Connected stats callback in set_on_state_change() and added _notify_stats_change() method

Test Files:

• test_conversation_stats.py (+31 lines): Added test_stats_change_callback_triggered()
• test_state_change_callback.py (+33 lines): Added test_stats_change_triggers_callback()

🎯 How It Works:

1. When ConversationState.set_on_state_change() is called, it also sets up the stats callback
2. When register_llm() is called (during LLM usage), it detects new registrations and triggers the callback
3. The callback creates a ConversationStateUpdateEvent with key="stats" and broadcasts it
4. RemoteState receives the event via WebSocket and updates its cached state
5. The visualizer now shows real-time cost updates instead of $0.00

The fix is backward compatible, fully tested, and ready for review!

_{View full conversation}