Skip to content

feat(galaxy): add LLMCallEvent and CostThresholdExceededEvent to event bus#319

Open
nuthalapativarun wants to merge 3 commits intomicrosoft:mainfrom
nuthalapativarun:feat/cost-tracking-issue-1
Open

feat(galaxy): add LLMCallEvent and CostThresholdExceededEvent to event bus#319
nuthalapativarun wants to merge 3 commits intomicrosoft:mainfrom
nuthalapativarun:feat/cost-tracking-issue-1

Conversation

@nuthalapativarun
Copy link
Copy Markdown

Feature: LLM Cost & Token Usage Tracking

UFO already computes cost per LLM call in ufo/llm/base.py and returns it from every provider, but the value is discarded by callers. This PR series wires that existing data through the event bus, aggregates it in the metrics observer, exposes it via a FastAPI router, and visualises it in the Galaxy Web UI.

This is PR 1 of 2. PR 2 (full backend wiring + frontend) depends on this PR and will be linked once this is reviewed.


What this PR does

Adds the event type definitions that the rest of the feature depends on — no runtime behaviour changes.

galaxy/core/events.py

  • Adds LLM_CALL_COMPLETED and COST_THRESHOLD_EXCEEDED to the EventType enum
  • Adds LLMCallEvent(Event) dataclass with fields:
    • agent_type: str — which agent made the call (e.g. HOST_AGENT)
    • model: str — model name (e.g. gpt-4o, claude-3-5-sonnet-20241022)
    • prompt_tokens: int
    • completion_tokens: int
    • cost: float — estimated USD cost
    • duration_ms: float — wall-clock time of the API call
  • Adds CostThresholdExceededEvent(Event) dataclass with fields:
    • session_id: str
    • total_cost: float
    • threshold: float

ufo/trajectory/parser.py

  • Replaces bare except: with except json.JSONDecodeError: to avoid masking unexpected exceptions during trajectory evaluation file parsing.

Architecture

LLM Provider (openai / claude / gemini)
  └─ returns (response, cost, tokens)
       └─ llm_call.py: emits LLMCallEvent          ← wired in PR 2
            └─ EventBus
                 └─ SessionMetricsObserver          ← wired in PR 2
                      └─ MetricsService
                           └─ /api/metrics/*        ← added in PR 2
                                └─ CostDashboard    ← added in PR 2

Testing

Pure data structure addition — no logic changes, no new dependencies. Existing tests unaffected.

vyokky and others added 3 commits January 6, 2026 15:22
…t bus

Add LLM_CALL_COMPLETED and COST_THRESHOLD_EXCEEDED to EventType enum.
Add LLMCallEvent dataclass (agent_type, model, prompt/completion tokens,
cost, duration_ms) and CostThresholdExceededEvent dataclass (session_id,
total_cost, threshold) to support cost tracking pipeline.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants