feat(galaxy): add LLMCallEvent and CostThresholdExceededEvent to event bus#319
Open
nuthalapativarun wants to merge 3 commits intomicrosoft:mainfrom
Open
feat(galaxy): add LLMCallEvent and CostThresholdExceededEvent to event bus#319nuthalapativarun wants to merge 3 commits intomicrosoft:mainfrom
nuthalapativarun wants to merge 3 commits intomicrosoft:mainfrom
Conversation
merge back to prelease
…t bus Add LLM_CALL_COMPLETED and COST_THRESHOLD_EXCEEDED to EventType enum. Add LLMCallEvent dataclass (agent_type, model, prompt/completion tokens, cost, duration_ms) and CostThresholdExceededEvent dataclass (session_id, total_cost, threshold) to support cost tracking pipeline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature: LLM Cost & Token Usage Tracking
UFO already computes cost per LLM call in
ufo/llm/base.pyand returns it from every provider, but the value is discarded by callers. This PR series wires that existing data through the event bus, aggregates it in the metrics observer, exposes it via a FastAPI router, and visualises it in the Galaxy Web UI.This is PR 1 of 2. PR 2 (full backend wiring + frontend) depends on this PR and will be linked once this is reviewed.
What this PR does
Adds the event type definitions that the rest of the feature depends on — no runtime behaviour changes.
galaxy/core/events.pyLLM_CALL_COMPLETEDandCOST_THRESHOLD_EXCEEDEDto theEventTypeenumLLMCallEvent(Event)dataclass with fields:agent_type: str— which agent made the call (e.g.HOST_AGENT)model: str— model name (e.g.gpt-4o,claude-3-5-sonnet-20241022)prompt_tokens: intcompletion_tokens: intcost: float— estimated USD costduration_ms: float— wall-clock time of the API callCostThresholdExceededEvent(Event)dataclass with fields:session_id: strtotal_cost: floatthreshold: floatufo/trajectory/parser.pyexcept:withexcept json.JSONDecodeError:to avoid masking unexpected exceptions during trajectory evaluation file parsing.Architecture
Testing
Pure data structure addition — no logic changes, no new dependencies. Existing tests unaffected.