feat: add pricing registry and cost metrics (#464)#882
Open
ajbozarth wants to merge 2 commits intogenerative-computing:mainfrom
Open
feat: add pricing registry and cost metrics (#464)#882ajbozarth wants to merge 2 commits intogenerative-computing:mainfrom
ajbozarth wants to merge 2 commits intogenerative-computing:mainfrom
Conversation
Contributor
|
The PR description has been updated. Please fill out the template for your PR to be reviewed. |
Adds a PricingRegistry backed by builtin_pricing.json for OpenAI and Anthropic models, a mellea.llm.cost.usd counter via record_cost(), and a CostMetricsPlugin that records estimated cost on each generation call. Custom pricing overrides are supported via MELLEA_PRICING_FILE. Assisted-by: Claude Code Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
ajbozarth
force-pushed
the
feat/pricing-464
branch
from
b72abc8 to
3dcc39c
Compare
Contributor
Author
|
This is parallel work to #883 which adds the other remaining metrics telemetry and whichever merges second will need a manual merge to deal with the overlapping git diffs. |
This was referenced Apr 17, 2026
psschwei
reviewed
Apr 20, 2026
- Use lazy _get_registry() init to avoid eager module-level load - Warn once per unknown model using _warned_models set - Use .get() with defaults to avoid KeyError on malformed pricing JSON - Fix CostMetricsPlugin docstring (cost is not skipped silently) - Switch pricing format from per-1k to per-1m tokens to match provider docs - Update tests and docs to reflect new per-1m format Assisted-by: Claude Code Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com>
Contributor
Author
|
@psschwei I've addressed all you comments with either a response or a fix in the last commit |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Misc PR
Type of PR
Description
Adds estimated LLM request cost tracking as part of the telemetry epic (#443).
mellea/telemetry/builtin_pricing.json— bundled pricing data for current Anthropic and OpenAI models, loaded viaimportlib.resourcesso it works correctly in installed packages and wheelsmellea/telemetry/pricing.py—PricingRegistryclass that merges built-in prices with optional custom overrides fromMELLEA_PRICING_FILE; module-levelcompute_cost()convenience functionmellea/telemetry/metrics.py— newmellea.llm.cost.usdcounter andrecord_cost()following the existing lazy-init patternmellea/telemetry/metrics_plugins.py— newCostMetricsPlugin(priority 53) that hooksgeneration_post_calland records cost when pricing data is availabledocs/docs/evaluation-and-observability/metrics.md— new Cost metrics section covering the instrument, attributes, built-in pricing, and custom pricing file formatTesting
Attribution