Skip to content

v0.1.9 - Complete Metrics Implementation

Choose a tag to compare

@AdityaSaroj AdityaSaroj released this 09 Feb 09:50
· 72 commits to main since this release
14e0c06

Summary

End‑to‑end metrics support for Traccia: standardized LLM + agent metrics across core SDK, integrations, and docs, with a central recorder, pricing awareness, and tests.

Highlights

Core metrics system

  • Introduced a unified metrics layer (metrics.metrics, metrics.recorder) to emit:
    • Token usage
    • Operation cost (USD)
    • Operation duration / latency
  • Added a global metrics recorder accessor so integrations can record metrics without tight coupling.

SDK & instrumentation

  • Updated core instrumentation (auto.py, config.py, instrumentation/openai.py, instrumentation/anthropic.py, instrumentation/requests.py) to:
    • Attach pricing‑aware cost metrics to each LLM call.
    • Record token usage and duration consistently under gen_ai.client.*.
  • Exposed metrics configuration and behavior at the top level (init.py) for easier imports and discoverability.

Framework integrations

  • LangChain: Extended callback handler to emit standardized metrics (tokens, cost, duration) for LangChain LLM calls.
  • CrewAI: Instrumented CrewAI integration to record LLM metrics via the central recorder.
  • OpenAI Agents SDK: Enhanced processor to:
    • Track span start times.
    • Record generation metrics (tokens, cost, duration) on span end using the new metrics layer.

CLI & configuration

  • Updated cli.py and config.py to:
    • Include metrics options in configuration handling.
    • Ensure traccia config init produces a config that is compatible with the new metrics system.

Testing & docs

  • Added tests/test_metrics.py with coverage for:
    • Metrics recorder behavior.
    • Emission of token, cost, and duration metrics under expected names/attributes.
  • Expanded README.md with:
    • An overview of built‑in metrics (LLM + agent).
    • How metrics relate to pricing and token usage.
    • Guidance on enabling metrics in typical setups.