Skip to content

Add 0083 token-budget WARNING surfaces#237

Merged
chris-colinsky merged 1 commit into
mainfrom
feature/0083-token-budget-warning-surfaces
Jul 25, 2026
Merged

Add 0083 token-budget WARNING surfaces#237
chris-colinsky merged 1 commit into
mainfrom
feature/0083-token-budget-warning-surfaces

Conversation

@chris-colinsky

Copy link
Copy Markdown
Member

Second of three PRs for proposal 0083 (per-prompt token-budget observability). Adds the two vendor-neutral over-budget WARNING surfaces on top of PR A's completion-path signal. The structured-output-failure path (fixture 131) and the implemented flip follow in PR C.

What it adds

  • Langfuse (§8.4.3): on a budget exceedance the successful Generation ends at level = "WARNING" with a statusMessage naming each breached bound (e.g. token budget exceeded: input 20 > 10), and metadata.token_budget carries the declared bounds. A hard ERROR still takes precedence on the failure path (the advisory WARNING only applies to a successful call).
  • §7 log: one WARNING record per over-budget attempt on the openarmature.observability logger, naming the prompt and the breached bound(s). It is its own surface, dispatched independently of the span (disable_llm_spans) and metric (enable_metrics) gates. The record correlates via correlation_id; it does not carry OTel trace_id/span_id, since OA's observer never activates its spans and the record is emitted outside any span context.
  • Shared helper: the per-bound evaluation used by the span, metrics, Langfuse WARNING, and §7 log is lifted to observability/llm_event.py so both observers share one implementation.

Conformance

Un-defers fixture 130 (the Langfuse WARNING). The §7 log is prose-pinned (no fixture) and covered by unit tests. conformance.toml stays not-yet until PR C lands the failure path.

Tests

Fixture 130 plus unit coverage for: the Langfuse WARNING (over/under budget, both-bounds statusMessage, metadata.token_budget on success and on a failed Generation) and the §7 log (prompt identity, multi-bound formatting, per-attempt cardinality, the structured-output-failure and no-usage cases, and independence from both the span and metric gates). Full suite green; ruff and pyright clean.

Add the two vendor-neutral over-budget WARNING surfaces from
proposal 0083, on top of the completion-path signal.

- observability: lift the shared _token_budget_evaluations helper
  to llm_event.py for both observers.
- langfuse: on a budget exceedance the success Generation ends at
  level WARNING with a statusMessage naming the breached bound(s),
  and metadata.token_budget carries the declared bounds. A hard
  ERROR still takes precedence on the failure path.
- otel: emit one section 7 WARNING log per over-budget attempt on
  the openarmature.observability logger, dispatched independently
  of the span (disable_llm_spans) and metric (enable_metrics)
  gates. The record correlates via correlation_id (no active span
  at emit time, so no trace_id/span_id).
- conformance: un-defer fixture 130 (Langfuse WARNING). The
  structured-output-failure path (131) and the implemented flip
  follow in PR C.
Copilot AI review requested due to automatic review settings July 25, 2026 00:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends OpenArmature’s proposal 0083 token-budget observability by adding two vendor-neutral “over budget” WARNING surfaces (Langfuse Generation level/statusMessage and a spec §7 warning log), and centralizes shared per-bound evaluation logic so observers cannot drift.

Changes:

  • Add §7 token-budget WARNING logging on openarmature.observability, emitted once per over-budget attempt and independent of span/metric gates.
  • Add Langfuse advisory WARNING mapping for successful completions that exceed declared token budget bounds, including statusMessage formatting and metadata.token_budget.
  • Move the shared token-budget per-bound evaluation helper into observability/llm_event.py and enable conformance fixture 130.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/unit/test_observability_otel.py Adds unit coverage for the §7 warning log surface (cardinality, formatting, gating independence, prompt identity, failure/no-usage cases).
tests/unit/test_observability_langfuse.py Adds unit coverage for Langfuse WARNING level/statusMessage and token budget metadata behavior on success and failure.
tests/conformance/test_observability_langfuse.py Wires token_budget fixture data into the conformance harness prompt objects and enables fixture 130.
tests/conformance/test_fixture_parsing.py Removes the “not implemented” deferral for fixture 130 while keeping 131 deferred.
src/openarmature/observability/otel/observer.py Implements the §7 warning log surface and switches to the shared token-budget evaluation helper.
src/openarmature/observability/llm_event.py Introduces the shared _token_budget_evaluations helper for consistent per-bound evaluation across observers.
src/openarmature/observability/langfuse/observer.py Implements Langfuse Generation WARNING level/statusMessage on budget breach and maps metadata.token_budget.
conformance.toml Updates proposal 0083 notes to reflect fixture 130 now running while 131 remains deferred.

@chris-colinsky
chris-colinsky merged commit 39fb843 into main Jul 25, 2026
7 checks passed
@chris-colinsky
chris-colinsky deleted the feature/0083-token-budget-warning-surfaces branch July 25, 2026 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants