Skip to content

v0.9.2

Choose a tag to compare

@coder-eval-release-bot coder-eval-release-bot released this 31 Jul 21:54
· 17 commits to main since this release

Bug Fixes

  • cost: A task timeout with no preserved turn is unrecorded spend, not free
    (#63,
    93c7fc0)

  • cost: Book spend on the error and timeout paths, flag what is unpriced
    (#63,
    93c7fc0)

  • cost: Flag every hard-killed task as a cost floor, not just the empty ones
    (#63,
    93c7fc0)

  • evalboard: Honest scoped counts, and one definition of a run's scope
    (#69,
    0bdac0b)

  • litellm: Gate cost_log_tags on agent capability, not route (fixes non-Claude crash)
    (#66,
    4131a2a)

  • litellm: Make the orphaned-spend warning actually fire
    (#66,
    4131a2a)

  • litellm: Per-attempt cost-log scoping + single run-id accessor + no-match warning
    (#66,
    4131a2a)

  • litellm: Pin each open-weight model to a vetted provider set (no silent fallback)
    (#66,
    4131a2a)

  • litellm: Proxy-authoritative token buckets + all-priced gate + transactional join
    (#66,
    4131a2a)

  • litellm: Sanitize cost headers, reject non-finite cost, drop debug scaffolding
    (#66,
    4131a2a)

  • orchestrator: Recover the in-flight turn's spend on a hard kill
    (#63,
    93c7fc0)

  • pricing: Add the claude-opus-5 rate so killed turns stop booking zero
    (#63,
    93c7fc0)

  • pricing: Add the five unpriced codex tiers still on OpenAI's rate card
    (#63,
    93c7fc0)

  • pricing: Correct every wrong rate-card entry and close the alias gaps
    (#63,
    93c7fc0)

  • pricing: Refresh the rate card and correct gemini-3-flash-preview
    (#63,
    93c7fc0)

  • reports: Count errors as misses and stop losing cost on error paths
    (#63,
    93c7fc0)

  • reports: Count errors as misses in one canonical pass rate
    (#63,
    93c7fc0)

Code Style

  • evalboard: Drop the swatch dots and the scope caption from the header
    (#69,
    0bdac0b)

Documentation

  • cost: Describe the per-turn backfill as the net it is
    (#63,
    93c7fc0)

  • cost: Describe the unpriced-crash mechanism accurately and keep comments framework-general
    (#63,
    93c7fc0)

  • litellm: Correct the cost contract after cutting per-message distribution
    (#66,
    4131a2a)

  • litellm: Document LITELLM_COST_LOG wiring + correct the reconciliation-cost contract
    (#66,
    4131a2a)

Features

  • cost: Publish one accurate total on every reporting surface
    (#63,
    93c7fc0)

  • docker: Bind-mount the LiteLLM cost log so --driver docker joins actual cost
    (#66,
    4131a2a)

  • evalboard: Compare every harness on the overview, and scope the whole page to one
    (#69,
    0bdac0b)

  • evalboard: Compare harnesses on the overview, and identify each run
    (#69,
    0bdac0b)

  • evalboard: Lift the harness scope to the page header, in vendor colors
    (#69,
    0bdac0b)

  • evalboard: Make each turn's provider-call table a collapsed dropdown
    (#66,
    4131a2a)

  • evalboard: Mark a partly-priced run total as a floor, not the bill
    (#63,
    93c7fc0)

  • evalboard: One set of pass-rate cutoffs, and a run table that pages through all history
    (#69,
    0bdac0b)

  • evalboard: Per-call cost/cache table from provider_call_costs (replaces inline)
    (#66,
    4131a2a)

  • evalboard: Read the canonical pass rate and surface incomplete cost
    (#63,
    93c7fc0)

  • evalboard: Say which harness, model, and framework version a run used
    (#69,
    0bdac0b)

  • litellm: Actual per-call cost + cache accounting for the open-weight backend
    (#66,
    4131a2a)

Refactoring

  • cost: Correct the simulator-cost bound and drop the unread variant error share
    (#63,
    93c7fc0)

  • cost: Cut the commentary and drop unreachable rate-card keys
    (#63,
    93c7fc0)

  • cost: Define the unpriced-row test once, and only for new runs
    (#63,
    93c7fc0)

  • cost: Total_cost_usd means the whole bill everywhere
    (#63,
    93c7fc0)

  • evalboard: Call the UiPath harness Delegate
    (#69,
    0bdac0b)

  • litellm: Cut per-message distribution; turn-level join + per-call audit record
    (#66,
    4131a2a)

  • litellm: Drop the provider field/column — unavailable on the streaming path
    (#66,
    4131a2a)

  • litellm: Stream the cost log + de-duplicate the OpenRouter config comment
    (#66,
    4131a2a)

Testing

  • litellm: Cover config shape, join ordering, and defensive cost branches
    (#66,
    4131a2a)