Proxytrace 1.3.0

Added

• Outlier detection for traces. Each ingested call is now flagged when it deviates from its
  agent's own recent behaviour on any of four per-call metrics — high token count (also the cost
  signal), high latency, low turn-2+ cache hit, and many tool calls. Detection is
  per-agent and adaptive: a call is flagged when a metric exceeds the agent's recent mean ± N
  standard deviations, so a cheap fast agent and an expensive reasoner each get their own "normal".
  The Traces list carries a dedicated Anomalies column (just before the timestamp) that shows
  an amber warning chip on each flagged call — hover it for the reasons — plus a new Outliers only
  toggle that filters the list to just the outliers, and the agent detail page gains a Recent outliers widget
  that lists why each recent call was flagged. Admins tune the sensitivity (enable/disable, sigma,
  minimum samples, baseline window) under Settings → Outlier detection. Existing traces are not
  retroactively flagged; detection applies to calls ingested from now on.
• Call distribution stats on the agent page. The agent detail view's Performance card now
  shows one small card per stat in a single grid that reflows to the available width: the window
  totals (pass rate, traces, tokens, cost, latency — each with a trend sparkline), then the
  mean ± standard deviation of an agent's successful calls over the selected range — input and
  output tokens and latency (per call), and cost, cache hit rate (turns after the
  first, which can't be cache hits) and tool calls (per conversation). Each distribution card draws
  a small density curve of the real sample shape — hover to read a slice's value range and how many
  calls (or conversations) fall in it — and metrics with no signal in the window (an agent that never
  caches or calls a tool) are dropped rather than shown empty. Everything shares one time-range selector
  that persists as you switch agents, and updates live as new traces arrive, so a single card shows
  not just the totals but how consistent — or skewed — your agent's calls are.
• Sample a test run multiple times. When you start a run you can now pick a sample count (1–5) —
  Proxytrace runs each selected endpoint that many times and averages the results per endpoint, so
  non-deterministic models don't hide flaky cases. The results matrix shows one column per endpoint with
  a per-case pass fraction (e.g. 4/5) and average score, a new Flaky filter surfaces cases whose
  samples disagree, and clicking a cell drills into the individual sample i/N runs. Anomaly detection
  and the auto-optimization loop operate on one representative run per endpoint, so sampling never
  fires duplicate anomalies or biases a proposal toward a single lucky sample. Existing single-sample
  runs look exactly as before.
• Two-factor authentication (TOTP). Protect your account with a second factor from an
  authenticator app (Google Authenticator, Authy, 1Password). Turn it on under Account security
  (in the account menu): scan the QR code, confirm a code, and save the 10 one-time backup codes
  you're shown. After that, signing in asks for a 6-digit code — or a backup code if you've lost your
  device. Disabling it requires your password. Admins can clear a locked-out user's MFA from
  Settings → Users (Reset MFA). MFA is opt-in per user, free on every tier, and applies to
  password (local) sign-in; SSO/OIDC handles MFA at your identity provider. Enabling, disabling, and
  failed code attempts are recorded in the audit log, and the verification endpoint is rate-limited.
• Forgot your password? Self-service reset. The sign-in screen now has a Forgot password?
  link. Enter your email and Proxytrace sends a one-time reset link — valid for 1 hour — that lets you
  choose a new password and signs you straight in. No SMTP? You're still covered: if outgoing
  email isn't configured, the reset link is written to the server log for the operator to relay, and
  an admin can mint a one-time reset link for any user from Settings → Users (the Reset
  password button). Reset requests and completions are recorded in the audit log, and the public
  reset endpoints are rate-limited.
• Cancel or reject an optimization theory. On the Optimization Theories board you can now
  dismiss a theory you don't want to pursue: Reject a Proposed theory to skip A/B validation
  entirely, or Cancel validation on a Validating theory to abort its in-flight A/B run. Either
  way the theory moves to Rejected and can still be reset later. Validation already runs one
  theory at a time, so these controls let you clear the queue and stop runs you no longer need.
• Much broader audit-log coverage. The audit log now records far more of what happens in
  Proxytrace: the whole optimization-theory loop (theory submitted/reset/rejected, plus the A/B
  pipeline's validated/invalidated decisions, the proposals it generates, and proposals it
  auto-adopts), the test-run lifecycle (cancel, optimize, delete) and recurring schedules
  (create/update/delete/run-now), trace deletions, agent-version moves, test-case edits, the
  destructive non-model data purge, and the one-time at-rest secrets backfill. New OIDC coverage: the first-time
  provisioning of an SSO user is now recorded. New failure visibility: a forbidden attempt to
  change something (HTTP 403) is logged as an Access denied failure, so privilege-probing shows up
  alongside failed sign-ins. All new action types are filterable on the audit-log page.

Changed

• The Evaluators page remembers your selected time range. The range selector (1h / 24h / 7d /
  30d) on the evaluator detail now persists, so it survives switching between evaluators and reloading
  the page instead of snapping back to 7d each time (matching the agent detail page).

• Consistent typography, spacing and corners across the app. Swept every screen onto the design
  system's type scale, spacing steps, corner radii and surface colours, replacing dozens of off-scale
  one-offs that left labels, paddings and rounded corners subtly mismatched between otherwise-identical
  panels. Purely visual — no behaviour changes.

• Live duration, cost and tokens during a test run. While a run is in progress its model cards now
  count duration, cost and tokens up as each case lands, instead of sitting at "—"/$0 until the
  run finished. Running runs are also highlighted in the left-hand Test Runs list with an animated
  accent ring and a pulsing Running tag, so an in-flight run is obvious at a glance.

• The Test Runs list loads incrementally. Instead of fetching a large batch up front, the runs rail
  now loads the most recent runs first and reveals older ones on demand via a Load more button —
  faster to open and lighter on projects with a long run history.

• Test-run model comparison, rebuilt around your production model. A run's results now open with
  the model you have in production (the agent's deployed endpoint) as the baseline — a
  highlighted champion card carrying the headline pass rate plus duration, cost, and token totals.
  Every other model is a candidate, read as deltas measured against production: pass-rate
  points, faster/slower, and cheaper/pricier, coloured green when the candidate wins that metric and
  red when it loses — so it's obvious at a glance whether a candidate is worth switching to. Three
  award medals call out the highest pass rate, the fastest, and the cheapest model, and the
  evaluator breakdown now highlights the leading model per evaluator. When a run doesn't include your
  deployed model, the best performer stands in as the baseline. Deltas and medals still appear only
  once the whole run group has finished.

• Tracey no longer cuts a reply short at a turn limit. The assistant's per-turn tool-step cap and
  its "Step limit reached" notice have been removed, so a complex request that needs many tool steps
  now runs to completion instead of stopping early and asking you to continue. (A high internal
  safety backstop still prevents a runaway loop.)

• Tracey's per-response stats now break down token usage. The quiet status row beneath each reply
  shows input tokens, the share of input served from cache, and output tokens instead of
  a single total — making it clear how much of a turn's cost was cached prompt vs. fresh input vs.
  generated output.

• Tracey now focuses on your own agents. Her data tools hide Proxytrace's internal system
  agents — Tracey herself and the evaluators that score your test runs — by default, so "list my
  agents", token-usage charts, recent test runs (the internal A/B validation runs are hidden too),
  and trace searches stay about your work rather than the platform's own activity. Ask explicitly
  (e.g. "include the Tracey agent" or "list system agents") and she'll add them back in.

Fixed

• Evaluator statistics now include historical evaluations. The Evaluators page (average score,
  evaluation count, pass rate, average latency, score distribution and the per-evaluator sparkline)
  reads a query-optimized projection that was only written for evaluations recorded after the feature
  shipped — so existing results showed a dash or zero even when an evaluator had plenty of past
  evaluations. A one-time, idempotent backfill now rebuilds that projection for older results on
  startup, so historical evaluations count toward the statistics.

• Test-run latency now measures the model, not the wall clock. The latency shown on a run's
  per-model comparison cards (the champion card, the candidate "Speed" deltas, and the "Fastest"
  medal) and on an optimization proposal's A/B-test card was computed as a wall-clock timer over the
  whole run (completed − started), so it also counted the time the run waited in the queue and the
  time spent running evaluators, and was compressed by cases running in parallel — none of which is
  the model's latency. It now reports the average per-case inference latency (the same figure the
  test-case matrix shows), so model speed comparisons reflect the model itself and a run that merely
  waited longer in the queue no longer looks slower.

• Polished several rough UI details. The test-run "Evaluation started" confirmation now uses a
  proper check icon instead of a typed character; the evaluator score legend no longer crams a full
  sentence into each coloured pill (short pills now sit beside their plain-text meaning); the Error
  Log and Audit Log page titles match the size of every other page header; a trace's ID in the
  detail panel renders at its intended size again; and the "move version" agent picker no longer shows
  a transparent, off-theme list. A further pixel-level pass squared up details across the app: long
  names, IDs, links and titles now shorten with an ellipsis instead of overflowing their card or
  shoving neighbouring controls off-screen (agent/suite/evaluator/run headers, notifications, global
  search results, password-reset and invite links, the member picker, Tracey tool cards and the
  playground tool list); section, dialog and entity-detail titles now share a single weight; the
  scheduled-runs "recent runs" strip lays out as a horizontal row again instead of a vertical
  stack; and assorted side-by-side inconsistencies were aligned — matching selected-row styling
  and panel framing in the evaluator bench, label casing in the playground parameters, row indentation
  and divider alignment between suite tabs, and date-column contrast and form-field labels on the
  admin and sign-up screens. Finally, status and agent chips now sit vertically centred against
  their heading across every detail header (test runs, suites, agents, evaluators, the evaluator
  bench and the proposals board) — previously the chips drooped a few pixels below the title — and
  the two chips in a run header now render at a single matching size instead of one larger than
  the other.

• Deleting a test run no longer flashes a "not found" error. Removing a run refreshed the whole
  run namespace, which re-fetched the just-deleted run's own detail and surfaced a 404. Delete now
  drops the run from the list immediately and skips re-fetching the gone detail, so it disappears
  cleanly.

• Interrupted test runs no longer hang in "Running" forever. A run in progress when the server
  restarts (deploy, crash, container recycle) could be stranded in Running/Pending
  indefinitely, since its work only lived in memory and can't be resumed. On startup the server now
  marks any such orphaned run Cancelled, so the list and headers reflect reality instead of a
  ghost run that never finishes.

• A running test run now shows a live duration. Each model card's Duration stayed at "—" for
  the whole run (the per-run state only flips once a case finishes and that transition wasn't streamed)
  and only filled in at the end. A run now reads as running as soon as its first case starts, with
  the duration ticking up live alongside cost and tokens.

• A finished test run now updates to "Completed" on its own. The run header could stay stuck on
  Running after the run had actually finished on the server, only flipping to Completed after a
  manual page refresh. The live view now keeps its event stream open until the group finishes (not
  just its individual runs) and flips the status the moment the completion event arrives, so the header,
  the Cancel button, and the live progress bar all settle without a refresh.

• Test-run pass rate no longer shows a long decimal. Averaged pass rates on the comparison cards
  rendered as e.g. 96.66666666666667%; they're now rounded to a whole percent like every other
  pass-rate readout.

• Outlier-detection changes show a proper label in the audit log. Tuning Settings → Outlier
  detection records an audit entry, but the Audit Log page rendered that action with a blank,
  uncoloured, unfilterable label. The Outlier Settings Updated action now shows its label and
  colour like every other audit action.

• Deleting an agent from its detail page works again. With an agent open, the detail view kept
  several live-update streams connected at once, which on the bundled (HTTP/1.1) setup could use up
  the browser's small per-site connection budget. A delete then had no connection left and silently
  never reached the server — the confirmation closed but the agent stayed in the list. The detail
  view now shares one connection across those streams, freeing capacity so Delete (and other actions
  taken while viewing an agent) go through reliably.

• Enabling MFA no longer fails when the setup request is sent twice. Two near-simultaneous
  "set up MFA" requests for the same account (e.g. a double-click or a retried request) raced on the
  one-enrollment-per-user rule and the second crashed with a server error. Setup now tolerates the
  race and returns the enrollment that took effect, so the QR code always matches the stored secret.

• Traces show their message preview again. Traces ingested before the list's denormalised preview
  column was introduced rendered with a blank message preview. A one-time, idempotent startup backfill
  now recomputes the preview (the first user message) for those rows in bounded batches, so every trace
  shows its preview after the next restart — no longer only the ones captured since the column was added.

• Password reset and invite links point at the right address. The emailed reset/invite links fell
  back to the API server's own host and port when no explicit frontend URL was configured, producing a
  link the browser couldn't open. They now use the configured frontend origin (Frontend:AllowedOrigin),
  so the links work out of the box in every environment.

• "No traces" message instead of setup instructions when filters exclude everything. The Traces
  page treated an empty list from the new Outliers only filter as an empty project and showed the
  first-time setup instructions. It now shows "No traces match your filters" when any filter (including
  Outliers only) is active, and keeps the setup instructions only for a project with no traces at all.

• Dashboard and statistics stay fast on large datasets. On a database with a lot of history the
  dashboard and statistics aggregates could take several seconds because PostgreSQL's query planner,
  working from out-of-date table statistics, chose a plan that scanned the whole traces table the slow
  way. Proxytrace now keeps the planner's statistics fresh on the high-volume traces table (more
  frequent auto-analysis), so the same queries run in a few hundred milliseconds. If you bulk-import or
  restore a large database, run ANALYZE once afterwards so the speed-up applies immediately rather
  than after the next automatic analysis.

• Tracey's own traces are captured reliably again. Tracey runs inside the app, but her captured
  calls were being routed through the same Redis message stream used to bridge the standalone
  ingestion proxy — so whenever that stream was unavailable, every Tracey trace was silently dropped
  (her replies still worked, but the trace link reported "still being captured" forever and nothing
  showed in Traces). In-app captures now persist directly, with no dependency on the proxy's
  transport.

• Test-run results stay readable with many evaluators. The test-case matrix used to scroll inside
  its own card, shrinking to an unusable height when a run had lots of evaluators. The whole results
  column now scrolls as one unit, so the matrix keeps its full height.

• Global search hides built-in system agents and their traces. The title-bar search and recent
  feed no longer surface internal system agents (Tracey, the optimization/A-B optimizer agents,
  agentic-evaluator agents) or the traces they generate — only your own agents, suites, traces,
  evaluators, and test cases. Any previously indexed system entities are purged on the next reindex.

• Global search again shows recent agents, suites, and evaluators. The title-bar search's default
  (empty-query) list was being crowded out by traces on busy projects, leaving only recent traces.
  Each entity type is now surfaced independently, so recent agents, test suites, evaluators, and
  traces all appear again.

• Evaluator playground shows tool-call responses. When a selected past evaluation's response was a
  tool call with no text, the reference showed "—" and the candidate was blank (the scoring
  itself was unaffected). Both now render the tool call (e.g. [tool call] get_weather({…})).

• Numeric evaluator scoring no longer depends on the server's locale. The numeric-match evaluator
  parsed expected and actual values using the server's regional settings, so a value like 3.14 could
  be read as 314 on a non-US host and silently flip a pass to a fail. Numbers are now parsed the same
  way everywhere (invariant format), and a tool message carrying more than one result no longer drops
  the extra results from its text.

• Operator error log keeps its full retention under bursty errors. When many errors shared the
  exact same timestamp, trimming the error log to its configured size could delete the whole group at
  the cutoff and leave fewer entries than intended. Trimming now breaks ties deterministically, so it
  keeps exactly the configured number of most-recent errors.

• Provider pricing with input cost ≥ output cost can be saved again. Activating or updating a model
  endpoint wrongly required input token cost ≤ output token cost, rejecting legitimate provider pricing
  (some cached, batch, and reasoning tiers price input at or above output). That rule is gone; cost
  calculation is unaffected. Proxytrace also now rejects nonsensical stored numbers — out-of-range pass
  rates and p-values, and invalid inference parameters such as negative max tokens or NaN/Infinity —
  instead of persisting them.

• Promoting a response-less trace returns a clear 400. Promoting a captured call that has no
  response into a test case failed with a generic server error (500); it now returns 400 (bad request),
  like the adjacent validation cases.

• A slow trace ingest no longer produces duplicate traces. When persisting a single captured call
  took unusually long (heavy database contention or a very large transcript), the ingestion worker could
  reclaim the still-in-flight item and process it a second time — creating two identical traces, two
  notifications, and two outlier evaluations. The worker now skips an item it is already processing and
  waits much longer before reclaiming, so a slow-but-live ingest is never double-counted.

• Sign-in stays fast and email is treated case-insensitively. Email addresses are now stored in a
  normalised (lower-case) form and looked up by exact match, so logging in uses the email index instead
  of scanning the whole users table, and Foo@x.com and foo@x.com can no longer become two separate
  accounts. Existing addresses are normalised once on upgrade. Creating a model or endpoint during a
  burst of traffic also retries cleanly instead of failing if two requests create it at the same moment.

• Cancelled playground and Tracey calls are no longer recorded as failed traces. Cancelling a model
  request (or shutting the app down mid-call) recorded a phantom HTTP-500 trace that polluted statistics
  and outlier detection. Cancellations are now ignored rather than captured.

• Latency percentiles honour the "exclude system agents" option on PostgreSQL. The p50/p95/p99
  latency and live-telemetry queries ignored the option that hides built-in system agents (Tracey, the
  optimizer/evaluator agents), so those calls could skew the percentiles. The option is now applied on
  PostgreSQL, matching every other statistic.

• Test-suite run statistics stay fast as history grows. Opening the test-suites list or a single
  suite read the entire run-statistics table and filtered it in memory; it now asks the database only
  for the suites in view. Archived-entity lists are likewise filtered in the database rather than after
  loading every row.

• Real-time streams clean up and stay bounded. The long-lived trace, proposal, theory, and
  notification streams now send a periodic keep-alive, so a connection that dies without notice (a
  half-open socket) releases its slot instead of lingering, and the two streams every client subscribes
  to now cap their subscriber count like the others — protecting the server from a flood of stream opens.

• Background backfills release database resources promptly. The one-time preview and secret-
  encryption backfills, which run in batches over potentially millions of rows, held onto a database
  context per batch for the lifetime of the process; they now dispose each batch's context as they go.

• Audit log shows a label for email-settings changes. Saving SMTP/email settings produced an audit
  entry the Audit Log page rendered with a blank action label and no colour; it now shows a proper
  "Email Settings Updated" label and badge.

Security

• Evaluator test bench no longer exposes another tenant's test case via a supplied id. Loading or
  running a test case on the bench verified access to the evaluator but not to the separately
  supplied test-case id, so a signed-in user could pass a test-case id from another project and read
  its conversation, expected and actual responses, and scores. The test case's owning project is now
  verified too, returning 404 on mismatch (no existence oracle).
• Internal error detail no longer leaks on a few self-handled responses. The playground's streamed
  error event and the admin email / SMTP connection tests echoed raw exception text — which can carry
  SQL, schema, or file-path detail — even in production, bypassing the global suppression. These now log
  the fault under an error id and return a generic message outside Development, matching the rest of the
  API.
• Closed cross-tenant access gaps on statistics, the playground, the evaluator test bench, trace
  promotion, and theory submission. Several endpoints accepted a project, agent, trace, or evaluator
  id without checking that the caller is a member of the owning project, so any signed-in user could
  read another tenant's data — or, worse, run a model completion on another tenant's provider
  credential (the playground and the evaluator test bench's run). All of these now resolve the
  owning project and return 404 when the caller lacks access (no existence oracle). The dashboard
  additionally refuses the unscoped, all-tenant aggregate to non-admins: a normal user must request a
  project they belong to (the app already does this), while administrators keep the global view.
• Password-reset links are no longer written to the log by default. When email is unconfigured or
  sending fails, Proxytrace previously logged the full one-time reset link (a live credential for an
  hour) so a sole administrator could still recover access. The log now records only a redacted hint by
  default; the full emergency link is logged only when an operator explicitly opts in with the new
  Authentication:EmergencyLogResetLink setting. The in-process auth, MFA, and rate-limit state is also
  now documented as single-instance by design — running multiple API replicas would split those limits
  per replica.