feat(dashboard): Σ_pol policy backtester — counterfactual replay of recorded traffic - #71
Merged
Conversation
…ecorded traffic Adds POST /dashboard/api/policy/backtest: groups the window's calls by model family (one bounded SQL query, top-50 by requests with truncation metadata), ranks each family through the router's /x/rank with the candidate policy, and prices the winner against the group's historical token volumes. Reports actual vs counterfactual cost, per-route winners with fallbacks, observed-latency proxies, unroutable traffic, and an explicit caveats array (current catalog/prices, per-family ranking). Per-family rank failures degrade gracefully: the group is flagged rank_error and excluded from totals; only all-families-failed aborts. Dashboard: Backtest button + window select (24h/7d/30d) in the Policy debugger, with a results panel of headline deltas and per-route rows.
|
Warning Review limit reached
Next review available in: 40 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. 📝 WalkthroughWalkthroughThis PR adds an admin-only Σ_pol policy backtesting feature. It introduces a new database aggregation function ChangesΣ_pol Policy Backtesting Feature
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant AuthProxy
participant HostStore
participant RankerAPI
Client->>AuthProxy: POST /dashboard/api/policy/backtest
AuthProxy->>AuthProxy: validate admin session, parse policy_ir
AuthProxy->>HostStore: policy_backtest_groups(since_ts, limit)
HostStore-->>AuthProxy: grouped route traffic and totals
AuthProxy->>AuthProxy: fetch current prices via _price_table()
loop each historical route group
AuthProxy->>RankerAPI: POST /x/rank(requirements)
RankerAPI-->>AuthProxy: ranked candidates or error
AuthProxy->>AuthProxy: compute counterfactual winner cost
end
AuthProxy-->>Client: window totals, per-group backtest results, caveats
Poem: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@auth_proxy.py`:
- Line 4658: The backtest table in bRenderBacktest only treats g.unroutable as a
special case, so rank_error groups look like normal zero-admitted rows and their
failure reason is hidden. Update the row rendering to also branch on
g.rank_error (using the backend-provided rank_error_status and rank_error_kind)
and show a visible bad/warn pill or message for the failure cause. Also apply
consistent highlighting in the row class logic, e.g. alongside the existing
unroutable check, so rank-API failures are visually distinct.
- Around line 2001-2040: The backtest ranking loop in the route-processing block
is making `_rank_policy_for_backtest` calls sequentially, which causes latency
to scale linearly with distinct routes. Refactor the `/x/rank` lookup path so
the per-route rank requests are dispatched in parallel and then merged back into
`rank_cache`, preserving the existing `first_rank_error`, `rank_error_groups`,
and `rank_successes` bookkeeping in the same loop structure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 25b9f523-ffd8-4fc6-9e72-2b15aa1c0a65
📒 Files selected for processing (3)
auth_proxy.pyhost_store.pytests/test_policy_backtest.py
… rows in UI Distinct families rank via semaphore-bounded (8) asyncio.gather instead of sequentially — worst case drops from ~50x10s to a few seconds behind proxy timeouts. Error semantics and group ordering unchanged. Rank-failed rows get the bad-pill treatment in the results table so they can't be misread as costed.
MuncleUscles
added a commit
that referenced
this pull request
Aug 6, 2026
The repo had no pre-merge checks. notify-ci.yml fires on push to main — after the merge button — and only dispatches to the private CI repo; PR checks were CodeRabbit alone, and branch protection required a review but zero status checks. tests/test_antseed_node.py was consequently red for a month across #63/#68/#69/#70/#71 and every one merged. Three jobs: - tests: pytest against a UTF8 Postgres service (SQL_ASCII makes psycopg return TEXT as bytes, which silently voids settings overrides), submodules checked out so the engine-backed tests are real, node present for the sidecar suite. - core-tests: the Lua policy core's unit + golden conformance vectors. - images: build both images, then smoke the artifact. The image smoke is the one that would have caught #95. A green suite proves the REPO is consistent and says nothing about what COPY put in the image — which is exactly how a control.js requiring ./ids.js shipped without it, died at import, and took every wallet endpoint down with a 502. scripts/check_sidecar_modules.js resolves (never executes) each shipped module's local imports inside the built image; verified to reproduce that failure against the old COPY list.
jmlago
pushed a commit
that referenced
this pull request
Aug 7, 2026
…#96) * ci: gate merges on tests, core conformance and a built-image smoke The repo had no pre-merge checks. notify-ci.yml fires on push to main — after the merge button — and only dispatches to the private CI repo; PR checks were CodeRabbit alone, and branch protection required a review but zero status checks. tests/test_antseed_node.py was consequently red for a month across #63/#68/#69/#70/#71 and every one merged. Three jobs: - tests: pytest against a UTF8 Postgres service (SQL_ASCII makes psycopg return TEXT as bytes, which silently voids settings overrides), submodules checked out so the engine-backed tests are real, node present for the sidecar suite. - core-tests: the Lua policy core's unit + golden conformance vectors. - images: build both images, then smoke the artifact. The image smoke is the one that would have caught #95. A green suite proves the REPO is consistent and says nothing about what COPY put in the image — which is exactly how a control.js requiring ./ids.js shipped without it, died at import, and took every wallet endpoint down with a 502. scripts/check_sidecar_modules.js resolves (never executes) each shipped module's local imports inside the built image; verified to reproduce that failure against the old COPY list. * ci: give the image smoke a database and fail fast on a dead container The router opens a host-store pool at startup, so booting it without Postgres proved only that it can fail to connect. Adds a postgres service to the images job and runs the container with --network host so it can reach it (a bridged container cannot see the runner's localhost). Also drops the '|| true' after docker run: a container that fails to start must fail the job immediately, not fall through to a curl loop that reports the same thing sixty seconds later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Answers the question the Policy debugger couldn't: "if I'd been running this policy, what would my recent traffic have cost, and where would it have routed?"
POST /dashboard/api/policy/backtest— body{policy_ir, timeframe (24h/7d/30d, default 7d), consumer?}:idx_calls_ts; no unbounded loads./x/rankcall per distinct family (cached within the request) ranks the candidate policy over the current catalog._price_table()fallback). Latency shown only where the winner has observed history in the window — nothing invented.rank_errorand skipped (totals only compare priceable groups); only all-families-failed aborts with the upstream error.caveatsarray: current catalog/prices/breaker state (not historical), ranking treated as request-independent per family.Dashboard: Backtest button + window select in the Policy debugger; results panel shows actual vs counterfactual cost, delta, unroutable share, and per-route winner rows with highlights.
Σ_flow is out of scope (clear 400).
Review
Spec + review by Claude, implementation by Codex, reviewed against internal review standards; the one Important finding (a single /x/rank failure aborted the whole backtest) was fixed with partial-result degradation and tests for both partial- and total-failure paths.
Tests
tests/test_policy_backtest.py(12 tests): auth, flow_ir rejection, exact group/pricing/delta math against a faked rank endpoint,_price_tablefallback, unroutable exclusion, rank-failure partial results, all-failed propagation, truncation metadata, empty-window shape, winner-latency null case. Full suite: 467 passed, 2 skipped, 1 pre-existing unrelated failure (test_antseed_node.py).Summary by CodeRabbit
New Features
Bug Fixes