feat: spend budgets

[marcorivm]

Per-agent and per-project spend budgets with metered enforcement in the gateway. 20 files, +2,523/−35 — ~1,960 lines of code plus ~560 of tests.

This PR is the tail of the split: it was the original 381-file open-edition PR, now reduced to just its last two commits (741a45c spend budgets, 36aebad gitignore). Everything else moved into the stack below.

gateway/hooks.rs gains the real budget implementation — prepare_request forces Accept-Encoding: identity for metered hosts so responses can be measured. Worth knowing for the next upstream bump: upstream v1.45.0 inserts a refuse_empty_scope hook at exactly the same anchor (see #9's review doc — it's an adjacent-insertion conflict, keep both).

Stack

Split out of the original 381-file #8. Upstream catch-up (v1.42.0 → v1.44.0) already landed as #10, so main is now v1.44.0 and everything below is our own code.

main (v1.44.0, after #10)
 └─ #14  01-tier1-ungating              23 files    +73/-442
     └─ #11  02-org-members-rbac        47 files  +5298/-36
         └─ #12  03-user-groups         16 files  +2963/-1
             └─ #13  04-project-access  33 files  +7701/-234
                 └─ #15  05-gateway-org-scope       6 files  +1080/-176
                     └─ #16  06-gateway-conditions  23 files  +2292/-211
                         └─ #17  07-gateway-resource-scope  13 files  +1551/-20
                             └─ #18  08-web-org-policy      23 files  +2421/-384
                                 └─ #8   spend budgets      20 files  +2523/-35
                                     └─ #9   upstream-sync tooling  8 files  +823/-0

Review and merge in order, top to bottom. Roughly half of each diff is tests.

[marcorivm]

What it actually does

Per-secret (Anthropic/OpenAI) spend caps, enforced in the Rust gateway with real metering.

• Unit: dollars, tracked as nano-dollars (i64, CENT_TO_NANOS = 10_000_000). Cost is input_tokens × price_in + output_tokens × price_out, plus Anthropic prompt-cache tokens at 1.25× (write) and 0.1× (read) of base input. Prices come from a static hand-curated table in budget.rs (price_per_token), matched by longest model-name prefix. An unrecognized model meters as 0 — logged, not blocked: a deliberate under-meter rather than a fabricated price on a control that can 402 real traffic.
• Where counting happens: hooks.rs::track_and_wrap wraps the upstream response in a MeteredStream when a metered binding applies. It tees a bounded 16 KiB (META_CAP) head and rolling tail as bytes stream — it does not buffer the whole body. At clean stream end (finalize()) it substring-scans (budget::parse_usage) for the provider's usage JSON (head catches SSE message_start, tail catches trailing usage/message_delta), prices it, and attaches a BudgetCharge to the RequestEvent. To keep non-streaming JSON scannable, prepare_request forces Accept-Encoding: identity whenever a metered binding is present — reqwest is built without gzip/brotli/deflate, but a client's own Accept-Encoding: gzip (common in httpx/urllib3) would otherwise return compressed bytes the scanner can't read, silently charging 0 forever.
• Where the charge lands: RequestEvents flow through the async telemetry channel into telemetry.rs::flush_loop, running every 5 seconds (FLUSH_INTERVAL_SECS) or every 500 buffered events. The loop sums charges per (secret_id, organization_id, period_key) and does one atomic Postgres upsert per key (db.rs::upsert_budget_spend, spent_nanos = spent_nanos + delta), then overwrites (set_raw, not increment) the hot cache counter with the new total.
• When exhausted: hooks::pre_forward — called from both forward.rs and websocket.rs, so before every proxied request including WS upgrades — reads the cache counter (rehydrating once from the durable Postgres floor on a miss) and returns a hard 402 Payment Required with {"error":"budget_exceeded","secretId","period","limitCents","spentCents"} once spend >= limit. A genuine block, not a warning. Enforcement fails open: any read error lets the request through.
• Reset: monthly budgets key off m:YYYY-MM (UTC), so a new month is automatically a new counter — no reset job. "Total" budgets use a constant key and never reset.

Why it exists

Cost control for org/project-owned LLM credentials proxied through the gateway — cap spend on a secret so a runaway agent or leaked key can't rack up unbounded provider bills.

Reading order

1. apps/gateway/src/budget.rs — module doc and pre_forward's doc comment first, then parse_usage/price/period_key/is_over. Pure logic core, best comments in the PR.
2. apps/gateway/src/gateway/hooks.rs — the wiring: prepare_request (Accept-Encoding), pre_forward (the 402 gate + read_running_total), track_and_wrap/MeteredStream (tee + finalize/Drop). Highest-stakes file.
3. apps/gateway/src/telemetry.rs — record_spend and the batch aggregation; where the lag between "cost computed" and "cap enforced" is introduced.
4. apps/gateway/src/db.rs — find_budgets_for_secrets, read_budget_spend, upsert_budget_spend. The ON CONFLICT … spent_nanos = spent_nanos + $4 is the one place true atomicity lives.
5. packages/api/src/services/budget-service.ts — CRUD plus currentSpentCents. Note the hand-duplicated constants (CENT_TO_NANOS, monthlyPeriodKey, METERED_TYPES) that must stay in lockstep with budget.rs by convention alone.
6. packages/api/src/routes/org/budgets.ts — same admin + org-scope guard pattern as feat(web): org policy page, identity picker, role-mappings UI #18's policy.ts; quick once you've seen that.
7. create-budget-dialog.tsx — worth reading for its honesty: the copy tells the admin outright that "one in-flight request may overshoot" and "streaming spend may under-count for some providers."

Low risk, skim: budgets-list.tsx, budget-usage-bar.tsx, budget-row-actions.tsx, use-budgets.ts, budgets/page.tsx, and the unrelated .gitignore one-liner.

What to scrutinise

The flush-lag race is larger than the code's comments suggest. budget.rs says "one request may overshoot — the cap blocks new requests once exceeded." That holds only for the single request crossing the line mid-flight. It is not true for concurrency inside the flush window: the cache counter is refreshed only when flush_loop runs (every 5s / 500 events) via a full set_raw overwrite, not an atomic increment. CacheStore does expose incr, used elsewhere in the codebase — budget metering doesn't use it. So any number of concurrent requests against the same budget within that window all read the same stale under-limit value in pre_forward and are all let through. Real overshoot is bounded by "how much can arrive in ≤5s across however many parallel agents share this secret," not by one request. For a shared org LLM secret behind many agents that is not a rounding error.

WebSocket-tunnelled usage is never metered. The 402 gate does run on WS upgrades, so an already-over budget correctly blocks a new connection. But once a tunnel is established, track_and_wrap/MeteredStream never runs on it — there's no equivalent tee for duplexed WS bytes. Any provider streaming LLM usage over persistent WebSocket (realtime/voice APIs) accrues real cost with zero spend recorded. Worth documenting as an explicit scope limitation.

Best-effort substring parsing is spoofable in principle. find_uint/find_str scan raw bytes rather than parsing JSON — deliberately tolerant of truncation, but it will match a "usage"-shaped key anywhere in the payload, including inside content the model generated (an agent whose output literally contains "output_tokens": 999999999, echoed back in a content field). Narrow, but it's a scanner that doesn't understand structure being manipulated by the content it measures. Worth deciding whether providers' real usage fields are structurally distinguishable enough for this to be impractical.

Accept-Encoding: identity is scoped but not free. Applied only when a metered binding exists for the host — but it strips compression for the entire response, not just enough to read the usage tail. Every request against a capped secret pays higher bandwidth/latency regardless of how far from the cap it is. No functional breakage (identity is always valid), but it changes wire behaviour broadly.

Error paths look defensible — verify the state machine. finalize() only fires on clean stream end; a Drop impl catches early termination (client disconnect, upstream error) and still emits the base telemetry event with no charge. Read the Drop impl closely: the Option::take() guard looks correct, but this is exactly where a double-take would silently double-emit or drop telemetry, and there is no test proving it.

Retries are not double-counting. A retried request is a genuinely new upstream call with its own MeteredStream, so it's correctly charged again — two calls, two charges. Don't mistake that for a bug.

Design decisions worth questioning

• Cache updated by periodic full overwrite rather than atomic incr per charge, despite incr existing and being used elsewhere — the direct cause of the race above. Incrementing at charge time (in finalize, which already runs per request) and reconciling against the Postgres floor periodically would tighten the window substantially.
• The pricing table is static and hand-maintained in Rust with no connection to any other pricing config; a new model silently meters as free until someone updates it.
• Cross-language constant duplication (CENT_TO_NANOS, METERED_TYPES, period_key) between budget.rs and budget-service.ts is enforced only by "must match" comments — a natural drift point.
• Fail-open is the right call for a cost control and is explicitly reasoned about — but combined with the flush lag, it absorbs more silent overshoot than the comments imply.

Test coverage reality

budget.rs has solid inline unit tests: Anthropic/OpenAI non-stream and SSE-split parsing, cache-token pricing math with exact-value assertions, longest-prefix matching, month rollover, and the is_over boundary (>=, not >). All labelled "pure unit — no DB/network."

Zero tests for hooks.rs, db.rs, or telemetry.rs — no #[cfg(test)]/#[test]/#[tokio::test] in any of the three. So pre_forward's 402 gate, read_running_total's cache-then-DB rehydration, MeteredStream's poll/finalize/drop state machine, the atomic upsert, and the per-flush aggregation are entirely unverified. Every mechanism flagged as risky above sits in that untested set.

routes/org/budgets.test.ts (559 lines) covers the admin API well — scope fencing, metered-type rejection, foreign-org secret 404, duplicate 409, non-positive cap 422, org-fencing on update/delete, 403 on project-scoped key. But it never touches the Rust code, so it says nothing about whether spend is actually metered or capped correctly in the proxy.

Untested paths worth naming: the 402 body's shape under load; concurrent-request overshoot; MeteredStream::Drop on a genuinely disconnected client mid-stream; cache/DB rehydration on a cold cache after restart.

---

Forward-looking: upstream v1.45.0 inserts a refuse_empty_scope hook at the same anchor in hooks.rs where this PR adds its pre_forward doc comment. Expect an adjacent-insertion conflict on the next upstream rebase — both sides must be kept; it's not a logical conflict, just two unrelated features landing on the same line.

---

_{Reviewer orientation guide — produced by analysing this PR's diff and surrounding code, not the commit messages. Claims about line numbers and behaviour are worth spot-checking as you read; where it says something is untested or risky, that was verified against the tree rather than inferred.}