feat(replay-vision): monthly observation quota#60357
Merged
Merged
Conversation
Contributor
|
🎭 Playwright didn't run on this PR — your changes touch code that could affect E2E behavior, but Playwright is opt-in via label now to keep CI cost down. Add the Most PRs don't need this. Real regressions still get caught on master and fix-forward. |
Contributor
|
Size Change: -2.25 kB (0%) Total Size: 80.3 MB 📦 View Changed
ℹ️ View Unchanged
|
Contributor
MCP UI Apps size report
|
TueHaulund
force-pushed
the
tue/replay-vision-quotas
branch
2 times, most recently
from
7484ae9 to
ae1818c
Compare
TueHaulund
changed the title
TueHaulund
force-pushed
the
tue/replay-vision-quotas
branch
from
1aa082b to
751ba6e
Compare
Contributor
|
🎭 Playwright report · View test results →
These issues are not necessarily caused by your changes. |
TueHaulund
force-pushed
the
tue/replay-vision-quotas
branch
from
751ba6e to
adcdc9b
Compare
TueHaulund
requested review from
a team,
arnohillen,
fasyy612,
ksvat and
nicowaltz
and removed request for
a team
Contributor
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
TueHaulund
force-pushed
the
tue/replay-vision-quotas
branch
from
adcdc9b to
673742d
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
PR overviewThe PR has one remaining security concern: the monthly observation quota can be bypassed through concurrent observation requests because the quota check and row creation are not atomic. This would let a user with scanner write access create more pending observations than the organization’s monthly cap allows, increasing resource usage beyond intended limits. One prior issue has already been addressed, so the remaining risk is focused and should be resolved with an atomic reservation or locked count-and-insert flow. Open issues (1)
Fixed/addressed: 1 · PR risk: 5/10 |
TueHaulund
force-pushed
the
tue/replay-vision-quotas
branch
from
673742d to
58a295d
Compare
Caps each org at 1,000 successful observations per calendar month and
blocks on-demand observations once that ceiling is reached.
- `quota.py` — single static beta quota (1000) and a Postgres-backed
`compute_quota_snapshot(org_id)` that counts succeeded observations
with `completed_at` in the current UTC month.
- `GET /api/environments/{team_id}/vision/quota/` returns
`{monthly_quota, usage_this_month, remaining, exhausted, period_start, period_end}`.
- `POST /api/environments/{team_id}/vision/scanners/{id}/observe/`
returns `HTTP 402 quota_exhausted` (with the next period boundary)
before starting any Temporal workflow when the org's snapshot is exhausted.
- Frontend `VisionQuotaMeter` now consumes real backend data; type
renamed to match (`monthly_quota` / `usage_this_month` / `remaining`
/ `exhausted`). Dropped the unused `policy` field and the placeholder
sparkline + day-range picker — no policy concept until billing is wired,
no per-day history from this endpoint.
The single source of truth — `compute_quota_snapshot` — is the swap
point for a future billing-service integration: no caller (workflow,
viewset, frontend) sees the static-vs-billed distinction.
TueHaulund
force-pushed
the
tue/replay-vision-quotas
branch
from
58a295d to
66feb1f
Compare
| f"User {inputs.triggered_by_user_id} is not a member of scanner {inputs.scanner_id}'s organization" | ||
| ) | ||
|
|
||
| if compute_quota_snapshot(scanner.team.organization_id).exhausted: |
There was a problem hiding this comment.
Medium: Quota check is not atomic
compute_quota_snapshot(...).exhausted runs before the observation row is inserted, with no lock or reservation. An attacker with scanner write access can submit many /observe/ calls for different sessions when the org is just under quota; the Temporal activities can all see the same pre-insert usage and create PENDING observations past the monthly cap. Move the quota check into an atomic reservation, such as a per-org/month counter row locked with select_for_update or an advisory lock around the count-and-insert path.
rorylshanks
pushed a commit
that referenced
this pull request
May 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Replay Vision needs a cap on observation volume before it can be turned on more broadly — a runaway scanner would happily burn through a month's Gemini budget.
Changes
A flat 1,000 successful observations per organization per UTC calendar month, enforced at
/observe/.compute_quota_snapshot(org_id)countssucceededobservations in the current monthGET /api/environments/{team_id}/vision/quota/returns{monthly_quota, usage_this_month, remaining, exhausted, period_start, period_end}POST .../scanners/{id}/observe/returnsHTTP 402 quota_limit_exceededbefore touching Temporal when exhaustedVisionQuotaMeterreads the real endpointHow did you test this code?
hogli test products/replay_vision/backend/tests/test_quota.py— 9 passed.hogli build:openapiclean.Publish to changelog?
no
🤖 Agent context
Tool: Claude Code.