fix(tasks): enforce gated model access server-side - #80331
Conversation
Preview models were gated only in the Desktop model pickers. A stored per-task model preference, an older client, or a direct API call all reached the task-run and loop write paths without the flag ever being consulted, so the picker was the only thing standing between a caller and the model. Add a MODEL_ACCESS_FLAGS registry mapping a model id to the flag a caller needs, and re-check entitlement in the write paths that persist a model selection. The check fails closed: only a registered model triggers an evaluation at all, so an evaluation outage withholds a preview model rather than opening it up. Kimi K3 is the first entry. Adding the next gated model is one line. Generated-By: PostHog Code Task-Id: 30de719b-d537-4496-869a-6b385b3c7dd0
|
😎 Merged successfully - details. |
|
Review found two ways past the serializer check, both because the caller omits `model` and it gets filled in later. Resume: run_task copies a missing model from the previous run's state, so the serializer only ever saw None. Re-check after inheritance, next to the existing reasoning-effort check that guards the same window. Only a gated model pays the distinct-id lookup. Warm: POST /tasks/warm boots a sandbox and starts the agent on a caller-supplied model, so it bills like a run and now gates like one. The create path's write-only model hint selects which warm Run gets activated, and that activation never reaches run_task, so it is gated too. Generated-By: PostHog Code Task-Id: 30de719b-d537-4496-869a-6b385b3c7dd0
🤖 CI report
|
| File | Patch | Uncovered changed lines |
|---|---|---|
products/tasks/backend/presentation/serializers_loops.py |
66.7% | 554 |
products/tasks/backend/feature_flags.py |
93.3% | 102 |
products/tasks/backend/presentation/serializers.py |
94.1% | 2635 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 31374680736 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
demo |
███████████░░░░░░░░░ 56.3% |
1,497 / 2,661 |
tasks |
██████████████░░░░░░ 70.4% |
35,677 / 50,672 |
signals |
█████████████████░░░ 82.6% |
27,316 / 33,075 |
cdp |
█████████████████░░░ 84.2% |
3,914 / 4,649 |
notebooks |
█████████████████░░░ 86.4% |
8,029 / 9,296 |
data_modeling |
█████████████████░░░ 86.5% |
8,459 / 9,782 |
actions |
█████████████████░░░ 86.6% |
717 / 828 |
cohorts |
██████████████████░░ 87.6% |
6,482 / 7,400 |
managed_warehouse |
██████████████████░░ 87.8% |
5,928 / 6,752 |
product_tours |
██████████████████░░ 87.9% |
1,303 / 1,482 |
data_warehouse |
██████████████████░░ 88.4% |
11,653 / 13,185 |
exports |
██████████████████░░ 89.1% |
7,490 / 8,406 |
engineering_analytics |
██████████████████░░ 89.3% |
6,759 / 7,570 |
dashboards |
██████████████████░░ 89.3% |
5,995 / 6,710 |
alerts |
██████████████████░░ 90.3% |
4,482 / 4,966 |
mcp_analytics |
██████████████████░░ 90.4% |
3,968 / 4,388 |
conversations |
██████████████████░░ 90.5% |
18,337 / 20,256 |
canvas |
██████████████████░░ 90.6% |
2,080 / 2,296 |
streamlit_apps |
██████████████████░░ 90.7% |
2,630 / 2,901 |
error_tracking |
██████████████████░░ 91.1% |
11,147 / 12,237 |
stamphog |
██████████████████░░ 91.3% |
4,505 / 4,936 |
slack_app |
██████████████████░░ 91.7% |
10,813 / 11,789 |
ai_observability |
███████████████████░ 92.9% |
17,116 / 18,426 |
early_access_features |
███████████████████░ 92.9% |
1,347 / 1,450 |
web_analytics |
███████████████████░ 93.1% |
15,963 / 17,154 |
marketing_analytics |
███████████████████░ 93.1% |
15,445 / 16,591 |
surveys |
███████████████████░ 93.2% |
5,919 / 6,349 |
posthog_ai |
███████████████████░ 93.3% |
1,327 / 1,423 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
approvals |
███████████████████░ 93.5% |
3,491 / 3,734 |
product_analytics |
███████████████████░ 93.5% |
7,095 / 7,587 |
workflows |
███████████████████░ 94.3% |
7,951 / 8,436 |
endpoints |
███████████████████░ 94.3% |
8,771 / 9,306 |
review_hog |
███████████████████░ 94.6% |
8,246 / 8,715 |
skills |
███████████████████░ 94.8% |
3,496 / 3,687 |
logs |
███████████████████░ 95.6% |
11,815 / 12,362 |
replay_vision |
███████████████████░ 95.7% |
18,220 / 19,043 |
experiments |
███████████████████░ 95.8% |
28,701 / 29,962 |
annotations |
███████████████████░ 96.2% |
732 / 761 |
revenue_analytics |
███████████████████░ 96.3% |
1,887 / 1,960 |
data_quality |
███████████████████░ 96.4% |
747 / 775 |
feature_flags |
███████████████████░ 96.4% |
17,770 / 18,429 |
user_interviews |
███████████████████░ 96.5% |
2,638 / 2,734 |
customer_analytics |
███████████████████░ 96.6% |
11,420 / 11,816 |
warehouse_sources |
███████████████████░ 97.5% |
380,712 / 390,563 |
data_catalog |
████████████████████ 97.9% |
2,677 / 2,734 |
pulse |
████████████████████ 98.4% |
2,017 / 2,049 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
Problem
A user who is not entitled to a preview model can still run one. The Desktop model picker hides it, but the picker is the only gate — nothing on the server checks entitlement, so a caller who gets the model id from anywhere else runs it.
KIMI_MODEL_FLAG(tasks-kimi-k3) is read only in the Desktop UI —ModelSelector.tsx,LoopModelFields.tsx,usePreviewConfig.ts, and the mobile task screens.modelthey are sent.validate_model_selectionchecks that the runtime adapter and reasoning effort agree with the model; it never asks who is calling.Observed in production: users for whom the flag evaluated
falseran Kimi K3 minutes later, and kept running it for days. Flag evaluations for this key come only from$lib = web; there are none from the server.Changes
MODEL_ACCESS_FLAGSinproducts/tasks/backend/constants.py, mapping a model id to the flag a caller needs to select it. Kimi K3 is the first and only entry.get_model_access_errorinproducts/tasks/backend/feature_flags.py, exposed to presentation through therun_configfacade.POST /tasks/{id}/runTaskRunCreateRequestSerializerTaskRunBootstrapCreateRequestSerializerLoopWriteSerializerrun_task, after inheritancePOST /tasks/warmWarmTaskRequestSerializerTaskWriteSerializerThe last three came out of review. The pattern is the same in each: the caller omits
model, so the serializer seesNoneand passes, and the real model is filled in later — fromprev_stateon resume, or from a warm Run that a write-only hint selected. Warming is gated because it boots a sandbox and starts the agent, so it bills like a run. Warm activation goes through_activate_warm_run, notrun_task, so it needed its own check.The check fails closed, unlike the telemetry flags next to it. Only a model listed in
MODEL_ACCESS_FLAGSreaches an evaluation at all, so an outage withholds a preview model from everyone rather than opening it to everyone. This gate decides spend, so that is the safer direction. Ungated models never trigger a flag roundtrip or a user lookup.