When the playground needs the newest revision of several workflows at once, one request to POST /workflows/revisions/query blocks an API worker for about two seconds. Several of those together stall the API long enough that unrelated requests time out. A user sees an agent turn fail with session <id> record log is unreadable; cannot rebuild the conversation, even though the record log is fine.
Steps to reproduce
- Open a project with several workflows, where at least one workflow has many revisions.
- Open the playground on a session and let the UI load the workflow list.
- Watch the API access log and the CPU of the API worker.
What happens
windowing.limit applies to the whole result set, not to each parent. A limit of 5 over 5 workflows can return 5 revisions of the first workflow and none of the others. The frontend has no safe way to ask for "the latest one each", so it drops the limit and takes everything. The comment in web/packages/agenta-entities/src/workflow/api/api.ts:1293 says so:
When fetching for a single workflow, limit to 1 (latest) to reduce payload.
With multiple workflows the global limit would cut across all, so skip it.
On the project measured, 19 workflows hold 269 revisions, and two of them hold 179 revisions and 30 MB between them. A batch over those two returns 177 revisions.
Measured inside the API container on that real data:
| Stage |
Cost |
Build models, including jsonschema.check_schema |
385 ms |
model_dump |
146 ms |
| Serialize to JSON |
92 ms, giving 76.4 MB |
| gzip at level 5 |
1293 ms, giving 18.9 MB |
| Total, none of it yielding to the event loop |
1916 ms |
Evidence from the incident
Session f5cf97a3-7e1a-46f0-a4f4-5f0a45946834 on the agenta-oss-team stack, images v0.114.5, 2026-09-03.
The runner asked for the session record log at 12:00:57.79 and gave up at 12:01:02.81, after its five second budget:
[sessions/records-query] query FAILED session=f5cf97a3-...: The operation was aborted due to timeout
[keepalive] evict key=...:f5cf97a3-... reason=no-park:failed
The API served no response at all between 12:00:59.6 and 12:01:05.7, then answered in a burst. Using the per-minute scheduler request as a traffic-independent probe, the API's latency was 0.01 s median over 983 ticks in 17 hours, with only two ticks above one second: 1.62 s at 11:40:00 and 4.90 s at 12:01:00. The second is this failure. Seven calls to /api/workflows/revisions/query landed in the 11:40 minute.
While the loop was blocked, the cache client hit its 0.5 s socket timeout, which is the visible warning:
[WARN.] [cache] GET namespace=check_action_access key={'permission': 'view_sessions', ...}
[WARN.] Timeout reading from redis-volatile:6379
Redis itself is healthy: 5 MB in use of 512 MB, no evictions, slowest command 73 ms. The database query is not the problem either: an EXPLAIN ANALYZE of the record log read runs in 4.7 ms.
A second, silent instance of the same gap
web/packages/agenta-entities/src/testset/api/api.ts:193 reaches for the missing capability a different way:
testset_refs: testsetIds.map((id) => ({id, limit: 1})),The API drops that limit. Reference is class Reference(Identifier, Slug, Version) with no model_config, so pydantic ignores extra keys. Checked on the running API:
Reference(**{"id": "...", "limit": 1})
-> version=None slug=None id=UUID('...')
-> dumped: {'id': UUID('...')}
So that batch fetches every revision of every testset. Testsets are small on the project measured, so nothing hurts today, but the defect is latent.
Scope
Six routers expose POST /<entity>/revisions/query with the same request shape, and all six reach GitDAO.query_revisions: workflows, testsets, evaluators, environments, applications, and queries. The fix belongs in the shared layer, and the request field should land on all six together.
Plan
The design workspace is at docs/design/revision-query-grouping/, covering the reasoning,
the measurements, the interface, and the shapes we rejected. It was reviewed by Codex and
the plan changed as a result.
Interface: one optional field, a sibling of windowing, on all six request models.
The first release returns at most one revision per requested parent. It carries no
"newest N per parent" option and no paging over parents. Both were cut because no caller
needs them and each adds a branch we cannot specify yet.
Do not land the field before the behavior
The workflows and environments routers read await request.json() and expand it into a
parser with an explicit keyword signature, inside a bare except Exception: pass. Verified
on the running API:
parse_workflow_revision_query_request_from_body(**{"workflow_refs": [...], "grouping": {...}})
-> TypeError: got an unexpected keyword argument 'grouping'
The router swallows that and leaves the body None, so the parent references are discarded
and the query becomes project-wide. Shipping the field before the parsers accept it would
make this incident worse, not better.
Two related costs found in the same investigation
Separate from this issue, and both reduced by shrinking the payload.
WorkflowRevisionData._validate runs jsonschema.check_schema every time a model is built, so every read re-checks a schema that was already validated at write time. 2.2 ms per revision.api/entrypoints/routers.py:503 adds GZipMiddleware(minimum_size=1000, compresslevel=5), which compresses synchronously on the event loop. 1293 ms on the 76 MB body, about two thirds of the total. This may be the more urgent of the two: the OSS nginx config already sets gzip on for application/json, so in that deployment the API duplicates the proxy's work. The stack where this incident happened runs Traefik rather than nginx, so each hosting path needs checking before the middleware is removed.
Fixed already
One correctness bug found while investigating this, now fixed in
web/packages/agenta-entities/src/testset/api/api.ts. fetchLatestRevisionsBatch sent no windowing, and query_revisions applies an ORDER BY only when windowing is present. With no ordering, the rows came back in unspecified order and the loop kept whichever revision arrived last, so a function named "latest" returned an arbitrary revision. It now asks for newest first, keeps the first revision per testset, and prefers a configured revision over a version 0 placeholder. The dead limit and the docstring describing a ReferenceWithLimit feature that never existed are gone. This does not stop the over-fetch, which needs the API change above.
When the playground needs the newest revision of several workflows at once, one request to
POST /workflows/revisions/queryblocks an API worker for about two seconds. Several of those together stall the API long enough that unrelated requests time out. A user sees an agent turn fail withsession <id> record log is unreadable; cannot rebuild the conversation, even though the record log is fine.Steps to reproduce
What happens
windowing.limitapplies to the whole result set, not to each parent. A limit of 5 over 5 workflows can return 5 revisions of the first workflow and none of the others. The frontend has no safe way to ask for "the latest one each", so it drops the limit and takes everything. The comment inweb/packages/agenta-entities/src/workflow/api/api.ts:1293says so:On the project measured, 19 workflows hold 269 revisions, and two of them hold 179 revisions and 30 MB between them. A batch over those two returns 177 revisions.
Measured inside the API container on that real data:
jsonschema.check_schemamodel_dumpEvidence from the incident
Session
f5cf97a3-7e1a-46f0-a4f4-5f0a45946834on theagenta-oss-teamstack, imagesv0.114.5, 2026-09-03.The runner asked for the session record log at 12:00:57.79 and gave up at 12:01:02.81, after its five second budget:
The API served no response at all between 12:00:59.6 and 12:01:05.7, then answered in a burst. Using the per-minute scheduler request as a traffic-independent probe, the API's latency was 0.01 s median over 983 ticks in 17 hours, with only two ticks above one second: 1.62 s at 11:40:00 and 4.90 s at 12:01:00. The second is this failure. Seven calls to
/api/workflows/revisions/querylanded in the 11:40 minute.While the loop was blocked, the cache client hit its 0.5 s socket timeout, which is the visible warning:
Redis itself is healthy: 5 MB in use of 512 MB, no evictions, slowest command 73 ms. The database query is not the problem either: an
EXPLAIN ANALYZEof the record log read runs in 4.7 ms.A second, silent instance of the same gap
web/packages/agenta-entities/src/testset/api/api.ts:193reaches for the missing capability a different way:The API drops that
limit.Referenceisclass Reference(Identifier, Slug, Version)with nomodel_config, so pydantic ignores extra keys. Checked on the running API:So that batch fetches every revision of every testset. Testsets are small on the project measured, so nothing hurts today, but the defect is latent.
Scope
Six routers expose
POST /<entity>/revisions/querywith the same request shape, and all six reachGitDAO.query_revisions: workflows, testsets, evaluators, environments, applications, and queries. The fix belongs in the shared layer, and the request field should land on all six together.Plan
The design workspace is at
docs/design/revision-query-grouping/, covering the reasoning,the measurements, the interface, and the shapes we rejected. It was reviewed by Codex and
the plan changed as a result.
Interface: one optional field, a sibling of
windowing, on all six request models.{ "workflow_refs": [{"id": "..."}, {"id": "..."}], "grouping": { "by": "artifact" } }The first release returns at most one revision per requested parent. It carries no
"newest N per parent" option and no paging over parents. Both were cut because no caller
needs them and each adds a branch we cannot specify yet.
auto-created placeholder or a real configured revision. Also settle whether grouping
means "newest matching revision" or "newest revision, returned only if it matches".
RevisionGroupingto the git core DTOs andgroupingto all six request models.groupingreturn a client error. See the warning below.
GitDAOInterface.query_revisions, the concrete DAO, and the six services, andapply the fold in SQL with
DISTINCT ONplus an outer ordering stage.web/oss/src/state/entities/testset/revisionEntity.ts.grouping.Do not land the field before the behavior
The workflows and environments routers read
await request.json()and expand it into aparser with an explicit keyword signature, inside a bare
except Exception: pass. Verifiedon the running API:
The router swallows that and leaves the body
None, so the parent references are discardedand the query becomes project-wide. Shipping the field before the parsers accept it would
make this incident worse, not better.
Two related costs found in the same investigation
Separate from this issue, and both reduced by shrinking the payload.
WorkflowRevisionData._validaterunsjsonschema.check_schemaevery time a model is built, so every read re-checks a schema that was already validated at write time. 2.2 ms per revision.api/entrypoints/routers.py:503addsGZipMiddleware(minimum_size=1000, compresslevel=5), which compresses synchronously on the event loop. 1293 ms on the 76 MB body, about two thirds of the total. This may be the more urgent of the two: the OSS nginx config already setsgzip onforapplication/json, so in that deployment the API duplicates the proxy's work. The stack where this incident happened runs Traefik rather than nginx, so each hosting path needs checking before the middleware is removed.Fixed already
One correctness bug found while investigating this, now fixed in
web/packages/agenta-entities/src/testset/api/api.ts.fetchLatestRevisionsBatchsent nowindowing, andquery_revisionsapplies anORDER BYonly when windowing is present. With no ordering, the rows came back in unspecified order and the loop kept whichever revision arrived last, so a function named "latest" returned an arbitrary revision. It now asks for newest first, keeps the first revision per testset, and prefers a configured revision over a version 0 placeholder. The deadlimitand the docstring describing aReferenceWithLimitfeature that never existed are gone. This does not stop the over-fetch, which needs the API change above.