-
Notifications
You must be signed in to change notification settings - Fork 43
Bulk storyboard validation needs backoff or concurrency control #1994
Copy link
Copy link
Closed
Description
Problem
Running all 21 storyboards sequentially against the training agent causes rate limit cascading. The creative_ad_server storyboard fails because earlier storyboards consume the rate limit budget.
The per-storyboard scenario filtering (#1985, #433) solved the single-storyboard case — each evaluation now runs 2-5 MCP calls instead of 30+. But bulk validation (running all storyboards against one agent) still accumulates calls and hits limits.
Current behavior:
- Each storyboard evaluation runs its own comply() call
- No delay between storyboards
- Agent rate limits (~20 calls) are hit by storyboard 5-6
- Later storyboards get 405/429 responses and cascade-fail
Options:
- Sequential with backoff: Add delay between storyboard evaluations when running bulk validation
- Concurrency limit: Run at most N storyboards concurrently
- Shared session context: Deduplicate discovery calls (get_products, list_creative_formats) across storyboards that share the same agent
- Agent-side rate limit headers: Respect Retry-After headers from the agent
Option 3 aligns with the long-term goal from #433 (option 4: deduplicate discovery).
Context
Discovered during storyboard validation (#1985). The creative_ad_server storyboard consistently fails when run after other storyboards but passes when run standalone.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels