Skip to content

Bulk storyboard validation needs backoff or concurrency control #1994

@bokelley

Description

@bokelley

Problem

Running all 21 storyboards sequentially against the training agent causes rate limit cascading. The creative_ad_server storyboard fails because earlier storyboards consume the rate limit budget.

The per-storyboard scenario filtering (#1985, #433) solved the single-storyboard case — each evaluation now runs 2-5 MCP calls instead of 30+. But bulk validation (running all storyboards against one agent) still accumulates calls and hits limits.

Current behavior:

  • Each storyboard evaluation runs its own comply() call
  • No delay between storyboards
  • Agent rate limits (~20 calls) are hit by storyboard 5-6
  • Later storyboards get 405/429 responses and cascade-fail

Options:

  1. Sequential with backoff: Add delay between storyboard evaluations when running bulk validation
  2. Concurrency limit: Run at most N storyboards concurrently
  3. Shared session context: Deduplicate discovery calls (get_products, list_creative_formats) across storyboards that share the same agent
  4. Agent-side rate limit headers: Respect Retry-After headers from the agent

Option 3 aligns with the long-term goal from #433 (option 4: deduplicate discovery).

Context

Discovered during storyboard validation (#1985). The creative_ad_server storyboard consistently fails when run after other storyboards but passes when run standalone.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions