feat(ai): deep research tests competing hypotheses in parallel

[sdolidze]

Closes PROD-9230

Summary

Deep Research previously ran one agent loop that could converge on a plausible correlation and present it as a causal explanation. This PR replaces that loop with deterministic phases: a structured planner produces N distinct falsifiable hypotheses, investigators examine each one in parallel with isolated context, and an independent judge compares the evidence and synthesizes the final report through the existing report path.

User question
     │
     ▼
Planner (structured call, forced submitResearchHypotheses)
     │ exactly maxHypotheses hypotheses
     ├──────────────┬──────────────┐
     ▼              ▼              ▼
Investigator 1  Investigator 2   … N     ← Promise.allSettled, isolated context,
     │              │              │        budget slices, shared AbortSignal
     └──────────────┴───────┬──────┘
                            ▼
Judge (compact reports only, forced submitResearchReport on retry)
                            ▼
        Existing report path (chart verification, snapshots, frontend)

How it works

Backend

• AiDeepResearchExecutor — orchestrates plan → parallel investigation → judge. Fan-out is application-driven (Promise.allSettled), never model-driven. Run-level budgets stay hard aggregate ceilings across every phase via shared counters; one AbortSignal reaches every child, so cancellation or budget exhaustion aborts all active investigators and prevents the judge from starting.
• AiAgentExecutionConfig gains a research role (planner / investigator / judge) and parentToolCallId. agentV2 shapes the toolset per role: planner and judge are single-purpose structured calls (only their submission tool, small step caps); investigators keep the agent's full toolset + the run's MCP selection with submitResearchReport swapped for submitInvestigationReport. execution.research === undefined preserves the previous single-loop behavior exactly.
• Structured handoffs follow the discoverFields pattern: zod-validated submission tools fire callbacks (onHypotheses / onReport), so schema enforcement is structural and the executor never parses child transcripts. Schemas cap string/array sizes so a runaway investigator cannot overflow the judge's context.
• Planner/investigator activity persists under parent_tool_call_id sentinels (deep-research:<runUuid>:planner / :<hypothesisId>): rebuilt model history filters them (investigators never see sibling hypotheses; later turns replay only the judge's report), while getToolCallsAndResultsForPrompt gains includeSubagentToolCalls so chart-evidence provenance still sees child rows.
• AiDeepResearchService — AiDeepResearchBudget.maxHypotheses per effort (low 2 / medium 3 / high 4 / xhigh 6); validation requires ≥2 and toolCalls > hypotheses; legacy budget snapshots normalize with a default of 3 and keep stripping legacy limits.
• Failure handling: a failed investigator reaches the judge as an explicit unavailable marker instead of discarding successes; a report submitted before a crash is salvaged; fewer than two completed investigations fails the run with an actionable reason; each investigator and the judge get one forced-submission retry.

Regression analysis

Group	Effect
Standard agent chat	No change — role shaping only activates under execution.mode: 'deep_research' with a research role set.
Deep research runs (new)	Planner → parallel investigators → judge; same report contract, chart verification, progress events (now planning/investigating/synthesizing), cancellation, heartbeat, and stale-run sweep.
Queued runs persisted before deploy	Budget snapshot lacks maxHypotheses → defaults to 3; legacy limits still stripped from API responses.
Graphile durability / MCP credentials	Untouched — same enqueue, claim, retry, and credential-resolution paths.

No migration: budget_snapshot is jsonb; the entity type marks maxHypotheses optional for legacy rows.

Test plan

• pnpm -F common typecheck && pnpm -F backend typecheck && pnpm -F frontend typecheck — clean
• pnpm -F common lint && pnpm -F backend lint — clean; pnpm -F common test — pass
• Deep-research suites — 77 tests: effort→hypotheses mapping (2/3/4/6), deterministic concurrency (deferreds prove every investigator starts before any resolves), judge gating (<2 reports → actionable failure) and judge input (successes + explicit failure markers), aggregate tool-call/warehouse budgets across parallel investigators, cancellation during fan-out (all children aborted, judge never starts), forced-submission retries, report salvage, legacy budget defaulting
• Regression sweep over ee/services/AiDeepResearchService, ee/services/ai, ee/services/AiAgentService, ee/models — 831 tests / 75 files pass
• pnpm generate-api — TSOA resolves the extended budget type (artifacts not committed)
• Manual e2e on a local instance (real Anthropic model, seeded warehouse, effort low): planner produced exactly 2 competing hypotheses; investigators ran interleaved under distinct deep-research:<run>:hypothesis-1/2 parent rows; judge submitted top-level; progress events planning → investigating → synthesizing; final report refused causation on a deliberately spurious correlation ("we cannot establish causation… inconclusive"), compared both hypotheses with per-section confidence tags, carried a failed investigation as an explicit caveat, and listed the missing evidence needed to establish causation

Follow-ups (separate PRs)

• Surface hypothesis/investigation structure in the run UI (effort remains the only user-facing control per the ticket).
• Deep-research coverage in the weekly AI eval workflow (ai-agent-integration-tests.yml has none today).

🤖 Generated with Claude Code

[linear-code]

PROD-9230

[github-actions]

🧪 Test Selection

✅ Tests that will run

Test	Description
Preview Environment	Deploys a preview environment for testing
Frontend E2E Tests	Runs Cypress app tests
Backend API Tests	Runs Vitest API tests
CLI Tests	Runs CLI integration and dbt version tests

⏭️ Tests skipped (no relevant file changes detected)

Test	How to trigger manually
Timezone Tests	Add test-timezone to PR description

Tip: Add test-all to your PR description to run all tests.

[github-actions]

🛡️ Upgrade safety for self-hosted customers

Comparing against main (3e0402aefb).

• ✅ Safe to upgrade normally. No downtime needed.
• No database changes in this release.
• ⚠️ Also: this makes a breaking change to the REST API — anyone running their own scripts or integrations against it may need to update.

What we looked at

What	Result
Database changes	none
REST API	2 breaking changes
MCP tools	no breaking changes
Upgrade notes	none

Technical details (raw JSON)

{
  "schemaVersion": "1",
  "version": "pr-26545",
  "previousVersion": "3e0402aefb",
  "releaseDate": "2026-07-30T12:57:36.577Z",
  "capabilities": [
    "migrations",
    "ai-review",
    "rest",
    "mcp",
    "upgrade"
  ],
  "migrations": {
    "present": false,
    "count": 0,
    "files": [],
    "ee": false
  },
  "compatibility": {
    "rollingUpdateSafe": true,
    "recommendedStrategy": "RollingUpdate",
    "notes": "AI rolling-update review: The only inputs are two REST response changes that add new toolName enum values (submitInvestigationReport, submitResearchHypotheses) to a thread-fetch endpoint. Widening a response enum is backward compatible for the in-flight frontend, which handles unknown tool names via default switch cases and an MCP fallback rather than throwing. No migrations or MCP changes were provided, so the release is rolling-update safe. This marker only reflects the checks listed in `capabilities`. It does NOT detect code-only or config-only breaking changes (env defaults, removed Helm values, serialization/protocol changes), which can also break old pods during a rolling update."
  },
  "api": {
    "rest": {
      "checked": true,
      "breaking": true,
      "changes": [
        "GET /api/v1/projects/{projectUuid}/aiAgents/{agentUuid}/threads/{threadUuid} — added the new `submitInvestigationReport` enum value to the `results/allOf[subschema #2]/messages/items/anyOf[#/components/schemas/AiAgentMessageAssistant]/toolResults/items/allOf[subschema #2]/anyOf[subschema #1]/toolName` response property for the response status `200`",
        "GET /api/v1/projects/{projectUuid}/aiAgents/{agentUuid}/threads/{threadUuid} — added the new `submitResearchHypotheses` enum value to the `results/allOf[subschema #2]/messages/items/anyOf[#/components/schemas/AiAgentMessageAssistant]/toolResults/items/allOf[subschema #2]/anyOf[subschema #1]/toolName` response property for the response status `200`"
      ]
    },
    "mcp": {
      "checked": true,
      "breaking": false,
      "changes": []
    }
  },
  "upgrade": {
    "minPreviousVersion": null,
    "requiredStop": false,
    "note": null,
    "sourceVersion": null,
    "kind": null,
    "requiredStops": []
  }
}

---

_{Automated upgrade-safety check. Once merged, it ships a small release-safety.json with the release so customers’ upgrade automation can read it. It covers database and API changes; it doesn’t yet catch config/env-var or data-format changes.}

[lightdash-bot]

🎉 This PR is included in version 1.42.0 🎉

The release is available on:

• 1.42.0
• GitHub release

Your semantic-release bot 📦🚀