feat(ai): persist Deep Research run metrics

[sdolidze]

Closes: https://linear.app/lightdash/issue/PROD-9376/persist-and-emit-complete-deep-research-run-metrics

Summary

Persists an authoritative operational-metrics snapshot for every terminal Deep Research run, exposes it through the API, and emits completion analytics from the same stored values.

Token classes now accumulate after every model step, so failed, cancelled, and pre-judge runs retain usage instead of depending on a final in-memory total.

How it works

Backend

model step ──► normalize usage ──► atomic run-row increment
                                         │
terminal transition ──► derive counts ───┼──► run snapshot
                                         └──► transactional analytics outbox

• agentV2 emits one ai.usage event per model step and attaches Deep Research run and phase provenance.
• AiDeepResearchRunModel atomically accumulates nullable token classes and permanently marks incomplete provider telemetry.
• Terminal transitions persist duration, attempted calls, deduplicated errors, warehouse queries, findings, and charts before inserting the exactly-once outbox event.
• The Deep Research API and completion analytics map the persisted snapshot without reconstructing values from prompt provenance.

Regression analysis

Group	Effect
New Deep Research runs	Receive incremental token usage and a complete terminal operational snapshot.
Legacy Deep Research rows	Return nullable metric fields without backfill or inferred values.
Standard agent runs	Continue emitting ai.usage, now once per completed model step instead of one aggregate event.
Failed or schema-invalid tool attempts	Count once as attempted calls and errors before terminal metrics are derived.

No authorization, tenant-scoping, report-generation, or retention behavior changes.

Migration

pnpm -F backend migrate

Adds 13 nullable columns to ai_deep_research_runs; the additive schema keeps existing rows backward-compatible.

Test plan

• Focused backend unit suite — 6 files and 124 tests pass.
• Common suite — 120 files and 3,225 tests pass with 1 existing skip.
• AiDeepResearchRunModel Postgres integration — 12 tests pass against an isolated cloned database.
• Backend/common typechecks and linters pass with only unrelated existing warnings.
• pnpm generate-api is stable and git diff --check passes.
• Development Postgres exposes all 13 nullable metric columns.

Example payloads

AiDeepResearchRun.metrics after completion:

{
  "durationMs": 42000,
  "inputTokens": 1200,
  "outputTokens": 480,
  "cacheReadTokens": 600,
  "cacheWriteTokens": 0,
  "reasoningTokens": 140,
  "totalTokens": 2420,
  "tokenUsageComplete": true,
  "toolCallCount": 8,
  "toolErrorCount": 1,
  "warehouseQueryCount": 3,
  "findingsCount": 4,
  "chartCount": 2
}

[linear-code]

PROD-9376

[github-actions]

🧪 Test Selection

✅ Tests that will run

Test	Description
Preview Environment	Deploys a preview environment for testing
Frontend E2E Tests	Runs Cypress app tests
Backend API Tests	Runs Vitest API tests
CLI Tests	Runs CLI integration and dbt version tests

⏭️ Tests skipped (no relevant file changes detected)

Test	How to trigger manually
Timezone Tests	Add test-timezone to PR description

Tip: Add test-all to your PR description to run all tests.

[github-actions]

🛡️ Upgrade safety for self-hosted customers

Comparing against main (c6f9ba024d).

• ✅ Safe to upgrade normally. No downtime needed.
• This changes the database, and the old version keeps working with those changes through the upgrade.

What we looked at

What	Result
Database changes	1 migration (incl. enterprise)
REST API	no breaking changes
MCP tools	no breaking changes
Upgrade notes	none

Technical details (raw JSON)

{
  "schemaVersion": "1",
  "version": "pr-26615",
  "previousVersion": "c6f9ba024d",
  "releaseDate": "2026-07-31T11:47:18.016Z",
  "capabilities": [
    "migrations",
    "sql-lint",
    "ai-review",
    "rest",
    "mcp",
    "upgrade"
  ],
  "migrations": {
    "present": true,
    "count": 1,
    "files": [
      "20260731120000_add_deep_research_run_metrics.ts"
    ],
    "ee": true
  },
  "compatibility": {
    "rollingUpdateSafe": true,
    "recommendedStrategy": "RollingUpdate",
    "notes": "AI rolling-update review: The single migration adds thirteen nullable metric columns to ai_deep_research_runs with no defaults, constraints, or backfill. The previous release's running code continues to insert/read without issue, and the new columns are only referenced by new-release code. Fully additive and rolling-update safe. This marker only reflects the checks listed in `capabilities`. It does NOT detect code-only or config-only breaking changes (env defaults, removed Helm values, serialization/protocol changes), which can also break old pods during a rolling update."
  },
  "api": {
    "rest": {
      "checked": true,
      "breaking": false,
      "changes": []
    },
    "mcp": {
      "checked": true,
      "breaking": false,
      "changes": []
    }
  },
  "upgrade": {
    "minPreviousVersion": null,
    "requiredStop": false,
    "note": null,
    "sourceVersion": null,
    "kind": null,
    "requiredStops": []
  }
}

---

_{Automated upgrade-safety check. Once merged, it ships a small release-safety.json with the release so customers’ upgrade automation can read it. It covers database and API changes; it doesn’t yet catch config/env-var or data-format changes.}

[github-actions]

Your preview environment pr-26615 has been deployed.

Preview environment endpoints are available at:

• https://lightdash-pr-26615.lightdash.okteto.dev
• https://lightdash-preview-pr-26615.lightdash.okteto.dev

[github-actions]

Preview Environment

🌐 URL: https://lightdash-preview-pr-26615.lightdash.okteto.dev

📋 Logs: View in GCP Console

🔧 SSH: ./scripts/okteto-ssh.sh 26615

[lightdash-bot]

🎉 This PR is included in version 1.55.0 🎉

The release is available on:

• 1.55.0
• GitHub release

Your semantic-release bot 📦🚀