feat(vnext): add incremental statement indexing

[Light2Dark]

Summary

• update the dialect-aware statement index incrementally from trusted analysis-coordinate edits while preserving the full scan as the oracle
• add a private lazy per-session index cache with atomic invalidation/reuse rules
• replace structural dialect definitions with opaque built-in factory handles, keeping lexical profiles and indexes out of the public API

Correctness and safety

• conservative restart at the old slot at or left of the earliest edit
• convergence only at an exact terminated boundary after the final edit
• exact full-oracle fallback for inconsistent metadata or absent convergence
• global 10,000-slot and opaque-suffix behavior preserved
• monotonic convergence cursor keeps resource-cap recovery linear
• no source text or history retained by the cache

Validation

• 833 unit tests passed; 1 governed expected failure
• 33 incremental tests with deterministic differential sequences across all four profiles
• 80,000 additional adversarial differential edits matched the full oracle
• changed coverage: 98.78% statements, 97.00% branches, 100% functions, 98.77% lines
• typecheck, oxlint, integrity, browser, demo, and packed-package smoke passed
• 1 MiB benchmark: full scan about 2.2-2.5 ms, middle edit about 0.028 ms, prefix shift about 0.11 ms
• resource-cap recovery about 1.35-1.46 ms after fixing the reviewed quadratic path

Review

Two independent exact-SHA reviewers approve 2e83d17 with no unresolved blocker, high, or medium findings.

---

Summary by cubic

Adds incremental statement indexing with a lazy, per-session cache so edits update indexes in microseconds, while a full scan remains the correctness fallback. Replaces caller-defined dialect objects with built-in opaque dialect handles that bind lexical behavior internally.

• New Features

  • Incremental updates from trusted analysis-coordinate edits; reuse unchanged prefix and reuse/shift suffix at exact terminated boundaries, with full-oracle fallback.
  • Private per-session statement-index cache: reused on context-only updates (same profile) and same-text replacements; incrementally updated on trusted identity-source changes; cleared on profile change or changed replacements; no source text or history retained.
  • Built-in dialect factories return frozen opaque singletons: duckdbDialect(), postgresDialect(), bigQueryDialect(), dremioDialect(); handles are validated, local to the package instance, and don’t cross workers/processes.
  • Bench and tests: pnpm run bench:statement-index; extensive unit, randomized differential, and session cache tests.

• Migration

  • Replace defineSqlDialect(...) and SqlDialectDefinition with built-in handles and SqlDialect.
    • Before: createSqlLanguageService({ dialects: [defineSqlDialect({ id: "duckdb", displayName: "DuckDB" })] })
    • After: createSqlLanguageService({ dialects: [duckdbDialect()] })
  • Keep using the handle’s .id string in document context: { dialect: dialect.id }.
  • Do not copy/serialize/fabricate dialects; pass authentic handles from the same @marimo-team/codemirror-sql/vnext instance (handles don’t cross workers/processes).

^{Written for commit ce94176. Summary will update on new commits.}

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	98.77% (🎯 91%)	727 / 736
🔵	Statements	98.78% (🎯 91%)	730 / 739
🔵	Functions	100% (🎯 90%)	79 / 79
🔵	Branches	97% (🎯 85%)	551 / 568

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
src/vnext/index.ts	0%	0%	0%	0%
src/vnext/session.ts	98.8%	97.16%	100%	98.79%	223-226, 249-252, 414, 457
src/vnext/statement-index.ts	98.72%	96.91%	100%	98.72%	475, 1019, 1040, 1128, 1179
src/vnext/types.ts	100%	100%	100%	100%

Generated in workflow #683 for commit ce94176 by the Vitest Coverage Report Action

[Copilot]

Pull request overview

This PR adds an incremental SQL statement-index update path (while keeping the full scan as the correctness oracle), introduces a lazy per-session statement-index cache with explicit invalidation/reuse rules, and updates the vNext dialect API to use opaque built-in factory handles rather than structural caller-defined dialect objects.

Changes:

• Add updateSqlStatementIndex plus supporting utilities to incrementally rescan from a conservative restart point and opportunistically reuse an unchanged suffix.
• Add a private, lazy statement-index cache to sessions and update it incrementally for trusted identity-source edits; invalidate on profile changes or non-trusted document mutations.
• Replace public structural dialect definitions with built-in opaque dialect singleton factories (duckdbDialect(), etc.) and reject fabricated/copied handles.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/vnext-types/session.test-d.ts	Updates type-level API expectations (opaque dialects; statement index remains non-public in types).
src/vnext/types.ts	Introduces SqlDialect as an opaque branded handle and updates service options to accept SqlDialect[].
src/vnext/statement-index.ts	Refactors scanning into a reusable core and adds updateSqlStatementIndex for incremental updates + suffix reuse.
src/vnext/session.ts	Adds built-in dialect factories + runtime metadata resolution; adds per-session statement-index cache and incremental updating.
src/vnext/index.ts	Exposes built-in dialect factories and updates exported types.
src/vnext/tests/statement-index.bench.ts	Adds benchmarks for full scans, incremental edits, prefix shifts, and resource-cap recovery.
src/vnext/tests/session.test.ts	Updates dialect tests for opaque handles and adds coverage for session statement-index caching/incremental behavior.
src/vnext/tests/incremental-statement-index.test.ts	Adds extensive differential and randomized tests to validate incremental updates against the full oracle.
SQL_EDITOR_RESEARCH.md	Updates research notes to reflect the opaque handle approach for stable API design.
scripts/package-smoke.mjs	Updates smoke tests/consumer snippets to use built-in dialect factories.
package.json	Adds a bench:statement-index script to run the new Vitest benchmark.
docs/vnext/statement-index.md	Documents incremental update behavior and session cache semantics.
docs/vnext/source-coordinates.md	Documents analysis-text/value-based reuse rules and constraints for transformed sources.
docs/vnext/session-primitives.md	Updates examples and documents dialect-handle registration semantics and cache behavior.
docs/adr/0001-language-service-and-session.md	Updates ADR to reflect opaque dialect handles and incremental statement-index/cache behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cubic-dev-ai]

All reported issues were addressed across 15 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>