Optimize traversal/query execution and add performance measurement guardrails#6
Merged
Merged
Conversation
dccabe9 to
248739a
Compare
Re-architect SQL compilation into plan/passes/emitter pipeline and split
monolithic Drizzle operations into concern-based modules with strategy
dispatch. Add prepared query support, batch insert/lookup APIs,
compilation caching, and execution adapters with SQLite prepared
statement caching.
Breaking API changes:
- selectAggregate() → aggregate()
- includeImplyingEdges → expand: "none"|"implying"|"inverse"|"all"
- Recursive traversal consolidated into recursive({ minHops, maxHops,
cyclePolicy, path, depth }) replacing chained methods
- EdgeTypeNames/NodeTypeNames → EdgeKinds/NodeKinds (and getter fns)
- Store class now type-only export (use createStore())
- StoreConfig replaced by StoreOptions
- Schema APIs extracted to new @nicia-ai/typegraph/schema entry point
- Removed from main: KindRegistry, Result utilities, date helpers,
validation utilities, compiler/profiler internals
New features:
- param()/prepare()/execute() for zero-recompilation prepared queries
- getByIds(), bulkInsert, bulkCreate, bulkUpsert on collections
- Node find({ where }) with full predicate support
- Bind-limit-aware batch chunking per dialect
Adds new test suites, rewrites benchmark framework with CI
regression guardrails, and adds schema evolution, testing, and query
execution documentation.
248739a to
4553aed
Compare
Merged
pdlug
added a commit
that referenced
this pull request
May 8, 2026
P1 — customizable uniques table for materializeRemovals cleanup: - Extended SqlTableNames to include `uniques`. SQLite + Postgres backends populate it from the dialect-specific schema. Cleanup deletes from the customized table instead of the hardcoded default. P2 #2 — reconciliation watermark for history walk: - New optional backend primitives: ensureReconciliationMarkersTable, getReconciliationMarker, setReconciliationMarker. New table typegraph_reconciliation_markers (graphId PK, reconciledToVersion). - reconcilePendingRemovals reads the marker, walks only transitions newer than it, then writes active.version as the new marker. - Backends without the primitives fall back to walking from version 1 (legacy behavior preserved for custom backends). - A 100-version graph drops 100 round-trips + 100 Zod parses to 0 on steady-state calls. P2 #3 — drop first validateGraphExtension call in mergeGraphExtension: - Single union-validate covers both input shape and cross-document invariants. Genuine evolves now pay one validator walk instead of two. Callers wanting input-precise errors call validateGraphExtension(document) themselves first. P2 #4 — per-kind compile cache: - WeakMap<ExtensionNodeDef, CompiledNode> scoped per kindName outer Map. - WeakMap<ExtensionEdgeDef, ZodObject> for edge schemas (from/to resolution stays per-call since it depends on the broader compile context's nodeTypeByName). - Partial-overlap evolves reuse compiled output for unchanged kinds. P2 #5 — strict mode threaded into index validation: - validateIndexEntry now accepts strict and applies rejectUnknownObjectKeys with NODE_INDEX_KEYS or EDGE_INDEX_KEYS allowlist. - Typos like `coveringField` (singular) now surface as INVALID_INDEX_DECLARATION instead of compiling to a weaker index. P2 #6 — vector-index signature uses actual embeddings table: - materialize-indexes.ts now reads backend.tableNames?.embeddings for the signature input. Fixes false drift detection on backends with custom embeddings table names. Test impact: custom-table-names.test.ts updated for the new `uniques` slot in SqlTableNames. All 3060 SQLite + 588 Postgres tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-architects the SQL compiler and Drizzle backend into modular, dialect-aware layers while delivering performance improvements across query compilation, execution, traversal, and ingestion paths. Simplifies and tightens the public API surface, adds prepared query support, and introduces benchmark regression guardrails in CI.
Breaking API Changes
selectAggregate(...)renamed toaggregate(...)includeImplyingEdgesreplaced withexpand: "none" | "implying" | "inverse" | "all"(default:"inverse").maxHops(),.minHops(),.collectPath(),.withDepth()) consolidated intorecursive({ minHops, maxHops, cyclePolicy, path, depth })cyclePolicy: "prevent" | "allow"(default:"prevent")MAX_RECURSIVE_DEPTH = 100; explicitmaxHopsvalidated up toMAX_EXPLICIT_RECURSIVE_DEPTH = 1000EdgeTypeNames/NodeTypeNamesrenamed toEdgeKinds/NodeKinds;getEdgeTypeNames()/getNodeTypeNames()renamed togetEdgeKinds()/getNodeKinds()Storeclass exported as type-only (preventsnew Store(), preservesStore<G>annotations)StoreConfigreplaced byStoreOptions(addsqueryDefaults.traversalExpansion)Public API Surface Reduction
Schema management extracted to new
@nicia-ai/typegraph/schemasub-export:serializeSchema,deserializeSchema,computeSchemaHash,initializeSchema,ensureSchema,migrateSchema,computeSchemaDiff,getMigrationActions,isBackwardsCompatible, all serialized types, and validation utilitiesRemoved from main entry point:
QueryAst,SetOperation,ValueType,VariableLengthSpec, logical plan types, predicate/temporal/recursive compiler functionsgetDialect,DialectAdapter(keptSqlDialectas type-only)extractPropertyAccesses,ProfileCollector,keyToPath/pathToKey,generateRecommendations,getUnindexedFiltersKindRegistry/buildKindRegistryok/err/isOk/isErr/unwrap/unwrapOr)encodeDate/decodeDatecreateValidationError,validateProps,wrapZodError)New Features
Prepared Queries
param(name)placeholders for scalar predicate positions.prepare()precompiles AST → SQL text once;prepared.execute(bindings)runs with zero recompilationCollection API Additions
getByIds(ids)on node/edge collections — batchedSELECT ... WHERE id IN (...), preserves input order, returnsundefinedfor missing IDsbulkInsert— void-returning fire-and-forget ingestionbulkCreate— multi-rowINSERT ... RETURNINGinstead of per-item insertsbulkUpsert(edges) — batchgetNodeslookup instead of N+1 sequential callsfind({ where })— delegates to query builder with full predicate supportcount()now acceptsQueryOptions(temporal mode, asOf)Store & Type Additions
StoreHooks,HookContext,OperationHookContext,QueryHookContextTypedEdgeCollection,TypedNodeReftype exportsCompilerInvariantError,DatabaseOperationErrorflatMap,map,mapErr,orElseCompiler Architecture
SQL compilation restructured into a three-stage pipeline:
plan/— Intermediate representation with 9 plan node types (Scan, Filter, Join, Aggregate, Sort, Limit, VectorKnn, RecursiveExpand, SetOp); lowering functions for standard, recursive, and set-operation queriespasses/— Multi-pass optimization framework with temporal, recursive, and vector passesemitter/— SQL generation from logical plans;plan-inspector.tsvalidates plan shape before emission;standard-builders.tsextracts CTE/projection/clause constructionstandard-pass-pipeline.ts— Orchestrates passes, predicate indexing, and column pruningCompiler introduces predicate pre-indexing, column pruning, and compilation caching to reduce per-query overhead.
Backend Architecture
Backend refactored from monolithic
operations.tsinto concern-based modules with dialect-polymorphic strategy dispatch..Benchmarks
Complete rewrite of
@nicia-ai/typegraph-benchmarks— replaced tinybench with a custom measurement framework:cli.ts,config.ts,backend.ts,measurements.ts,guardrails.ts,seed.tsperf:checkandperf:check:postgressteps in GitHub ActionsTesting
Expanded test coverage for existing public APIs and added extensive new tests for the new APIs and backend refactor
Documentation
New pages: Schema Evolution guide, Testing guide, Query Execution reference
Major expansions: Performance overview (+411 lines), recursive queries, predicates reference, schemas & stores, traversal docs
Updated for new APIs:
aggregate(), traversalexpandmodes, recursive options,bulkInsert,bulkCreate,getByIds, nodefind({ where }), prepared queries