Skip to content

Release: IR pipeline, SPARQL conversion layer, and agent workflow#15

Merged
flyon merged 78 commits intomainfrom
dev
Mar 2, 2026
Merged

Release: IR pipeline, SPARQL conversion layer, and agent workflow#15
flyon merged 78 commits intomainfrom
dev

Conversation

@flyon
Copy link
Member

@flyon flyon commented Mar 2, 2026

Summary

This PR brings dev into main with three major feature areas developed across multiple PRs:

1. Canonical Intermediate Representation (IR) pipeline

Replaced query internals with a backend-agnostic IR AST. All DSL queries now compile through a three-stage pipeline (desugar → canonicalize → lower) into canonical IR objects that any store can consume.

  • getCanonicalIR() exposed on select factories
  • Canonical mutation IR for create/update/delete
  • IR documentation and golden test coverage

2. SPARQL conversion layer with SparqlStore base class

Full SPARQL pipeline: IR → SPARQL algebra → SPARQL string → execute → map results back to typed DSL objects.

New exports from @_linked/core/sparql:

  • SparqlStore — abstract base class for SPARQL-backed stores. Extend it and implement two methods:

    import {SparqlStore} from '@_linked/core/sparql';
    
    class MyStore extends SparqlStore {
      protected async executeSparqlSelect(sparql: string): Promise<SparqlJsonResults> { /* ... */ }
      protected async executeSparqlUpdate(sparql: string): Promise<void> { /* ... */ }
    }
  • IR → SPARQL string (full pipeline in one call): selectToSparql, createToSparql, updateToSparql, deleteToSparql

  • IR → SPARQL algebra (for stores that want to inspect/optimize before serialization): selectToAlgebra, createToAlgebra, updateToAlgebra, deleteToAlgebra

  • Algebra → string serialization: selectPlanToSparql, insertDataPlanToSparql, deleteInsertPlanToSparql, deleteWherePlanToSparql, plus serializeAlgebraNode, serializeExpression, serializeTerm

  • Result mapping: mapSparqlSelectResult, mapSparqlCreateResult, mapSparqlUpdateResult — handles flat/nested/aggregated results with XSD type coercion

  • All algebra types re-exported: SparqlTerm, SparqlTriple, SparqlAlgebraNode, SparqlExpression, SparqlSelectPlan, SparqlPlan, SparqlOptions, etc.

Key design decisions:

  • SPARQL algebra aligned with SPARQL 1.1 spec §18 — inspectable/optimizable data structure before serialization
  • Property triples wrapped in OPTIONAL (LeftJoin) so missing values produce nulls, not row elimination
  • VariableRegistry with collision detection for SPARQL variable management
  • NestingDescriptor tree for reconstructing nested result objects from flat SPARQL bindings
  • XSD/RDF constants imported from ontology modules (no hardcoded URIs)

Bug fix: isNodeReference() in MutationQuery.ts — nested creates with predefined IDs now correctly insert entity data.

3. Agent workflow skills

Structured mode-based workflow for agent-assisted development: ideation → plan → tasks → implementation → review → wrapup.

Documentation

  • SPARQL Algebra Layer — full type reference, conversion rules, store implementation guide
  • Intermediate Representation — updated with SparqlStore reference
  • README — pipeline walkthrough, type inference highlight, SparqlStore section
  • Ideation docs for future work: named graphs, computed expressions, advanced query patterns

Test coverage

  • ~200 unit tests: algebra, serialization, result mapping, golden SPARQL comparison, negative/edge cases
  • 80 Fuseki integration tests: all query types end-to-end through SparqlStore (requires Docker)
  • All existing tests pass (394 total non-Fuseki)

Changesets

  • major: first major release
  • minor: canonical IR pipeline
  • minor: selectAll(), property shape deduplication, override guards
  • minor: SPARQL conversion layer (SparqlStore, all conversion/mapping functions, algebra types)

Test plan

  • npx tsc --noEmit passes
  • All unit tests pass (394 passing)
  • All Fuseki integration tests pass (80/80, requires Docker)
  • Documentation reviewed and consistent with implementation

🤖 Generated with Claude Code

flyon and others added 30 commits February 23, 2026 15:32
Refactored intermediate representation

Enhance Jest configuration and documentation for Intermediate Representation

- Added new test patterns for Intermediate Representation related tests in jest.config.js.
- Updated README to include a section on Intermediate Representation with a link to the documentation.
Refactored intermediate representation

Enhance Jest configuration and documentation for Intermediate Representation

- Added new test patterns for Intermediate Representation related tests in jest.config.js.
- Updated README to include a section on Intermediate Representation with a link to the documentation.
Phase 0: plan document covering 8 phases to complete the IR refactor —
expand desugar, lift to full AST, test parity, production wiring,
cleanup, and documentation sync.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Added 6 new desugared types (DesugaredCountStep, DesugaredTypeCastStep,
DesugaredSubSelect, DesugaredCustomObjectSelect, DesugaredEvaluationSelect,
DesugaredMultiSelection) to handle sub-selects, custom result objects, type
casting, preload, and count/aggregation patterns. Expanded desugar tests from
3 to 35 covering all query patterns in query.test.ts. Updated downstream
consumers (IRPipeline, IRProjection) for new union types.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract shared QueryCaptureStore to src/test-helpers/query-capture-store.ts,
merge ir-pipeline-parity tests into ir-select-golden, document pipeline
architecture in documentation/intermediate-representation.md.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite IR docs with comprehensive examples for all variant types,
add migration section to README, mark docs 002/003 as superseded.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…/DeleteQuery are now IR types

Old interfaces renamed to LegacySelectQuery/LegacyCreateQuery/LegacyUpdateQuery/LegacyDeleteQuery.
IQuadStore and LinkedStorage now import canonical query types from their query files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…Object/getIR

All four query factories now expose build() as the primary method.
getQueryObject() and getIR() are deprecated aliases. QueryParser and
internal callers (getPropertyPath, isValidResult) updated.
getLegacyQueryObject() remains internally for pipeline input (Phase 11)
and test helpers (Phase 12).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…peline

Define RawSelectInput as the internal pipeline input type capturing exactly
what the desugar pass needs. SelectQueryFactory.build() now constructs
RawSelectInput directly from factory state, bypassing getLegacyQueryObject().
IRDesugar.ts and IRPipeline.ts no longer import LegacySelectQuery.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ery.test.ts

- Delete getLegacyQueryObject() from all four factories
- Add toRawInput() on SelectQueryFactory for pipeline/test access
- Delete LegacySelectQuery, LegacyCreateQuery, LegacyUpdateQuery, LegacyDeleteQuery
- Define mutation input types locally in IRMutation.ts
- Update query-capture-store to use toRawInput() for select, build() for mutations
- Simplify ir-mutation-parity.test.ts (capture store now returns IR directly)
- Update core-utils.test.ts to use toRawInput()
- Delete query.test.ts (superseded by ir-select-golden.test.ts)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…SelectQuery, update docs

- Delete SelectQueryIR type alias from IRPipeline.ts
- Rename buildSelectQueryIR to buildSelectQuery
- Update ir-select-golden.test.ts to use SelectQuery type and build()
- Update documentation/intermediate-representation.md for current pipeline
- Update README.md migration section for build() API

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ion type discriminators

- Delete getQueryObject() from all 4 factories and QueryFactory base class
- Delete getIR() from SelectQueryFactory
- Delete LinkedQuery interface (no longer needed)
- Remove dead buildCanonicalMutationIR dispatcher and CanonicalMutationIR type
- Remove type:'create'|'update'|'delete' discriminators from mutation input types
- Fix as any in BoundComponent.getPropertyPath() with proper Array.isArray narrowing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…p redundant structure

- Top-level kinds: select_query→select, create_mutation→create, update_mutation→update, delete_mutation→delete
- Graph pattern: exists_pattern→exists
- Flatten IRShapeRef {shapeId: string} and IRPropertyRef {propertyShapeId: string} to plain strings
- Rename description→data and updates→data on mutations
- Rename IRNodeDescription→IRNodeData, IRNodeFieldUpdate→IRFieldUpdate
- Drop kind from projection items, order by items (unambiguous from context)
- Flatten resultMap from {kind: 'result_map', entries: [...]} to just [...]
- Remove IRResultMap wrapper type
- Update all pipeline stages, tests, documentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…arser

This commit completes the IR refactoring branch with final improvements for downstream implementers:

### Dead code removal
- QueryFactory.ts: removed ~50 lines of commented-out type variants
- SelectQuery.ts: removed commented-out QueryBoolean stub

### Documentation improvements
- Added JSDoc comments to all 13 exported IR pipeline functions
- Functions documented: buildSelectQuery, desugarSelectQuery, canonicalizeWhere, canonicalizeDesugaredSelectQuery, lowerSelectQuery, projectionKeyFromPath, lowerSelectionPathExpression, buildCanonicalProjection, and mutation IR builders
- Added JSDoc for IRAliasBinding type and validateAliasReference function

### QueryParser simplification
- Deleted IQueryParser interface (existed only for swappability with one impl)
- Shape now calls QueryParser.selectQuery/updateQuery/createQuery/deleteQuery directly
- Removed Shape.queryParser static property and queryParser from method this constraints
- Updated QueryCaptureStore to use jest.spyOn instead of implementing IQueryParser
- Tests updated to use new captureQuery API (6 test files updated)
- Result: cleaner module boundaries, one less abstraction, no runtime circular dependency

### Consolidation
- Merged docs 002, 003, 004 into single docs/003-ir-refactoring.md
- All 26 implementation phases summarized with key decisions and final state

All 147 tests pass. TypeScript clean. Ready for production.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Replaces the internal query representation with a canonical backend-agnostic IR AST. The public Shape DSL (Shape.select(), .create(), .update(), .delete()) is unchanged — what changed is what IQuadStore implementations receive.

Why
Stores previously received ad-hoc nested arrays and objects (query.select[0][0].property.label). This made downstream store implementations brittle, hard to reason about, and tightly coupled to internal DSL structure. A normalized IR with typed discriminated unions gives store implementers a stable, well-documented contract to build against.

What

    SelectQuery, CreateQuery, UpdateQuery, DeleteQuery ARE the IR types. No separate "IR" vs "query" type hierarchy — factories emit IR directly via build().
    Full IR AST for selects: root (shape scan pattern), projection (expression-based), where (expression tree), orderBy, limit, subjectId, singleResult, resultMap.
    Typed mutations: Same kind discriminators (create, update, delete) with IRFieldUpdate[], IRNodeDescription, and set modification support (add/remove).
    Compact references: IR carries shapeId and propertyShapeId strings only — no embedded shape/property objects.
    Early normalization: some() → exists(), every() → not exists(not ...) during canonicalization. Boolean trees flattened.
    Explicit aliasing: Deterministic alias generation (a0, a1, ...) with lexical scope validation.

How

A multi-stage pipeline converts DSL traces to IR:

SelectQueryFactory.build()
  → toRawInput()             (extract factory state)
  → desugarSelectQuery()     (IRDesugar — flatten DSL tree)
  → canonicalize()           (IRCanonicalize — normalize booleans, rewrite quantifiers)
  → lowerSelectQuery()       (IRLower — produce full AST with aliases, patterns, expressions)
  → SelectQuery

Mutations convert directly via buildCanonical*MutationIR() in IRMutation.ts.

New modules: IRDesugar.ts, IRCanonicalize.ts, IRLower.ts, IRPipeline.ts, IRProjection.ts, IRAliasScope.ts, IRMutation.ts. All exported functions have JSDoc documentation.
Breaking changes for downstream stores

IQuadStore method signatures are unchanged, but the objects they receive are structurally different. Select queries are completely new (query.root, query.projection, query.where). Mutation queries are structurally similar but properly typed. Store result types are now exported: ResultRow, SelectResult, CreateResult, UpdateResult, SetOverwriteResult, SetModificationResult.

Full IR reference: documentation/intermediate-representation.md.
Other improvements

    QueryParser simplified: Deleted IQueryParser interface. Shape calls QueryParser directly — no swappable indirection.
    Dead code removed: ~50 lines of commented-out type variants cleaned up.
    Test coverage: 147 tests across 9 suites. Every select pattern (56) and mutation pattern (18) from the original query.test.ts has IR assertions. Old legacy test file deleted.
    Docs consolidated: Four plan docs merged into docs/003-ir-refactoring.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update AGENTS workflow for ideation and phased implementation
flyon and others added 27 commits February 28, 2026 06:49
Phase 10: Recursive nesting in result mapping
Phase 11: Tighten Fuseki assertions to OLD test depth
Phase 12: Inline where filter lowering (IR pipeline)
Phase 13: Fix invalid SPARQL generation (3 fixtures)
Phase 14: Fix edge cases (eval projection, context path)

Each phase includes root cause analysis, approach options,
files changed, and open questions for user decision.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each phase now includes:
- Exact code locations and line numbers for changes
- Before/after code examples with concrete implementations
- Expected SPARQL output after fixes
- Precise root cause analysis with code path traces
- Architectural considerations and edge cases

Key findings from deep dive:
- Phase 13b: root cause is line 158 of IRCanonicalize.ts calling
  toComparison() instead of checking for SOME/EVERY on the `first`
  element of a where_boolean compound
- Phase 13c: SparqlSelectPlan already has `having` field and
  algebraToString.ts already serializes it — only irToAlgebra.ts
  needs changes
- Phase 14b: arg_path.subject already carries context entity IRI —
  fix is to use it as a fixed triple subject instead of using the
  query's root alias

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tasks mode: defines concrete tasks, dependency graph, parallel
execution strategy, stub boundaries, and detailed validation
specifications with test data assertions for each phase.

Key structure:
- Phases 10, 12, 13, 14 run in parallel (different files)
- Phase 11 runs last (assertion tightening after fixes land)
- Shared file risk in irToAlgebra.ts documented with section ranges
- Integration verification checklist after all phases merge

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrites resultMapping.ts to support arbitrary nesting depth in
SPARQL result grouping. Introduces buildAliasChain() to walk the
traversal map, insertIntoTree() for recursive NestedGroup construction,
and collectNestedGroup() for recursive binding collection. All 452
tests pass.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
13a: Wrap NOT expressions in parentheses in algebraToString.ts to
     fix precedence (!(expr) instead of !expr).
13b: Add canonicalizeComparison() in IRCanonicalize.ts to detect
     SOME/EVERY quantifiers in the first element of boolean compounds.
13c: Route aggregate-containing WHERE filters to HAVING in
     irToAlgebra.ts via containsAggregate() detection.

All 452 tests pass with updated golden expectations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
14a: Add 'expression' variant to SparqlProjectionItem for computed
     projections like (expr AS ?alias). Fixes customResultEqualsBoolean
     which now correctly projects the boolean comparison.
14b: Add context_property_expr IR expression type to resolve context
     entity property paths via separate triples instead of reusing
     the same variable. Fixes whereWithContextPath tautology.

All 452 tests pass with updated golden expectations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Process DesugaredPropertyStep.where predicates that were silently
dropped during IR lowering. When a property step has .where(), force
a traversal creation, canonicalize and lower the where predicate, and
attach the resulting filter to the traverse pattern. In irToAlgebra,
filtered traversals produce OPTIONAL blocks with the traverse triple,
filter property triples, and FILTER co-located inside.

Fixes 6 fixtures: whereFriendsNameEquals, whereHobbyEquals, whereAnd,
whereOr, whereAndOrAnd, whereAndOrAndNested.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 3-level nesting and alias_expr tests to sparql-result-mapping.test.ts.
Tighten Fuseki test assertions from defensive patterns (>=1, toBeDefined)
to precise values matching test data. Update inline where section (now
lowered via Phase 12) and invalid SPARQL section (now valid via Phase 13).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ested create with predefined ID (Phases 15-17)

- Fix DELETE variable reuse via SparqlDeleteInsertPlan refactor
- Fix context variable naming with sanitizeVarName()
- Fix aggregate alias collision in irToAlgebra
- Fix UPDATE OPTIONAL wrapping for safe WHERE clause
- Add operator parenthesization for OR inside AND expressions
- Fix literal property inline where result mapping with detectLiteralTraversals()
- Fix isNodeReference() to distinguish references from nested creates
- Handle predefined ID in convertNodeDescription()
- All 456 tests pass, 75/75 Fuseki end-to-end

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add comprehensive review section to plan doc with remaining gaps categorized by priority
- Create documentation/sparql-algebra.md covering the full SPARQL pipeline:
  algebra types, IR→algebra conversion, string serialization, result mapping,
  public API, store implementation guide, and known limitations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…sion detection (Phases 18-20)

Phase 18: Add escapeSparqlString() for SPARQL literal escaping (quotes,
backslashes, newlines, tabs, CRs). Fix language-tagged literal bypass in
serializeTerm(). Remove unused varCounter in updateToAlgebra().

Phase 19: Rewrite convertExistsPattern() with full recursive support for
optional (→ left_join), union (→ union), and nested exists patterns.
Fix join case to handle mixed sub-pattern types instead of only traverse.

Phase 20: Improve detectLiteralTraversals() to scan all bindings and
handle mixed types safely. Add duplicate result key detection in
buildNestingDescriptor(). Fix context property collision by using raw IRI
as registry key + variable name deduplication with counter.

All 471 tests pass, 75/75 Fuseki, TypeScript clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove string literal escaping and EXISTS limitations (now fixed).
Add escapeSparqlString to utilities table. Update known limitations
to reflect current state.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…pe casts

- Add SparqlStore abstract base class (IR → SPARQL → execute → map pipeline)
- Add FusekiStore concrete implementation with 5 integration tests (80/80 Fuseki)
- Replace hardcoded XSD/RDF URI strings with ontology module imports
- Fix `as any` casts: use `as never` for exhaustive switches, explicit ResultRow[] for arrays
- Remove dead commented code in MutationQuery.ts
- Update plan review section with resolved nice-to-have items

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…dvanced patterns

- 005: Named graph support (.from()/.into() for GRAPH clauses)
- 006: Computed expressions & update functions (L.times, L.concat, etc.)
- 007: Advanced query patterns (MINUS set difference + DELETE WHERE bulk delete)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Walks through Shape → DSL → IR → SPARQL Algebra → SPARQL String → Result
with abbreviated examples at each stage. Links to IR and SPARQL algebra docs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…type inference

- README: highlight fully inferred result types as top feature, add AST optimization
  mention, remove stray code block, update store packages list
- sparql-algebra.md: fix "no base classes" statement, add SparqlStore.ts to file
  structure, rewrite store implementation section to use SparqlStore base class,
  add SparqlStore to public API section
- intermediate-representation.md: update reference implementations to describe
  built-in SparqlStore instead of "coming soon"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rewrote report with full architecture, design decisions, API surface,
  conversion rules, resolved gaps, and test coverage
- Added changeset (minor) with detailed user-facing changelog
- Updated wrapup skill with report quality and changeset guidelines

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Summary

Adds a complete SPARQL conversion pipeline to @_linked/core — compiles the canonical IR into executable SPARQL and maps results back to fully typed DSL objects.
What's new

    Three-layer pipeline: IR → SPARQL algebra (typed AST) → SPARQL string → execute → result mapping
    SparqlStore abstract base class: Concrete stores only implement executeSparqlSelect() and executeSparqlUpdate() — the base class handles everything else
    SPARQL algebra types: Formal algebra aligned with SPARQL 1.1 spec §18 (BGP, LeftJoin, Filter, Union, Minus, Extend, Graph)
    Prefix-aware serialization: Automatic PREFIX block generation from registered ontologies
    Result mapping: SPARQL JSON → typed DSL objects with XSD coercion, nested grouping, and literal traversal detection
    Fully inferred result types: TypeScript return type is automatically inferred from selected paths — highlighted as a top-level feature in README
Resolve conflicts:
- AGENTS.md: keep dev's workflow-skill approach
- src/shapes/Shape.ts: keep dev's selectAll() method, restore SelectAllQueryResponse type
- src/tests/query.test.ts: removed (tests reorganized in dev)
- docs/reports/002-select-all-property-shapes.md: keep dev's renamed location

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@socket-security
Copy link

socket-security bot commented Mar 2, 2026

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedulid@​3.0.210010010084100

View full report

The merge from main auto-resolved several files incorrectly, taking
main's reverted versions instead of dev's. Restores:
- SHACL.ts: getUniquePropertyShapes(), override validation, explicit symbol tracking
- SelectQuery.ts: selectAll() on QueryShapeSet and QueryShape, SelectAllQueryResponse type
- Shape.ts: SelectAllQueryResponse import
- query-fixtures.ts: Employee class, selectAll query factories
- query.types.test.ts: selectAll type tests
- metadata.test.ts: override validation tests
- README.md: selectAll documentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@flyon flyon merged commit 6d32d8a into main Mar 2, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant