Document the TransformationSpecification-as-IR architectural vision

[amc-corey-cox]

Context

A design discussion clarified the architectural direction of linkml-map:

• The TransformationSpecification is the core value — a declarative IR for data model mappings
• It can be executed by multiple backends (Python interpreter today, SQL compiler experimentally)
• Output serialization (YAML, JSON, JSONL, TSV, CSV) is intentionally text-based — loading into
  analytical stores (DuckDB, Parquet, etc.) is a downstream pipeline concern, not linkml-map's job
• The SQL/DuckDB backend is a growth direction for the execution layer, not the output layer

Done

Initial documentation updates have been made:

• docs/index.md — added Architecture section framing the two backends
• README.md — updated opening to lead with the TransformationSpecification as core concept
• docs/faq.md — added multi-backend framing paragraph

Remaining

• Consider adding a backend feature parity matrix (which spec features are supported by Python vs SQL)
• The compliance suite could potentially track per-backend support
• As the SQL backend matures (Expand SQLCompiler to cover more of the TransformationSpecification #150, Flesh out DuckDBTransformer as a full execution backend #151), documentation should grow to match

Related

• Expand SQLCompiler to cover more of the TransformationSpecification #150 (SQLCompiler expansion)
• Flesh out DuckDBTransformer as a full execution backend #151 (DuckDBTransformer as full backend)
• Update documentation for recent features before RC #137 (documentation update tracking)

[github-actions]

Issue Summary (auto-generated)

This issue documents the TransformationSpecification-as-IR architectural vision — the conceptual framing that positions TransformationSpecification as a declarative intermediate representation (IR) for data model mappings, analogous in spirit to how LLVM IR decouples frontends from backends.

---

Architectural Vision

The core idea is a clean separation of concerns:

Layer	Responsibility
TransformationSpecification	Declarative IR: what to transform (class derivations, slot derivations, expressions, enum mappings)
Python ObjectTransformer	Execution backend #1: row-by-row interpretation (currently full spec support)
SQL SQLCompiler + DuckDBTransformer	Execution backend #2: set-based SQL compilation (currently limited subset)
Output serialization	Text-based (YAML, JSON, JSONL, TSV, CSV) — loading into DuckDB/Parquet is a downstream concern

The key design decision: DuckDB/SQL is a execution backend, not an output format. The map-data CLI writes text; consumers load it into analytical stores as a separate step.

---

Current Code State (as of review)

SQLCompiler (src/linkml_map/compiler/sql_compiler.py, ~140 lines):

• ✅ populated_from slot renames → column AS alias
• ✅ JSON stringification → CAST(col AS TEXT)
• ✅ Delimiter-based aggregation → STRING_AGG(col, 'delim')
• ✅ DDL generation for source/target schemas (with DuckDB array types)
• ❌ Expression compilation (expr field)
• ❌ Enum/permissible value derivations → SQL CASE
• ❌ Cross-class join support (dot-notation org_id.name)
• ❌ Unit conversion arithmetic
• ❌ copy_all / exclude → SELECT * EXCEPT

DuckDBTransformer (src/linkml_map/transformer/duckdb_transformer.py):

• ✅ map_database() — functional, delegates to SQLCompiler
• ❌ map_object() — raises NotImplementedError("DuckDBTransformer.transform")

docs/index.md: Has good general documentation but does not yet contain an explicit "Architecture" section describing the two-backend model — that content is what this issue is requesting.

---

Related Issues (all currently open)

Issue	Title	Status
#150	Expand SQLCompiler to cover more of the TransformationSpecification	Open (enhancement)
#151	Flesh out DuckDBTransformer as a full execution backend	Open (enhancement, help wanted)
#137	Update documentation for recent features before RC	Open (documentation)

#150 proposes incremental SQLCompiler expansion in priority order: expressions → enum derivations → join support → unit conversion → copy directives. Each as a separate PR with explicit errors for unsupported features.

#151 proposes: wiring expanded SQLCompiler into map_database(), deciding map_object() semantics (or making database-only explicit), potential --backend duckdb CLI flag, and a backend parity matrix in docs.

#137 is the broader pre-RC documentation tracking issue: expression language guide, uuid5() docs, multi-output (-O) examples, cross-table join docs, multiple class_derivations examples.

---

Remaining Work for This Issue

The "Done" section lists documentation updates to docs/index.md, README.md, and docs/faq.md. The remaining items are:

1. Backend feature parity matrix — a table showing which TransformationSpecification features are supported by Python vs SQL backend. This would live in docs and potentially be validated by the compliance suite.
2. Compliance suite backend tracking — per-backend test annotations as Expand SQLCompiler to cover more of the TransformationSpecification #150/Flesh out DuckDBTransformer as a full execution backend #151 mature.
3. Documentation growth — as SQL backend capabilities expand, the architecture docs should be updated to reflect current parity.

The parity matrix is the highest-value deliverable here: it gives users a clear signal of what they can rely on in each backend, and gives contributors a roadmap for #150/#151.

[amc-corey-cox]

Closed by #153.