docs: add ENGINE_REFACTORING.md, an R-numbered structural debt log

[TimelordUK]

Docs only — no code. Sits on its own branch so it doesn't restart CI on #31.

What

A companion to SQL_PARITY.md. The two tracks answer different questions:

	Question	Driven by
SQL_PARITY.md (P-numbers)	Do we return the right answer?	Differential testing vs DuckDB
ENGINE_REFACTORING.md (R-numbers)	Can we keep changing the engine safely?	Findings from doing the work

A parity gap is a wrong result. An R-finding is a place where the engine's shape makes the next change disproportionately expensive or risky.

Why now

The engine grew organically and on demand, which is why it does as much as it does. The cost is that some structure was never designed, only accreted — and that bill arrives exactly when we want to go further (richer functions, deeper nesting, correlated scoping).

This is deliberately not a rewrite proposal. Principle 1 in the doc is that every entry must reduce to independently shippable steps. The point is to distinguish "this is awkward" from "this is blocking", and to record why something was deferred so the decision doesn't get relitigated.

Seeded findings

All seven came out of scoping parity P3, and each carries evidence rather than opinion:

	Finding	Status
R1	Two FROM representations	🟡 partial (#30) — migration deferred
R2	446 hand-rolled SqlExpression matches across 40 files	🟡 partial (#31)
R3	Catch-all arms swallow new variants	🔴 open
R4	Transformer fixtures use ASTs the parser never produces	🔴 open
R5	Dead code accumulates undetected	🔴 open
R6	CorrelatedSubqueryAnalyzer unwired, tests hollow	🔴 open
R7	Subqueries substituted, not evaluated	🔴 open — root cause under P3

Two worth calling out:

• R3 is backed by a probe, not a guess: temporarily adding an Exists variant showed only five match sites in the entire codebase fail to compile. Everything else would have silently ignored it.
• R4 explains why R1's desync survived a green test suite — the fixtures set from_source: None, an input the parser never produces. That's the kind of thing worth writing down once so it isn't rediscovered.

R1 records the deferral as explicitly as the work: the full FROM migration is blocked on TableSource lacking a table-function variant and being unable to carry an alias, which drags JoinClause in. Assessed as larger than P3 itself.

Sequencing

The doc makes the load-bearing dependency explicit — the advanced work is gated on R2/R3, because every new expression variant currently costs a 40-file hand audit with silent failure as the default outcome.

R2 walkers ──┬─→ R3 catch-alls retired
             │
             └─→ P3 Exists variant + scope spine ──→ correlated subqueries
                        ↑
                   R6 analyzer wiring
                        ↑
                   R7 per-row evaluation

Also

• Cross-links P3 → R7/R2/R6 in SQL_PARITY.md, and adds the reverse pointer.
• Surfaces both books of work in CLAUDE.md so future sessions find them before touching the engine.

🤖 Generated with Claude Code