rewriter-go

Native Go implementation of the ClickHouse SQL Rewriter interface — an in-process (no gRPC hop) alternative to the C++ rewriter-grpc service, built on tobilg/polyglot (Rust SQL transpiler, ClickHouse dialect) via its PureGo SDK (CGO_ENABLED=0).

Target: full behavioral parity with rewriter-grpc across Rewrite + RewriteErrorMessage, validated by a differential oracle harness that runs both engines and diffs their responses.

Status

Phase 0 complete (engine + harness + fidelity spike). Verdict: GO for full parity via a dual-dialect + raw-SQL architecture — see the fidelity report. Phases 1–5 (the statement handlers) land on feat/* branches via PR.

Phase 1 complete (SELECT handler, branch feat/phase-1-select). Implemented:

• Table resolution — static 3-map (table_map, remote_table_map, table_with_database_map) + dynamic (logical→physical, buildDynamicTableName, known-physical passthrough), including remote() wrapping for remote upstreams.
• Option pipeline — LimitRewrite (force + replace), OffsetRewrite, SettingsRewrite.
• CTE injection (CommonTableExprRewrite) — parse-and-inject bodies before the table walk, failing aliases recorded and sorted.
• GLOBAL cross-shard pass (ForceGlobalForRemoteAsymmetry) to handle mixed local/remote JOIN patterns.
• Full response population (table_rewrites, original_accessed_tables, failed_cte_aliases, sql_after_rewrite).
• Validated by internal/harness golden tests (10 cases covering dynamic rename, static maps, remote, CTE injection, limit, JOIN, passthrough) and, when REWRITER_ORACLE_ADDR is set, an env-gated differential against the live C++ oracle.

Known limitation: a GLOBAL JOIN whose left operand is a remote() function cannot be synthesised through polyglot's generator — such cases are left un-GLOBAL and allow-listed in the harness corpora.

Layout

Path	What
rewriter.go / native.go / service.go	Public Rewriter interface + RewriteResult; the per-connection NativeRewriter; the stateless Service (request/response shape mirroring the gRPC contract — the embedding entry point for hosts like housegate)
internal/engine	The polyglot seam — the ONLY package that imports the polyglot SDK (Engine interface, NodeKind/CommandSQL, fidelity metric)
internal/corpus	SQL corpus loader
internal/harness	Differential comparator + env-gated rewriter-grpc oracle client
cmd/fidelity-spike	Round-trip fidelity probe over a corpus
rewriter-proto	Versioned protobuf contract and generated Go types imported by this module

Protobuf package migration

Releases through v0.3.0 exposed generated types at github.com/housegate/rewriter-go/gen/pb. The shared-contract migration removes that package; consumers upgrading to the first release containing this change must switch imports to github.com/housegate/rewriter-proto/gen/pb. Treat that upgrade as a breaking source migration.

Build & test

polyglot is a git submodule at third_party/polyglot-src (pinned commit = single source of truth for both the Go bindings, via the replace in go.mod, and the Rust FFI lib). Clone with --recurse-submodules, or run git submodule update --init after a plain clone. The FFI library itself is not vendored — build it once from the submodule (cargo build, a few minutes):

make ffi      # builds third_party/lib/libpolyglot_sql_ffi.<ext> from the submodule
make test     # sets POLYGLOT_SQL_FFI_PATH and runs `go test ./...`

Tests that need the engine skip themselves when POLYGLOT_SQL_FFI_PATH is unset, so go test ./... alone still runs the pure-Go units (comparator, corpus, fidelity-metric, contract tests).

Oracle differential (optional, local-only)

When REWRITER_ORACLE_ADDR points at a live rewriter-grpc gRPC server, the internal/harness tests additionally diff every golden corpus field-by-field against it:

make ffi
POLYGLOT_SQL_FFI_PATH="$PWD/third_party/lib/libpolyglot_sql_ffi.$(uname | grep -qi darwin && echo dylib || echo so)" \
  REWRITER_ORACLE_ADDR=localhost:50051 go test ./internal/harness -count=1

A small, documented allow-list of intentional divergences is carved out per-case in the corpora (allow_*_divergence flags): the IN(>50) OR-batch preprocess we skip (§6f), admin statements polyglot parses but ClickHouse's parser rejects (COPY / ATTACH GRANT → SyntaxError), the GRANT … TO ALL marker backtick-quoting, and the static-table_map db-qualifier (a C++-internal inconsistency, flagged for review).

Design

See docs/superpowers/specs/2026-06-03-native-go-rewriter-design.md for the full design: architecture, the polyglot engine seam (incl. the dual-dialect parse strategy, §3.1), the parity risk register, the differential-harness validation strategy, and the phased build plan. The Phase 0 implementation plan is in docs/superpowers/plans/.