Skip to content

feat(ast): implement comprehensive SQL AST formatting #4205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Nov 30, 2025
Merged

feat(ast): implement comprehensive SQL AST formatting #4205

merged 9 commits into from
Nov 30, 2025
+851 −107

Conversation

@kyleconroy
Copy link
Collaborator

@kyleconroy kyleconroy commented Nov 30, 2025

Summary

  • Implements comprehensive Format() methods for SQL AST nodes to convert parsed AST back to valid SQL strings
  • Adds fmt_test.go which validates that formatting produces semantically equivalent SQL (via fingerprint comparison)
  • All format tests now pass (started with 90+ failures)

Key Features Implemented

Expression Formatting:

  • Named parameters (@param) without space after @
  • NULLIF expressions
  • BETWEEN, LIKE, ILIKE, SIMILAR TO expressions
  • IS DISTINCT FROM / IS NOT DISTINCT FROM
  • Scalar array operations (ANY/ALL)
  • CASE expressions

Type Handling:

  • Internal type name mapping (int4→integer, timestamptz→timestamp with time zone, etc.)
  • Array type support (text[])
  • Type modifiers (varchar(32))
  • pg_catalog schema handling

Statement Formatting:

  • CREATE FUNCTION with parameters, options (AS, LANGUAGE, STRICT, SECURITY), and modes (IN, OUT, INOUT, VARIADIC)
  • CREATE EXTENSION with IF NOT EXISTS
  • DO $$ ... $$ anonymous code blocks
  • INSERT with proper spacing before VALUES
  • UPDATE with array subscript access (names[$1])

Clause Formatting:

  • WITHIN GROUP for ordered-set aggregates
  • NULLS FIRST/LAST in ORDER BY
  • CROSS JOIN detection (JOIN without ON/USING)
  • LATERAL in subselects and function calls
  • FOR UPDATE/SHARE locking clauses with options (NOWAIT, SKIP LOCKED)
  • ON CONFLICT with inference and actions
  • Window definitions (PARTITION BY, ORDER BY, frame clauses)

Identifier Quoting:

  • Automatic quoting for SQL reserved words (SELECT, CASE, etc.)
  • Mixed-case identifier preservation ("deviceId")

Test plan

  • All TestFormat tests pass (validates AST→SQL→fingerprint matches original)
  • All existing go test ./... tests pass
  • No regressions in other packages

🤖 Generated with Claude Code

kyleconroy and others added 5 commits November 29, 2025 22:30
@kyleconroy @claude
refactor(fmt_test): use config-based engine detection and parser for …
ab9ce9e 
…statement boundaries

- Parse sqlc config file to determine database engine instead of
  hardcoding pgx/v5 path filter
- Use parser's StmtLocation/StmtLen for proper statement boundaries
  instead of naive semicolon splitting
- Handle both file and directory paths in queries config
- Only test PostgreSQL for now (formatting support is PostgreSQL-only)

This fixes issues with multi-query files containing semicolons in
strings, PL/pgSQL functions, or DO blocks.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@kyleconroy @claude
feat(ast): add and improve Format methods for SQL AST nodes
7731b59 
Add Format methods:
- A_ArrayExpr: Format ARRAY[...] literals
- NullIfExpr: Format NULLIF(arg1, arg2) function calls
- OnConflictClause: Format ON CONFLICT ... DO UPDATE/NOTHING
- InferClause: Format conflict target (columns) or ON CONSTRAINT
- IndexElem: Format index elements for conflict targets
- WindowDef: Format window definitions with PARTITION BY, ORDER BY, and frame clauses

Improve existing Format methods:
- A_Expr: Add BETWEEN, NOT BETWEEN, ILIKE, SIMILAR TO, IS DISTINCT FROM handling
- A_Expr_Kind: Add all expression kind constants
- CaseExpr: Handle CASE with test argument and optional ELSE
- DeleteStmt: Add USING clause formatting
- FuncCall: Add DISTINCT, ORDER BY, FILTER, and OVER clause support
- InsertStmt: Delegate to OnConflictClause.Format
- JoinExpr: Add RIGHT JOIN, FULL JOIN, NATURAL, and USING clause
- LockingClause: Add OF clause, SKIP LOCKED, NOWAIT, and fix strength values
- RangeFunction: Add LATERAL support and fix alias spacing
- SelectStmt: Add HAVING clause formatting

These changes reduce test failures from 135 to 102.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@kyleconroy @claude
feat(ast): add more Format methods for SQL AST nodes
c355232 
Add Format methods:
- NullTest: Format IS NULL / IS NOT NULL expressions
- ScalarArrayOpExpr: Format scalar op ANY/ALL (array) expressions
- CommonTableExpr: Add column alias list support

Improve existing Format methods:
- WithClause: Fix spacing after WITH and RECURSIVE keywords

These changes reduce test failures from 102 to 91.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@kyleconroy @claude
feat(postgresql): add custom Deparse wrapper with bug fixes
25ee705 
- Switch fmt_test.go to use postgresql.Deparse instead of ast.Format
- Add deparse.go and deparse_wasi.go with Deparse wrapper function
- Fix pg_query_go bug: missing space before SKIP LOCKED
- Skip tests with parse errors (e.g., syntax_errors test cases)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@kyleconroy @claude
feat(ast): complete SQL AST formatting implementation
c4e885f 
Fixes all ast.Format test failures by implementing comprehensive
Format methods for SQL AST nodes. Key improvements include:

- Named parameters (@param) formatting without space after @
- NULLIF expression support in A_Expr
- NULLS FIRST/LAST in ORDER BY clauses
- Type name mapping (int4→integer, timestamptz→timestamp with time zone)
- Array type support (text[]) and type modifiers (varchar(32))
- CREATE FUNCTION with parameters, options (AS, LANGUAGE), and modes
- CREATE EXTENSION statement formatting
- DO $$ ... $$ anonymous code blocks
- WITHIN GROUP clause for ordered-set aggregates
- Automatic quoting for SQL reserved words and mixed-case identifiers
- CROSS JOIN detection (JOIN without ON/USING clause)
- LATERAL keyword in subselects and function calls
- Array subscript access in UPDATE statements (names[$1])
- Proper AS keyword before aliases

Also removes unused deparse files and cleans up fmt_test.go to use
ast.Format directly.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. 🔧 golang labels Nov 30, 2025
kyleconroy and others added 4 commits November 30, 2025 13:05
@kyleconroy @claude
refactor(postgresql): use existing convert functions instead of trans…
0e7fa5a 
…late helpers

Replace custom translate functions (translateTypeNameFromPG, translateOptions,
translateNode, translateDefElem) with existing convert.go functions
(convertTypeName, convertSlice) to maintain architectural consistency.

Both parse.go and convert.go import the same pg_query_go/v6 package, so the
types are compatible and the existing convert functions can be used directly.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@kyleconroy @claude
refactor(format): add Formatter interface for SQL dialect-specific qu…
dc44702 
…oting

- Create internal/sql/format package with Formatter interface
- Add QuoteIdent method to TrackedBuffer that delegates to Formatter
- Implement QuoteIdent on postgresql.Parser using existing IsReservedKeyword
- Update all Format() methods to use buf.QuoteIdent() instead of local quoteIdent()
- Remove duplicate reserved word logic from ast/column_ref.go
- Update ast.Format() to accept a Formatter parameter

This allows each SQL dialect to provide its own identifier quoting logic
based on its reserved keywords and quoting rules.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@kyleconroy @claude
refactor(format): add TypeName method to Formatter interface
8194d94 
- Add TypeName(ns, name string) string method to Formatter interface
- Implement TypeName on postgresql.Parser with pg_catalog type mappings
- Add TypeName method to TrackedBuffer that delegates to Formatter
- Update ast.TypeName.Format to use buf.TypeName()
- Remove mapTypeName from ast package (moved to postgresql package)

This allows each SQL dialect to provide its own type name mappings
(e.g., pg_catalog.int4 -> integer for PostgreSQL).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@kyleconroy @claude
fix(postgresql): restore parseRelationFromNodes for column type resol…
7a49c3a 
…ution

The convertTypeName function populates extra fields (Names, ArrayBounds,
Typmods) on the TypeName struct which breaks the catalog's type equality
check used for ALTER TYPE RENAME operations.

This change:
- Reverts to using parseRelationFromNodes + rel.TypeName() which only
  populates Catalog, Schema, Name fields needed for type resolution
- Updates ColumnDef.Format to use IsArray field for array formatting
  since TypeName.ArrayBounds is no longer set

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@kyleconroy kyleconroy merged commit 405a905 into main Nov 30, 2025
13 checks passed
@kyleconroy kyleconroy deleted the kyle/sql-ast-format branch November 30, 2025 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

size:XL This PR changes 500-999 lines, ignoring generated files. 🔧 golang

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kyleconroy