Hard fork the TiDB parser into pkg/parser (MySQL-only)

[morgo]

Hard forks the TiDB parser into pkg/parser and removes the go.mod require+replace of github.com/pingcap/tidb/pkg/parser. Spirit only targets MySQL, so the fork strips everything TiDB- or MariaDB-specific and keeps a parser we can extend quickly for our narrower use case. See the new pkg/parser/README.md for the full story.

Fork base: block/tidb@e528fd979fc8 (upstream pingcap/tidb master as of 2026-05-04, plus our spatial type/index support). The first commit is a verbatim copy with imports rewritten, so reviewing commit-by-commit gives readable diffs for everything that was actually changed.

What was stripped

• TiDB/MariaDB grammar and keywords: TiDB statements (ADMIN, BRIE, TRACE, SET SESSION_STATES, placement/attributes/resource-groups, FLUSH TIDB PLUGINS, FLUSH CLIENT_ERRORS_SUMMARY), MariaDB SYSTEM_TIME partitioning, TiDB INTERVAL partitioning, ILIKE, TiDB system functions (tidb_*, vitess_hash, ...), TiDB-only optimizer hints, GLOBAL TEMPORARY tables, sequences, etc. The goyacc parse table shrinks from ~1.57M to ~1.52M entries.
• The driver indirection and Datum/expression glue TiDB layered on top of the AST.
• terror + github.com/pingcap/errors + zap: replaced with stdlib wrapped errors (errors.Is/As work) and no logging dependency. pingcap/* and go.uber.org/zap are no longer in spirit's direct dependency graph.
• The legacy Format(io.Writer) pretty-printer (parallel to Restore, zero consumers).
• Charset machinery: gbk/gb18030 transcoders, custom charset registration, TiFlash charset lists, and the TiDB-invented utf8mb4_zh_pinyin_tidb_as_cs collation. Charset names are still recognized in DDL.
• Dead code: after the sweep, deadcode -test ./... reports only interface-conformance marker methods.

MySQL fidelity

• reserved_words_test.go (build-tagged, requires a live server) now passes against MySQL 8.0.45: the grammar's reserved-word set matches MySQL's exactly, modulo two documented exceptions (CURRENT_ROLE, ARRAY).
• MySQL-compat fixes that landed upstream after the fork base are ported: parser-depth DoS guard (e2b6ce7333), INSERT ... AS row_alias (551d10a652), dual-password syntax (b254c43931), SET_VAR decimal hints (9bbc86e96e), and GROUP_CONCAT separator charset handling (2b285ed389). A review of all upstream pkg/parser commits through 2026-08-13 found nothing else MySQL-relevant to pick up.
• Fixes --statement rewrites column default expression without parentheses #542: parenthesized default values now survive the parse/restore round trip. MySQL 8.0.13+ treats DEFAULT ('{}') (expression default, required on BLOB/TEXT/JSON/GEOMETRY) and DEFAULT '{}' (literal default) as different DDL; the parser used to discard the parens, so --statement mode rewrote valid ALTERs into DDL MySQL rejects with Error 1101. The upstream parser still has this bug (Parser cannot pass default value with an expression pingcap/tidb#57768). The statement layer now also distinguishes the two forms in declarative diffs, and extraction was validated to converge with MySQL's SHOW CREATE TABLE rendering (DEFAULT (_utf8mb4'{}')).
• Fixes keyword-named function calls (first half of parser: adopt upstream's precedence-aware parentheses canonicalizer (RestoreSkipRedundantParentheses) #1128): DEFAULT (point(0,0)) — the MySQL manual's own expression-default example — now parses (upstream special-cases only REPLACE), and the spatial constructors (linestring(), polygon(), multipoint(), ...) now parse as function calls in every expression context (SELECT lists, generated columns, CHECK constraints). As keyword tokens they had been broken everywhere since the GIS fork, making any table that uses one invisible to spirit. Validated against MySQL 8.0.45 including SHOW CREATE TABLE convergence.

Validation

• Full test suite green against MySQL 8.0.45 (privileges tests excluded locally; covered in CI).
• golangci-lint run reports 0 issues repo-wide (the fork previously surfaced ~518).
• make parser regenerates parser.go/hintparser.go reproducibly with zero grammar conflicts; goyacc is a nested module so its dependencies stay out of the main graph.

🤖 Generated with Claude Code

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

[aparajon]

🤖 Adversarial correctness and security review at 210b33ce, looking specifically for anything that could take a process down or make Spirit execute DDL the author did not write. Verdict: no blocking findings, and nothing I found is a regression introduced by this PR. Every issue below reproduces identically on pre-fork main, so this reads as a faithful port; they belong in follow-up issues rather than in this diff.

A 59k-line fork cannot be reviewed line by line, so I reviewed it by behavior instead: the 50-file integration delta in full, then differential testing of the new parser against pre-fork main and against a live MySQL 8.0.45. Details of what that covered are in the collapsed section at the bottom.

1. A numeric literal longer than 81 digits panics the process. pkg/parser/ast/mydecimal.go carries upstream's five panic(panicInfo) branches in MyDecimal.FromString and fixWordCntError. They are reachable from both public entry points with ordinary DDL — a column default, a table option, or a partition bound:

statement.ParseCreateTable("CREATE TABLE t (a DECIMAL(65,0) DEFAULT " + strings.Repeat("9", 82) + ")")
statement.New("ALTER TABLE t1 AUTO_INCREMENT=" + strings.Repeat("9", 100))
statement.ParseCreateTable("CREATE TABLE t (id INT PRIMARY KEY) PARTITION BY RANGE (id) (PARTITION p0 VALUES LESS THAN (" + strings.Repeat("9", 100) + "))")

Each panics with This branch is not implemented...; 81 digits parses fine, 82 panics. This is not new — I ran the same three inputs against pre-fork main and they panic identically, because the upstream test_driver MyDecimal has the same stubs and Spirit has always imported it. What changes is ownership: the code is now ours, and any long-running process that parses a schema file out of a pull request can be crashed by a one-line change to that file. TiDB's real types.MyDecimal returns an overflow/truncation error on these branches instead of panicking; porting that, or simply returning an error, closes it. This is the one I would file today.

2. Nothing in CI verifies the generated parser matches the grammar. pkg/parser/README.md says "CI verifies they are in sync with the grammar", but no workflow references pkg/parser/Makefile. I regenerated at this commit and it is clean — make parser reproduces parser.go and hintparser.go byte for byte with no conflicts — so the tree is correct today. The gap is forward-looking: an edit to parser.y that skips regeneration ships a grammar file that no longer describes the parser, and for the component that decides which DDL Spirit will execute, that divergence is worth catching mechanically. A make parser && git diff --exit-code step would do it; the README sentence should either become true or be softened.

3. Bare and parenthesized literal defaults are now a real diff — expect one-time churn on the first plan. This is the intended consequence of #542, but it is worth stating in operational terms. For a live column created as DEFAULT 'x' against a schema file that writes DEFAULT ('x'), pre-fork main produced no statements; this branch produces:

ALTER TABLE `t1` MODIFY COLUMN `v` varchar(10) NULL DEFAULT ('x')

That is correct — MySQL genuinely stores the two forms differently — and it converges in one shot (I verified round-trip idempotency against a live server). But it means a previously-converged table whose schema file wraps a literal default in parentheses on a non-BLOB/TEXT/JSON column will surface an unexpected MODIFY COLUMN on the first plan after this lands. Callers rolling the new parser out may want to sweep for DEFAULT (' in managed schema roots ahead of time rather than discover it one table at a time.

4. Two constructs MySQL accepts still fail to parse. INT DEFAULT ((1+2)) and JSON DEFAULT (CAST('{}' AS JSON)) are both accepted by MySQL 8.0.45 and both rejected here with a syntax error. Identical on pre-fork main, so no regression — but since the parser is now in-repo, these are fixable here rather than upstream, which is the stated point of the fork. Failing closed on unparseable DDL is the safe behavior, so this is a compatibility gap, not a safety one.

5. DEFAULT (ABS(-1)) never converges. MySQL stores it as (abs(-1)) and the differ re-emits MODIFY COLUMN ... DEFAULT (abs(-1)) on every run. Pre-existing and unchanged by this PR — I mention it only because the same round-trip harness that verified the new paren behavior surfaced it, and a permanently-diffing column is the kind of thing that erodes trust in a declarative plan.

6. _gbk'…' and _gb18030'…' introducers are now a parse error. Consistent with the documented charset reduction: the lexer returns Unsupported character introducer: 'gbk' where the TiDB parser would have transcoded. This fails closed, which is the right direction, but a table whose schema declares a gbk column with a non-ASCII literal default will stop parsing rather than silently mis-decode. Note that CHARACTER SET gbk on a column still parses — only the literal introducer is rejected — so the failure mode is narrow and loud.

Action items (all follow-ups; nothing here should hold this PR):

1. File an issue for the MyDecimal panic in (1) and replace the panic(panicInfo) branches with returned errors.
2. Add a CI step that runs make parser and fails on a diff, and correct or soften the README claim in (2).
3. Optionally file (4) as a parser compatibility gap and (5) as a differ convergence bug.

Verified

Read the full 50-file non-parser integration delta (the import rewrite is mechanical and complete — no pingcap imports remain outside two comment references). Confirmed the dependency change is a genuine reduction and not a loosening: the block/tidb replace directive is gone, pingcap/errors and pingcap/tidb/pkg/parser are now indirect-only through go-mysql, pingcap/failpoint is dropped, golang.org/x/text is promoted to direct, and nothing was downgraded. Apache-2.0 LICENSE and per-file copyright headers are present with attribution in the README. Exercised the DoS surface with hostile input: 100k nested parentheses, 500k chained NOT, 200k-term OR, 300k unary minus, 50k nested function calls, deep CASE/CHECK/generated-column nesting and nested hint comments all terminate with a clean depth error (parentheses nesting depth exceeds maximum 10000 / AST nesting depth exceeds maximum 10064) rather than a stack overflow; invalid UTF-8 in identifiers and literals, NUL bytes, unterminated strings/comments/identifiers, unknown charset introducers, 100k-character identifiers, and raw binary garbage all either parse or error cleanly, and every AST that parsed restored to text that re-parses. Verified the new default handling against a live MySQL 8.0.45 across 22 column forms (JSON/BLOB/TEXT/VARCHAR/INT/TIMESTAMP/DATETIME parenthesized and bare, NOW(), UUID(), JSON_OBJECT(), hex and _binary literals, and defaults containing quotes, backslashes, newlines and % sequences): create in MySQL, read back SHOW CREATE TABLE, parse both sides, diff — all idempotent except the two pre-existing cases above, which fail identically pre-fork. Mutation-checked the lint_zero_date.go change by deleting the unwrap loop: DEFAULT ('0000-00-00') then escapes the zero-date linter entirely, so that hunk is closing a real safety-gate blind spot the paren preservation would otherwise have opened, and it is the only such consumer — ColumnOptionDefaultValue has exactly two readers outside the parser and both are handled. Confirmed the reserved-word set matches a live MySQL 8.0.45 (-tags reserved_words_test), parsed the repo's MySQL DDL corpus with no new failures, and ran the parser, statement, lint and fmt suites green (the FK and transaction-compression failures I saw are local server configuration and reproduce on pre-fork main). Checked for unsafe usage, mutable package-level state and concurrency hazards in the new package: all inherited from upstream, read-only maps built at init. CI is green including govulncheck and lint; Copilot left no inline comments.

This review was generated by Claude Code (claude-opus-5).

[morgo]

🤖 Addressing the review feedback:

(1) MyDecimal panic — fixed in the stacked PR #1142 (parser: stop panicking on decimal literals wider than 81 digits). It ports the upstream TiDB clamping semantics: int-part overflow clamps to the max decimal with a warning, fraction overflow truncates with a warning, and the remaining panic(panicInfo) branches return errors. All three repro statements from the review now parse without panicking, and a 197k-statement mysql-test corpus run went from 16 panics to 0. Happy to move that commit down into this PR instead if you'd rather not gate the fix on the stack.

(2) CI regen verification — added in this PR (e17f8a2): a parser-regen workflow deletes parser.go/hintparser.go, rebuilds them with make -C pkg/parser, and fails on any diff, so a .y edit can't ship without its regenerated output. The README sentence now describes the job that actually exists. It's green on this PR (22s).

(4)/(5) — both are on the known-gaps list from the fork audit: DEFAULT ((1+2)) / DEFAULT (CAST(...)) parse support is a parser compatibility follow-up, and DEFAULT (ABS(-1)) non-convergence is a differ canonicalization follow-up (related to the paren canonicalization work in #1134). Neither is fixed in this stack yet; say the word and I'll file both as issues.

(3)/(6) — agreed these are rollout notes rather than code changes: the one-time MODIFY COLUMN churn on parenthesized literal defaults is intended (#542), and the gbk/gb18030 introducer rejection fails closed by design.

Also while here: merged main forward twice (through #1138 and #1037) — #1037's new errors.Errorf call sites landed in the de-pingcap'd sqlescape, which auto-merged into undefined: errors; fixed in 336ba2f. The stacked branches (#1134, #1142) are merged forward and green as well.