Add lazy native parser node facade by adamziel · Pull Request #386 · WordPress/sqlite-database-integration

adamziel · 2026-04-28T15:21:52Z

Summary

Adds WP_MySQL_Native_Parser_Node, a WP_Parser_Node subclass the native MySQL parser extension constructs when it produces an AST. Read methods delegate into the Rust-owned AST so children aren't copied into PHP unless something walks the tree. On the first append_child() / merge_fragment(), the node copies its native children into the inherited $children array and behaves like a plain WP_Parser_Node from then on — that's what was_mutated() tracks.

Mutation matters because WP_PDO_MySQL_On_SQLite rewrites parsed queries (e.g. synthetic count(*) expressions) by appending children to existing nodes.

Other change

WP_Parser_Node::$children private → protected so the subclass can populate it during materialization.

JanJakes

Looks good! Thanks!

Forward-port the renamed has_unmaterialized_native_ast helper and expanded class docs from PR #386 to the tip of the stack. Earlier mid-stack commits temporarily reverted the lazy hedge, masking the new name during the rebase.

When a native parser is in use, expose query results through a node class that defers child materialization until callers actually walk the tree. The base WP_Parser_Node::$children visibility is loosened to protected so the facade can populate it on demand.

## Summary - reuse one `WP_MySQL_Parser` instance inside the SQLite driver and reset its token stream per query - add `reset_tokens()` to the PHP parser polyfill and the Rust native parser - restore native parser-node accessor fast paths in `WP_MySQL_Native_Parser_Node`, while keeping PHP child materialization for mutation - fix the local native extension build helper for Nix/libclang bindgen by undefining `__SSE2__` during binding generation ## Stack This is the top PR in the native MySQL lexer/parser stack. The stack is split so each GitHub diff shows one reviewable concern: 1. [#384 Extract MySQL lexer and parser polyfills](#384) - `trunk` -> `codex/native-parser-php-facade` - extraction-only PHP refactor - moves the existing PHP lexer/parser implementations into polyfill classes - keeps public `WP_MySQL_Lexer` and `WP_MySQL_Parser` as thin PHP subclasses 2. [#385 Add optional native parser routing](#385) - `codex/native-parser-php-facade` -> `codex/native-parser-class-routing` - adds fallback `WP_MySQL_Native_*` PHP classes - routes the public lexer/parser classes through native classes when the Rust extension provides them - adds the minimal PHP grammar-export bridge for the native parser 3. [#386 Add lazy native parser node facade](#386) - `codex/native-parser-class-routing` -> `codex/native-parser-node-facade` - keeps `WP_Parser_Node` as the plain PHP tree node - adds `WP_MySQL_Native_Parser_Node extends WP_Parser_Node` for native-backed lazy AST nodes - keeps native AST handles and native accessor delegation out of the base node class 4. [#381 Add lazy native AST facade](#381) - `codex/native-parser-node-facade` -> `codex/native-lazy-ast-facade` - implements the Rust lexer/parser extension and lazy native AST facade - makes the Rust extension instantiate `WP_MySQL_Native_Parser_Node` - adds native-extension CI coverage for the SQLite driver and WordPress PHPUnit tests - includes the local SQLite facade smoke benchmark 5. [#387 Cache native grammar on parser grammar object](#387) - `codex/native-lazy-ast-facade` -> `codex/native-parser-object-grammar-cache` - restores the object-attached native grammar cache - adds only `WP_Parser_Grammar::$native_grammar` on the PHP side - removes the Rust content-hash cache that walked the whole exported grammar on every parser construction 6. This PR, [#388 Speed up native AST materialization](#388) - `codex/native-parser-object-grammar-cache` -> `codex/native-parser-bulk-materialization` - optimizes native-to-PHP AST access after the grammar-cache performance restoration - reuses the SQLite driver's parser instance instead of constructing it per query ## Why The native lexer/parser itself is fast, but the PHP-facing path can lose that benefit if each query repeatedly rebuilds native parser state or forces full PHP AST materialization. On the current stack, #387 already removes the large grammar export/hash cost. This PR removes the remaining per-query parser construction churn and restores the native AST accessor path for descendant-heavy SQLite driver workloads. ## Measurements Environment: local PHP 8.2 via the native build helper, release Rust extension, current top of this PR. Focused constructor/reset benchmark over 5000 unique SELECT queries: | Phase | Time | | --- | ---: | | native tokenize | 22.62 us/query | | fresh native parser constructor only | 2.31 us/query | | reusable parser `reset_tokens()` only | 0.32 us/query | | reusable parser reset + parse + `get_descendants()` | 157.06 us/query | | constructor/reset ratio | 7.3x | The previously reported ~622 us/query constructor cost does not reproduce on this stack because #387 already caches the native grammar on the PHP grammar object. Parser reuse still removes most of the remaining constructor overhead. SQLite facade smoke workload: Command: ```bash TMP_TEST_NATIVE_QUERY_COUNT=250 ./tmp-test-native/run.sh ``` | Workload | PHP fallback | Native extension | Speedup | | --- | ---: | ---: | ---: | | 250 generated queries, including 1 x 2000-row insert | 4.060s | 0.525s | 7.73x | ## Testing - `cargo fmt --check` - `git diff --check` - `composer run check-cs` - `composer run test` from `packages/mysql-on-sqlite` - `php -d extension=packages/mysql-on-sqlite/ext/wp-mysql-parser/target/release/libwp_mysql_parser.so packages/mysql-on-sqlite/vendor/bin/phpunit -c packages/mysql-on-sqlite/phpunit.xml.dist` - `TMP_TEST_NATIVE_QUERY_COUNT=250 ./tmp-test-native/run.sh`

adamziel mentioned this pull request Apr 28, 2026

Speed up native AST materialization #388

Merged

adamziel force-pushed the codex/native-parser-class-routing branch from 6c17f23 to 0dda506 Compare April 30, 2026 11:52

adamziel force-pushed the codex/native-parser-node-facade branch from b20499f to 77a45df Compare April 30, 2026 11:52

adamziel force-pushed the codex/native-parser-class-routing branch from 0dda506 to 0f3cbb6 Compare April 30, 2026 11:59

adamziel force-pushed the codex/native-parser-node-facade branch from 77a45df to 830a9b2 Compare April 30, 2026 11:59

adamziel force-pushed the codex/native-parser-class-routing branch from 0f3cbb6 to c265cf5 Compare April 30, 2026 12:15

adamziel force-pushed the codex/native-parser-node-facade branch from 830a9b2 to e66bab3 Compare April 30, 2026 12:16

adamziel force-pushed the codex/native-parser-class-routing branch from c265cf5 to 00c2687 Compare April 30, 2026 12:21

adamziel changed the base branch from codex/native-parser-class-routing to trunk April 30, 2026 12:22

adamziel force-pushed the codex/native-parser-node-facade branch from e66bab3 to bb3b8e6 Compare April 30, 2026 12:24

JanJakes approved these changes Apr 30, 2026

View reviewed changes

adamziel force-pushed the codex/native-parser-node-facade branch from bb3b8e6 to 89bc6a4 Compare April 30, 2026 12:37

adamziel force-pushed the codex/native-parser-node-facade branch from 89bc6a4 to cd22199 Compare April 30, 2026 12:40

adamziel force-pushed the codex/native-parser-node-facade branch from cd22199 to 23b1c02 Compare April 30, 2026 12:43

adamziel marked this pull request as ready for review April 30, 2026 12:49

adamziel merged commit b6473ef into trunk Apr 30, 2026
16 checks passed

adamziel mentioned this pull request Apr 30, 2026

Add native Rust-based MySQL parser extension #381

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add lazy native parser node facade#386

Add lazy native parser node facade#386
adamziel merged 1 commit intotrunkfrom
codex/native-parser-node-facade

adamziel commented Apr 28, 2026 •

edited

Loading

Uh oh!

JanJakes left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

adamziel commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Other change

Uh oh!

JanJakes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adamziel commented Apr 28, 2026 •

edited

Loading