Speed up native AST materialization#388
Merged
adamziel merged 8 commits intocodex/native-parser-object-grammar-cachefrom Apr 30, 2026
Merged
Speed up native AST materialization#388adamziel merged 8 commits intocodex/native-parser-object-grammar-cachefrom
adamziel merged 8 commits intocodex/native-parser-object-grammar-cachefrom
Conversation
b7a83a9 to
a6f897d
Compare
eaa3b8c to
38ebab5
Compare
a6f897d to
826dd24
Compare
38ebab5 to
00e0ac8
Compare
826dd24 to
74f1c3f
Compare
00e0ac8 to
9925824
Compare
74f1c3f to
1bce38a
Compare
4c45ada to
b62b39b
Compare
1bce38a to
dcf5c96
Compare
b62b39b to
859da96
Compare
dcf5c96 to
e9a0923
Compare
859da96 to
4bf2305
Compare
4bf2305 to
a516b67
Compare
f0bb626 to
56e2f94
Compare
a516b67 to
a0a093d
Compare
56e2f94 to
f333a57
Compare
a0a093d to
780e349
Compare
f333a57 to
c1dc9b3
Compare
780e349 to
0ff98d2
Compare
c1dc9b3 to
76aebba
Compare
0ff98d2 to
3123e7a
Compare
76aebba to
b0af9dc
Compare
3123e7a to
2371b7a
Compare
b0af9dc to
31a2778
Compare
2371b7a to
b92dc18
Compare
31a2778 to
468d8ce
Compare
b92dc18 to
c1f5074
Compare
The previous helper name (has_unmaterialized_native_ast) implied a runtime check for native-extension presence. It's actually a per-instance state flag tracking whether this node's children have been copied into PHP. was_mutated() reads that intent more directly.
When a native parser is in use, expose query results through a node class that defers child materialization until callers actually walk the tree. The base WP_Parser_Node::$children visibility is loosened to protected so the facade can populate it on demand.
When a native parser is in use, expose query results through a node class that defers child materialization until callers actually walk the tree. The base WP_Parser_Node::$children visibility is loosened to protected so the facade can populate it on demand.
When a native parser is in use, expose query results through a node class that defers child materialization until callers actually walk the tree. The base WP_Parser_Node::$children visibility is loosened to protected so the facade can populate it on demand.
468d8ce to
09b9c1a
Compare
c1f5074 to
57e5c3e
Compare
57e5c3e to
12e984f
Compare
076b7e5
into
codex/native-parser-object-grammar-cache
21 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WP_MySQL_Parserinstance inside the SQLite driver and reset its token stream per queryreset_tokens()to the PHP parser polyfill and the Rust native parserWP_MySQL_Native_Parser_Node, while keeping PHP child materialization for mutation__SSE2__during binding generationStack
This is the top PR in the native MySQL lexer/parser stack. The stack is split so each GitHub diff shows one reviewable concern:
#384 Extract MySQL lexer and parser polyfills
trunk->codex/native-parser-php-facadeWP_MySQL_LexerandWP_MySQL_Parseras thin PHP subclasses#385 Add optional native parser routing
codex/native-parser-php-facade->codex/native-parser-class-routingWP_MySQL_Native_*PHP classes#386 Add lazy native parser node facade
codex/native-parser-class-routing->codex/native-parser-node-facadeWP_Parser_Nodeas the plain PHP tree nodeWP_MySQL_Native_Parser_Node extends WP_Parser_Nodefor native-backed lazy AST nodes#381 Add lazy native AST facade
codex/native-parser-node-facade->codex/native-lazy-ast-facadeWP_MySQL_Native_Parser_Node#387 Cache native grammar on parser grammar object
codex/native-lazy-ast-facade->codex/native-parser-object-grammar-cacheWP_Parser_Grammar::$native_grammaron the PHP sideThis PR, #388 Speed up native AST materialization
codex/native-parser-object-grammar-cache->codex/native-parser-bulk-materializationWhy
The native lexer/parser itself is fast, but the PHP-facing path can lose that benefit if each query repeatedly rebuilds native parser state or forces full PHP AST materialization. On the current stack, #387 already removes the large grammar export/hash cost. This PR removes the remaining per-query parser construction churn and restores the native AST accessor path for descendant-heavy SQLite driver workloads.
Measurements
Environment: local PHP 8.2 via the native build helper, release Rust extension, current top of this PR.
Focused constructor/reset benchmark over 5000 unique SELECT queries:
reset_tokens()onlyget_descendants()The previously reported ~622 us/query constructor cost does not reproduce on this stack because #387 already caches the native grammar on the PHP grammar object. Parser reuse still removes most of the remaining constructor overhead.
SQLite facade smoke workload:
Command:
Testing
cargo fmt --checkgit diff --checkcomposer run check-cscomposer run testfrompackages/mysql-on-sqlitephp -d extension=packages/mysql-on-sqlite/ext/wp-mysql-parser/target/release/libwp_mysql_parser.so packages/mysql-on-sqlite/vendor/bin/phpunit -c packages/mysql-on-sqlite/phpunit.xml.distTMP_TEST_NATIVE_QUERY_COUNT=250 ./tmp-test-native/run.sh