feat: Compile queries to Rust#558
Merged
Merged
Conversation
The new crate hosts everything query execution needs independently of how the query program is delivered: cursor navigation, checkpoint/frame stacks, the effect log, regex DFAs, execution limits, and the vocabulary shared with the compiler (`Nav`, id newtypes, `NodeClass`/`SkipClass`). The VM consumes it via re-exports, so `plotnik-lib` paths are unchanged. `plotnik-lib` gains a default-on `vm` feature gating the tree-sitter dependency (plus the tree renderers); `--no-default-features` builds the compiler without linking tree-sitter's C runtime. Both build scripts now compile against the C-free lib. `make check` and CI guard the new configuration.
Lowering now splits at the executor fork point: `lower_semantic` produces `SemanticNfa` — the optimized post-dedup IR every backend shares — and `pack_lowered` packs it for the wire. The bytecode path composes both; code generation will consume the fork-point IR directly. Every label records the compilation window that allocated it (definition body, consuming-body variant, or entrypoint wrapper). Post-build passes only delete and rewire, so allocation-time origins stay valid; pack truncates the origins of dead labels whose ids it reuses for cascades, keeping wire artifacts distinguishable from source-attributed states. The new label-space dump mirrors the bytecode dump's visual grammar while keeping what the wire format erases: real member names on `Set`/`EnumOpen`, callee names on calls, definition-name section headers, and inline predicate text. Exposed as `CompiledQuery::dump_nfa`; `04-emit` fixtures pin it in an `nfa` section beside the packed bytecode.
`compiler/typegen/rust` renders every definition's output as a Rust
item: structs, enums with struct variants for the anonymous payloads,
`pub type` aliases for scalar and wrapper outputs, and a marker comment
for void definitions. Unlike the TypeScript backend it consumes the
analysis-level type model, where `Ref` recursion is still explicit —
`Box` lands at exactly the occurrences that close a by-value cycle
(`Vec` already provides indirection), and the `'t` node lifetime
appears only on types that transitively hold a node. Exposed as
`CompiledQuery::to_rust`.
Rust spelling is hygienic without renaming the data: keyword captures
render as raw identifiers (unrawable ones get a trailing underscore),
std paths are absolute so a definition named `Option` cannot shadow the
prelude, and serialized keys and tags keep the original query strings.
Opt-in serde support generates `SerializeWithSource` impls against the
new `plotnik-rt` `serde` feature — plain `Serialize` is impossible
because node text lives in the source buffer. The serialized shape is
byte-for-byte the VM's materialized JSON (nodes as `{kind, text, span}`,
one flat null per absent value), which is the value-level conformance
channel generated matchers will be diffed through.
`05-typegen` now renders `typescript` and `rust` sections from the one
authored corpus (the section formerly named `types` is `typescript`
everywhere), and a trybuild test compiles every golden `rust` section
against the real `plotnik-rt`.
A non-quantified child pattern committed to the first kind-matching sibling: descent failure had no choice point to resume the sibling loop, silently dropping matches ((program (expression_statement (number) @n)) over 'a; 5;' found nothing). The Call twin: a field-constrained callee retry checked the field only at the immediate next sibling, so an anonymous separator ended the search. Uniform choice-point discipline, in the shared engine layer: - Checkpoint resume is now an enum: Branch | Call(CallResume) | Match. Accepting a candidate in an engine-owned sibling search (non-exact Down*/Next* nav, Nav::is_sibling_search) pushes a match-retry checkpoint when the skip policy admits the candidate into the gap; backtracking re-runs the same match's candidate search past it, replaying effects and branches exactly as the original acceptance. Branch checkpoints stack above the resume, so ordered-choice priority is preserved. - Call retry now loops the field scan under its skip policy, mirroring the navigate-time search, instead of dying at the first non-field sibling. - Nav::skip_policy() is the single source of the nav-to-policy mapping (navigate() and backtrack resume both derive from it). - Lowering keeps every search under exactly one retry owner: emit_wildcard_nav exact-ifies position-search internals, which the engine's resume gate then ignores; wrapped bodies are StayExact. Behavior of these steps is unchanged (Any constraint never consults the policy). Three fixtures froze the hole as expected output and are renamed to assert the corrected semantics; field_call_retry_skips_separator.txt pins the Call-retry case. Alternation-as-field-value wrappers gain correct multi-candidate retry for free (candidate_matches re-verifies the field on resume).
A named-node capture binds its own field alongside the captures that bubble up from inside it (`(named (child) @x) @x`). The node-capture merge added its field with a raw `BTreeMap::insert`, silently overwriting a colliding child capture: the query compiled and ran, emitting two JSON entries under the same key while the inferred type declared only one. Route that insert through the same duplicate-capture gate the sibling sequence path already uses. Extract `insert_scope_field` (add one field to a scope, reporting `DuplicateCaptureInScope` on a name clash) and fold `merge_scope_fields` through it per field. Detection is by interned capture name; the node path draws the caret on its `@x` suffix.
clap fills the query slot from the first positional and the source slot from the second. When `-q`/`-s` also supplies that role inline, the loaders silently prefer the inline text and drop the positional, so `run -q '...' q.ptk app.js` ran the inline query and ignored `q.ptk`. Add `reject_ambiguous_inputs` and call it from the query+source commands (run, trace, inspect, ast) before loading: a query given both ways, or a source given both ways, is now a fatal usage error (exit 2). Query-only commands (check, dump, infer) are untouched — they deliberately ignore an extra positional so a run-shaped command line still works.
The spec restricts a field value to a node, an alternation, or a
quantifier of those; sequences `{...}` are not allowed. The field-arity
check caught multi-element sequences (arity Many) but let a single-element
`field: {(x)}` through, since its arity is one — the query compiled and
matched as if the field held `(x)` directly.
Reject any sequence core in the field-value check, alongside the existing
multi-node arity rejection, reusing the same `FieldSequenceValue`
diagnostic. Multi-element sequences and multi-node references keep their
exact message and span; only the single-element gap closes.
The step tracer had two rendering defects:
- Naming: only entrypoints carry a name, so every call into an internal
definition body rendered as `(?)` and its section header as `?:`. Fall
back to the target address `@{ip}`, exactly as the bytecode dump renders
the same call.
- Call-stack desync: `trace_backtrack` popped the checkpoint-IP stack but
never the call stack, while the VM restores its frame arena in one bulk
move. After backtracking across a call the tracer still believed it was
inside the callee, so later returns popped the wrong frame and a
top-level return lost its `◼` marker, trailing a dangling `?:` header.
Pass the checkpoint's restored `recursion_depth` to the hook and truncate
the call stack to it.
The `Tracer` trait gains the depth argument; NoopTracer and RecordingTracer
(which tracks its own shadow stack) ignore it. Regenerates 06-vm fixtures.
`(ERROR)` lowered to `Named(None)` — the same wildcard as `(_)` — so it matched the first named node anywhere rather than a parser error node. The `parse_error_node` fixture only passed because the error node happened to be the first named child. Lower `(ERROR)` to tree-sitter's builtin error symbol `Named(Some(NodeKindId::ERROR))` (`0xFFFF`, always named): the VM's existing named-kind arm then matches exactly `is_named() && kind_id() == 0xFFFF`, which is `is_error()`. The symbol has no grammar entry, so the bytecode and NFA renderers special-case it back to `ERROR`. `(MISSING)` still lowers to a wildcard; its orthogonal flag lands separately.
`(MISSING …)` lowered to a bare wildcard, dropping both the missing-ness and the kind argument: `(MISSING "}")` matched any node — in the VM fixture it captured an `expression_statement` instead of the inserted brace. A missing node is a normal-kind node carrying tree-sitter's orthogonal `is_missing()` flag (unlike `(ERROR)`, which has its own symbol), so it needs its own bit. Claim one of the Match counts word's three reserved bits for a `missing` flag (forcing at least `Match16`, since `Match8` has no counts word); the VM gates on `is_missing()` after the usual kind check. Resolve the argument too: `(MISSING)` → any missing node, `(MISSING identifier)` → a missing named token, `(MISSING ";")` → a missing anonymous token. Validate the argument at compile time: an unknown kind reports `UnknownNodeKind` like any node reference, and a non-leaf kind such as `(MISSING binary_expression)` is rejected — error recovery only ever inserts leaf tokens, so it could never match.
The new codegen backend compiles the optimized pre-pack NFA to a self-contained Rust module over `plotnik_rt`: one dispatch loop with dense u16 state ids, per-state arms with every operand const-folded (node kinds, field ids, member indices, predicate literals, embedded regex DFAs), and the VM's checkpoint discipline transcribed handler-for-handler. State consts mirror the NFA dump's labels and every arm carries its instruction in dump format, so generated code lines up with `dump_nfa` 1:1. The execution-state core (cursor, frames, checkpoints, effect log, suppression) moves from the VM into `plotnik_rt::Engine`, shared by both executors; the VM refactor is behavior-identical (zero fixture churn). Conformance: `codegen_conformance` compiles every runnable 06-vm fixture twice, runs the bytecode VM in-process as the oracle, and bakes its committed effect stream plus the generated matcher into one trybuild program that is built and executed — 241 fixtures, trace equality node-for-node. The new `07-codegen` golden stage snapshots emitted matchers, and `rust_sections_compile` (née `rust_typegen_compile`) compile-gates both RUST and MATCHER golden sections against the real runtime.
The generated module is now the complete proc-macro payload: typed output items, per-type trace readers, parse/try_parse (matches for void definitions), and the matcher machinery shielded in a nested mod so engine names can never collide with query type names. Readers replay the committed effect stream as a generated deserializer: struct scopes peek the balancing Set to dispatch value-first fields, arms match the union of nominal twins' absolute member indices, and nested optionals read as one flat null. Metering is compiled in (const-generic driver transcribing the VM loop head; rt::LimitError; limits chosen at generation time via MatcherConfig). Conformance now checks two channels per 06-vm fixture: the effect stream against the VM, and the serialized typed value against the VM's materialized JSON, with try_parse required to equal parse and a limit-trip module pinning the compiled-in budget. 244 modules conform; the corpus grows twin-annotation and empty-list-dispatch fixtures that pin the reader-specific edges.
The crates.io name `plotnik` moves to a new facade crate (`query!` +
`rt` + `tree_sitter` re-exports); the CLI package becomes `plotnik-cli`
(binary name unchanged). `plotnik-macros` compiles queries at build time
with no built-in language registry: `grammar = "..."` resolves any
dependency-graph package shipping a grammar.json (tree-sitter-*,
arborium-*, or a local crate) via cargo metadata, a subgrammar selector
("pkg/name") for multi-grammar packages, or a grammar.json path. Every
generated module now bakes expected kind/field-id tables and verifies
the tree's language on first run, panicking on grammar version skew.
Post-review hardening of the fork-point codegen from the previous three commits. No matcher transcription bugs surfaced; these close robustness gaps that contradicted the runtime's own stated guarantees. - Verify the tree's language per run, not once per process. A long-lived multi-language host (LSP, lint daemon) that fed a foreign tree after the first match previously skipped the check and compared field ids against the wrong grammar. The check now pointer-compares the language on every call. - Meter replay depth as a first-class resource. The typed replay recurses once per nested value, so hostile input could abort the process past the native stack even under the memory limit. Metered `try_*` entry points now refuse a match that nests past a compiled-in ceiling (`REPLAY_DEPTH_AUTO`, 1024) before replay touches the stack, exposed via the `depth` macro arg and `Config::depth`. - Make `TraceReader` replay linear. A precomputed `Set` index replaces the per-field subtree rescan that made replay `O(n*depth)`. - Box recursive fields per occurrence, not per referenced type. `Q.expr` is no longer boxed when `Q` sits on no cycle; only genuinely on-cycle edges (e.g. `Paren.inner`) carry the `Box`, and the readers stay in sync. - Reject definition names that collapse to one snake_case spelling with a real diagnostic instead of a rustc duplicate-definition error from inside the expansion, and keep the `F_*` field-id consts injective. - Cache grammar parsing per rustc process and scope package resolution to the invoking crate's dependency closure. - Drop the doubled `error: error:` prefix on macro diagnostics, fix the stale `deserialize_dfa` SAFETY note, and name endianness (not "the compiler") in the big-endian panic message. - Document the feature: README roadmap tick plus a compile-time-queries section, and runtime-engine.md.
- Examples showed `query!` followed by runtime statements, implying function-body usage; that fails to compile since the macro must sit at module scope. Move `query!` to module scope and the runtime code into `fn main`, so the snippets are copy-pasteable. - Note at the call site that `parse` is the trusted-input path and `try_parse` applies the compiled-in step/memory/depth limits for untrusted source. - Put the query string first in every example; the argument parser already accepts any order, so this is convention only. - Add a trybuild fixture pinning that a syntax error in a `file =` query renders the full annotated diagnostic, not a bare span on the argument.
…etering Complete the typed Rust surface for generated matchers: - Three tiers keyed on arity + captures: `parse` + `matches` for nominal struct/enum outputs, `matches`-only for void tags, no top-level surface for `Arity::Many` fragments. - Inherent methods plus `Matches`/`Parse` traits and `plotnik::parse`/`matches` free functions — porcelain over plumbing. - Step and memory ceilings meter independently through `run`'s const generics: an unbounded resource folds its check out at monomorphization, so a fully unbounded policy never counts steps or samples `heap_bytes`. - Replay-depth guard lives in recursive readers, decoupled from metering, with the default ceiling scaled by the emitter's max reader-frame estimate; `depth = unbounded` is rejected at the macro surface. - `matches` runs with data effects suppressed (OOM-safe) and never reports `Depth`; `LimitExceeded` is the single safe-run error. Removes the engine's `effect_depth` tracking (superseded by data suppression) and reshoots the codegen goldens.
zharinov
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
query!proc-macro. Give it a query and a grammar; it generates Rustthat walks the parsed tree and returns typed results. The generated code
doesn't use the VM.
parse— returns the match as typed data, orNoneif nothing matched.matches— returnstrue/false.plotnik-rt(the runtime engine, shared by the VM and thegenerated code) and
plotnik(facade that re-exportsquery!, the runtime,and
tree_sitter).Why
A fixed query compiled to Rust at build time runs faster than going through the
VM, and hands back typed results directly.
Notes
Foo::parse(&tree, src)orplotnik::parse::<Foo>(&tree, src).memory. Set limits in the macro:
stepsandmemory:auto(scales with input size), a number, orunbounded.depth(recursion depth when building typed results):autoor a number.Not
unbounded— a deep recursive value overflows the stack when it drops.matches. Adefinition that matches many nodes (
Ids = (id)*) isn't callable on its own;you nest it inside another pattern.
matchesbuilds no output, so a yes/no check can't be pushed into using lotsof memory.
refactors; the feature is thefeatcommits.