feat(worms): create a functional beta for worms - #2
Merged
Conversation
Matches what we call them everywhere else. The key was parsed but unimplemented, so this only changes which name errors with its milestone and which one is unknown. The schema entry was still the old shape, an array of local Luau paths, so it moves to the settled one, a map of GitHub pins with an expanded form for an asset that is not named after the key. Leaving the old shape under a new name would have handed editors wrong completions.
The manifest and the second guest form, so both are interchangeable behind one Worm type. Nothing above this module learns which form it is talking to, which is the whole point of offering two. A Luau worm is an ordinary function. mlua marshals strings natively, so none of the ptr and len work the wasm form needs has an equivalent here, and a transform somebody writes in an afternoon needs no toolchain and no wasm32 target. It runs under Luau's own sandbox with a memory ceiling and an interrupt, so a worm cannot reach the filesystem, leak, or loop forever. Validation refuses the cases we agreed should be loud. An api mismatch names both versions, a run_order on a worm with no rules says there is nothing to order, and a worm declaring neither a frontend nor rules is refused rather than loaded and never called.
Three subcommands, all usable with no project and nothing installed. run loads a worm from a directory and pushes one file through it, so an author can see output before publishing anything. It reports the line count either way, because retain lines is the property most easily broken by accident and the hardest to notice. info prints what a worm declared without running it. types writes worm.d.luau, so a worm written in Luau typechecks and autocompletes under luau-lsp. Verified against the real language server, a compile that returns a number and a misspelled frontend key are both caught. frontend is required in the type for now, which is what makes the typo an error rather than a silently absent role.
Groundwork for the structured tier, which is the half of the worm API that does not exist yet. Our tree is Box and Vec of borrowed nodes, so it cannot cross a boundary. mlua userdata has to be static, and handing a guest a raw pointer is undefined the moment it stashes one in a global. So each file walks once into an owned table addressed by pre-order index, which is the node id the plan reserved in section 4.1 for this. Spans resolve to byte ranges, parents and children link both ways, and kind names are what filter in worm.toml will name. A worm never receives our node types, only ids and accessors, which is what lets the AST change without breaking every pinned worm.
The structured tier. A rule walks real nodes and queues real edits, rather than only getting the file as a string. Handles are two integers and carry the epoch they were minted under. Keeping one across files is caught with a sentence instead of silently reading the wrong tree, which there is a test for. Accessors reach the flattened table through Lua app data, so nothing of ours is ever a pointer the guest holds. Edits are deferred and measured against the original bytes, the same as the builtin rules, so a worm never sees another transform's output and everything lands in one splice. remove leaves newlines behind, so a worm keeps retain lines without having to know about it. Settings arrive once through init rather than per file, which is what the design promised and what the rules tier needs, since larvae has to know which rules are on before deciding whether to build a tree. A Luau worm gets real tables, the wasm form will get the TOML string, each what is natural for it. init is a field on the returned table and not a global, because the sandbox correctly makes _G readonly.
…oles kind() returns the union of literal kinds rather than string, so comparing against one autocompletes and a typo fails to typecheck instead of quietly never matching. Verified against luau-lsp, "CallExp" is rejected and names every valid kind in the error. The union is written by hand so it reads well, which means it can drift from the AST, so a test walks it in both directions and names whichever side is missing a kind. The Worm docs now show a front-end, a rule, and one worm holding both, instead of only a rule visit.
Host functions under a larvae import module, so a rule compiled from Rust reads kind, text, span, parent and children and queues the same deferred edits. No WASI, so the only thing a worm can reach is what we handed it. Strings come back in two calls rather than one. An accessor stages its text host side and answers with a length, the guest allocates that much and asks for the copy. A wasm function returns one number, and this keeps the host free of any allocator on the guest's side of the boundary. Handles carry the same u64 epoch the Luau form uses, checked before any accessor answers, so a node from a file we have moved past is refused rather than read out of the wrong tree. larvae-worm grows a Node type and a rules macro, so none of that is visible to a worm author. The fixture worm now ships three rules and the tests drive them through a real wasm module.
[worms] accepts a pin, an expanded table, or a path. The path form exists because a worm author has to be able to run their own before publishing it, and so does anyone debugging one. The registry is where checks spanning more than one worm live. A manifest name has to equal its key, since that key namespaces the worm's rules and is what [config.<key>] attaches to, so a disagreement would leave someone configuring a worm that never reads their settings. Two front-ends claiming one extension is refused rather than merged, because there is no sensible order and picking one quietly would be worse. Rule values resolve user over manifest default, and an off rule is simply absent so a worm never sees one it should not run. run_order resolves the three levels, user first, then the worm, then after larvae. A release pin says fetching is not implemented yet and points at path, rather than failing somewhere stranger.
A claimed file's bytes are replaced in a pre-pass and renamed to .luau, so every stage below receives ordinary Luau and none of them learn a worm was involved. That ordering is what the stage being separate enforces, rather than something call order has to remember, since markup reaching our lexer would report an error nobody could act on. The pre-pass is serial on purpose. A front-end touches the files that use its syntax and not the whole tree, and keeping worms off the parallel loop means no worm instance is ever shared between threads. Three things had to learn about the rename. Discovery, or a claimed file is copied through untouched and looks fine until Studio tries to run markup. process_file, which was reading the file again and would have handed a lexer the markup the worm was there to remove. And pruning, which computed its set from the pre-rename path and deleted the output the build had just written. Rule name validation now happens after worms load, because a worm's rules sit in [rules] beside ours and config is read before worms exist. A name neither side owns is still an error.
A worm's rules move out of [rules] and under [worms.<name>] rules. In the old shape a worm declaring const_requires would have been eaten by our typed config and never seen it, silently. Now [rules] is ours alone, a worm's are its own, and the two can share a name without meaning the same thing. That also removes the two phase validation the old shape needed, since config no longer has to wait for worms to know whether a rule name is real. A rule the user turns on that its worm does not declare is named rather than ignored. run_order now takes "before" or "after" as well as a number, and an author declares which side of our native rules their worm wants. Nobody should have to know what number our own stage is to say "run me first". A user writing an explicit number still wins over both.
A pin resolves to a release asset, downloads once, unpacks into the project cache under name and version, and records the sha256 so a later build does no network and a changed asset is caught. GitHub lets an asset be replaced under an unmoved tag, which a pin alone does not cover. Unpacking refuses any entry whose path would land outside the worm directory rather than sanitising it, because a half understood path is not one worth writing to. Absolute paths, .., drive prefixes and empty results are all rejected, and there is a test with a crafted archive proving nothing escapes. Tags resolve with or without a leading v, since projects tag both ways.
mlua::Lua is !Send, so a worm cannot be moved into a rayon closure at all, let alone shared. The registry now keeps artifacts and settings, which are shareable, and each worker builds its own instances the first time it meets a file needing one. A worker that never does pays nothing. Serial was not an option and the numbers say so rather than intuition. A rule crossing costs about 1.2us in Luau and 6.9us in wasm, so three thousand files would be 0.6s and 2.7s of dispatch against a build that is otherwise 25ms. Measured on 1000 files: 22ms with a rule on at 873% cpu, against 14ms with no worms at all. The boundary dominates rather than the parse, three times for Luau and sixteen for wasm, which is the opposite of what I assumed before measuring. So filter is the real cost lever: nodes are bucketed by kind once per file, a rule sees only what it declared, and a file matching nothing never builds a tree. Cross file worm state is undefined and documented as such. Work stealing decides which files a worker sees, so state surviving a file would make output depend on scheduling. A rule gets a node and a context and nowhere to put anything else.
larvae schema becomes larvae self code, and it stops assuming the only way to get completion is a URL at the top of somebody's config. If Even Better TOML is installed it writes a schema association into the editor's user settings instead. The pattern matches larvae.toml wherever it sits, so one entry covers every project a person will ever open rather than a .vscode folder per repo. Without the extension we fall back to the schema line, which is what any Taplo based editor reads. init no longer writes that line at all. Deciding how a project gets editor support belongs to the command that knows what is installed. worm types now adds worm.d.luau to the project's luau-lsp settings, so a worm typechecks as it is written. Project settings this time rather than the user's, because a definitions file lives beside the worm it describes. Settings are read as JSONC, since VS Code allows comments, and are only rewritten when something actually changed. Reserializing drops comments, so an unnecessary write would cost somebody their notes. The schema itself moves under crates/larvae beside the crate that ships it, and the milestone markers for work that has landed are gone.
The setting takes package name to path. documentationFiles sitting right beside it is an array, which is what made the wrong shape look right, and the extension ignores a list without complaining. Read the declared type out of the installed extension rather than guessing again, and pinned it with a test so the next person does not have to.
Three keys were declared and inert, which is the failure the plan refuses everywhere else. All three work now. run_order is real staging rather than a sorting of edits. Our own rules sit at [process] run_order, default 1, so a worm names a side instead of a number. Between two slots the buffer is spliced and re-lexed, so a rule at "after" genuinely reads our rewritten requires and one at "before" reads the original. There are tests for both, asserting on what the worm was handed. requires = "worm" now skips scanning entirely for files that worm produces, so we neither rewrite nor complain about requires we were told not to own. The front-end was running before the cache was consulted, so every build recompiled every claimed file. It moved behind the check, which works because the rename follows from the extension without asking the worm. It also moved into the parallel loop, since the per worker instances the rule half needed made a serial pass pointless. Forty markup files with a slow front-end went from 217ms every build to 37ms cold and 2ms warm. Two bugs found while measuring rather than reading. The cache retain set was keyed on the pre-rename path, so every cached entry for a worm's output was evicted and builds alternated between fast and full. And a line count change was dropping the file rather than warning about it, which is a warning's job.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.