fix(web/guides): parse-compile {test:compile} blocks instead of executing them in a bare engine - #3052
Conversation
…tead of bare-engine execution The compile driver's native mode executed each snippet in a bare Lucee engine with no framework or app context, failing 312/317 live-tree blocks on missing context rather than content (#3041). Rewrite the native path to verify what {test:compile} promises — parse/compile, never execute: - component bodies: strip the declaration header, wrap each inner body in a never-invoked function shell (the engine compiles the whole script before executing, so syntax errors fail while framework functions never need to resolve) - tag bodies: close the engine's implicit <cfscript> wrapper and emit the body inside <cfif false> — compiled in template context, never executed, never double-wrapped - script fragments (config/spec/plain): single never-invoked function shell, which also legalizes top-level var declarations Add a file-based expected-failure allowlist (scripts/verify-docs/expected-failures.json, override via VERIFY_DOCS_ALLOWLIST) keyed on file path + body sha256 with mandatory reason + issue refs, so CI can gate the live tree incrementally. The report prints each failing block's body hash for copy-paste, flags stale entries that now pass, and flags orphan entries on full-tree runs. Invalid allowlists exit 2. cli and tutorial drivers are untouched. Fixes #3041 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
![wheels-bot[bot]](https://avatars.githubusercontent.com/in/3657588?s=60&v=4){kind=small}
There was a problem hiding this comment.
Wheels Bot — Reviewer
TL;DR: This PR rewrites the verify-docs compile driver's native mode to parse/compile {test:compile} blocks by sniffed wrap kind (component / tag / script) instead of executing them in a bare engine, and adds a validated, hash-keyed expected-failure allowlist. The design is sound, the wrap builders are pure and unit-tested, e2e tests cover both pass and fail directions per kind, and the allowlist/exit-code plumbing is correct (hard failures only drive exit 1; invalid allowlist exits 2 before any work). Verdict: comment — no correctness, security, or cross-engine blockers; two minor findings below.
Correctness
Nothing blocking. Items I checked and confirmed clean:
verify-docs.mjs:99hashesr.bodyfor all results including tutorial/cli cumulative ones — safe, becauseextractExamples(lib/extract.mjs:58-65) always setsbodyand drivers spread{ ...ex, ...result }.lib/report.mjs:13-17classifiesexpectedbefore countingfail, so allowlisted failures never reach the exit code;verify-docs.mjs:105exits on hard failures only.- The e2e fixture body in
test/orchestrator.test.mjs:874is unbalanced, so it fails in both native and fallback modes as the test comment claims (in native mode it returns theunbalanced braceserror fromextractDeclarationInners). - CI (
.github/workflows/docs-verify.yml:116) runspnpm verify:docswith no args, soisFullDefaultRun(verify-docs.mjs:36) correctly enables orphan warnings on the real CI invocation.
One minor nit:
-
drivers/compile.mjs:177-182—neutralizeComponentOnlyStatementsmatches any line whose first word isproperty, not just property declarations:/^([ \t]*)property\b[^;\n]*;?[ \t]*$/gimA line like
property = "x";(identifier assignment) or a lonepropertyargument in a multi-line call also gets commented out. In practice this silently excludes those lines from the parse check rather than causing false failures, and zero live blocks are affected — but tightening the regex to require a declaration shape (e.g. reject when the next non-space char is=,.,(or[) would keep the neutralization scoped to what the doc comment says it targets.
Tests
drivers/compile.mjs:86claimsinterfacesupport for the component wrap kind, andtest/compile.test.mjs:648asserts the sniff (interface { function handle(req, next); }→component), but there is no end-to-end compile test for an interface body. Interface methods are bodiless (function handle(req, next);), and a bodiless function declaration inside the never-invoked function shell is a plausible parse error — i.e. the first doc page that adds aninterface{test:compile}block may false-fail. No live block usesinterfacetoday (verified by grep oversrc/content/docs/v4-0-0), so nothing is broken now. Suggestion: add a native-mode e2e test for an interface block (and handle bodiless declarations if it fails), or document the limitation inVALIDATION.mdnext to the existing header-typo caveat.
Otherwise coverage is good: per-kind pass/fail e2e tests with clean fallback-mode skips, 10 allowlist unit tests covering shape validation, matching, stale and orphan warnings, and two entrypoint e2e tests for masking and invalid-allowlist exit 2.
Docs
VALIDATION.md is updated thoroughly and honestly (the "what this does and does not verify" section is exactly the right caveat). No changelog fragment is needed — this is internal docs-harness tooling, not user-facing framework behavior, consistent with prior harness/CI changes.
Commits
Single commit dc1b46c81: header is 98 chars (≤ 100), valid type fix, optional scope, DCO sign-off present and matching the author. PR title is also a valid conventional-commit header for the squash merge. Clean.
Security
The change strictly reduces execution of doc-supplied code (the old driver executed every body; the new one compiles behind never-invoked shells / <cfif false>). runExec spawns without a shell, and the allowlist requires attributable reason + issue refs, so masked failures can't accumulate silently. No findings.
1 participant
Fixes #3041
What
The verify-docs compile driver's native mode handed each
{test:compile}body towheels cfml <body>, which executes the snippet in a bare Lucee engine with no framework and no app context. Valid Wheels examples cannot pass that bar — 306 of 364 live-tree blocks failed on missing context, zero on content (per the triage atdocs/superpowers/audits/2026-06-verify-docs-triage.md). This PR makes the native path verify what{test:compile}promises: parse/compile, never execute, plus a file-based expected-failure allowlist so CI can gate incrementally.How
drivers/compile.mjssniffs each body into one of three wrap kinds and builds a program the engine compiles without running (buildNativeProgram, exported and unit-tested):getComponentMetadata/new/createObjectare all unusable in the LuCLI script-engine context (every component-path lookup dies onNativeException: "searchLocal" is null, verified against brew wheels 4.0.3), so a temp-.cfcfixture-app compile is not reachable from this binary. Instead the declaration header is stripped (comment/string-aware brace scanner, handles multiple declarations per block) and each inner body is wrapped in its own never-invoked function shell. The engine compiles the entire script before executing anything, so syntax errors fail whilevalidatesPresenceOf/hasMany/describenever need to resolve. Limitation (documented in VALIDATION.md): typos in the header itself (e.g. theextendstarget) are not checked; top-levelpropertydeclarations are neutralized (zero live blocks use them).<cfscript>…</cfscript>; the driver closes that wrapper and emits the body inside<cfif false>…</cfif>, so tag bodies compile in template context (mismatched tags, bad<cfoutput>expressions are caught) without execution and without the double-<cfscript>crash.var(how the framework runsconfig/services.cfm).Fallback mode (old CLIs where
wheels cfmlalways exits 0) is unchanged: bracket-balance only. cli and tutorial drivers are untouched.Allowlist (
scripts/verify-docs/expected-failures.json, override viaVERIFY_DOCS_ALLOWLIST): entries keyed on file-path suffix + first 12 hex of the body sha256, with mandatory non-emptyreasonandissue(#NNNNor GitHub URL). The report prints every failing block's hash for copy-paste, warns on stale entries that now pass, warns on orphan entries during full-tree runs, and exits 2 on an invalid allowlist. Ships empty — the live tree needs no entries.Acceptance evidence (issue #3041)
All runs used the brew-installed wheels CLI 4.0.3 at
/opt/homebrew/bin/wheels(the binarylib/exec.mjsresolves). The mode probe (wheels cfml 'throw(message="probe")') exits 1 on this binary, so every run below was native mode — not the bracket-balance fallback.testing/index.mdx:40WheelsTest spec) passes — dedicated test incompile.test.mjs.runCompile passes a valid CFC blocktest, which failed before this change, now passes.compile.test.mjs(componentx = ;, scriptset(dataSourceName=);, mismatched<cfloop>/<cfif>), plus a spot check corrupting the live G1 block in a copy oftesting/index.mdx→ exit 1 withnative(component wrap): … Invalid identifier, at [8:38] … expect(post.valid())..toBeFalse(;.WHEELS_FRAMEWORK_PATHset):31b9f0e1f{test:cli}/{test:tutorial}coverage.Tests
pnpm test:docs-harness: 56 / 57 pass (was 26/29 at baseline). New: 18 compile-driver tests (sniffing + program building + per-kind end-to-end through the engine, native-only assertions skip cleanly in fallback mode), 10 allowlist unit tests, 2 verify-docs end-to-end tests (allowlist masks a failure → exit 0 with "expected failure" section; invalid allowlist → exit 2). The one failure,tutorial driver walks mini-tutorial end to end, fails identically on the baseline before this change (wheels server did not listen on port … within 60000ms— the brew CLI cannot boot a server from a.claude/worktreescheckout on this machine); the tutorial driver is untouched by this PR.Out of scope
.github/workflows/docs-verify.ymltrigger paths +continue-on-errorremoval — the issue notes that fix ships separately.{test:compile}) — they all parse, so they pass; the tagging-policy cleanup remains a docs decision.🤖 Generated with Claude Code