fix(website): llms corpus leaves JS escapes in fenced code samples

[vivek7405]

Every line anchor below was re-verified against HEAD 79fc28fc. The anchors in the first version of this body had drifted by five lines and are corrected here.

Problem

The llms corpus builder folds JS escapes out of PROSE template holes but not out of FENCED code samples, so prose and code on the SAME docs page disagree about how to write a form binding.

#1261 (PR #1331, merged as 3db6e6e9) taught the prose pipeline that a docs page's source is a JS template literal, so \${x} there is escaped literal text and a backtick has to be written \`. Both prose keep-passes run their text through unescapeJs. The fenced-sample capture was left alone, and it copies sample text straight out of the same source with only decodeEntities applied.

Re-measured at HEAD 79fc28fc by walking every fenced region of all 44 doc pages (485 fenced blocks). These match the numbers the first version of this body carried, so nothing here has drifted.

	count
escaped backticks (\`) inside fences	512
escaped \${ inside fences	263
pages affected	29 of 44
fenced form bindings rendering with a stray backslash	5

Every backslash-escape kind that occurs inside a fence today, so the fold's whole blast radius is known:

escape in a fence	count	folds to	correct?
\`	512	`	yes
\$	265 (263 of them \${)	$	yes
\\	5	\	yes
\s	2	s	see below
\}	1	}	yes

No \n, \t, \x or \u occurs inside any fence, which matters because unescapeJs is a naive \\(.) fold and would turn those into the bare letter rather than the control character. The corpus does not exercise that gap today.

The five form bindings are the sharpest case, because root AGENTS.md invariant 12 governs exactly which shapes a bound form may take, and the corpus is the surface whose only reader is an LLM. Verified source anchors:

• website/app/docs/progressive-enhancement/page.ts:28 and :162 emit <form action=\${createPost}>
• website/app/docs/server-actions/page.ts:429 emits html`<form action=\${createPost}> ... </form>` and :458 emits <form action=\${createPost}>
• website/app/docs/file-storage/page.ts:76 emits <form action=\${saveAvatar} class="flex flex-col gap-3">

In every one of those the RENDERED docs page shows <form action=${createPost}>, which is the correct shape. Only the corpus carries the backslash. Confirmed on the captured corpus, where grep -cF 'form action=\${' is 5 and grep -cF 'form action=${' is 24.

website/app/docs/components/page.ts shows the inconsistency inside one file. Its prose .fallback= line was fixed by #1331 and now reads .fallback=${html`…`} (corpus lines 2566 and 3241), while a fenced sample teaching the same thing still carries the escapes.

This is not a regression from #1331. It was measured as unchanged in both directions across that merge. #1331 fixed the prose half and deliberately deferred this half rather than growing further, and its review thread on website/lib/docs-llms.server.ts says so.

One thing the first version of this body did not know

Two docs pages author a regex with a SINGLE backslash inside a code sample, which is already broken on the rendered page and which the fold makes visible in the corpus:

• website/app/docs/backend-only/page.ts:336 writes replace(/\s+/g, '-'). A template literal cooks \s to s, so the LIVE docs page already renders replace(/s+/g, '-'). Verified by calling the page's default export through renderToString.
• website/app/docs/websockets/page.ts:80 writes header.split(/;\s*/). The live page already renders /;s*/. Verified the same way.
• website/app/docs/api-routes/page.ts:296 gets it right, writing replace(/\\s+/g, '-'), which cooks to \s and renders correctly.

So the fold is faithful in both cases, and the faithful output is a broken regex, because the pages are wrong. The fix for those two lines is one character each and belongs in this PR, since this change is what puts the damage in front of a reader. Without it the corpus would ship replace(/s+/g, '-') as a teaching sample.

Design / approach

Apply the same fold the prose path already uses, at the sample-capture site, and settle the ordering question against how a browser actually processes the page.

unescapeJs already exists in the module at website/lib/docs-llms.server.ts:308, is a hoisted function declaration (so it is callable from the capture site 88 lines above it), and reads exactly s.replace(/\\(.)/g, '$1'). A docs page body is a JS template literal, so inside it every backslash is an escape, which is what makes the fold correct rather than lossy.

The settled decision: unescapeJs FIRST, decodeEntities SECOND

codeBlocks.push(decodeEntities(unescapeJs(String(code))).replace(/\n+$/, ''));

Why that order and not the other. The extractor is reconstructing what a reader of the rendered page sees, and the rendered page is produced by two folds in a fixed order. JS cooks the template literal first, and only then does the HTML parser see the resulting text and decode its entities. WebJs's html tag receives the COOKED strings array (packages/core/src/html.js:13 stores strings as handed to it and never touches .raw), exactly as lit-html does at ~/Documents/Projects/frameworks/lit/packages/lit-html/src/lit-html.ts:826, where getTemplateHtml reads strings[i] from the cooked array. Matching the browser therefore means unescaping first.

Both orders were prototyped and rendered over the whole real corpus, and they produce byte-identical output today (991,312 bytes before, 990,527 bytes for both orders). So the real corpus cannot settle this, and the decision rests on the two adversarial shapes below, both run through both prototypes.

source inside a fence	unescape then decode	decode then unescape	what the browser shows
a \& b	a & b	a & b	a & b
path \n here	path \n here	path \n here	path \n here
\ literal	\ literal	\ literal	\ literal
&am\p; joined	& joined	& joined	& joined
html\`<form action=\${createPost}>\`	html`<form action=${createPost}>`	same	same
x = /a\\s+/;	x = /a\s+/;	same	same

The last column was established independently by cooking the literal in Node, which turns `&am\p; joined` into & joined, a string the HTML parser then decodes to & joined.

Two safety properties, both checked rather than assumed:

• No order can resurrect an escape. decodeEntities at website/lib/docs-llms.server.ts:312-324 decodes only &, <, >, {, }, ', ', ", …, —,  . None of them yields a backslash, and \ is not in the table at all, so decoding can never manufacture an escape for a later fold to eat. Verified with the \ and \ rows above.
• No double-fold is possible. The captured sample is parked behind the U+E000 sentinel at website/lib/docs-llms.server.ts:221 and restored at :285, which is AFTER the prose decodeEntities(body) at :282 and after both prose keep-passes at :266-267. The fenced text therefore passes through exactly one decode and, with this change, exactly one unescape.

Matching the prose precedent, deliberately. The prose path at website/lib/docs-llms.server.ts:266-267 calls unescapeJs while parking the kept hole, and the parked text is restored at :278-280 before decodeEntities(body) runs at :282. That is unescape-then-decode. This change matches it rather than differing from it, so the module has one rule for both paths.

Alternatives considered and rejected

• Rewriting the docs pages to avoid the escapes. Rejected. The pages are correct as authored, and a sample inside a JS template literal has no choice but to escape its backticks (root AGENTS.md invariant 9). The extractor is what is wrong. This does not cover the two single-backslash regex lines above, which are genuine authoring errors visible on the live site and are fixed here.
• Decode then unescape. Rejected on the &am\p; row, where it ships & and the browser shows &.
• Teaching unescapeJs the real JS escape table (\n, \t, \xNN, \uNNNN). Rejected as unused scope. No fence carries one today, the prose path has lived with the same naive fold since #1331, and a divergence between the two folds would be worse than the shared gap.
• Doing this inside #1261. Rejected there and still right. It is a different code path, 785 more corpus positions, and it forces a rethink of an existing test, so it earns its own before-and-after.

Implementation plan

All anchors are at HEAD 79fc28fc.

1. Fold the escapes at the fenced-capture site

website/lib/docs-llms.server.ts, inside bodyToMarkdown, line 220. Today:

  const codeBlocks: string[] = [];
  body = body.replace(/<(?:pre|code-block)(?=[\s>])[^>]*>([\s\S]*?)<\/(?:pre|code-block)>/g, (_m, code) => {
    codeBlocks.push(decodeEntities(String(code)).replace(/\n+$/, ''));
    return `\uE000CODE${codeBlocks.length - 1}\uE000`;
  });

After:

  const codeBlocks: string[] = [];
  body = body.replace(/<(?:pre|code-block)(?=[\s>])[^>]*>([\s\S]*?)<\/(?:pre|code-block)>/g, (_m, code) => {
    // A sample is copied out of page SOURCE, which is a JS template literal,
    // so a backtick in it is written `\`` and a literal hole `\${`. Without
    // this fold the corpus taught `<form action=\${createPost}>` where the
    // rendered page shows `<form action=${createPost}>`, on the exact shape
    // invariant 12 governs, in the one surface whose reader is an LLM.
    //
    // Escapes fold BEFORE entities decode, which is the order the browser
    // applies them: JS cooks the literal first, and the HTML parser sees only
    // the cooked text. The two orders agree on every sample in the repo and
    // differ on one shape, an escape splitting an entity, where `&am\p;`
    // cooks to `&amp;` and the parser then shows `&`. The reverse order can
    // never help, because the entity table below yields no backslash, so
    // decoding cannot manufacture an escape for this fold to eat. Same order
    // the prose keep-passes use below.
    codeBlocks.push(decodeEntities(unescapeJs(String(code))).replace(/\n+$/, ''));
    return `\uE000CODE${codeBlocks.length - 1}\uE000`;
  });

Leave the ordering-invariant comment at :213-217 in place and do not disturb the "strip at every stage, decode exactly once, at the end" rule. This change adds a stage BEFORE that single decode on this one path, so the rule still holds verbatim.

Do NOT paste literal private-use characters into the file. The sentinels are written as \uE000 and \uE001 escapes on purpose, and the reason is recorded at :181-185 and :224-227. If you touch that region, verify with grep -n plus cat -A.

2. Fix the two docs pages whose regex is authored with a single backslash

website/app/docs/backend-only/page.ts:336. Today:

    title, body, slug: title.toLowerCase().replace(/\s+/g, '-'), authorId,

After:

    title, body, slug: title.toLowerCase().replace(/\\s+/g, '-'), authorId,

website/app/docs/websockets/page.ts:80. Today:

  for (const part of header.split(/;\s*/)) {

After:

  for (const part of header.split(/;\\s*/)) {

The precedent for the doubled form is website/app/docs/api-routes/page.ts:296, which already writes replace(/\\s+/g, '-'). Both edits fix the LIVE rendered page as well as the corpus. Confirm by grepping the after-corpus for /\s+/g and /;\s*/ and finding the backslash present on all three lines.

3. Teach the corpus-walk test's mirror the same fold

website/test/ssr/docs-llms.test.ts. Add the mirror above the existing decodeEntities mirror at :113, so the two sit together:

/**
 * Mirrors the extractor's own source-escape fold, which is module-private. A
 * sample is copied out of page SOURCE, where it is a JS template literal, so
 * the corpus carries it cooked and a comparison against the raw source has to
 * cook it the same way.
 */
function unescapeJs(s: string): string {
  return s.replace(/\\(.)/g, '$1');
}

/** Mirrors the extractor's own entity decoding, which is module-private. */
function decodeEntities(s: string): string {

Then change line 105, inside a sample that reaches the corpus reaches it whole. Today:

      const text = decodeEntities(m[1]).replace(/\n+$/, '');

After:

      const text = decodeEntities(unescapeJs(m[1])).replace(/\n+$/, '');

Keep the assertion exactly as it is. The if (m[1].includes('${')) continue; skip at :103 stays, and it matters for scoping: \${ contains ${, so every escaped-hole sample is ALREADY exempt and only escaped-BACKTICK samples are newly involved.

The blast radius here is 27 samples, not 300. Measured against the prototype: 348 samples are compared today, 27 of them contain a backslash, and those 27 are exactly the set that reds if the mirror and the extractor disagree in either direction. The compared > 300 gate at :109 is unaffected, since the skip set does not change.

4. Add the fixtures

website/test/ssr/docs-llms.test.ts, in the fixture section that starts at :128. Put them immediately after a kept hole is unescaped, since the source is a template literal (:207-219), which is the prose half of the same rule. Both strings below were run against a scratchpad prototype and pass.

test('a fenced sample is unescaped, since the source is a template literal', () => {
  // The prose half of this rule is pinned above. The fenced half was left out,
  // so the corpus taught `<form action=\${createPost}>` on 5 lines while the
  // rendered docs page showed `<form action=${createPost}>`, disagreeing with
  // itself about the one shape invariant 12 governs.
  const md = bodyToMarkdown('html`<code-block>html\\`&lt;form action=\\${createPost}&gt;&lt;/form&gt;\\`</code-block>`');
  assert.equal(md, '```\nhtml`<form action=${createPost}></form>`\n```');
});

test('a fenced sample folds its escapes before it decodes its entities', () => {
  // Order matters on exactly one shape, and the browser settles it: JS cooks
  // the template literal first, so `&am\p;` reaches the HTML parser as
  // `&amp;`, which it decodes to `&`. Decoding first would ship `&amp;`.
  // No docs page carries this shape, so only a fixture can hold the rule.
  assert.equal(bodyToMarkdown('html`<code-block>a &am\\p; b</code-block>`'), '```\na & b\n```');
});

5. Update the website/AGENTS.md inventory entry

website/AGENTS.md:124-139, the docs-llms.server.ts entry in the lib/ inventory. It already records the strip-then-decode invariant and the three prose hole shapes. Add the escape-folding rule so the two paths are described together, in the same indented style as the surrounding block. Content to add, wording yours:

• A sample and a kept prose hole are both copied out of page SOURCE, which is a JS template literal, so both are unescaped before the single entity decode.
• That order is the browser's own, since JS cooks the literal and the HTML parser only ever sees cooked text.
• The fold is why the corpus stopped printing <form action=\${createPost}> on five lines.

6. Capture the accounted before-and-after diff

Same shape #1331 used.

cd website && node --input-type=module -e "const m = await import('./lib/docs-llms.server.ts'); process.stdout.write(await m.renderLlmsFull());" > /tmp/llms-before.txt

Apply steps 1 and 2, re-run into /tmp/llms-after.txt, then diff. Measured expectations, taken from a scratchpad prototype of step 1 alone (step 2 changes 2 further positions on top of these):

	value
changed lines	459
line count before and after	16,038 both, so no line is added or removed
bytes removed	785
backslashes in the corpus	853 before, 68 after
lines whose only change is not a backslash removal	0
grep -cF 'form action=\${'	5 before, 0 after
grep -cF 'form action=${'	24 before, 29 after
fence parity, authored blocks versus fenced blocks	equal on every page before and after
triple-backtick appearing INSIDE a fence	0 before and after, so no fold breaks a fence

The 785 removed backslashes account exactly to the escape table in the Problem section (512 plus 265 plus 5 plus 2 plus 1).

Tests

Unit, website/test/ssr/docs-llms.test.ts. The only layer that applies, and it carries three changes.

• The mirror update in step 3, inside a sample that reaches the corpus reaches it whole at :81-111. Confirm compared stays above the > 300 gate. It is 348 at HEAD and 348 after, because the skip set is unchanged.
• The two fixtures in step 4.
• Run test/lib/doc-headings.test.ts and test/ssr/docs-search.test.ts too. They consume this markdown and pin the fence predicate, and fence parity is unchanged, so they should stay green with no edit.

Counterfactual, two independent ones. Revert step 1 only and both must red.

1. The two new fixtures fail. Verified against the unchanged extractor in a scratchpad prototype.
2. a sample that reaches the corpus reaches it whole fails with 27 mangled samples, once its mirror carries the fold. Measured, not assumed: the folded mirror against the unchanged extractor reports 27 of 348, and the folded mirror against the changed extractor reports 0.

That second one is worth noting in the PR body. Updating the mirror does not merely keep the test passing, it turns the corpus walk itself into a counterfactual for this change.

test/docs/llms.test.mjs. Route-level transport test, asserts nothing about escaping, needs no change. Run it anyway. It is 11 passing tests at HEAD.

Layers that do not apply, with the reason for each.

• Browser. docs-llms.server.ts is a .server.ts module whose output is served as text/plain. Nothing hydrates, no custom element is involved, no DOM is produced.
• e2e. Same reason. The observable is the bytes of a text response, and the unit layer holds the exact bytes.
• Bun parity. Does not apply, and this was verified against the hook rather than asserted. .claude/hooks/require-bun-parity-with-runtime-src.sh:64 scopes its runtime-sensitive match to ^packages/([^/]+/src|editors/[^/]+/src|cli/lib)/. A change under website/ cannot match that pattern, so the gate does not fire. On the merits it is a pure string-to-string transform over two String.prototype.replace calls with no runtime-divergent surface (no listener, no serializer, no node:crypto, no stream).
• Smoke. test/examples/*/smoke/* covers the example apps, not the marketing site.

Commands to run and report.

cd website && npm test && npm run typecheck && npx webjs check && npx webjs doctor
cd .. && node --test test/docs/llms.test.mjs

Docs

• website/AGENTS.md:124-139, described in step 5. This is the doc surface for this change.
• Every other surface is N/A. This is website-internal server code exporting no public API. No @webjsdev/* export changes, no CLI flag, no webjs config key, no html hole prefix, no lifecycle hook, no convention an app author follows. So AGENTS.md at the repo root, the skill at .agents/skills/webjs/, README.md, CONVENTIONS.md, the docs site pages under website/app/docs/ (other than the two regex fixes in step 2, which are content corrections rather than doc-surface sync) and the scaffold templates under packages/cli/templates/ all stay untouched.
• No escape hatch is needed. .claude/hooks/require-docs-with-src.sh:59 gates only on staged paths matching ^packages/([^/]+/src|editors/[^/]+/src|cli/lib)/, which this change does not touch, and ^website/ counts as a doc surface anyway (:74). WEBJS_NO_DOC_GATE=1 is not required and should not be used.

Acceptance criteria

• A fenced sample carrying \` or \${ reaches the corpus unescaped, matching what the rendered docs page shows
• The fold runs BEFORE the entity decode at the capture site, and the source comment states the browser-order reason
• The 5 fenced form bindings emit <form action=${createPost}> and <form action=${saveAvatar} ...>, so grep -cF 'form action=\${' on the corpus is 0 and grep -cF 'form action=${' is 29
• Prose and fenced code on /docs/components agree about .fallback=${html`…`}
• website/app/docs/backend-only/page.ts and website/app/docs/websockets/page.ts author their regex with \\s, so both the live page and the corpus show \s
• No corpus line loses anything but a backslash, proven by an accounted before-and-after diff, with the line count unchanged at 16,038
• Fence parity per page is unchanged and no triple-backtick appears inside a fence
• Neither sentinel (U+E000, U+E001) leaks into the output, and neither is written as a literal character in the source
• a sample that reaches the corpus reaches it whole still compares more than 300 samples with 0 mangled, with its decode mirror extended rather than its assertion weakened
• Reverting the capture-site change reds the two new fixtures AND reds the corpus walk with 27 mangled samples
• cd website && npm test, npm run typecheck, node --test test/docs/llms.test.mjs, npx webjs check and npx webjs doctor all pass, and the results are reported
• website/AGENTS.md records the escape-folding rule and its ordering alongside the existing decode-once invariant

Out of scope

• Resolving template holes inside a fence. 40 fence lines carry a ${'…'} string-literal hole that the fenced path copies verbatim, so website/app/docs/data-fetching/page.ts:36 reaches the corpus as <webjs-suspense .fallback=${'${html`<p>Loading section…</p>`}'}> where the rendered page shows .fallback=${html`<p>Loading section…</p>`}. That is a real and separate defect, it is not an escape problem, and fixing it would collide with a fenced sample keeps the interpolation holes and indentation the prose pipeline would eat at website/test/ssr/docs-llms.test.ts:45-57, which depends on a genuine ${children} surviving a fence verbatim. Do not touch it here and do not file it as a follow-up. Mention it in the PR body and let the owner decide.
• Teaching unescapeJs the real JS escape table. No fence carries \n, \t, \x or \u today, and changing the shared helper would change the prose path at the same time.
• The double-decode already latent in decodeEntities. Its replaces are chained, so < decodes twice to < regardless of any ordering decided here. Pre-existing, unrelated to this change, and no docs page triggers it.
• The non-docs-page sections of /llms-full.txt. renderLlmsFull folds in the repo-root WebJs skill markdown verbatim, without passing it through bodyToMarkdown, so its backticks are outside this change's reach. Corpus lines around 13,997 and 14,655 look similar and are not affected. Do not chase them.
• Widening the change past website/lib/docs-llms.server.ts, website/test/ssr/docs-llms.test.ts, website/AGENTS.md and the two one-character docs-page regex fixes.