fix(website): fold JS escapes out of llms corpus prose too

The fenced-sample fold left prose alone, so the corpus still taught
`html\`...\`` on 50 lines across 8 pages where the rendered page shows a
plain backtick. Prose is copied out of the same template literal as a
sample or a hole, so it folds the same way, and the module now has one
rule instead of three paths that disagree.

The fold runs after the two hole passes because those are what tell a
literal `\${x}` from a render-time `${x}`, and before the restore so
text they already folded cannot fold twice. The single entity decode
still comes last.

The two page regex fixes had nothing that would notice a regression,
which matters because the fold's whole point is that it now copies an
authoring error into the corpus verbatim. A letter after a backslash is
never meaningful in a template literal, so a test rejects one across
every docs page.

Author: vivek7405

website/AGENTS.md

@@ -136,15 +136,26 @@ website/
                        `${x}` is render-time and gets dropped. Treating all
                        three alike printed `<form action=\>` on 12 corpus
                        lines, teaching an LLM the one shape invariant 12
-                       exists to rule out. A fenced sample and a kept prose
-                       hole are both copied out of page SOURCE, which is a JS
-                       template literal, so both fold their backslash escapes
-                       BEFORE that single entity decode. That order is the
-                       browser's own: JS cooks the literal first and the HTML
-                       parser only ever sees cooked text. The fenced half was
-                       missing, which is why the corpus printed
-                       `<form action=\${createPost}>` on 5 lines while the
-                       rendered page showed `<form action=${createPost}>`.
+                       exists to rule out. EVERYTHING it emits is copied out
+                       of page SOURCE, which is a JS template literal, so
+                       every path folds its backslash escapes BEFORE that
+                       single entity decode: a fenced sample at capture, a
+                       kept hole as it parks, and ordinary prose in one pass
+                       once the hole passes have run. That order is the
+                       browser's own, since JS cooks the literal first and the
+                       HTML parser only ever sees cooked text. The prose fold
+                       runs AFTER the hole passes because those are what tell
+                       `\${x}` (literal text) from `${x}` (render-time, and
+                       dropped), so folding first would drop the literal.
+                       Only the hole half existed, which is why the corpus
+                       printed `<form action=\${createPost}>` on 5 lines and
+                       ``html\`...\` `` on 50 more, where the rendered page
+                       shows `<form action=${createPost}>` and a plain
+                       backtick. A backslash surviving into the corpus is now
+                       one an author WROTE as `\\`, so a page that escapes a
+                       letter (`/\s+/g`, which cooks to `/s+/g` and rendered
+                       that way live on two pages) is a page bug, guarded by
+                       a test rather than by convention.
   modules/
     ui/components/     GITIGNORED mirror of the @webjsdev/ui registry sources,
                        written by scripts/copy-registry.mjs. NEVER hand-write

website/lib/docs-llms.server.ts

@@ -283,6 +283,20 @@ export function bodyToMarkdown(raw: string): string {
     // behind a nested hole.
     .replace(/\$\{(?:[^{}]|\{[^}]*\})*\}/g, '');
 
+  // Prose is copied out of the same JS template literal as a hole or a
+  // sample, so it folds its escapes too. Without this the corpus taught
+  // ``export a function returning html\`...\` `` on 50 lines across 8 pages,
+  // where the rendered page shows a plain backtick, which is the same
+  // disagreement the two hole passes above and the sample capture were
+  // written to remove.
+  //
+  // AFTER those hole passes, never before: the escaped-hole pass is what
+  // tells `\${x}` (literal text a reader sees) apart from `${x}` (render-time
+  // and dropped), so folding first would erase the distinction and drop the
+  // literal. BEFORE the restore below, so the text those passes already
+  // folded cannot fold a second time. The single decode still comes last.
+  body = unescapeJs(body);
+
   // Restore kept holes. A kept hole can itself contain a parked sentinel (an
   // escaped hole nested inside a string-literal one), so this repeats until
   // none is left. Replacing once emitted the inner sentinel verbatim, which

website/test/ssr/docs-llms.test.ts

@@ -245,6 +245,42 @@ test('a fenced sample folds its escapes before it decodes its entities', () => {
   assert.equal(bodyToMarkdown('html`<code-block>a &am\\p; b</code-block>`'), '```\na & b\n```');
 });
 
+test('prose is unescaped too, since it comes out of the same template literal', () => {
+  // A hole and a sample were folded before prose was, so the corpus still
+  // taught ``returning html\`...\` `` on 50 lines across 8 pages where the
+  // rendered page shows a plain backtick.
+  assert.equal(bodyToMarkdown('html`<p>returning <code>html\\`...\\`</code></p>`'), 'returning html`...`');
+
+  // The fold runs AFTER the hole passes, never before. Folding first would
+  // turn `\${x}` into `${x}`, which the dynamic-hole pass then drops, losing
+  // the literal text a reader actually sees.
+  assert.equal(bodyToMarkdown('html`<p>a \\${x} b ${y} c</p>`'), 'a ${x} b c');
+});
+
+test('no docs page escapes a letter, which the extractor would fold away', async () => {
+  // A page body is a JS template literal, so `\s` cooks to a bare `s`: the
+  // LIVE page rendered `replace(/s+/g, '-')` on /docs/backend-only and
+  // `/;s*/` on /docs/websockets until those two were corrected to `\\s`.
+  // The extractor is faithful, so it copies that damage into the corpus as a
+  // teaching sample, and the corpus's only reader is an LLM.
+  //
+  // Escaping a letter or a digit is never meaningful here. `\``, `\$` and
+  // `\\` are the escapes a template literal genuinely needs, and punctuation
+  // escapes are at worst redundant. So a letter or digit after a backslash
+  // is always the mistake above, and this is the only thing that would
+  // notice the next one.
+  const offenders: string[] = [];
+  for (const page of await getDocPages()) {
+    const src = await readFile(new URL(`../../app${page.path}/page.ts`, import.meta.url), 'utf8');
+    // Pair-consuming, so the `\\` in a correctly authored `\\s` is eaten as
+    // one escape and its `s` is never read as escaped.
+    for (const m of src.matchAll(/\\([\s\S])/g)) {
+      if (/[A-Za-z0-9]/.test(m[1])) offenders.push(`${page.path}: \\${m[1]} in ${JSON.stringify(src.slice(Math.max(0, m.index - 30), m.index + 30))}`);
+    }
+  }
+  assert.deepEqual(offenders.slice(0, 5), [], `${offenders.length} docs-page escapes fold to a bare letter`);
+});
+
 test('a kept hole nested inside another leaves no sentinel in the output', () => {
   // The two keep passes run in sequence, so a string-literal hole can park
   // text that already contains an escaped hole's sentinel. Restoring once