Skip to content

Keep modulepreload hints within the servable set#161

Merged
vivek7405 merged 3 commits into
mainfrom
fix/preload-hints-nonservable-urls
May 31, 2026
Merged

Keep modulepreload hints within the servable set#161
vivek7405 merged 3 commits into
mainfrom
fix/preload-hints-nonservable-urls

Conversation

@vivek7405

@vivek7405 vivek7405 commented May 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

Closes #158
Closes #159

The SSR preload emitter and the source-serving authorization gate walked the module graph differently, so the framework emitted <link rel="modulepreload"> hints for URLs it then 404s. Found by auditing all four in-repo dogfood apps for broken preloads.

App Broken preloads (before) Cause
example-blog /modules/posts/utils/slugify.ts, /modules/posts/types.ts, /modules/auth/types.ts preload walk crossed the .server.* boundary (#158)
docs /app/docs/components/counter.ts scanner counted an import shown as code inside a template literal (#159)
website none already clean
ui-website none already clean

What changed

transitiveDeps stops at .server.* (#158). reachableFromEntries (the auth gate) already stops at .server.{js,ts,mjs,mts}, because a client import of a server file resolves to an RPC / throw-at-load stub and the server file's own imports never reach the browser. transitiveDeps (which feeds the preload hints) did not, so it walked through a .server.ts into its server-only deps and preloaded them. It now applies the same SERVER_FILE_RE stop, so a preload set is always a subset of the servable set. The .server.* file itself stays preloadable (its stub URL is fetched on demand), and a file reachable through both a server file and a real client path is still reached via the client path.

Import scanner masks string/template content (#159). parseFile ran the import regexes on raw source, so an import/export … from printed as example code inside an html`` template (a

block on the docs getting-started page) became a phantom graph edge and a preload for a file that does not exist. It now computes a position-preserving mask viaredactStringsAndTemplatesand skips any match whose keyword sits inside a blanked literal. The specifier is still read from raw source, so real top-level imports and multi-lineexport … from` re-exports are unaffected.

Deliberately excluded

import type { T } from './x.ts' still creates a graph edge (the regex treats type as part of the binding list). It did not cause any of the observed 404s (a types-only file strips to an empty module served 200, not 404), and the .server.* boundary already keeps the blog's type-only files off the wire. Narrowing type-only imports is a separate concern; not in scope here.

Test plan

Coverage added at every layer the change affects, each with a verified counterfactual or a real network probe:

  • Unit (packages/server/test/module-graph/module-graph.test.js): transitiveDeps stops at the .server.* boundary; buildModuleGraph ignores import/export shown as code inside a template literal. Both fail when their guard is removed.
  • Integration (test/ssr/ssr.test.js): ssrPage does not emit a preload for a server-only util reached THROUGH a .server file, while a real client dep still is. Counterfactually verified (fails against the pre-fix module-graph.js).
  • E2E (test/e2e/e2e.test.mjs): probe every modulepreload href on the real blog, assert none 404. Suite 52/52.
  • Smoke (test/examples/blog/smoke/blog-smoke.test.js): same probe, browserless. Suite 6/6.
  • Browser (wtr): N/A because this is server-side graph logic with no client-runtime/hydration/DOM surface; ran it anyway (268/0/1) to confirm no regression.

Server suite 628/628. Dogfood, all four apps (prod mode): blog / 10 preloads / 0 broken (was 13/3); docs /docs/getting-started 5/0 (was 6/1); website + ui-website 200, 0 broken. The 21 red packages/core/test/**/browser/*.test.js under node --test are pre-existing wtr tests run under the wrong runner, unrelated to this change.

Docs

  • packages/server/AGENTS.md: invariant 1 and the module-graph.js module-map line now state that transitiveDeps shares the .server.* stop (preloads are a subset of servable) and that the scanner ignores imports shown inside template literals.
  • User-facing docs: N/A because the public contract (preloads cover transitive deps; only graph-reachable files are servable) is unchanged; this fixes an internal inconsistency, it does not alter documented behavior.
  • Scaffold templates: N/A because generated app code is unaffected.

vivek7405 added 2 commits June 1, 2026 01:53
The SSR preload emitter walks transitiveDeps, the source-serving
authorization gate walks reachableFromEntries. The two disagreed about
the .server.* boundary, so the framework emitted modulepreload hints for
URLs it then 404s. Two root causes, both surfaced by auditing the four
dogfood apps for broken preloads:

1. transitiveDeps walked THROUGH .server.* files into their server-only
   deps (slugify.ts, the two types.ts on the blog). The gate stops at
   that boundary; the preload walk did not. Add the same SERVER_FILE_RE
   stop so a preload set is always a subset of the servable set. The
   .server.* file itself stays preloadable (its stub URL is fetched); a
   file reachable through both a server file and a real client path is
   still reached via the client path. Closes #158.

2. The import scanner matched import/export statements that appear as
   TEXT inside a template literal (example code in a <pre> inside an
   html`` template, as the docs getting-started page does), producing a
   phantom edge and a preload for a file that does not exist
   (/app/docs/components/counter.ts on docs). Reuse js-scan's
   redactStringsAndTemplates as a position-preserving mask and skip any
   match whose keyword was blanked (sits inside a literal). The specifier
   is still read from raw source, so real top-level imports and
   multi-line export-from re-exports are unaffected. Closes #159.

Audit result across the four apps: blog had the 3 server-only 404s,
docs had the 1 phantom 404, website and ui-website were clean. Both
fixes verified against the real app graphs; counterfactual tests fail
when either guard is removed.
… masks literals

Invariant 1 and the module-graph.js module-map line now state that
transitiveDeps stops at .server.* like the gate (preloads are a subset
of servable), and that the import scanner ignores import/export shown as
example code inside a template literal.
@vivek7405 vivek7405 self-assigned this May 31, 2026

@vivek7405 vivek7405 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went through the correctness of both guards end to end. The mask is genuinely position-preserving (one char out per char in, newlines kept), so masked[m.index] lands on the same keyword byte in raw source even past the multi-byte ellipsis/arrow chars in this file, and I confirmed real imports after division, postfix ++, and regex literals still resolve while template-embedded and comment-embedded import text gets blanked at the keyword. The server-boundary stop sits after the skip check and the result push but before the queue push, so the .server.* URL stays preloadable while its imports are not walked, and it is a no-op for the two elision callers since they already carry serverFiles in their skip set. Both new tests fail when their own guard is pulled, which is what I wanted to see.

Two things I am explicitly leaving alone because they predate this change and the change only ever shrinks the phantom-edge set: a single-line untagged backtick literal containing an import, and a from-specifier sitting inside a block comment between a real export keyword and its real specifier, can still register a stray edge. Neither caused any of the four-app 404s and neither is made worse here. Last round was clean.

@vivek7405 vivek7405 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second pass, focused on the one thing that worried me: the guard only checks the keyword position, not where the captured specifier lands. Walked the nasty cases. A real export keyword whose lazy from-gap binds a specifier inside a block comment or template does still miss-bind, but that is EXPORT_FROM_RE's lazy gap and it behaved identically before any mask existed, so this change neither fixes nor worsens it. Going the other way, I could not construct a valid import whose keyword offset gets blanked: the redactor is one-char-per-char so the offset always lands on the real keyword byte, and the word-boundary anchors keep the match off mid-token positions like reimport. Cache path is fine too, masked is only computed on a miss and matchAll always hands back a numeric index, and if it somehow did not the comparison fails toward keeping an edge, never dropping one. Clean.

@vivek7405

Copy link
Copy Markdown
Collaborator Author

How these were found, and the reusable audit

Both bugs are the same class (a modulepreload hint pointing at a URL the auth gate refuses to serve), and the way to catch them is to diff the two sets the framework computes: the preload set (from transitiveDeps) and the servable set (from reachableFromEntries). Whenever those two graph walks diverge, you get a preload that 404s.

The live audit that surfaced them: fetch each app's HTML, pull every <link rel="modulepreload" href>, and request each href. One gotcha worth writing down so the next sweep does not trip on it: request with GET, not HEAD. The dev/prod server only serves source files on GET, so a HEAD probe returns 404 for every source URL and makes the whole app look broken. Filtering to URLs that 404 on GET is what isolated the four real ones (3 on the blog, 1 on docs) from the noise.

This is also why both fixes live in one PR rather than two: they are one invariant ("the preload set is a subset of the servable set"), violated in two different spots. Keeping the invariant stated once, in transitiveDeps and the scanner that feeds it, is the durable fix; a future caller that walks the graph for preloads inherits the boundary for free.

The unit layer already covers transitiveDeps + the scanner mask. This
adds the higher layers the change affects:

- Integration (test/ssr/ssr.test.js): ssrPage must not emit a preload
  for a server-only util reached THROUGH a .server file (the slugify.ts
  shape), while a real client dep is still preloaded. The existing test
  only covered the .server file itself, not its transitive dep.
- E2E (test/e2e/e2e.test.mjs): probe every modulepreload href on the
  real blog and assert none 404 (the live symptom of #158/#159).
- Smoke (test/examples/blog/smoke): same probe, browserless and fast.

@vivek7405 vivek7405 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the higher test layers and went back over them with a skeptic's eye, mostly worried the e2e and smoke probes could pass vacuously. They do not: each guards on preloads.length > 0 before probing, the same-origin filter drops the CDN preloads without emptying the set, and the gate returns a real 404 for a non-servable source URL rather than a redirect, so fetch following redirects cannot quietly turn a 404 into a 200. The href-extraction regexes match the actual link rel=modulepreload href= shape ssr.js emits. Integration test carries a positive assertion (the real client dep IS still preloaded) so an over-eager filter would fail it, not pass it. Fixture dirs are unique-suffixed and cleaned. Last round clean.

@vivek7405 vivek7405 merged commit 04f6cef into main May 31, 2026
5 checks passed
@vivek7405 vivek7405 deleted the fix/preload-hints-nonservable-urls branch May 31, 2026 21:05
vivek7405 added a commit that referenced this pull request May 31, 2026
* chore: release @webjsdev/core 0.7.3 and @webjsdev/server 0.8.3

Release debt accumulated since the 0.7.2 / 0.8.2 release (#149):

core 0.7.3 (patch): the two client-router fixes from #151 and #157
(JS-handled forms and links were hijacked by the router despite
e.preventDefault).

server 0.8.3 (patch): #161 (modulepreload hints no longer point at
server-only or template-embedded files the auth gate 404s) and #156
(webjs check no longer leaks a git env var across worktrees).

Both stay in a single minor line; all in-repo dependents pin ^0.7.0 /
^0.8.0, so no dependent range edits are needed. Lockfile regenerated.

* chore: release @webjsdev/cli 0.10.0 (webjs vendor surface)

cli 0.9.1 shipped without the `webjs vendor` command surface that
#105 and #89 added (pin / unpin / list / audit / outdated / update,
the --from multi-CDN selector, and --download caching). That is
feature-level work, so a minor bump.

create-webjs and webjsdev are thin shims that delegate to cli; widen
their `@webjsdev/cli` range from ^0.9.0 to ^0.10.0 so the workspace
keeps linking the local cli (a minor bump falls outside the old caret).
They are not in the changelog auto-publish system, so they carry no
changelog entry of their own. Lockfile regenerated.
vivek7405 added a commit that referenced this pull request May 31, 2026
The no-build page's modulepreload section said preloads cover "transitive
dependencies" and only carved out .server.* files themselves, which read
as the pre-#161 behavior (a plain util imported through a server action
would be preloaded). #161 made the walk stop at the .server boundary, so
those transitive server-only deps are excluded too. Document that the
preload set is exactly the set the page fetches.
vivek7405 added a commit that referenced this pull request May 31, 2026
The no-build page's modulepreload section said preloads cover "transitive
dependencies" and only carved out .server.* files themselves, which read
as the pre-#161 behavior (a plain util imported through a server action
would be preloaded). #161 made the walk stop at the .server boundary, so
those transitive server-only deps are excluded too. Document that the
preload set is exactly the set the page fetches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Import scanner counts imports shown as code inside template literals Modulepreload hints emitted for server-only files the gate 404s

1 participant