feat(web): add web generator #285

avivkeller · 2025-05-28T21:53:14Z

Fixes #7.

This PR adds the web generator.

Tasks / Issues

P1 – Must Complete Before Merge

Add more items (anyone can do this as they review¹)

P2 – Must complete before migration

ChangeHistory component doesn’t render Markdown correctly. (See entry.changes aren't converted to HTML #326)
Add more items (anyone can do this as they review¹)

P3 – Can Be Done in a Follow-up

Remove mustache dependency
Add more items (anyone can do this as they review¹)

Get a preview

node bin/cli.mjs generate -t orama-db -i "/node/doc/api/*.md" -o "./out" # If you want search functionality
node bin/cli.mjs generate -t web -i "/node/doc/api/*.md" -o "./out" --index "/node/doc/api/index.md" # Specifying `--index` is optional, but recommended
npx serve out # You can serve the output, or just open one of the files in your browser. Serving is required for using search.

Add things as they appear, or leave review comments. ↩ ↩² ↩³

codecov-commenter · 2025-05-28T21:53:59Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
165	1	164	0

View the top 1 failed test(s) by shortest run time

test::should process entries and include JSX wrapper elements

Stack Traces | 0.00573s run time

Error [ERR_TEST_FAILURE]: Missing "Article". Found: NotificationProvider, NavBar
    at async Promise.all (index 0) {
  code: 'ERR_TEST_FAILURE',
  failureType: 'testCodeFailure',
  cause: AssertionError [ERR_ASSERTION]: Missing "Article". Found: NotificationProvider, NavBar
      at file:.../utils/__tests__/buildContent.test.mjs:31:7
      at Array.forEach (<anonymous>)
      at TestContext.<anonymous> (file:.../utils/__tests__/buildContent.test.mjs:30:19)
      at async Test.run (node:internal/test_runner/test:1054:7)
      at async Promise.all (index 0)
      at async Suite.run (node:internal/test_runner/test:1442:7)
      at async startSubtestAfterBootstrap (node:internal/test_runner/harness:296:3) {
    generatedMessage: false,
    code: 'ERR_ASSERTION',
    actual: false,
    expected: true,
    operator: '=='
  }
}

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

src/generators/web/components/SearchBox/constants.mjs

src/generators/web/components/CodeBox.jsx

src/generators/legacy-html/utils/tableOfContents.mjs

src/generators/web/components/NavBar.jsx

package.json

avivkeller · 2025-05-30T14:54:47Z

🎉 The code now dehydrates to the client so it can render without JavaScript!

AugustinMauroy · 2025-05-30T15:28:54Z

🎉 The code now dehydrates to the client so it run without JavaScript!

Wow 😵‍💫 and what about codetab

avivkeller · 2025-05-30T15:43:36Z

It rehydrates and runs with JS, but if you don't have JS, you can still view the docs. I used React's SSRing

src/constants.mjs

avivkeller · 2025-05-31T21:54:29Z

@AugustinMauroy and I got search to finally work 🎉

src/generators/web/client/components/MetaBar.jsx

src/generators/web/index.mjs

src/generators/web/client/components/MetaBar.jsx

src/generators/web/client/components/SearchBox/index.jsx

avivkeller · 2025-06-03T23:05:06Z

@nodejs/nodejs-website @nodejs/web-infra ChangeHistory and SideBar aren't implemented yet, so this is still a draft, but it's ready for you to take a look, so feel free to review :-)

ovflowd · 2025-07-09T10:11:55Z

@avivkeller I found the issue, the files on .vercelignore get deleted before the build cache gets restored 🤦

ovflowd · 2025-07-09T10:18:59Z

[12:14:00.889] Running build in Portland, USA (West) – pdx1
[12:14:00.889] Build machine configuration: 4 cores, 8 GB
[12:14:00.906] Cloning github.com/nodejs/api-docs-tooling (Branch: feat/web/gen, Commit: 89b924a)
[12:14:01.526] Cloning completed: 620.000ms
[12:14:01.583] Found .vercelignore
[12:14:01.585] Removed 0 ignored files defined in .vercelignore
[12:14:01.733] Restored build cache from previous deployment (C4uxXMmxowy7HiVuJMfKhbr7PiC1)
[12:14:02.304] Running "vercel build"
[12:14:02.723] Vercel CLI 44.2.13
[12:14:03.292] Running "install" command: `./scripts/vercel-prepare.sh`...
[12:14:03.297] Cloning into 'node'...
[12:14:22.742] Updating files:  26% (11087/41733)
Updating files:  27% (11268/41733)
Updating files:  28% (11686/41733)
Updating files:  29% (12103/41733)
Updating files:  30% (12520/41733)
Updating files:  31% (12938/41733)
Updating files:  32% (13355/41733)
Updating files:  33% (13772/41733)
Updating files:  34% (14190/41733)
Updating files:  35% (14607/41733)
Updating files:  36% (15024/41733)
Updating files:  37% (15442/41733)
Updating files:  38% (15859/41733)
Updating files:  39% (16276/41733)
Updating files:  40% (16694/41733)
Updating files:  41% (17111/41733)
Updating files:  42% (17528/41733)
Updating files:  43% (17946/41733)
Updating files:  44% (18363/41733)
Updating files:  45% (18780/41733)
Updating files:  46% (19198/41733)
Updating files:  47% (19615/41733)
Updating files:  48% (20032/41733)
Updating files:  49% (20450/41733)
Updating files:  50% (20867/41733)
Updating files:  51% (21284/41733)
Updating files:  52% (21702/41733)
Updating files:  53% (22119/41733)
Updating files:  54% (22536/41733)
Updating files:  55% (22954/41733)
Updating files:  56% (23371/41733)
Updating files:  57% (23788/41733)
Updating files:  58% (24206/41733)
Updating files:  59% (24623/41733)
Updating files:  60% (25040/41733)
Updating files:  61% (25458/41733)
Updating files:  62% (25875/41733)
Updating files:  63% (26292/41733)
Updating files:  63% (26462/41733)
Updating files:  64% (26710/41733)
Updating files:  65% (27127/41733)
Updating files:  66% (27544/41733)
Updating files:  67% (27962/41733)
Updating files:  68% (28379/41733)
Updating files:  69% (28796/41733)
Updating files:  70% (29214/41733)
Updating files:  71% (29631/41733)
Updating files:  72% (30048/41733)
Updating files:  73% (30466/41733)
Updating files:  74% (30883/41733)
Updating files:  75% (31300/41733)
Updating files:  76% (31718/41733)
Updating files:  77% (32135/41733)
Updating files:  78% (32552/41733)
Updating files:  79% (32970/41733)
Updating files:  80% (33387/41733)
Updating files:  81% (33804/41733)
Updating files:  82% (34222/41733)
Updating files:  83% (34639/41733)
Updating files:  84% (35056/41733)
Updating files:  85% (35474/41733)
Updating files:  86% (35891/41733)
Updating files:  87% (36308/41733)
Updating files:  88% (36726/41733)
Updating files:  89% (37143/41733)
Updating files:  90% (37560/41733)
Updating files:  91% (37978/41733)
Updating files:  92% (38395/41733)
Updating files:  93% (38812/41733)
Updating files:  94% (39230/41733)
Updating files:  95% (39647/41733)
Updating files:  96% (40064/41733)
Updating files:  97% (40482/41733)
Updating files:  98% (40899/41733)
Updating files:  99% (41316/41733)
Updating files:  99% (41610/41733)
Updating files: 100% (41733/41733)
Updating files: 100% (41733/41733), done.
[12:14:32.004] 
[12:14:32.004] > prepare
[12:14:32.004] > husky
[12:14:32.004] 
[12:14:32.068] 
[12:14:32.069] added 704 packages, and audited 705 packages in 9s
[12:14:32.069] 
[12:14:32.069] 292 packages are looking for funding
[12:14:32.069]   run `npm fund` for details
[12:14:32.071] 
[12:14:32.071] 1 low severity vulnerability
[12:14:32.071] 
[12:14:32.071] To address all issues, run:
[12:14:32.071]   npm audit fix
[12:14:32.071] 
[12:14:32.071] Run `npm audit` for details.
[12:14:32.222] Generating man page (node.1)
[12:14:34.759] Generating addon tests
[12:14:36.785] Generating API doc links
[12:14:40.500] Generating API docs: orama-db, legacy-json, llms-txt, web
[12:16:16.972] Build Completed in /vercel/output [2m]
[12:16:17.005] Deploying outputs...
[12:16:19.467] 
[12:16:19.834] Deployment completed

Takes around 2m to build all these generators on CI. Eventually I'd be good to check:

If there are any bottlenecks on legacy-json generator
Further reduce web generator bottleneck (on my PC around 63 seconds, so I bet on CI I'd be around 90-120 seconds at least)

src/generators/web/template.html

avivkeller · 2025-07-09T17:12:56Z

Rebasing to include #347

ovflowd · 2025-07-10T01:23:15Z

Using Rolldown appears to reduce compile time significantly... if I can get it to work 😅

What piece is not working? 👀

avivkeller · 2025-07-10T01:23:53Z

What piece is not working? 👀

(Oops, I accidentally deleted the comment)

It's causing a SyntaxError in some of the compiled code, which I'm investigating. It has to do with the string interpolation 🤔

If you look at https://api-docs-tooling-7fvr17mt4-openjs.vercel.app/, it doesn't generate most of the documentation.

avivkeller · 2025-07-10T01:36:37Z

🤦 I was using Math.random().toString(36).slice(2) to generate unique variable names, but forgot that that doesn't generate safe JavaScript variables (since it can start with a number). I needed to prefix it with _.

avivkeller · 2025-07-10T01:41:58Z

Based on my estimates, it should decrease the compile time, but we will see.

ovflowd · 2025-07-10T10:35:48Z

Based on my estimates, it should decrease the compile time, but we will see.

Is the latest version using Rolldown?

ovflowd · 2025-07-10T10:38:12Z

node bin/cli.mjs generate -t web -i "../node/doc/api/*.md" -o "./out" --index  52.77s user 15.80s system 202% cpu 33.882 total

ovflowd · 2025-07-10T12:54:19Z

node bin/cli.mjs generate -t web -i "../node/doc/api/*.md" -o "./out" --index  52.77s user 15.80s system 202% cpu 33.882 total

Note that this is around 10s faster than the previous one. Are we still thinking on LightningCSS?

BTW, I still find impressive building all these react pages under 1 minute. They are huge long massive pages.

avivkeller · 2025-07-10T13:05:24Z

Note that this is around 10s faster than the previous one. Are we still thinking on LightningCSS?

The 10s improvement comes from LightningCSS + Rolldown, yes. Based on my testing, Rolldown is slightly faster than ESBuild, the LightningCSS is much faster than PostCSS, hence the improvement.

BTW, I still find impressive building all these react pages under 1 minute. They are huge long massive pages.

🎉 (My goal to make them even faster 🚀)

avivkeller marked this pull request as ready for review May 28, 2025 22:13

avivkeller requested a review from a team as a code owner May 28, 2025 22:13

avivkeller commented May 29, 2025

View reviewed changes

src/generators/web/components/SearchBox/constants.mjs Outdated Show resolved Hide resolved

avivkeller marked this pull request as draft May 29, 2025 16:36

AugustinMauroy reviewed May 29, 2025

View reviewed changes

src/generators/web/components/CodeBox.jsx Outdated Show resolved Hide resolved

src/generators/legacy-html/utils/tableOfContents.mjs Show resolved Hide resolved

src/generators/web/components/NavBar.jsx Outdated Show resolved Hide resolved

ovflowd reviewed May 29, 2025

View reviewed changes

package.json Show resolved Hide resolved

avivkeller force-pushed the feat/web/gen branch from 1eb144a to 492a696 Compare May 29, 2025 20:30

avivkeller force-pushed the feat/web/gen branch from d8ab24c to 9ebcb95 Compare May 30, 2025 14:57

avivkeller mentioned this pull request May 30, 2025

chore(ui): simplify CodeBox-like UI components nodejs/nodejs.org#7793

Merged

avivkeller added the enhancement label May 30, 2025

avivkeller added this to Node.js API Documentation Tooling May 31, 2025

avivkeller moved this to In Progress in Node.js API Documentation Tooling May 31, 2025

avivkeller removed this from Node.js API Documentation Tooling May 31, 2025

AugustinMauroy reviewed May 31, 2025

View reviewed changes

src/constants.mjs Outdated Show resolved Hide resolved

flakey5 reviewed May 31, 2025

View reviewed changes

src/generators/web/client/components/MetaBar.jsx Outdated Show resolved Hide resolved

avivkeller force-pushed the feat/web/gen branch from 705f9e2 to 76ef14a Compare May 31, 2025 22:18

AugustinMauroy reviewed Jun 1, 2025

View reviewed changes

src/generators/web/index.mjs Show resolved Hide resolved

AugustinMauroy reviewed Jun 1, 2025

View reviewed changes

src/generators/web/client/components/MetaBar.jsx Outdated Show resolved Hide resolved

AugustinMauroy reviewed Jun 1, 2025

View reviewed changes

src/generators/web/client/components/SearchBox/index.jsx Outdated Show resolved Hide resolved

avivkeller force-pushed the feat/web/gen branch 2 times, most recently from 024bbea to dbfe55d Compare June 3, 2025 21:40

avivkeller force-pushed the feat/web/gen branch from e814300 to ba07721 Compare June 5, 2025 20:27

avivkeller linked an issue Jun 6, 2025 that may be closed by this pull request

Write React Components #10

Closed

avivkeller force-pushed the feat/web/gen branch from ca72ed0 to 99176ba Compare June 6, 2025 20:33

vercel bot deployed to Production July 9, 2025 10:10 View deployment

vercel bot deployed to Production July 9, 2025 10:19 View deployment

vercel bot deployed to Production July 9, 2025 14:03 View deployment

canerakdas reviewed Jul 9, 2025

View reviewed changes

src/generators/web/template.html Show resolved Hide resolved

avivkeller force-pushed the feat/web/gen branch from 9f98edf to cbfe925 Compare July 9, 2025 17:13

vercel bot deployed to Production July 9, 2025 17:15 View deployment

vercel bot deployed to Production July 9, 2025 17:19 View deployment

vercel bot deployed to Production July 9, 2025 17:24 View deployment

vercel bot deployed to Production July 10, 2025 01:17 View deployment

vercel bot deployed to Production July 10, 2025 01:22 View deployment

feat(web): add web generator

e035823

avivkeller force-pushed the feat/web/gen branch from baf2a94 to e035823 Compare July 10, 2025 01:39

vercel bot deployed to Production July 10, 2025 01:42 View deployment

fix fonts, signatures

2cd9e56

vercel bot deployed to Production July 10, 2025 13:36 View deployment

fixup!

6e4b373

vercel bot deployed to Production July 10, 2025 14:36 View deployment

start working on notifications

8d11258

vercel bot deployed to Production July 10, 2025 17:15 View deployment

feat(web): add web generator #285

Are you sure you want to change the base?

feat(web): add web generator #285

Conversation

avivkeller commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks / Issues

P1 – Must Complete Before Merge

P2 – Must complete before migration

P3 – Can Be Done in a Follow-up

Get a preview

Footnotes

Uh oh!

codecov-commenter commented May 28, 2025 • edited by codecov bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

avivkeller commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AugustinMauroy commented May 30, 2025

Uh oh!

avivkeller commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

avivkeller commented May 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

avivkeller commented Jun 3, 2025

Uh oh!

ovflowd commented Jul 9, 2025

Uh oh!

ovflowd commented Jul 9, 2025

Uh oh!

Uh oh!

avivkeller commented Jul 9, 2025

Uh oh!

ovflowd commented Jul 10, 2025

Uh oh!

avivkeller commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avivkeller commented Jul 10, 2025

Uh oh!

avivkeller commented Jul 10, 2025

Uh oh!

ovflowd commented Jul 10, 2025

Uh oh!

ovflowd commented Jul 10, 2025

Uh oh!

ovflowd commented Jul 10, 2025

Uh oh!

avivkeller commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

avivkeller commented May 28, 2025 •

edited

Loading

codecov-commenter commented May 28, 2025 •

edited by codecov bot

Loading

avivkeller commented May 30, 2025 •

edited

Loading

avivkeller commented May 30, 2025 •

edited

Loading

avivkeller commented Jul 10, 2025 •

edited

Loading

avivkeller commented Jul 10, 2025 •

edited

Loading