Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consolidate manifest writing for Turbopack in a single function #59663

Conversation

timneutkens
Copy link
Member

@timneutkens timneutkens commented Dec 15, 2023

What?

I'm working on consolidating a bunch of the file writing related pieces in the Turbopack handling in the dev server so that it can be abstracted out as it's needed for next build too.

These changes make sure that there is a single writeManifests() instead of picking specific manifests to write.

We can optimize this later but for now the overhead of writing them to disk separately is negligible.

Closes NEXT-1884

@timneutkens
Copy link
Member Author

Current dependencies on/for this PR:

This stack of pull requests is managed by Graphite.

@@ -552,7 +552,7 @@ async function startWatcher(opts: SetupOpts) {
hmrBuilding = true
}
hmrPayloads.set(`${key}:${id}`, payload)
hmrEventHappend = true
hmrEventHappened = true
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noticed a typo here, corrected it.

@ijjk
Copy link
Member

ijjk commented Dec 15, 2023

Stats from current PR

Default Build
General
vercel/next.js canary vercel/next.js 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function Change
buildDuration 12.8s 12.9s ⚠️ +150ms
buildDurationCached 7s 6.1s N/A
nodeModulesSize 200 MB 200 MB N/A
nextStartRea..uration (ms) 420ms 428ms N/A
Client Bundles (main, webpack)
vercel/next.js canary vercel/next.js 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function Change
170-HASH.js gzip 26.8 kB 26.8 kB N/A
199.HASH.js gzip 181 B 185 B N/A
3f784ff6-HASH.js gzip 53.3 kB 53.3 kB
framework-HASH.js gzip 45.2 kB 45.2 kB
main-app-HASH.js gzip 241 B 241 B
main-HASH.js gzip 31.7 kB 31.7 kB N/A
webpack-HASH.js gzip 1.7 kB 1.7 kB N/A
Overall change 98.7 kB 98.7 kB
Legacy Client Bundles (polyfills)
vercel/next.js canary vercel/next.js 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function Change
polyfills-HASH.js gzip 31 kB 31 kB
Overall change 31 kB 31 kB
Client Pages
vercel/next.js canary vercel/next.js 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function Change
_app-HASH.js gzip 195 B 194 B N/A
_error-HASH.js gzip 183 B 182 B N/A
amp-HASH.js gzip 501 B 501 B
css-HASH.js gzip 321 B 321 B
dynamic-HASH.js gzip 2.5 kB 2.5 kB N/A
edge-ssr-HASH.js gzip 255 B 255 B
head-HASH.js gzip 349 B 350 B N/A
hooks-HASH.js gzip 368 B 369 B N/A
image-HASH.js gzip 4.27 kB 4.27 kB N/A
index-HASH.js gzip 255 B 256 B N/A
link-HASH.js gzip 2.61 kB 2.6 kB N/A
routerDirect..HASH.js gzip 311 B 309 B N/A
script-HASH.js gzip 384 B 384 B
withRouter-HASH.js gzip 307 B 306 B N/A
1afbb74e6ecf..834.css gzip 106 B 106 B
Overall change 1.57 kB 1.57 kB
Client Build Manifests
vercel/next.js canary vercel/next.js 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function Change
_buildManifest.js gzip 484 B 482 B N/A
Overall change 0 B 0 B
Rendered Page Sizes
vercel/next.js canary vercel/next.js 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function Change
index.html gzip 528 B 529 B N/A
link.html gzip 542 B 541 B N/A
withRouter.html gzip 524 B 524 B
Overall change 524 B 524 B
Edge SSR bundle Size
vercel/next.js canary vercel/next.js 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function Change
edge-ssr.js gzip 93.7 kB 93.7 kB N/A
page.js gzip 146 kB 146 kB N/A
Overall change 0 B 0 B
Middleware size
vercel/next.js canary vercel/next.js 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function Change
middleware-b..fest.js gzip 627 B 623 B N/A
middleware-r..fest.js gzip 151 B 151 B
middleware.js gzip 37.4 kB 37.4 kB N/A
edge-runtime..pack.js gzip 1.92 kB 1.92 kB
Overall change 2.07 kB 2.07 kB
Next Runtimes
vercel/next.js canary vercel/next.js 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function Change
app-page-exp...dev.js gzip 168 kB 168 kB
app-page-exp..prod.js gzip 94.1 kB 94.1 kB
app-page-tur..prod.js gzip 94.8 kB 94.8 kB
app-page-tur..prod.js gzip 89.4 kB 89.4 kB
app-page.run...dev.js gzip 138 kB 138 kB
app-page.run..prod.js gzip 88.7 kB 88.7 kB
app-route-ex...dev.js gzip 24 kB 24 kB
app-route-ex..prod.js gzip 16.6 kB 16.6 kB
app-route-tu..prod.js gzip 16.6 kB 16.6 kB
app-route-tu..prod.js gzip 16.2 kB 16.2 kB
app-route.ru...dev.js gzip 23.4 kB 23.4 kB
app-route.ru..prod.js gzip 16.2 kB 16.2 kB
pages-api-tu..prod.js gzip 9.37 kB 9.37 kB
pages-api.ru...dev.js gzip 9.64 kB 9.64 kB
pages-api.ru..prod.js gzip 9.37 kB 9.37 kB
pages-turbo...prod.js gzip 21.9 kB 21.9 kB
pages.runtim...dev.js gzip 22.5 kB 22.5 kB
pages.runtim..prod.js gzip 21.9 kB 21.9 kB
server.runti..prod.js gzip 49.4 kB 49.4 kB
Overall change 930 kB 930 kB
Commit: f3a2287

@@ -1041,6 +1041,18 @@ async function startWatcher(opts: SetupOpts) {
)
}

async function writeManifests(): Promise<void> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is This A Pigeon

@@ -1166,6 +1178,8 @@ async function startWatcher(opts: SetupOpts) {
'edge'
)
await loadMiddlewareManifest('instrumentation', 'instrumentation')
await writeManifests()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noticed this bug where it didn't write the manifest with instrumentation

Comment on lines +1045 to +1053
await writeBuildManifest(opts.fsChecker.rewrites)
await writeAppBuildManifest()
await writePagesManifest()
await writeAppPathsManifest()
await writeMiddlewareManifest()
await writeActionManifest()
await writeFontManifest()
await writeLoadableManifest()
await writeFallbackBuildManifest()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want to parallelize these? I have not idea if they have side effects so maybe not safe but seemingly the io parts could be done in parallel if they do not interact

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall we'll likely want to make these sync fs actually, but kept it as-is for now, can optimize later 👍

@timneutkens timneutkens merged commit 8c4649e into canary Dec 15, 2023
69 checks passed
@timneutkens timneutkens deleted the 12-15-Consolidate_manifest_writing_for_Turbopack_in_a_single_function branch December 15, 2023 16:21
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants