feat(next): trace build dependencies for turborepo

[tknickman]

What?

This adds a method to optionally detect disk, network, and env access access during a build and write this to a file.

Why?

This will be consumed by Turborepo to determine if a build can be cached, and if so, how it should be cached.

How?

This implements a proxy that can be selectively applied to functions to watch their access to disk, network, and env. For Turborepo, we only care about what can alter the build output, to achieve this, we trace the config load, and the static page generation.

Closes https://linear.app/vercel/issue/TURBO-1904/support-build-access-tracing-in-nextjs

Performance Testing:

Repo - https://github.com/tknickman/nextjs-turborepo-build-trace-performance-testing

- - - - - - - - - - - - - - - - - Performance - - - - - - - - - - - - - - - - -
pages-single-app:
  44629.53301 ms (+/- 6313.82757 ms) from 5 iterations (223318.53692 ms total)
local-pages-single-app-with-trace:
  45684.00503 ms (+/- 6570.85083 ms) from 5 iterations (228585.01729 ms total)
local-pages-single-app-without-trace:
  45938.76231 ms (+/- 6816.13022 ms) from 5 iterations (229830.15925 ms total)
approuter-single-app:
  49517.23156 ms (+/- 7820.20687 ms) from 5 iterations (247725.16713 ms total)
local-approuter-single-app-with-trace:
  50019.98247 ms (+/- 9443.82537 ms) from 5 iterations (250224.62496 ms total)
local-approuter-single-app-without-trace:
  50124.07092 ms (+/- 9656.22047 ms) from 5 iterations (250726.35688 ms total)
local-large-pages-single-app-without-trace:
  51582.92992 ms (+/- 12179.09895 ms) from 5 iterations (258469.36942 ms total)
local-large-pages-single-app-with-trace:
  51740.25813 ms (+/- 12601.02878 ms) from 5 iterations (259329.50946 ms total)
local-xlarge-pages-single-app-with-trace:
  234111.66735 ms (+/- 74629.30509 ms) from 5 iterations (1191018.63608 ms total)
local-xlarge-pages-single-app-without-trace:
  239621.11837 ms (+/- 84629.16374 ms) from 5 iterations (1218078.41921 ms total)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Test case keys

• local-xlarge-pages-single-app-without-trace: this branch, built and linked to a local project with 5000 SSG pages and trace disabled
• local-xlarge-pages-single-app-with-trace: this branch, built and linked to a local project with 5000 SSG pages and trace enabled
• local-large-pages-single-app-with-trace: this branch, built and linked to a local project with 100 SSG pages and trace enabled
• local-large-pages-single-app-without-trace: this branch, built and linked to a local project with 100 SSG pages and trace disabled
• local-approuter-single-app-without-trace: this branch, built and linked to a local default approuter starter and trace disabled
• local-approuter-single-app-with-trace: this branch, built and linked to a local default approuter starter and trace enabled
• local-pages-single-app-with-trace: this branch, built and linked to a local default pages starter and trace enabled
• local-pages-single-app-without-trace: this branch, built and linked to a local default pages starter and trace disabled
• approuter-single-app: next@latest running on the default approuter starter
• pages-single-app: next@latest running on the default pages starter

TODO:

• Telemetry events

[ijjk]

Tests Passed

[ijjk]

Stats from current PR

Default Build

General Overall increase ⚠️

	vercel/next.js canary	tknickman/next.js tomknickman/turbo-1904-support-build-access-tracing-in-nextjs	Change
buildDuration	11.6s	11.7s	N/A
buildDurationCached	6.3s	5.3s	N/A
nodeModulesSize	200 MB	200 MB	⚠️ +70.1 kB
nextStartRea..uration (ms)	425ms	423ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	tknickman/next.js tomknickman/turbo-1904-support-build-access-tracing-in-nextjs	Change
3f784ff6-HASH.js gzip	53.4 kB	53.4 kB	✓
423.HASH.js gzip	185 B	181 B	N/A
68-HASH.js gzip	29.9 kB	29.9 kB	N/A
framework-HASH.js gzip	45.2 kB	45.2 kB	✓
main-app-HASH.js gzip	238 B	239 B	N/A
main-HASH.js gzip	31.8 kB	31.8 kB	N/A
webpack-HASH.js gzip	1.7 kB	1.7 kB	✓
Overall change	100 kB	100 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	tknickman/next.js tomknickman/turbo-1904-support-build-access-tracing-in-nextjs	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	tknickman/next.js tomknickman/turbo-1904-support-build-access-tracing-in-nextjs	Change
_app-HASH.js gzip	194 B	195 B	N/A
_error-HASH.js gzip	182 B	181 B	N/A
amp-HASH.js gzip	502 B	502 B	✓
css-HASH.js gzip	320 B	322 B	N/A
dynamic-HASH.js gzip	2.5 kB	2.5 kB	N/A
edge-ssr-HASH.js gzip	255 B	256 B	N/A
head-HASH.js gzip	350 B	349 B	N/A
hooks-HASH.js gzip	368 B	369 B	N/A
image-HASH.js gzip	4.18 kB	4.18 kB	N/A
index-HASH.js gzip	257 B	256 B	N/A
link-HASH.js gzip	2.61 kB	2.61 kB	N/A
routerDirect..HASH.js gzip	310 B	311 B	N/A
script-HASH.js gzip	384 B	383 B	N/A
withRouter-HASH.js gzip	306 B	308 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	608 B	608 B	✓

Client Build Manifests

	vercel/next.js canary	tknickman/next.js tomknickman/turbo-1904-support-build-access-tracing-in-nextjs	Change
_buildManifest.js gzip	484 B	484 B	✓
Overall change	484 B	484 B	✓

Rendered Page Sizes

	vercel/next.js canary	tknickman/next.js tomknickman/turbo-1904-support-build-access-tracing-in-nextjs	Change
index.html gzip	528 B	527 B	N/A
link.html gzip	541 B	540 B	N/A
withRouter.html gzip	523 B	522 B	N/A
Overall change	0 B	0 B	✓

Edge SSR bundle Size

	vercel/next.js canary	tknickman/next.js tomknickman/turbo-1904-support-build-access-tracing-in-nextjs	Change
edge-ssr.js gzip	94 kB	94 kB	N/A
page.js gzip	150 kB	150 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	tknickman/next.js tomknickman/turbo-1904-support-build-access-tracing-in-nextjs	Change
middleware-b..fest.js gzip	623 B	623 B	✓
middleware-r..fest.js gzip	151 B	149 B	N/A
middleware.js gzip	37.6 kB	37.6 kB	N/A
edge-runtime..pack.js gzip	1.92 kB	1.92 kB	✓
Overall change	2.55 kB	2.55 kB	✓

Next Runtimes

	vercel/next.js canary	tknickman/next.js tomknickman/turbo-1904-support-build-access-tracing-in-nextjs	Change
app-page-exp...dev.js gzip	170 kB	170 kB	✓
app-page-exp..prod.js gzip	95.8 kB	95.8 kB	✓
app-page-tur..prod.js gzip	96.4 kB	96.4 kB	✓
app-page-tur..prod.js gzip	91 kB	91 kB	✓
app-page.run...dev.js gzip	142 kB	142 kB	✓
app-page.run..prod.js gzip	90.3 kB	90.3 kB	✓
app-route-ex...dev.js gzip	22.2 kB	22.2 kB	✓
app-route-ex..prod.js gzip	14.9 kB	14.9 kB	✓
app-route-tu..prod.js gzip	14.9 kB	14.9 kB	✓
app-route-tu..prod.js gzip	14.5 kB	14.5 kB	✓
app-route.ru...dev.js gzip	21.7 kB	21.7 kB	✓
app-route.ru..prod.js gzip	14.5 kB	14.5 kB	✓
pages-api-tu..prod.js gzip	9.43 kB	9.43 kB	✓
pages-api.ru...dev.js gzip	9.7 kB	9.7 kB	✓
pages-api.ru..prod.js gzip	9.43 kB	9.43 kB	✓
pages-turbo...prod.js gzip	22 kB	22 kB	✓
pages.runtim...dev.js gzip	22.7 kB	22.7 kB	✓
pages.runtim..prod.js gzip	22 kB	22 kB	✓
server.runti..prod.js gzip	49.7 kB	49.7 kB	✓
Overall change	933 kB	933 kB	✓

Commit: de9e4a3

[tknickman]

@wyattjoh all your comments should be addressed. I also wanted to get rid of the process.binding usage to make this a bit more resilient to node version changes (turns out this was the right instinct - node 20 deprecates it). TCP was simple, Env was already using Proxy, but Fs was non-trivial. The solution I have is the least intrusive one that I could come up with that works well. It's not 100% foolproof, but should capture a good amount of usage.

The alternative to that FS proxy method is throwing it out and just not trying to catch Fs access outside the repo. This should be ok, as access outside of the repo isn't going to be portable anyway and would cause issues when moving the repo between machines. So let me know what you think about that Fs proxy setup.