performance improvement of static generation #25035

sokra · 2021-05-12T09:48:32Z

move all access to built pages into worker pool

to allow parallelizing and avoid loading the bundles in the main thread

This improves performance of the static check step a bit and helps reducing memory load in main thread

enable splitChunks for server build in webpack 5

This improves performance for static generation by loading less code due to reduced duplication

to allow parallelizing and avoid loading the bundles in the main thread

this allows to deduplicate code in the server logic

ijjk · 2021-05-17T07:09:57Z

Failing test suites

Commit: 4b6b9cd

test/integration/app-document/test/index.test.js

Document and App > Client side > should keep state between page navigations

Expand output

● Document and App › Client side › should keep state between page navigations

thrown: "Exceeded timeout of 300000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

  71 |     })
  72 |
> 73 |     it('should keep state between page navigations', async () => {
     |     ^
  74 |       const browser = await webdriver(context.appPort, '/')
  75 |
  76 |       const randomNumber = await browser.elementByCss('#random-number').text()

  at integration/app-document/test/client.js:73:5
  at _default (integration/app-document/test/client.js:8:3)
  at integration/app-document/test/index.test.js:52:3
  at Object.<anonymous> (integration/app-document/test/index.test.js:26:1)

test/integration/scroll-back-restoration/test/index.test.js

Scroll Restoration Support > dev mode > should restore the scroll position on navigating back

Expand output

● Scroll Restoration Support › dev mode › should restore the scroll position on navigating back

TIMED OUT: /hi from another/

<head><meta charset="utf-8"><meta name="viewport" content="width=device-width"><meta name="next-head-count" content="2"><noscript data-n-css=""></noscript><link rel="preload" href="/_next/static/chunks/webpack.js?ts=1621235467147" as="script"><link rel="preload" href="/_next/static/chunks/main.js?ts=1621235467147" as="script"><link rel="preload" href="/_next/static/chunks/pages/_app.js?ts=1621235467147" as="script"><link rel="preload" href="/_next/static/chunks/pages/index.js?ts=1621235467147" as="script"><noscript id="__next_css__DO_NOT_USE__"></noscript></head><body><div id="__next"><div style="width:10000px;height:10000px;background:orange"></div><a id="to-another" style="margin-left:5000px;width:95000px;display:block" href="/another">to another</a><div id="end-el">the end</div></div><script src="/_next/static/chunks/react-refresh.js?ts=1621235467147"></script><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"loaded":true},"__N_SSP":true},"page":"/","query":{},"buildId":"development","isFallback":false,"gssp":true}</script><script nomodule="" src="/_next/static/chunks/polyfills.js?ts=1621235467147"></script><script src="/_next/static/chunks/webpack.js?ts=1621235467147"></script><script src="/_next/static/chunks/main.js?ts=1621235467147"></script><script src="/_next/static/chunks/pages/_app.js?ts=1621235467147"></script><script src="/_next/static/chunks/pages/index.js?ts=1621235467147"></script><div id="__next-build-watcher" style="position: fixed; bottom: 10px; right: 20px; width: 0px; height: 0px; z-index: 99999;"></div><script src="/_next/static/development/_buildManifest.js?ts=1621235467147"></script><script src="/_next/static/development/_ssgManifest.js?ts=1621235467147"></script><next-route-announcer><p aria-live="assertive" id="__next-route-announcer__" role="alert" style="border: 0px; clip: rect(0px, 0px, 0px, 0px); height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; width: 1px; white-space: nowrap; overflow-wrap: normal;"></p></next-route-announcer><script src="/_next/static/chunks/pages/another.js"></script></body>

  395 |
  396 |   if (hardError) {
> 397 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content)
      |           ^
  398 |   }
  399 |   return false
  400 | }

  at check (lib/next-test-utils.js:397:11)
  at Object.<anonymous> (integration/scroll-back-restoration/test/index.test.js:40:5)

test/integration/scroll-forward-restoration/test/index.test.js

Scroll Restoration Support > dev mode > should restore the scroll position on navigating forward

Expand output

● Scroll Restoration Support › dev mode › should restore the scroll position on navigating forward

TIMED OUT: /the end/

<head><meta charset="utf-8"><meta name="viewport" content="width=device-width"><meta name="next-head-count" content="2"><noscript data-n-css=""></noscript><link rel="preload" href="/_next/static/chunks/webpack.js?ts=1621235475193" as="script"><link rel="preload" href="/_next/static/chunks/main.js?ts=1621235475193" as="script"><link rel="preload" href="/_next/static/chunks/pages/_app.js?ts=1621235475193" as="script"><link rel="preload" href="/_next/static/chunks/pages/another.js?ts=1621235475193" as="script"><noscript id="__next_css__DO_NOT_USE__"></noscript></head><body><div id="__next"><p id="another">hi from another</p><a id="to-index" href="/">to index</a></div><script src="/_next/static/chunks/react-refresh.js?ts=1621235475193"></script><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{}},"page":"/another","query":{},"buildId":"development","nextExport":true,"autoExport":true,"isFallback":false}</script><script nomodule="" src="/_next/static/chunks/polyfills.js?ts=1621235475193"></script><script src="/_next/static/chunks/webpack.js?ts=1621235475193"></script><script src="/_next/static/chunks/main.js?ts=1621235475193"></script><script src="/_next/static/chunks/pages/_app.js?ts=1621235475193"></script><script src="/_next/static/chunks/pages/another.js?ts=1621235475193"></script><div id="__next-build-watcher" style="position: fixed; bottom: 10px; right: 20px; width: 0px; height: 0px; z-index: 99999;"></div><script src="/_next/static/development/_buildManifest.js?ts=1621235475193"></script><script src="/_next/static/development/_ssgManifest.js?ts=1621235475193"></script><next-route-announcer><p aria-live="assertive" id="__next-route-announcer__" role="alert" style="border: 0px; clip: rect(0px, 0px, 0px, 0px); height: 1px; margin: -1px; overflow: hidden; padding: 0px; position: absolute; width: 1px; white-space: nowrap; overflow-wrap: normal;"></p></next-route-announcer><script src="/_next/static/chunks/pages/index.js"></script></body>

  395 |
  396 |   if (hardError) {
> 397 |     throw new Error('TIMED OUT: ' + regex + '\n\n' + content)
      |           ^
  398 |   }
  399 |   return false
  400 | }

  at check (lib/next-test-utils.js:397:11)
  at Object.<anonymous> (integration/scroll-forward-restoration/test/index.test.js:25:5)

test/integration/trailing-slashes-href-resolving/test/index.test.js

href resolving trailing-slash > dev mode > should route to /blog/first-post/ correctly
href resolving trailing-slash > dev mode > should route to /catch-all/hello/world/ correctly
href resolving trailing-slash > dev mode > should route to /top-level-slug/ correctly

Expand output

● href resolving trailing-slash › dev mode › should route to /blog/first-post/ correctly

thrown: "Exceeded timeout of 120000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

  27 |   })
  28 |
> 29 |   it('should route to /blog/first-post/ correctly', async () => {
     |   ^
  30 |     const browser = await webdriver(appPort, '/')
  31 |     await browser.elementByCss('#to-blog-post').click()
  32 |

  at runTests (integration/trailing-slashes-href-resolving/test/index.test.js:29:3)
  at integration/trailing-slashes-href-resolving/test/index.test.js:105:5
  at integration/trailing-slashes-href-resolving/test/index.test.js:98:3
  at Object.<anonymous> (integration/trailing-slashes-href-resolving/test/index.test.js:97:1)

● href resolving trailing-slash › dev mode › should route to /catch-all/hello/world/ correctly

thrown: "Exceeded timeout of 120000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

  37 |   })
  38 |
> 39 |   it('should route to /catch-all/hello/world/ correctly', async () => {
     |   ^
  40 |     const browser = await webdriver(appPort, '/')
  41 |     await browser.elementByCss('#to-catch-all-item').click()
  42 |

  at runTests (integration/trailing-slashes-href-resolving/test/index.test.js:39:3)
  at integration/trailing-slashes-href-resolving/test/index.test.js:105:5
  at integration/trailing-slashes-href-resolving/test/index.test.js:98:3
  at Object.<anonymous> (integration/trailing-slashes-href-resolving/test/index.test.js:97:1)

● href resolving trailing-slash › dev mode › should route to /top-level-slug/ correctly

thrown: "Exceeded timeout of 120000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

  65 |   })
  66 |
> 67 |   it('should route to /top-level-slug/ correctly', async () => {
     |   ^
  68 |     const browser = await webdriver(appPort, '/')
  69 |     await browser.elementByCss('#to-slug').click()
  70 |

  at runTests (integration/trailing-slashes-href-resolving/test/index.test.js:67:3)
  at integration/trailing-slashes-href-resolving/test/index.test.js:105:5
  at integration/trailing-slashes-href-resolving/test/index.test.js:98:3
  at Object.<anonymous> (integration/trailing-slashes-href-resolving/test/index.test.js:97:1)

ijjk · 2021-05-17T08:52:18Z

Stats from current PR

Default Server Mode (Decrease detected ✓)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
buildDuration	16.4s	17.3s	⚠️ +893ms
buildDurationCached	5.1s	4.8s	-334ms
nodeModulesSize	46.6 MB	46.7 MB	⚠️ +1.39 kB

Page Load Tests Overall decrease ⚠️

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.827	2.887	⚠️ +0.06
/ avg req/sec	884.39	866.01	⚠️ -18.38
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.69	1.724	⚠️ +0.03
/error-in-render avg req/sec	1479.28	1450.5	⚠️ -28.78

Client Bundles (main, webpack, commons)

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
framework-HASH.js gzip	39.3 kB	39.3 kB	✓
main-HASH.js gzip	19.4 kB	19.4 kB	✓
webpack-HASH.js gzip	994 B	994 B	✓
Overall change	59.7 kB	59.7 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
polyfills-HASH.js gzip	31.1 kB	31.1 kB	✓
Overall change	31.1 kB	31.1 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
_app-HASH.js gzip	1.02 kB	1.02 kB	✓
_error-HASH.js gzip	3.06 kB	3.06 kB	✓
amp-HASH.js gzip	526 B	526 B	✓
css-HASH.js gzip	334 B	334 B	✓
hooks-HASH.js gzip	890 B	890 B	✓
index-HASH.js gzip	262 B	262 B	✓
link-HASH.js gzip	1.65 kB	1.65 kB	✓
routerDirect..HASH.js gzip	331 B	331 B	✓
withRouter-HASH.js gzip	329 B	329 B	✓
bb14e60e810b..30f.css gzip	125 B	125 B	✓
Overall change	8.54 kB	8.54 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
_buildManifest.js gzip	390 B	390 B	✓
Overall change	390 B	390 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
index.html gzip	560 B	560 B	✓
link.html gzip	569 B	569 B	✓
withRouter.html gzip	557 B	557 B	✓
Overall change	1.69 kB	1.69 kB	✓

Serverless Mode (Decrease detected ✓)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
buildDuration	20.8s	19s	-1.8s
buildDurationCached	7.2s	5.9s	-1.3s
nodeModulesSize	46.6 MB	46.7 MB	⚠️ +1.39 kB

Client Bundles (main, webpack, commons)

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
framework-HASH.js gzip	39.3 kB	39.3 kB	✓
main-HASH.js gzip	19.4 kB	19.4 kB	✓
webpack-HASH.js gzip	994 B	994 B	✓
Overall change	59.7 kB	59.7 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
polyfills-HASH.js gzip	31.1 kB	31.1 kB	✓
Overall change	31.1 kB	31.1 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
_app-HASH.js gzip	1.02 kB	1.02 kB	✓
_error-HASH.js gzip	3.06 kB	3.06 kB	✓
amp-HASH.js gzip	526 B	526 B	✓
css-HASH.js gzip	334 B	334 B	✓
hooks-HASH.js gzip	890 B	890 B	✓
index-HASH.js gzip	262 B	262 B	✓
link-HASH.js gzip	1.65 kB	1.65 kB	✓
routerDirect..HASH.js gzip	331 B	331 B	✓
withRouter-HASH.js gzip	329 B	329 B	✓
bb14e60e810b..30f.css gzip	125 B	125 B	✓
Overall change	8.54 kB	8.54 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
_buildManifest.js gzip	390 B	390 B	✓
Overall change	390 B	390 B	✓

Serverless bundles Overall decrease ✓

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
_error.js	1.34 MB	16.9 kB	-1.33 MB
404.html	2.42 kB	2.42 kB	✓
500.html	2.41 kB	2.41 kB	✓
amp.amp.html	10.8 kB	10.8 kB	✓
amp.html	1.61 kB	1.61 kB	✓
css.html	1.79 kB	1.79 kB	✓
hooks.html	1.67 kB	1.67 kB	✓
index.js	1.34 MB	17.2 kB	-1.33 MB
link.js	1.4 MB	17.4 kB	-1.39 MB
routerDirect.js	1.4 MB	17.4 kB	-1.38 MB
withRouter.js	1.4 MB	17.4 kB	-1.38 MB
Overall change	6.91 MB	107 kB	-6.8 MB

Webpack 4 Mode (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
buildDuration	13.6s	14.3s	⚠️ +677ms
buildDurationCached	5.8s	5.8s	⚠️ +8ms
nodeModulesSize	46.6 MB	46.7 MB	⚠️ +1.39 kB

Page Load Tests Overall increase ✓

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.874	2.831	-0.04
/ avg req/sec	869.87	883.24	+13.37
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.688	1.584	-0.1
/error-in-render avg req/sec	1481.43	1577.8	+96.37

Client Bundles (main, webpack, commons)

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
677f882d2ed8..HASH.js gzip	13.3 kB	13.3 kB	✓
framework.HASH.js gzip	39 kB	39 kB	✓
main-HASH.js gzip	7.26 kB	7.26 kB	✓
webpack-HASH.js gzip	751 B	751 B	✓
Overall change	60.3 kB	60.3 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
polyfills-HASH.js gzip	31.3 kB	31.3 kB	✓
Overall change	31.3 kB	31.3 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
_app-HASH.js gzip	1.28 kB	1.28 kB	✓
_error-HASH.js gzip	3.74 kB	3.74 kB	✓
amp-HASH.js gzip	536 B	536 B	✓
css-HASH.js gzip	339 B	339 B	✓
hooks-HASH.js gzip	887 B	887 B	✓
index-HASH.js gzip	227 B	227 B	✓
link-HASH.js gzip	1.64 kB	1.64 kB	✓
routerDirect..HASH.js gzip	303 B	303 B	✓
withRouter-HASH.js gzip	302 B	302 B	✓
e025d2764813..52f.css gzip	125 B	125 B	✓
Overall change	9.38 kB	9.38 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
_buildManifest.js gzip	420 B	420 B	✓
Overall change	420 B	420 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js perf/static-check-avoid-main	Change
index.html gzip	614 B	614 B	✓
link.html gzip	620 B	620 B	✓
withRouter.html gzip	606 B	606 B	✓
Overall change	1.84 kB	1.84 kB	✓

Commit: e706ae6

timneutkens

Looks good to land now 👍

### move all access to built pages into worker pool to allow parallelizing and avoid loading the bundles in the main thread This improves performance of the static check step a bit and helps reducing memory load in main thread ### enable splitChunks for server build in webpack 5 This improves performance for static generation by loading less code due to reduced duplication

ijjk added the type: next label May 12, 2021

move all access to built pages into worker pool

c62678b

to allow parallelizing and avoid loading the bundles in the main thread

sokra force-pushed the perf/static-check-avoid-main branch from 346d41e to c62678b Compare May 12, 2021 09:54