Generate static html for bots

[huozhi]

x-ref: https://github.com/vercel/next.js/pull/34972/files/8abceb912d4f80b8dc7227d32e654caf8a6fe96c#r818204484

Web server should generate static response for bots, and also etag.
Currently we're unable to have etag in the edge runtime due to the implementation of etag library (it's relying on the nodejs libs). Leave it as TODO temporarily.

For testing, use the streaming chunk flushing count to detect if the response is flushed at once.

Bug

• Related issues linked using fixes #number
• Integration tests added
• Errors have helpful link attached, see contributing.md

[devknoll]

Just minor nits remaining

[ijjk]

Stats from current PR

Default Build (Increase detected ⚠️)

General Overall decrease ✓

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
buildDuration	16s	15.7s	-352ms
buildDurationCached	6s	5.9s	-117ms
nodeModulesSize	372 MB	372 MB	-138 B

Page Load Tests Overall increase ✓

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
/ failed reqs	0	0	✓
/ total time (seconds)	3.019	3.066	⚠️ +0.05
/ avg req/sec	828.15	815.52	⚠️ -12.63
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.395	1.368	-0.03
/error-in-render avg req/sec	1791.66	1828.05	+36.39

Client Bundles (main, webpack)

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
925.HASH.js gzip	179 B	179 B	✓
framework-HASH.js gzip	42 kB	42 kB	✓
main-HASH.js gzip	27.9 kB	27.9 kB	✓
webpack-HASH.js gzip	1.44 kB	1.44 kB	✓
Overall change	71.5 kB	71.5 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
_app-HASH.js gzip	1.36 kB	1.36 kB	✓
_error-HASH.js gzip	192 B	192 B	✓
amp-HASH.js gzip	309 B	309 B	✓
css-HASH.js gzip	327 B	327 B	✓
dynamic-HASH.js gzip	2.57 kB	2.57 kB	✓
head-HASH.js gzip	351 B	351 B	✓
hooks-HASH.js gzip	920 B	920 B	✓
image-HASH.js gzip	5.06 kB	5.06 kB	✓
index-HASH.js gzip	263 B	263 B	✓
link-HASH.js gzip	2.26 kB	2.26 kB	✓
routerDirect..HASH.js gzip	320 B	320 B	✓
script-HASH.js gzip	387 B	387 B	✓
withRouter-HASH.js gzip	319 B	319 B	✓
85e02e95b279..7e3.css gzip	107 B	107 B	✓
Overall change	14.8 kB	14.8 kB	✓

Client Build Manifests

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
_buildManifest.js gzip	460 B	460 B	✓
Overall change	460 B	460 B	✓

Rendered Page Sizes

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
index.html gzip	532 B	532 B	✓
link.html gzip	545 B	545 B	✓
withRouter.html gzip	525 B	525 B	✓
Overall change	1.6 kB	1.6 kB	✓

---

Default Build with SWC (Increase detected ⚠️)

General Overall decrease ✓

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
buildDuration	19.2s	19s	-191ms
buildDurationCached	5.9s	6.4s	⚠️ +492ms
nodeModulesSize	372 MB	372 MB	-138 B

Page Load Tests Overall increase ✓

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
/ failed reqs	0	0	✓
/ total time (seconds)	3.352	3.028	-0.32
/ avg req/sec	745.85	825.63	+79.78
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.457	1.39	-0.07
/error-in-render avg req/sec	1715.32	1798.18	+82.86

Client Bundles (main, webpack)

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
925.HASH.js gzip	178 B	178 B	✓
framework-HASH.js gzip	42.3 kB	42.3 kB	✓
main-HASH.js gzip	28.1 kB	28.1 kB	✓
webpack-HASH.js gzip	1.45 kB	1.45 kB	✓
Overall change	72.1 kB	72.1 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
_app-HASH.js gzip	1.36 kB	1.36 kB	✓
_error-HASH.js gzip	179 B	179 B	✓
amp-HASH.js gzip	313 B	313 B	✓
css-HASH.js gzip	324 B	324 B	✓
dynamic-HASH.js gzip	2.56 kB	2.56 kB	✓
head-HASH.js gzip	351 B	351 B	✓
hooks-HASH.js gzip	921 B	921 B	✓
image-HASH.js gzip	5.2 kB	5.2 kB	✓
index-HASH.js gzip	261 B	261 B	✓
link-HASH.js gzip	2.33 kB	2.33 kB	✓
routerDirect..HASH.js gzip	322 B	322 B	✓
script-HASH.js gzip	388 B	388 B	✓
withRouter-HASH.js gzip	317 B	317 B	✓
85e02e95b279..7e3.css gzip	107 B	107 B	✓
Overall change	14.9 kB	14.9 kB	✓

Client Build Manifests

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
_buildManifest.js gzip	458 B	458 B	✓
Overall change	458 B	458 B	✓

Rendered Page Sizes

	vercel/next.js canary	huozhi/next.js static-rendering-4bot	Change
index.html gzip	530 B	530 B	✓
link.html gzip	543 B	543 B	✓
withRouter.html gzip	525 B	525 B	✓
Overall change	1.6 kB	1.6 kB	✓

Commit: 52dff55