Add crawler blocking for fallback: true

[ijjk]

This implements automatic blocking for fallback: true when a crawler requests the page which is described in this RFC #28180.

Feature

• Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR.
• Related issues linked using fixes #number
• Integration tests added
• Documentation added
• Telemetry added. In case of a feature if it's used or not.
• Errors have helpful link attached, see contributing.md

x-ref: #28180

[ijjk]

Stats from current PR

Default Build (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
buildDuration	15.7s	15.7s	⚠️ +66ms
buildDurationCached	3.8s	3.8s	-19ms
nodeModulesSize	182 MB	182 MB	⚠️ +285 B

Page Load Tests Overall increase ✓

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
/ failed reqs	0	0	✓
/ total time (seconds)	3.301	3.3	0
/ avg req/sec	757.29	757.54	+0.25
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.953	1.943	-0.01
/error-in-render avg req/sec	1280.09	1286.49	+6.4

Client Bundles (main, webpack, commons)

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
745.HASH.js gzip	179 B	179 B	✓
framework-HASH.js gzip	42.2 kB	42.2 kB	✓
main-HASH.js gzip	26.6 kB	26.6 kB	✓
webpack-HASH.js gzip	1.45 kB	1.45 kB	✓
Overall change	70.4 kB	70.4 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
polyfills-a4..dd70.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
_app-HASH.js gzip	979 B	979 B	✓
_error-HASH.js gzip	194 B	194 B	✓
amp-HASH.js gzip	312 B	312 B	✓
css-HASH.js gzip	329 B	329 B	✓
dynamic-HASH.js gzip	2.67 kB	2.67 kB	✓
head-HASH.js gzip	351 B	351 B	✓
hooks-HASH.js gzip	918 B	918 B	✓
image-HASH.js gzip	4.14 kB	4.14 kB	✓
index-HASH.js gzip	261 B	261 B	✓
link-HASH.js gzip	1.66 kB	1.66 kB	✓
routerDirect..HASH.js gzip	318 B	318 B	✓
script-HASH.js gzip	387 B	387 B	✓
withRouter-HASH.js gzip	320 B	320 B	✓
bb14e60e810b..30f.css gzip	125 B	125 B	✓
Overall change	13 kB	13 kB	✓

Client Build Manifests

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
_buildManifest.js gzip	492 B	492 B	✓
Overall change	492 B	492 B	✓

Rendered Page Sizes

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
index.html gzip	540 B	540 B	✓
link.html gzip	553 B	553 B	✓
withRouter.html gzip	534 B	534 B	✓
Overall change	1.63 kB	1.63 kB	✓

---

Default Build with SWC (Decrease detected ✓)

General Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
buildDuration	8.1s	8s	-43ms
buildDurationCached	3.8s	3.8s	⚠️ +5ms
nodeModulesSize	182 MB	182 MB	⚠️ +285 B

Page Load Tests Overall decrease ⚠️

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
/ failed reqs	0	0	✓
/ total time (seconds)	3.386	3.33	-0.06
/ avg req/sec	738.42	750.78	+12.36
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.82	1.844	⚠️ +0.02
/error-in-render avg req/sec	1373.67	1356.11	⚠️ -17.56

Client Bundles (main, webpack, commons)

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
264.HASH.js gzip	179 B	179 B	✓
675-HASH.js gzip	13.8 kB	13.8 kB	✓
framework-HASH.js gzip	50.7 kB	50.7 kB	✓
main-HASH.js gzip	34.4 kB	34.4 kB	✓
webpack-HASH.js gzip	1.65 kB	1.65 kB	✓
Overall change	101 kB	101 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
polyfills-a4..dd70.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
_app-HASH.js gzip	1.32 kB	1.32 kB	✓
_error-HASH.js gzip	179 B	179 B	✓
amp-HASH.js gzip	315 B	315 B	✓
css-HASH.js gzip	330 B	330 B	✓
dynamic-HASH.js gzip	2.8 kB	2.8 kB	✓
head-HASH.js gzip	356 B	356 B	✓
hooks-HASH.js gzip	638 B	638 B	✓
image-HASH.js gzip	575 B	575 B	✓
index-HASH.js gzip	261 B	261 B	✓
link-HASH.js gzip	2.2 kB	2.2 kB	✓
routerDirect..HASH.js gzip	327 B	327 B	✓
script-HASH.js gzip	392 B	392 B	✓
withRouter-HASH.js gzip	323 B	323 B	✓
bb14e60e810b..30f.css gzip	125 B	125 B	✓
Overall change	10.1 kB	10.1 kB	✓

Client Build Manifests

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
_buildManifest.js gzip	512 B	512 B	✓
Overall change	512 B	512 B	✓

Rendered Page Sizes

	vercel/next.js canary	ijjk/next.js add/crawler-blocking	Change
index.html gzip	539 B	539 B	✓
link.html gzip	551 B	551 B	✓
withRouter.html gzip	532 B	532 B	✓
Overall change	1.62 kB	1.62 kB	✓

Commit: ff9ba07