perf: add bundled rendering runtimes (#52997)

## What?

In Next, rendering a route involves 3 layers:
- the routing layer, which will direct the request to the correct route to render
- the rendering layer, which will take a route and render it appropriately
- the user layer, which contains the user code 

In #51831, in order to optimise the boot time of Next.js, I introduced a change that allowed the routing layer to be bundled. In this PR, I'm doing the same for the rendering layer. This is building up on @wyattjoh's work that initially split the routing and the rendering layer into separate entry-points.

The benefits of having this approach is that this allows us to compartmentalise the different part of Next, optimise them individually and making sure that serving a request is as efficient as possible, e.g. rendering a `pages` route should not need code from the `app router` to be used.

There are now 4 different rendering runtimes, depending on the route type:
- app pages: for App Router pages
- app routes: for App Router route handlers
- pages: for legacy pages
- pages api: for legacy API routes

This change should be transparent to the end user, beside faster cold boots.

## Notable changes

Doing this change required a lot of changes for Next.js under the hood in order to make the different layers play well together.

### New conventions for externals/shared modules

The big issue of bundling the rendering runtimes is that the user code needs to be able to reference an instance of a module/value created in Next during the render. This is the case when the user wants to access the router context during SSR via `next/link` for example; when you call `useContext(value)` the value needs to be the exact same reference to one as the one created by `createContext` earlier.

Previously, we were handling this case by making all files from Next that were affected by this `externals`, meaning that we were marking them not to be bundled.

**Why not keep it this way?**

The goal of this PR as stated previously was to make the rendering process as efficient as possible, so I really wanted to avoid extraneous fs reads to unoptimised code. 

In order to "fix" it, I introduced two new conventions to the codebase:
- all files that explicitly need to be shared between a rendering runtime and the user code must be suffixed by `.shared-runtime` and exposed via adding a reference in the relevant `externals` file. At compilation time, a reference to a file ending with this will get re-written to the appropriate runtime.
- all files that need to be truly externals need to be suffixed by `.external`. At compilation time, a reference to it will stay as-is. This special case is needed mostly only for the async local storages that need to be shared with all three layers of Next.

As a side effect, we should be bundling more of the Next code in the user bundles, so it should be slightly more efficient.

### App route handlers are compiled on their own layer

App route handlers should be compiled in their own layer, this allows us to separate more cleanly the compilation logic here (we don't need to run the RSC logic for example).

### New rendering bundles

We now generate a prod and a dev bundle for:
- the routing server
- the app/pages SSR rendering process
- the API routes process

The development bundle is needed because:
- there is code in Next that relies on NODE_ENV
- because we opt out of the logic referencing the correct rendering runtime in dev for a `shared-runtime` file. This is because we don't need to and that Turbopack does not support rewriting an external to something that looks like this `require('foo').bar.baz` yet. We will need to fix that when Turbopack build ships.

### New development pipeline

Bundling Next is now required when developing on the repo so I extended the taskfile setup to account for that. The webpack config for Next itself lives in `webpack.config.js` and contains the logic for all the new bundles generated.

### Misc changes

There are some misc reshuffling in the code to better use the tree shaking abilities that we can now use.

fixes NEXT-1573

Co-authored-by: Alex Kirszenberg <1621758+alexkirsz@users.noreply.github.com>

.eslintrc.json

@@ -162,7 +162,10 @@
     },
     {
       "files": ["packages/**"],
-      "excludedFiles": ["packages/next/taskfile.js"],
+      "excludedFiles": [
+        "packages/next/taskfile*.js",
+        "packages/next/webpack.config.js"
+      ],
       "rules": {
         "no-shadow": ["warn", { "builtinGlobals": false }],
         "import/no-extraneous-dependencies": [

bench/basic-app/app/api/app/route.js

@@ -0,0 +1,5 @@
+export function GET() {
+  return { name: 'John Doe' }
+}
+
+export const dynamic = 'force-dynamic'

bench/basic-app/app/layout.js

@@ -0,0 +1,12 @@
+import React from 'react'
+
+export default function Layout({ children }) {
+  return (
+    <html>
+      <head>
+        <title>My App</title>
+      </head>
+      <body>{children}</body>
+    </html>
+  )
+}

bench/basic-app/app/page.js

@@ -0,0 +1,7 @@
+import React from 'react'
+
+export default function Page() {
+  return <h1>My Page</h1>
+}
+
+export const dynamic = 'force-dynamic'

bench/basic-app/next.config.js

@@ -0,0 +1,5 @@
+module.exports = {
+  experimental: {
+    serverMinification: true,
+  },
+}

bench/basic-app/pages/api/index.js

@@ -0,0 +1,3 @@
+export default function handler(req, res) {
+  res.status(200).json({ name: 'John Doe' })
+}

bench/basic-app/pages/pages/index.js

@@ -0,0 +1,7 @@
+export default () => 'Hello World'
+
+export function getServerSideProps() {
+  return {
+    props: {},
+  }
+}

packages/next-swc/crates/next-core/js/src/entry/app-edge-renderer.tsx

@@ -2,6 +2,8 @@
 // the other imports
 import startOperationStreamHandler from '../internal/operation-stream'
 
+import 'next/dist/server/node-polyfill-fetch'
+
 import { join } from 'path'
 import { parse as parseUrl } from 'node:url'
 

packages/next-swc/crates/next-core/js/src/entry/app-renderer.tsx

@@ -3,13 +3,15 @@
 import startOperationStreamHandler from '../internal/operation-stream'
 
 import '../polyfill/app-polyfills.ts'
+// TODO: when actions are supported, this should be removed/changed
+process.env.__NEXT_PRIVATE_PREBUNDLED_REACT = 'next'
+import 'next/dist/server/require-hook'
 
 import type { IncomingMessage } from 'node:http'
 
 import type { RenderData } from 'types/turbopack'
 import type { RenderOpts } from 'next/dist/server/app-render/types'
 
-import { renderToHTMLOrFlight } from 'next/dist/server/app-render/app-render'
 import { RSC_VARY_HEADER } from 'next/dist/client/components/app-router-headers'
 import { headersFromEntries, initProxiedHeaders } from '../internal/headers'
 import { parse, ParsedUrlQuery } from 'node:querystring'
@@ -23,6 +25,10 @@ import { join } from 'node:path'
 import { nodeFs } from 'next/dist/server/lib/node-fs-methods'
 import { IncrementalCache } from 'next/dist/server/lib/incremental-cache'
 
+const {
+  renderToHTMLOrFlight,
+} = require('next/dist/compiled/next-server/app-page.runtime.dev')
+
 installRequireAndChunkLoad()
 
 const MIME_TEXT_HTML_UTF8 = 'text/html; charset=utf-8'

packages/next-swc/crates/next-core/js/src/entry/app/hydrate.tsx

@@ -6,7 +6,7 @@ import { createFromReadableStream } from 'next/dist/compiled/react-server-dom-we
 import { callServer } from 'next/dist/client/app-call-server'
 import { linkGc } from 'next/dist/client/app-link-gc'
 
-import { HeadManagerContext } from 'next/dist/shared/lib/head-manager-context'
+import { HeadManagerContext } from 'next/dist/shared/lib/head-manager-context.shared-runtime'
 
 import { initializeHMR } from '@vercel/turbopack-next/dev/client'
 

packages/next-swc/crates/next-core/js/src/internal/page-server-handler.tsx

@@ -3,11 +3,12 @@
 import { IPC } from '@vercel/turbopack-node/ipc/index'
 
 import 'next/dist/server/node-polyfill-fetch.js'
+import 'next/dist/server/require-hook'
 import './shims'
 
 import type { IncomingMessage, ServerResponse } from 'node:http'
 
-import { renderToHTML, RenderOpts } from 'next/dist/server/render'
+import type { RenderOpts } from 'next/dist/server/render'
 import { getRedirectStatus } from 'next/dist/lib/redirect-status'
 import { PERMANENT_REDIRECT_STATUS } from 'next/dist/shared/lib/constants'
 import { buildStaticPaths } from 'next/dist/build/utils'
@@ -21,6 +22,9 @@ import type { RenderData } from 'types/turbopack'
 import type { ChunkGroup } from 'types/next'
 import type { NextComponentType } from 'next/types'
 import { parse } from 'node:querystring'
+const {
+  renderToHTML,
+} = require('next/dist/compiled/next-server/pages.runtime.dev')
 
 const ipc = IPC as Ipc<IpcIncomingMessage, IpcOutgoingMessage>
 

packages/next-swc/crates/next-core/src/app_source.rs

@@ -2,7 +2,7 @@ use std::{collections::HashMap, io::Write as _, iter::once};
 
 use anyhow::{bail, Result};
 use indexmap::indexmap;
-use indoc::indoc;
+use indoc::formatdoc;
 use serde_json::Value as JsonValue;
 use turbo_tasks::Vc;
 use turbopack_binding::{
@@ -968,13 +968,18 @@ impl AppRenderer {
             .emit();
         }
 
-        let mut result = RopeBuilder::from(indoc! {"
-                \"TURBOPACK { chunking-type: isolatedParallel; transition: next-edge-server-component }\";
+        let mut result = RopeBuilder::from(
+            formatdoc!(
+                "
+                \"TURBOPACK {{ chunking-type: isolatedParallel; transition: {rsc_transition} }}\";
                 import GlobalErrorMod from \"next/dist/client/components/error-boundary\"
-                const { GlobalError } = GlobalErrorMod;
-                \"TURBOPACK { chunking-type: isolatedParallel; transition: next-edge-server-component }\";
+                const {{ GlobalError }} = GlobalErrorMod;
+                \"TURBOPACK {{ chunking-type: isolatedParallel; transition: {rsc_transition} }}\";
                 import base from \"next/dist/server/app-render/entry-base\"\n
-            "});
+            "
+            )
+            .into_bytes(),
+        );
 
         for import in loader_tree_module.imports {
             writeln!(result, "{import}")?;

packages/next-swc/crates/next-core/src/next_edge/context.rs

@@ -96,10 +96,9 @@ pub async fn get_edge_resolve_options_context(
     ];
 
     match ty {
-        ServerContextType::AppRSC { .. } | ServerContextType::AppRoute { .. } => {
-            custom_conditions.push("react-server".to_string())
-        }
-        ServerContextType::Pages { .. }
+        ServerContextType::AppRSC { .. } => custom_conditions.push("react-server".to_string()),
+        ServerContextType::AppRoute { .. }
+        | ServerContextType::Pages { .. }
         | ServerContextType::PagesData { .. }
         | ServerContextType::AppSSR { .. }
         | ServerContextType::Middleware { .. } => {}

packages/next-swc/crates/next-core/src/next_edge/route_transition.rs

@@ -58,24 +58,26 @@ impl Transition for NextEdgeRouteTransition {
 
     #[turbo_tasks::function]
     async fn process_module(
-        &self,
+        self: Vc<Self>,
         asset: Vc<Box<dyn Module>>,
         context: Vc<ModuleAssetContext>,
     ) -> Result<Vc<Box<dyn Module>>> {
+        let new_context = self.process_context(context);
+        let this = self.await?;
         let new_asset = route_bootstrap(
             asset,
-            Vc::upcast(context),
-            self.base_path,
-            self.bootstrap_asset,
+            Vc::upcast(new_context),
+            this.base_path,
+            this.bootstrap_asset,
             Vc::cell(indexmap! {
-                "NAME".to_string() => self.entry_name.clone(),
+                "NAME".to_string() => this.entry_name.clone(),
             }),
         );
 
         let asset = ChunkGroupFilesAsset {
             module: Vc::upcast(new_asset),
-            client_root: self.output_path,
-            chunking_context: self.edge_chunking_context,
+            client_root: this.output_path,
+            chunking_context: this.edge_chunking_context,
             runtime_entries: None,
         };
 

packages/next-swc/crates/next-core/src/next_import_map.rs

@@ -216,32 +216,34 @@ pub async fn get_next_server_import_map(
     let ty = ty.into_value();
 
     insert_next_server_special_aliases(&mut import_map, ty, mode, NextRuntime::NodeJs).await?;
-    let external = ImportMapping::External(None).cell();
+    let external: Vc<ImportMapping> = ImportMapping::External(None).cell();
 
+    import_map.insert_exact_alias("next/dist/server/require-hook", external);
     match ty {
         ServerContextType::Pages { .. } | ServerContextType::PagesData { .. } => {
-            import_map.insert_exact_alias("next", external);
-            import_map.insert_wildcard_alias("next/", external);
             import_map.insert_exact_alias("react", external);
             import_map.insert_wildcard_alias("react/", external);
             import_map.insert_exact_alias("react-dom", external);
             import_map.insert_wildcard_alias("react-dom/", external);
             import_map.insert_exact_alias("styled-jsx", external);
             import_map.insert_wildcard_alias("styled-jsx/", external);
-            import_map.insert_exact_alias("react-server-dom-webpack/", external);
+            import_map.insert_wildcard_alias("react-server-dom-webpack/", external);
+            // TODO: we should not bundle next/dist/build/utils in the pages renderer at all
+            import_map.insert_wildcard_alias("next/dist/build/utils", external);
         }
         ServerContextType::AppSSR { .. }
         | ServerContextType::AppRSC { .. }
         | ServerContextType::AppRoute { .. } => {
             match mode {
-                NextMode::Development | NextMode::Build => {
+                NextMode::Build => {
                     import_map.insert_wildcard_alias("next/dist/server/", external);
                     import_map.insert_wildcard_alias("next/dist/shared/", external);
                 }
                 NextMode::DevServer => {
                     // The sandbox can't be bundled and needs to be external
                     import_map.insert_exact_alias("next/dist/server/web/sandbox", external);
                 }
+                NextMode::Development => {}
             }
             import_map.insert_exact_alias(
                 "next/head",
@@ -377,6 +379,11 @@ async fn insert_next_server_special_aliases(
         NextRuntime::Edge => request_to_import_mapping(context_dir, request),
         NextRuntime::NodeJs => external_request_to_import_mapping(request),
     };
+    let passthrough_external_if_node =
+        move |context_dir: Vc<FileSystemPath>, request: &str| match runtime {
+            NextRuntime::Edge => request_to_import_mapping(context_dir, request),
+            NextRuntime::NodeJs => ImportMapping::External(None).cell(),
+        };
     match (mode, ty) {
         (_, ServerContextType::Pages { pages_dir }) => {
             import_map.insert_exact_alias(
@@ -413,12 +420,7 @@ async fn insert_next_server_special_aliases(
         (_, ServerContextType::PagesData { .. }) => {}
         // In development, we *always* use the bundled version of React, even in
         // SSR, since we're bundling Next.js alongside it.
-        (
-            NextMode::DevServer,
-            ServerContextType::AppSSR { app_dir }
-            | ServerContextType::AppRSC { app_dir, .. }
-            | ServerContextType::AppRoute { app_dir },
-        ) => {
+        (NextMode::DevServer, ServerContextType::AppSSR { app_dir }) => {
             import_map.insert_exact_alias(
                 "@opentelemetry/api",
                 // TODO(WEB-625) this actually need to prefer the local version of
@@ -427,28 +429,40 @@ async fn insert_next_server_special_aliases(
             );
             import_map.insert_exact_alias(
                 "react",
-                request_to_import_mapping(app_dir, "next/dist/compiled/react"),
+                passthrough_external_if_node(app_dir, "next/dist/compiled/react"),
             );
             import_map.insert_wildcard_alias(
                 "react/",
-                request_to_import_mapping(app_dir, "next/dist/compiled/react/*"),
+                passthrough_external_if_node(app_dir, "next/dist/compiled/react/*"),
             );
             import_map.insert_exact_alias(
                 "react-dom",
-                request_to_import_mapping(
+                passthrough_external_if_node(
                     app_dir,
                     "next/dist/compiled/react-dom/server-rendering-stub.js",
                 ),
             );
             import_map.insert_wildcard_alias(
                 "react-dom/",
-                request_to_import_mapping(app_dir, "next/dist/compiled/react-dom/*"),
+                passthrough_external_if_node(app_dir, "next/dist/compiled/react-dom/*"),
+            );
+            import_map.insert_exact_alias(
+                "styled-jsx",
+                passthrough_external_if_node(app_dir, "next/dist/compiled/styled-jsx"),
+            );
+            import_map.insert_wildcard_alias(
+                "styled-jsx/",
+                passthrough_external_if_node(app_dir, "next/dist/compiled/styled-jsx/*"),
             );
             import_map.insert_wildcard_alias(
                 "react-server-dom-webpack/",
-                request_to_import_mapping(app_dir, "next/dist/compiled/react-server-dom-webpack/*"),
+                passthrough_external_if_node(
+                    app_dir,
+                    "next/dist/compiled/react-server-dom-webpack/*",
+                ),
             );
         }
+
         // NOTE(alexkirsz) This logic maps loosely to
         // `next.js/packages/next/src/build/webpack-config.ts`, where:
         //
@@ -460,7 +474,7 @@ async fn insert_next_server_special_aliases(
         // * passes through (react|react-dom|react-server-dom-webpack)/(.*) to
         //   next/dist/compiled/$1/$2
         (
-            NextMode::Build | NextMode::Development,
+            NextMode::Build | NextMode::Development | NextMode::DevServer,
             ServerContextType::AppRSC { app_dir, .. } | ServerContextType::AppRoute { app_dir },
         ) => {
             import_map.insert_exact_alias(
@@ -469,10 +483,20 @@ async fn insert_next_server_special_aliases(
                 // @opentelemetry/api
                 request_to_import_mapping(app_dir, "next/dist/compiled/@opentelemetry/api"),
             );
-            import_map.insert_exact_alias(
-                "react",
-                request_to_import_mapping(app_dir, "next/dist/compiled/react/react.shared-subset"),
-            );
+            if matches!(ty, ServerContextType::AppRSC { .. }) {
+                import_map.insert_exact_alias(
+                    "react",
+                    request_to_import_mapping(
+                        app_dir,
+                        "next/dist/compiled/react/react.shared-subset",
+                    ),
+                );
+            } else {
+                import_map.insert_exact_alias(
+                    "react",
+                    request_to_import_mapping(app_dir, "next/dist/compiled/react"),
+                );
+            }
             import_map.insert_exact_alias(
                 "react-dom",
                 request_to_import_mapping(