Add `next experimental-test` command #64352

Ethan-Arrowood · 2024-04-11T16:12:06Z

This PR adds a new experimental-test command to Next.js CLI.

It has 3 main functions:

auto installing missing playwright dependencies
auto generating missing playwright config
running tests via playwright test

I'm currently working on sharing a public RFC that will have more information. I will link that here when its available.

Closes NEXT-3076 NEXT-3032

ijjk · 2024-04-11T16:18:16Z

Tests Passed

ijjk · 2024-04-11T19:29:41Z

Stats from current PR

Default Build (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
buildDuration	16.7s	15.2s	N/A
buildDurationCached	8s	6.7s	N/A
nodeModulesSize	238 MB	238 MB	⚠️ +58.8 kB
nextStartRea..uration (ms)	398ms	390ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
2453-HASH.js gzip	31.5 kB	31.5 kB	N/A
3304.HASH.js gzip	169 B	169 B	✓
3f784ff6-HASH.js gzip	53.7 kB	53.7 kB	N/A
8299-HASH.js gzip	5.09 kB	5.09 kB	✓
framework-HASH.js gzip	45.2 kB	45.2 kB	✓
main-app-HASH.js gzip	228 B	228 B	✓
main-HASH.js gzip	31.5 kB	31.6 kB	N/A
webpack-HASH.js gzip	1.64 kB	1.65 kB	N/A
Overall change	50.7 kB	50.7 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
_app-HASH.js gzip	193 B	194 B	N/A
_error-HASH.js gzip	193 B	191 B	N/A
amp-HASH.js gzip	510 B	510 B	✓
css-HASH.js gzip	342 B	343 B	N/A
dynamic-HASH.js gzip	2.51 kB	2.52 kB	N/A
edge-ssr-HASH.js gzip	265 B	265 B	✓
head-HASH.js gzip	365 B	364 B	N/A
hooks-HASH.js gzip	389 B	391 B	N/A
image-HASH.js gzip	4.28 kB	4.28 kB	N/A
index-HASH.js gzip	269 B	268 B	N/A
link-HASH.js gzip	2.68 kB	2.69 kB	N/A
routerDirect..HASH.js gzip	328 B	326 B	N/A
script-HASH.js gzip	395 B	397 B	N/A
withRouter-HASH.js gzip	323 B	323 B	✓
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	1.2 kB	1.2 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
_buildManifest.js gzip	482 B	484 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
index.html gzip	529 B	528 B	N/A
link.html gzip	541 B	541 B	✓
withRouter.html gzip	524 B	523 B	N/A
Overall change	541 B	541 B	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
edge-ssr.js gzip	108 kB	108 kB	✓
page.js gzip	3.05 kB	3.04 kB	N/A
Overall change	108 kB	108 kB	✓

Middleware size

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
middleware-b..fest.js gzip	624 B	626 B	N/A
middleware-r..fest.js gzip	155 B	156 B	N/A
middleware.js gzip	27.8 kB	27.8 kB	N/A
edge-runtime..pack.js gzip	839 B	839 B	✓
Overall change	839 B	839 B	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
app-page-exp...dev.js gzip	171 kB	171 kB	✓
app-page-exp..prod.js gzip	97.6 kB	97.6 kB	✓
app-page-tur..prod.js gzip	99.4 kB	99.4 kB	✓
app-page-tur..prod.js gzip	93.6 kB	93.6 kB	✓
app-page.run...dev.js gzip	145 kB	145 kB	✓
app-page.run..prod.js gzip	92.1 kB	92.1 kB	✓
app-route-ex...dev.js gzip	21.5 kB	21.5 kB	✓
app-route-ex..prod.js gzip	15.1 kB	15.1 kB	✓
app-route-tu..prod.js gzip	15.1 kB	15.1 kB	✓
app-route-tu..prod.js gzip	14.9 kB	14.9 kB	✓
app-route.ru...dev.js gzip	21.2 kB	21.2 kB	✓
app-route.ru..prod.js gzip	14.9 kB	14.9 kB	✓
pages-api-tu..prod.js gzip	9.55 kB	9.55 kB	✓
pages-api.ru...dev.js gzip	9.82 kB	9.82 kB	✓
pages-api.ru..prod.js gzip	9.55 kB	9.55 kB	✓
pages-turbo...prod.js gzip	21.4 kB	21.4 kB	✓
pages.runtim...dev.js gzip	22.1 kB	22.1 kB	✓
pages.runtim..prod.js gzip	21.4 kB	21.4 kB	✓
server.runti..prod.js gzip	65.3 kB	65.3 kB	✓
Overall change	960 kB	960 kB	✓

build cache Overall increase ⚠️

	vercel/next.js canary	vercel/next.js next-experimental-test	Change
0.pack gzip	1.61 MB	1.61 MB	⚠️ +627 B
index.pack gzip	107 kB	107 kB	N/A
Overall change	1.61 MB	1.61 MB	⚠️ +627 B

Diff details

Diff for middleware.js

Diff too large to display

Diff for image-HASH.js

@@ -1,7 +1,7 @@
 (self["webpackChunk_N_E"] = self["webpackChunk_N_E"] || []).push([
   [8358],
   {
-    /***/ 1552: /***/ (
+    /***/ 4070: /***/ (
       __unused_webpack_module,
       __unused_webpack_exports,
       __webpack_require__
@@ -9,7 +9,7 @@
       (window.__NEXT_P = window.__NEXT_P || []).push([
         "/image",
         function () {
-          return __webpack_require__(5237);
+          return __webpack_require__(396);
         },
       ]);
       if (false) {
@@ -18,7 +18,7 @@
       /***/
     },
 
-    /***/ 2016: /***/ (module, exports, __webpack_require__) => {
+    /***/ 8490: /***/ (module, exports, __webpack_require__) => {
       "use strict";
       /* __next_internal_client_entry_do_not_use__  cjs */
       Object.defineProperty(exports, "__esModule", {
@@ -40,15 +40,15 @@
         __webpack_require__(422)
       );
       const _head = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(6074)
+        __webpack_require__(2457)
       );
-      const _getimgprops = __webpack_require__(9571);
-      const _imageconfig = __webpack_require__(6567);
-      const _imageconfigcontextsharedruntime = __webpack_require__(419);
-      const _warnonce = __webpack_require__(4486);
-      const _routercontextsharedruntime = __webpack_require__(162);
+      const _getimgprops = __webpack_require__(7932);
+      const _imageconfig = __webpack_require__(5706);
+      const _imageconfigcontextsharedruntime = __webpack_require__(9483);
+      const _warnonce = __webpack_require__(9035);
+      const _routercontextsharedruntime = __webpack_require__(4829);
       const _imageloader = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(6996)
+        __webpack_require__(7240)
       );
       // This is replaced by webpack define plugin
       const configEnv = {
@@ -379,7 +379,7 @@
       /***/
     },
 
-    /***/ 9571: /***/ (
+    /***/ 7932: /***/ (
       __unused_webpack_module,
       exports,
       __webpack_require__
@@ -395,9 +395,9 @@
           return getImgProps;
         },
       });
-      const _warnonce = __webpack_require__(4486);
-      const _imageblursvg = __webpack_require__(133);
-      const _imageconfig = __webpack_require__(6567);
+      const _warnonce = __webpack_require__(9035);
+      const _imageblursvg = __webpack_require__(2642);
+      const _imageconfig = __webpack_require__(5706);
       const VALID_LOADING_VALUES =
         /* unused pure expression or super */ null && [
           "lazy",
@@ -772,7 +772,7 @@
       /***/
     },
 
-    /***/ 133: /***/ (__unused_webpack_module, exports) => {
+    /***/ 2642: /***/ (__unused_webpack_module, exports) => {
       "use strict";
       /**
        * A shared function, used on both client and server, to generate a SVG blur placeholder.
@@ -827,7 +827,7 @@
       /***/
     },
 
-    /***/ 4085: /***/ (
+    /***/ 503: /***/ (
       __unused_webpack_module,
       exports,
       __webpack_require__
@@ -854,10 +854,10 @@
         },
       });
       const _interop_require_default = __webpack_require__(7456);
-      const _getimgprops = __webpack_require__(9571);
-      const _imagecomponent = __webpack_require__(2016);
+      const _getimgprops = __webpack_require__(7932);
+      const _imagecomponent = __webpack_require__(8490);
       const _imageloader = /*#__PURE__*/ _interop_require_default._(
-        __webpack_require__(6996)
+        __webpack_require__(7240)
       );
       function getImageProps(imgProps) {
         const { props } = (0, _getimgprops.getImgProps)(imgProps, {
@@ -889,7 +889,7 @@
       /***/
     },
 
-    /***/ 6996: /***/ (__unused_webpack_module, exports) => {
+    /***/ 7240: /***/ (__unused_webpack_module, exports) => {
       "use strict";
 
       Object.defineProperty(exports, "__esModule", {
@@ -924,7 +924,7 @@
       /***/
     },
 
-    /***/ 5237: /***/ (
+    /***/ 396: /***/ (
       __unused_webpack_module,
       __webpack_exports__,
       __webpack_require__
@@ -941,8 +941,8 @@
 
       // EXTERNAL MODULE: ./node_modules/.pnpm/react@18.2.0/node_modules/react/jsx-runtime.js
       var jsx_runtime = __webpack_require__(1527);
-      // EXTERNAL MODULE: ./node_modules/.pnpm/file+..+main-repo+packages+next+next-packed.tgz_react-dom@18.2.0_react@18.2.0/node_modules/next/image.js
-      var next_image = __webpack_require__(1577);
+      // EXTERNAL MODULE: ./node_modules/.pnpm/file+..+diff-repo+packages+next+next-packed.tgz_react-dom@18.2.0_react@18.2.0/node_modules/next/image.js
+      var next_image = __webpack_require__(73);
       var image_default = /*#__PURE__*/ __webpack_require__.n(next_image); // CONCATENATED MODULE: ./pages/nextjs.png
       /* harmony default export */ const nextjs = {
         src: "/_next/static/media/nextjs.cae0b805.png",
@@ -972,12 +972,8 @@
       /***/
     },
 
-    /***/ 1577: /***/ (
-      module,
-      __unused_webpack_exports,
-      __webpack_require__
-    ) => {
-      module.exports = __webpack_require__(4085);
+    /***/ 73: /***/ (module, __unused_webpack_exports, __webpack_require__) => {
+      module.exports = __webpack_require__(503);
 
       /***/
     },
@@ -987,7 +983,7 @@
     /******/ var __webpack_exec__ = (moduleId) =>
       __webpack_require__((__webpack_require__.s = moduleId));
     /******/ __webpack_require__.O(0, [2888, 9774, 179], () =>
-      __webpack_exec__(1552)
+      __webpack_exec__(4070)
     );
     /******/ var __webpack_exports__ = __webpack_require__.O();
     /******/ _N_E = __webpack_exports__;

Diff for 2453-HASH.js

Diff too large to display

Diff for main-HASH.js

Diff too large to display

Commit: c4a36ce

packages/next/src/cli/next-test.ts

github-actions · 2024-04-16T18:19:39Z

All broken links are now fixed, thank you!

packages/next/src/experimental/testmode/playwright/index.ts

wyattjoh · 2024-04-22T23:54:45Z

packages/next/src/cli/next-test.ts

+  await checkRequiredDeps(baseDir, 'playwright')
+
+  const playwrightConfigFile = await findUp(
+    ['playwright.config.js', 'playwright.config.ts'],


Does playwright also support .mjs or .json files for config?

I don't think so. I can only find references to these two in the docs

wyattjoh · 2024-04-23T16:49:58Z

packages/next/src/cli/next-test.ts

+  if (deps.missing.length > 0) {
+    await installDependencies(baseDir, deps.missing, true)
+
+    const playwright = spawn(


Nit for future, maybe swap with execa, it already does promises!

wyattjoh · 2024-04-23T16:53:43Z

packages/next/src/experimental/testmode/playwright/index.ts

-  return base.defineConfig<T>(config)
+  return base.defineConfig<T>(
+    defaultPlaywrightConfig as base.PlaywrightTestConfig<T>,
+    config


Maybe should be an optional config?

kevva · 2024-04-25T08:46:51Z

packages/next/src/cli/next-test.ts

+    )
+  } else {
+    const playwright = spawn(
+      path.join(baseDir, 'node_modules', '.bin', 'playwright'),


This doesn't work in monorepos where the node_modules directory lives at top level.

kevva · 2024-04-26T09:05:44Z

After this PR I can't start a Next.js server from Playwright anymore. I don't even use the experimental-test command, just next dev (configured in the Playwright config). It works fine in version 14.3.0-canary.21.

Error: http://localhost:3000 is already used, make sure that nothing is running on the port/url or set reuseExistingServer:true in config.webServer.

It works if I set reuseExistingServer but this sounds like a regression nonetheless.

cc @wyattjoh @dvoytenko

Ethan-Arrowood · 2024-04-26T13:37:56Z

Interesting, this seems to be fault of Playwright's defineConfig merge behavior. We now include a default webServer configuration so you don't have to specify it.

For now, unless you're specifying something different than our default in webServer, can you remove it from your config? That should restore functionality.

Fixes: #64352 (comment) Closes NEXT-3238

kevva · 2024-04-29T14:13:26Z

@Ethan-Arrowood, I can confirm that seems to have fixed the issue. Works in v14.3.0-canary.27 and onwards 👍.

wip

852c078

ijjk added created-by: Next.js team PRs by the Next.js team type: next labels Apr 11, 2024

share supported test runner types

c7881df

Ethan-Arrowood and others added 3 commits April 11, 2024 11:25

.

34b22bc

.

2a5934c

chore(cli): uncomment .usage()

e47cdb3

Ethan-Arrowood added 2 commits April 11, 2024 16:33

popsicle sticks and duct tape

5e71cda

Merge branch 'canary' into next-experimental-test

abb1434

eps1lon reviewed Apr 12, 2024

View reviewed changes

packages/next/src/cli/next-test.ts Outdated Show resolved Hide resolved

Ethan-Arrowood added 5 commits April 12, 2024 13:34

fix

cd41ce2

Merge branch 'canary' into next-experimental-test

2601aa5

Merge branch 'canary' into next-experimental-test

bd63c40

Merge branch 'canary' into next-experimental-test

99de698

start on integration tests

424a2ed

ijjk added the area: tests label Apr 15, 2024

Ethan-Arrowood added 2 commits April 15, 2024 18:49

test-runner validation

0ebe076

beep boop

932ea29

move tests to e2e

c44f927

Ethan-Arrowood marked this pull request as ready for review April 16, 2024 19:07

Merge branch 'canary' into next-experimental-test

2a3c8a3

Ethan-Arrowood requested review from timneutkens, ijjk, shuding, samcx, huozhi and feedthejim as code owners April 16, 2024 19:07

Ethan-Arrowood added 4 commits April 22, 2024 16:24

more fixes, resolve type issue

2687eb3

oh typescript

1276382

oh typescript

e947996

Merge branch 'canary' into next-experimental-test

adcc378

dvoytenko approved these changes Apr 22, 2024

View reviewed changes

packages/next/src/experimental/testmode/playwright/index.ts Show resolved Hide resolved

Ethan-Arrowood added 2 commits April 22, 2024 17:33

fix defineConfig one more time

6902027

Merge branch 'canary' into next-experimental-test

c8afd70

wyattjoh reviewed Apr 22, 2024

View reviewed changes

Merge branch 'canary' into next-experimental-test

3b7a17f

Ethan-Arrowood requested review from wyattjoh and ztanner April 23, 2024 02:38

Ethan-Arrowood added 2 commits April 23, 2024 08:46

Merge branch 'canary' into next-experimental-test

310ef22

Merge branch 'canary' into next-experimental-test

1c93145

wyattjoh reviewed Apr 23, 2024

View reviewed changes

update config

1b44741

wyattjoh approved these changes Apr 23, 2024

View reviewed changes

Ethan-Arrowood enabled auto-merge (squash) April 23, 2024 16:57

Ethan-Arrowood added 3 commits April 23, 2024 12:03

Merge branch 'canary' into next-experimental-test

fa4fef1

Merge branch 'canary' into next-experimental-test

1596290

Merge branch 'canary' into next-experimental-test

c4a36ce

wyattjoh disabled auto-merge April 24, 2024 19:16

wyattjoh merged commit 32dcec7 into canary Apr 24, 2024
81 checks passed

wyattjoh deleted the next-experimental-test branch April 24, 2024 19:21

kevva reviewed Apr 25, 2024

View reviewed changes

Ethan-Arrowood mentioned this pull request Apr 26, 2024

Fix playwright config merging for webServer property #65090

Merged

Ethan-Arrowood added a commit that referenced this pull request Apr 26, 2024

Fix playwright config merging for webServer property (#65090)

0382162

Fixes: #64352 (comment) Closes NEXT-3238

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `next experimental-test` command #64352

Add `next experimental-test` command #64352

Ethan-Arrowood commented Apr 11, 2024 •

edited

ijjk commented Apr 11, 2024 •

edited

ijjk commented Apr 11, 2024 •

edited

github-actions bot commented Apr 16, 2024 •

edited

wyattjoh Apr 22, 2024

Ethan-Arrowood Apr 23, 2024

wyattjoh Apr 23, 2024

wyattjoh Apr 23, 2024

kevva Apr 25, 2024 •

edited

kevva commented Apr 26, 2024 •

edited

Ethan-Arrowood commented Apr 26, 2024

kevva commented Apr 29, 2024

Add next experimental-test command #64352

Add next experimental-test command #64352

Conversation

Ethan-Arrowood commented Apr 11, 2024 • edited

ijjk commented Apr 11, 2024 • edited

Tests Passed

ijjk commented Apr 11, 2024 • edited

Stats from current PR

github-actions bot commented Apr 16, 2024 • edited

wyattjoh Apr 22, 2024

Choose a reason for hiding this comment

Ethan-Arrowood Apr 23, 2024

Choose a reason for hiding this comment

wyattjoh Apr 23, 2024

Choose a reason for hiding this comment

wyattjoh Apr 23, 2024

Choose a reason for hiding this comment

kevva Apr 25, 2024 • edited

Choose a reason for hiding this comment

kevva commented Apr 26, 2024 • edited

Ethan-Arrowood commented Apr 26, 2024

kevva commented Apr 29, 2024

Add `next experimental-test` command #64352

Add `next experimental-test` command #64352

Ethan-Arrowood commented Apr 11, 2024 •

edited

ijjk commented Apr 11, 2024 •

edited

ijjk commented Apr 11, 2024 •

edited

github-actions bot commented Apr 16, 2024 •

edited

kevva Apr 25, 2024 •

edited

kevva commented Apr 26, 2024 •

edited