PoC: Run E2E tests with WP Playground #62692

WunderBart · 2024-06-20T10:38:57Z

What?

A simple PoC for running E2E tests against a Playground instance (circumventing wp-env entirely).

How?

Checkout this branch.

Playground part

Clone WP Playground and install deps:

git clone -b trunk --single-branch --depth 1 git@github.com:WordPress/wordpress-playground.git
cd wordpress-playground
npm install

Run the following CLI command to start the Playground server:

GUTENBERG_PATH=../path/to/your/gutenberg/repo

bun packages/playground/cli/src/cli.ts server \
    --wp=6.5 \
    --mount=$GUTENBERG_PATH:/wordpress/wp-content/plugins/gutenberg \
    --mount=$GUTENBERG_PATH/test/gutenberg-test-themes:/wordpress/wp-content/themes/gutenberg-test-themes \
    --mount=$GUTENBERG_PATH/packages/e2e-tests/mu-plugins:/wordpress/wp-content/mu-plugins \
    --mount=$GUTENBERG_PATH/packages/e2e-tests/plugins:/wordpress/wp-content/plugins/gutenberg-test-plugins \
    --blueprint=$GUTENBERG_PATH/test/e2e/playground.blueprint.json

Note the server URL. By default it's http://127.0.0.1:9400.

Gutenberg part

Run npm run build to build the local Gutenberg.
Assign the Playground server via WP_BASE_URL and run the tests as usual. For example:
```
WP_BASE_URL=http://127.0.0.1:9400/ npm run test:e2e
```
No wp-env, no Docker. Profit?

This reverts commit 582c72c.

This reverts commit 12a8ac7.

WunderBart · 2024-06-21T15:36:37Z

@adamziel, many tests are failing because the database randomly disconnects, causing the app to throw 500 errors. I have yet to find a consistent way to reproduce the issue, but I'll continue working on it when I'm back (around July 2nd). Once this issue is resolved, the legit failures will provide valuable information on Playground's current limitations compared to wp-env.

It's been nice to see how easy it is to circumvent wp-env. Spinning up the WP Playground instance is really fast, too. 💪

cc: @griffbrad @dmsnell

swissspidy · 2024-07-02T08:14:05Z

FWIW for https://github.com/swissspidy/media-experiments I always use wp-now for running tests locally (as I can't use Docker) and I constantly run into the same issue with 500 errors. Curious to hear what you find out.

dmsnell · 2024-07-02T16:06:22Z

.github/workflows/end2end-test.yml

+
+                  # Install the dependencies
+                  cd playground
+                  npm install


I know this keeps the Playground updated, but the build artifact from the Playground should be an executable you can download and run directly. That would surely skip a lot of runtime and build steps.

Would that simply be https://www.npmjs.com/package/@wp-playground/cli?

dmsnell · 2024-07-02T16:06:51Z

.github/workflows/end2end-test.yml

-                  npm run wp-env start
+                  cd playground
+                  ~/.bun/bin/bun packages/playground/cli/src/cli.ts server \
+                      --wp=6.5 \


would it be preferable to use nightly here?

dmsnell · 2024-07-02T16:08:18Z

@WunderBart @swissspidy are you using a MySQL connection or relying on SQLite when you get the 500s? How do we know it's a database disconnect? I didn't think that is possible with SQLite, and I wonder also if we're seeing concurrency issues with multiple requests? Mostly the Playground handles requests serially, but I think it should be fine with concurrent requests as well.

swissspidy · 2024-07-02T18:37:35Z

Hmm I don't think it's a database issue in my case. Also no idea how to make it use MySQL, so I think it's just the default SQLite setup.

Just did another test and it's actually more a timeout than a proper error 500.

The server stops responding and in the terminal I get errors like this:

Error: PHP.run() failed with exit code 255 and the following output: 
    at _NodePHP.run (/Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@php-wasm/node/index.cjs:72937:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async #dispatchToPHP (/Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@php-wasm/node/index.cjs:72470:14)
    at async PHPRequestHandler.request (/Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@php-wasm/node/index.cjs:72385:14)
    at async PHPBrowser.request (/Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@php-wasm/node/index.cjs:72170:22)
    at async file:///Users/pascalb/Workspace/WordPress/media-experiments/node_modules/@wp-now/wp-now/main.js:945:20 {
  output: {
    stdout: '<!DOCTYPE html>\n' +
      '<html lang="en-US">\n' +
      '<head>\n' +
      '\t<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />\n' +
      '\t<meta name="viewport" content="width=device-width">\n' +
      '\t\t',
    stderr: ''
  },
  source: 'request'
}

But I don't wanna hijack the discussion here :-)

github-actions · 2024-07-11T14:33:31Z

Warning: Type of PR label mismatch

To merge this PR, it requires exactly 1 label indicating the type of PR. Other labels are optional and not being checked here.

Type-related labels to choose from: [Type] Automated Testing, [Type] Breaking Change, [Type] Bug, [Type] Build Tooling, [Type] Code Quality, [Type] Copy, [Type] Developer Documentation, [Type] Enhancement, [Type] Experimental, [Type] Feature, [Type] New API, [Type] Task, [Type] Technical Prototype, [Type] Performance, [Type] Project Management, [Type] Regression, [Type] Security, [Type] WP Core Ticket, Backport from WordPress Core.
Labels found: Needs Author's Reply.

Read more about Type labels in Gutenberg. Don't worry if you don't have the required permissions to add labels; the PR reviewer should be able to help with the task.

WunderBart added 2 commits June 20, 2024 12:15

modify config to run against Playground

0399391

upload e2e blueprint

e226d20

WunderBart force-pushed the try/run-e2e-tests-against-playground branch 2 times, most recently from 26f785c to f60c136 Compare June 20, 2024 13:38

WunderBart added 2 commits June 20, 2024 15:56

try running e2e tests w/ Playground in CI

467a732

make the test report verbose

5416734

WunderBart force-pushed the try/run-e2e-tests-against-playground branch from f60c136 to 5416734 Compare June 20, 2024 13:59

WunderBart changed the title ~~PoC: Run E2E tests against WP Playground~~ PoC: Run E2E tests with WP Playground Jun 20, 2024

WunderBart added 7 commits June 20, 2024 16:27

run single suite for testing

12a8ac7

try checking out PG repo aside

582c72c

Revert "try checking out PG repo aside"

2991126

This reverts commit 582c72c.

restore global setup for pre-auth

187acdf

add missing dir change

4e2de6d

Revert "run single suite for testing"

a54920c

This reverts commit 12a8ac7.

disable retries to finish faster avoid flaky reports

a34384e

dmsnell reviewed Jul 2, 2024

View reviewed changes

adamziel assigned WunderBart Jul 3, 2024

adamziel mentioned this pull request Jul 10, 2024

Intermittent SQLite Database Disconnects WordPress/wordpress-playground#1596

Open

adamziel added the Needs Author's Reply label Jul 11, 2024

adamziel removed the Needs Author's Reply label Jul 11, 2024

adamziel mentioned this pull request Jul 16, 2024

Performance testing next steps (adoption) WordPress/performance#1093

Open

adamziel added the Needs Author's Reply label Jul 16, 2024

swissspidy mentioned this pull request Jul 19, 2024

Blueprint support swissspidy/wp-performance-action#71

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PoC: Run E2E tests with WP Playground #62692

PoC: Run E2E tests with WP Playground #62692

WunderBart commented Jun 20, 2024 •

edited

Loading

WunderBart commented Jun 21, 2024

swissspidy commented Jul 2, 2024

dmsnell Jul 2, 2024

swissspidy Jul 19, 2024

dmsnell Jul 2, 2024

dmsnell commented Jul 2, 2024

swissspidy commented Jul 2, 2024

github-actions bot commented Jul 11, 2024 •

edited

Loading

PoC: Run E2E tests with WP Playground #62692

Are you sure you want to change the base?

PoC: Run E2E tests with WP Playground #62692

Conversation

WunderBart commented Jun 20, 2024 • edited Loading

What?

How?

Playground part

Gutenberg part

WunderBart commented Jun 21, 2024

swissspidy commented Jul 2, 2024

dmsnell Jul 2, 2024

Choose a reason for hiding this comment

swissspidy Jul 19, 2024

Choose a reason for hiding this comment

dmsnell Jul 2, 2024

Choose a reason for hiding this comment

dmsnell commented Jul 2, 2024

swissspidy commented Jul 2, 2024

github-actions bot commented Jul 11, 2024 • edited Loading

WunderBart commented Jun 20, 2024 •

edited

Loading

github-actions bot commented Jul 11, 2024 •

edited

Loading