Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Dockerized testing infrastructure #1389

borekb opened this issue Mar 7, 2019 · 3 comments


1 participant
Copy link

commented Mar 7, 2019

Full tests (npm run tests:full) currently take over an hour to run on a relatively powerful machine (Core i7 MacBook Pro) and there are some other issues related to tests as we reviewed them with @JanVoracek today.

UPDATE: In the end, this issue is a major update of the dockerized dev / testing environment last updated in #1329.


  • Full tests now run much faster thanks to volumes instead of host-mounted folders. On my machine, the time dropped from 1 hour 15 minutes to just 11 minutes. #1396
  • Permissions between containers (Debian vs. Alpine) have been fixed. #1397
  • New run-tests.ts script that orchestrates how tests run. For example, it decides when to start a WordPress stack (if at all), cleans up after itself, etc. #1401
  • The main testing scripts is now just npm run tests (the :full suffix has been dropped).
    • Also, all the tests:custom scripts have been dropped as run-tests.ts now accepts parameters, for example:
      npm run tests -- --testsuite Unit --filter CursorTest
  • Docker images are now fixed using their digests. No more surprises if upstream images change. #1398
  • Some effort has been put into testing and documenting how test work on Docker Toolbox. #1403

During the work, I've also encountered several other things that are noted in the "Deferred items" section below.

TODO list

  • Split testing documentation from Dev Setup, it's quite long on its own. #1392
    • ... and smaller related content changes. #1393
  • Split docker-compose services into two files, docker-compose.yml and docker-compose.tests.yml. #1395
  • ⭐ Permissions: Run CLI image as UID 33 (Debian's "www-data") #1397
  • ⭐ Volumes: use named volumes (i.e., don't map to host OS) & have some way to inspect the data later. #1396
  • Use Docker image digests instead of tags. #1398
    • They are uglier, e.g., busybox:latest becomes busybox@sha256:38a203e1986cf79639cfb9b2e1d6e773de84002feea2d4eb006b52004ee8502d, but it's the only way to make runs predictable (docker-compose lacks a concept of a lock file).
    • This has already been considered in #1329 but we eventually didn't go for it.
    • docker-compose config --resolve-image-digests can help with that.
    • Awesome post about the problem and solutions: Overcoming Docker’s mutable image tags
      • (BTW, it's by Renovate, a tool that we should consider in the future anyway to keep dependencies up to date.)
  • Stop containers after the tests finish running. #1399 (package.json impl), then #1401 (run-tests.ts impl)
  • ⭐ Introduce run-tests.ts and make test runs more reliable. #1400 (PR #1401)
  • Verify how things work in Docker Toolbox. #1403
  • Possibly remove "test:custom" npm scripts, just document how to do it. Done as part of #1401.
  • Explore using Samba to speed up files inspection, see #1389 (comment). Explored but not successfully, see #1411.
  • Document how to stop tests on Mac – Ctrl+C doesn't work, I have to open another terminal, type docker ps, copy the container ID and do docker stop <id>. Resolved as part of #1401 – Ctrl+C now kills the Node.js script.
  • ⭐ Make npm run tests pass. #1400

Deferred items

These should be done separately in future PRs.

  • Rename images to something like versionpress/wordpress and versionpress/cli, i.e., make "CLI" part of the image name so that it's clear when digests are used.
  • Only enable Xdebug for ...:debug scripts.
    • Right now, it's always enabled and just not configured with the right idekey in non-debug scripts which makes things slower. Here and here.
    • Update Xdebug to 2.7.0. (Let's still stay on PHP 7.2 for tests for now.)
  • Make PHP 7.2 the minimum required version? (We don't have an issue for it yet, will create it.)
  • The WP-CLI installation into the WP image looks a bit weird, review it.
  • Try updating PHPUnit (I remember it wasn't so easy due to supported PHP versions or something but it's worth looking again).
  • Don't base our CLI image on wordpress:cli, it does a lot more than we need.
  • Explore re-running failed tests: sebastianbergmann/phpunit#46
  • Close #1284 (Fix remaining tests under Docker, post-beta) and port those to a new issue
  • Review backlog from the "Noted for future work" section in #1329
  • Another big one is #1258 (WP-CLI / Selenium worker switching). That issue is a bit old but overall, we should still be able to run all tests with both workers, or pick a worker with a flag like npm run tests --worker=selenium. Things like a worker or a specific test site to use shouldn't be hardcoded in test-config.yml.

This comment has been minimized.

Copy link
Member Author

commented Mar 10, 2019

Good progress this weekend:


@pavelevap, could you please try how tests currently work on Windows? For me, npm run tests:full (after npm run tests:cleanup!) take about 10 minutes to run and produce only 1 failure in CloneMergeTest::cloneLooksExactlySameAsOriginal ("Tests: 738, Assertions: 950, Failures: 1, Skipped: 16").

If we could fix that issue, that would be a big win.

@borekb borekb added this to In Progress in 4.0-beta → 4.0-beta2 Mar 11, 2019


This comment has been minimized.

Copy link
Member Author

commented Mar 11, 2019

Named volumes implemented in #1396 made things much faster but also harder to inspect files locally. The current solution depends on copying files from a named volumed to a folder that will be bind-mount to host but this is quite slow (several minutes of heavy I/O on my Mac).

This SO answer suggests using Samba, we could explore that as an alternative.

UPDATE: I tried to add Samba in #1411 but didn't manage to.


This comment has been minimized.

Copy link
Member Author

commented Mar 26, 2019

The key items are done and the tests now fully pass 🎉.

I've updated the original description to contain all the relevant details.

@borekb borekb closed this Mar 26, 2019

4.0-beta → 4.0-beta2 automation moved this from In Progress to Done Mar 26, 2019

@borekb borekb referenced this issue Mar 26, 2019


Testing infrastructure – backlog #1412

0 of 10 tasks complete

@borekb borekb referenced this issue Mar 26, 2019


Travis CI #1259

3 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.