Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
New `run-tests.ts`, making test runs more reliable #1401
The main goal of this PR is to improve reliability of our test runs –
Full TODO list:
Noted for followup PRs:
c1a1fbb deals with another problem that @pavelevap frequently encounters – tests run before the Docker containers are fully started (WP + MySQL take around 20 seconds to start on my machine, can be longer depending on a specific device).
The commit shows an approach where a waiting container is added to
Running the commit as it is with
But when I comment out the waiting line in
So that proves that the waiting container works.
Note that on my machine, to make the test fail, it was necessary to start the MySQL from an entirely clean state – with the volume deleted, otherwise, it was too quick to start to demonstrate the issue. That's why the
referenced this pull request
Mar 17, 2019
The recent commits implemented
The script has this "API" for now:
BTW I'm pretty happy with the test reliability now:
I'm getting consistent results between several machines now.
MacBook Pro with Docker Desktop
MacBook Pro (2015)
Tests took 11 minutes.
Windows laptop with Docker Desktop
Dell XPS 13 9350 (2015)
Tests took 18 minutes. (The CLI environment is using this MSYS2 setup.)
Windows laptop with Docker Toolbox
HP ProBook 650 G1 (2014)
Tests took 37 minutes.
I tried with both the default VM configuration (1GB RAM, 1 CPU) and an upgraded VM with 4GB RAM and 2 CPU but it didn't have any effect, the tests still took about 37 minutes. It's probably limited by the virtualization technology; the tests cannot utilize the more resources:
On all machines, the console output was this:
Note: the above is omits the failing test with many empty lines, see #1400 (comment).