Experimental run-tests.php parallelisation #2822

hikari-no-yume · 2017-10-08T03:44:45Z

Moved to #3838.

====⚡️===========================================================⚡️====
====⚡️==== WELCOME TO THE FUTURE: run-tests PARALLEL EDITION ====⚡️====
====⚡️===========================================================⚡️====

You know it.

Only look at changes after the first commit if you want to understand the diff; the first commit is merely a minor refactoring, but because it moves code around it creates a lot of git diff noise. Alternatively, look at the whitespace-ignoring diff.

You should probably read my email to internals: http://news.php.net/php.internals/100838

hikari-no-yume · 2017-10-08T04:48:00Z

Argh, I notice now this has mixed tabs and spaces. Thanks PHPStorm.

hikari-no-yume · 2017-10-08T04:50:39Z

run-tests.php

+    global $workerID;
+
+    @unlink(__DIR__ . "/../worker$workerID.log");
+    ini_set("error_log", __DIR__ . "/../worker$workerID.log");


I forgot to remove debug code here, perhaps because it never worked in the first place.

I'd like to add an error handler that sends a message with the error.

hikari-no-yume · 2017-10-08T04:52:26Z

run-tests.php

@@ -2655,34 +2955,32 @@ function show_summary()
 }

 function show_redirect_start($tests, $tested, $tested_file)
-{
-	global $html_output, $html_file, $line_length, $SHOW_ONLY_GROUPS;
+    {


kelunik · 2017-10-08T07:53:01Z

run-tests.php

+
+    $workerID = 0;
+    if (getenv("TEST_PHP_WORKER")) {
+        $workerID = intval(getenv("TEST_PHP_WORKER"), 10);


You can just use $workerID = (int) getenv("TEST_PHP_WORKER");.

Oh. Right, yeah. I think I was thinking of JS parseInt() when I wrote this.

kelunik · 2017-10-08T07:56:03Z

run-tests.php

+            if (substr(PHP_OS, 0, 3) == "WIN") {
+                $pass_options .= " -c " . escapeshellarg($conf_passed);
+            } else {
+                $pass_options .= " -c '$conf_passed'";


Why no escapeshellarg here?

I didn't write this code, so I don't know.

kelunik · 2017-10-08T08:00:34Z

run-tests.php

+    echo "Spawning workers… ";
+    for ($i = 1; $i <= $workers; $i++) {
+        $proc = proc_open(
+            $thisPHP . ' ' . escapeshellarg($thisScript),


This should probably set some ini options, such as those, to prevent unexpected messages on STDOUT.

Yes, it would make sense to log to stderr.

That said, wouldn't writing a user error handler that sends errors as messages work? This is my current plan.

hikari-no-yume · 2017-10-08T13:03:33Z

Did some rebase-fu to fix the accidental use of spaces rather than tabs. At least, all the usages that were my fault ;)

kelunik · 2017-10-08T18:13:47Z

Could you try enabling it on Travis and see whether it can speed up the tests there, too?

hikari-no-yume · 2017-10-08T18:15:23Z

@kelunik Very sensible idea. I'll add it maybe as an extra task or whatever so I can see what's broken by the parallelism.

hikari-no-yume · 2017-10-08T22:06:27Z

@kelunik I'm happy to report that, judging by build #27751, it does indeed make Travis faster, at 530s to run tests with 2 worker children versus 908s in normal mode. Double the number of worker children to four and they can do it in 420s, which seems like a case of diminishing returns (I think Travis maybe only provides 2 logical CPUs, if that).

There's some weird issues in Travis, though, which tripped up my code's sanity checks. Apparently the workers are sending back more test results than the test files they were given to execute. I'm not sure why that is, maybe there's some run-tests.php feature I failed to account for, or maybe somehow messages are getting repeated (how would that happen, though?). The test summary totals look the same (aside from extra failures) as non-parallel builds' though, so probably it's the former and I shouldn't be worried. Nikita says it might be REDIR tests so I'll look into that. (Spoiler alert: I didn't know what REDIR tests do. If you run a REDIR test, it goes and runs more tests. Thus, the “total” number of tests to run is actually only a subset once you count the tests run by REDIR tests.)

hikari-no-yume · 2017-10-08T22:30:16Z

run-tests.php

+		$toRead = array_values($workerStdouts);
+		$toWrite = NULL;
+		$toExcept = NULL;
+		if (stream_select($toRead, $toWrite, $toExcept, 0, 50 * 1000)) {


@kelunik I'd guess you know what you're doing here better than I do, so: Would you see any disadvantage to having a higher timeout here, say 1 second? Checking the reads on these is supposedly non-blocking, and there's nothing else the host process needs to do except for wait on reads, so reducing the number of syscalls very slightly would make sense, right?

Yes, you can totally increase that to 10 seconds or even more. It will be interrupted by any signals or actionable streams.

hikari-no-yume · 2017-10-09T00:23:40Z

There's now a flag file (you just touch it, in traditional UNIX fashion) you can put in a test directory, !CAN_BE_PARALLELISED, which tells run-tests.php that there's nothing wrong with executing different files in that directory concurrently, and it will divide that directory into chunks which can be dispatched to different workers.

I've added that file to a bunch of large directories that seemed safe, and it hasn't broken anything. Unfortunately, it also hasn't improved effective test execution time for me, even on my 16-thread machine; no matter how fast you execute everything else, ext/standard/tests/file/ will drag things out. So the next step will be to break that directory up a bit by hand, if possible, and have a look at the other directories I haven't flagged for now.

jhdxr · 2017-10-09T14:36:54Z

run-tests.php

+	$workerStdouts = [];
+	$workerStderrs = [];
+
+	echo "====⚡️===========================================================⚡️====\n";


I don't think the emoji here is a good idea. it will cause problems in Windows/CMD.

I wouldn't recommend proc_open on Windows anyway, because STDIO are file handles there and block.

Quick testing on Windows 10 shows that in cmd, the emoji don't cause problems, they just show up as empty rectangles. That's fine, really. Maybe it can serve as an incentive to support Unicode terminal output.

And @kelunik is right about proc_open blocking… it seemingly ignores all all the workers but one because of it. :/

empty rectangles IS a problem, isn't it? If you insist on these emoji, I suggest you add something like --no-ansi, and turn it on on windows.

It's just cosmetic, so why bother? And actually, it's just a font support problem. Copy and paste them into another window and it shows up fine. I guess PHP does use Unicode after all.

I began modifying the code so it can also communicate over sockets, but this is a little more painful than I expected. If I get that working, it should solve the Windows problem.

…oh, I was using the wrong socket API >.>

staabm · 2017-10-10T05:48:02Z

@hikari-no-yume IIRC @sgolemon has a list of all tests which can ran in parallel (which was created for hhvm in the past)

hikari-no-yume · 2017-10-11T19:39:06Z

Since, as @kelunik pointed out, Windows's proc_open pipes don't support non-blocking I/O, I added a slightly different code-path that uses IPv6 loopback TCP sockets instead in that case. Now parallelised run-tests works on Windows – and quite well :)

KalleZ · 2017-10-17T13:31:09Z

@hikari-no-yume I suggest we perhaps do a cleanup of tests in Zend/ and put them in sub directories and then merge this but that can even be done after, so lets have a go at this in master.

Personally the only negative I have about this, as I stated on Twitter, is the annoying square boxes on my CLI

jhdxr · 2017-10-17T17:43:29Z

Again, I suggest to remove, or at least add an option for the emoji. I fully agree they are very cute and I like these cute things. But they are not cute any more when they become squares. The worse thing is for people without related knowledge, they may treat this as a problem of their cmd / php tests, because the abnormal character usually means something wrong.

jhdxr · 2017-10-17T17:46:14Z

Anyway, the parallel testing itself is awosome. Let RM make the decision.

staabm · 2017-10-17T18:10:54Z

run-tests.php

+			$workerInput = $workerSock;
+			$workerOutput = $workerSock;
+		} else {
+			$workerInput = $pipes[0];


Would it make sense to use sockets on all platforms to reduce code?

Maybe. Not using sockets means a simpler codepath less likely to face problems, though.

jhdxr · 2017-10-17T19:07:54Z

run-tests.php

+
+	if ($useSockets) {
+		// IPv6 because nobody uses it, so less chance of collisions, right? ;)
+		$port = (ord('<') << 8) + ord('?');


No need for a magic constant port. use stream_socket_server("tcp://[::1]:0"), and an unused port will be selected.

$a=stream_socket_server("tcp://[::1]:0"); var_dump(stream_socket_get_name($a, false)); //php shell code:1: //string(9) "::1:52491"

and then we can use ipv4. or at least use it as a fallback.

Yes, I thought about using :0, I'll try implementing it.

I don't see any value in adding IPv4. Who needs to test PHP 7.3 in parallel on a system with no support for IPv6? It's not like IPv6 connectivity is required, just loopback.

@hikari-no-yume There's 0 reason to use IPv6. Binding to IPv4 will usually bind both anyway, see http://php.net/manual/en/context.socket.php + ipv6_v6only.

You know there is almost no IPv6 in China, so sometimes the system administrator will disable all the IPv6 components because they believe it can speed up the system. It's silly but it happens.

@hikari-no-yume side note, IPv6 support is ALWAYS available on Windows, but can be disabled in custom builds for some odd reason I do not understand

hikari-no-yume · 2017-10-17T19:18:31Z

@KalleZ Zend's fine, it doesn't need breaking up. I mean, it would be nice to, but this thing already subdivides it automatically with few (no? I don't remember) problems.

KalleZ · 2017-10-17T19:58:27Z

@hikari-no-yume lets give it a try in master, then we can all hack on it together, there is nothing else holding it back

@jhdxr though we keep run-tests.php in sync across all branches, it should be perfectly fine to play around with in master, I'm sure the RMs for lower branches would merge it once it had some time there

hikari-no-yume · 2017-10-17T19:59:07Z

@KalleZ we do keep it in sync? Oh, okay. There's changes that aren't synced right now, and this code will only work on 7.2 as-is…

…trix

hikari-no-yume · 2018-07-05T21:22:32Z

Rebased for current master.

hikari-no-yume · 2018-07-05T21:55:20Z

The CI builds fail because it seems to choke on very large test output… fgets is only fetching 8192 bytes. A suspicious number. I'll raise the line length limit and see if that fixes it.

hikari-no-yume · 2018-07-05T22:19:59Z

run-tests.php

+					error("Could not find worker stdout in array of worker stdouts, THIS SHOULD NOT HAPPEN.");
+				}
+				// Some tests have *very* long output and fgets() seems to truncate to 8KiB unless we say we want more
+				while (FALSE !== ($rawMessage = fgets($workerSock, 64 * 1024))) {


This doesn't work… can fgets not handle non-blocking streams or something? :/

hikari-no-yume · 2018-07-05T23:00:06Z

run-tests.php

-				while (FALSE !== ($rawMessage = fgets($workerSock, 64 * 1024))) {
+				$rawMessageBuffer = '';
+				while (FALSE !== ($rawMessage = fgets($workerSock))) {
+					// work around fgets truncating things


This also apparently doesn't work. I bet there's a non-blocking I/O error in here somewhere.

hikari-no-yume · 2018-07-05T23:29:48Z

Woo, it doesn't choke on huge test output now.

hikari-no-yume · 2018-07-05T23:33:08Z

And perhaps more importantly, very few tests fail with parallelised tests enabled under Travis. However I/O capture tests are skipped, so we couldn't safely use only parallel testing right now…

nikic · 2019-02-12T11:50:18Z

From what I remember, this PR implements parallelization on the directory level and adds marker files to exclude directories from parallelization.

I wanted to propose an alternative approach for handling tests that can't be run together, which I think should be both relatively simple to implement and allows a smooth transition from coarse-grained (to start with) to fine-grained (once we have time to review individual tests) handling:

Add a new --CONFLICTS-- section to the phpt format, which contains a list of strings (or possibly just one string to start with). All tests which have a non-empty intersection in CONFLICTS cannot be run together (but may still be run together with other tests). For example, ext/mysqli and ext/pdo_mysql would use --CONFLICTS-- \n mysql, while ext/pgsql and ext/pdo_pgsql would use --CONFLICTS -- \n pgsql. This prevents MySQL tests from running in parallel, but they can still run in parallel with PgSQL tests.

Additionally a CONFLICTS file can be added to a whole directory, so this doesn't need to be specified per-file.

carusogabriel · 2019-02-12T11:53:38Z

@nikic Might be something, so we can delivery this :)

@hikari-no-yume

Prerequisite for parallelised testing: #2822 Quoth Nikita @ #3789 (comment): > @hikari-no-yume Please feel free to directly commit the first commit > (to PHP-7.4). That's probably a big and unnecessary source of > conflicts, and makes reviewing harder as well.

hikari-no-yume · 2019-02-16T00:12:39Z

As implied by 41fbeb6 I'm working on this again (thanks @KalleZ and @nikic for the reminder ^^). Rebasing it is gnarly though and I don't want to trash history for this branch again; I'm going to hand-rebase this on a new branch and then open a new PR once I'm done and close this one. Leaving this one open for the moment.

hikari-no-yume · 2019-02-16T01:40:02Z

Superseding with #3838.

Motivation: As an extension author, I want to speed up running tests in php <=7.3, both locally and in CI (e.g. with valgrind). This can be done by manually copying php 7.4's run-tests.php script to replace the one generated by `phpize` - list() doesn't work in php 7.0 - negative string offset doesn't work in php 7.2 If run-tests.php can be copied from php-src without any manual patches, that would be the easiest. Related to php#2822 - I didn't see any discussion for/against compatibility with older php versions

Motivation: As an extension author, I want to speed up running tests in php <=7.3, both locally and in CI (e.g. with valgrind). This can be done by manually copying php 7.4's run-tests.php script to replace the one generated by `phpize` - list() doesn't work in php 7.0 - negative string offset doesn't work in php 7.2 If run-tests.php can be copied from php-src without any manual patches, that would be the easiest. Related to #2822 - I didn't see any discussion for/against compatibility with older php versions

hikari-no-yume force-pushed the parallelised-run-tests branch from 3807427 to 0d1e4b5 Compare October 8, 2017 03:53

hikari-no-yume commented Oct 8, 2017

View reviewed changes

kelunik reviewed Oct 8, 2017

View reviewed changes

hikari-no-yume force-pushed the parallelised-run-tests branch from 0d1e4b5 to 91d19cb Compare October 8, 2017 13:00

hikari-no-yume force-pushed the parallelised-run-tests branch 2 times, most recently from b8be2ec to d713b17 Compare October 8, 2017 21:33

hikari-no-yume force-pushed the parallelised-run-tests branch from d713b17 to 9a825e3 Compare October 8, 2017 22:14

hikari-no-yume commented Oct 8, 2017

View reviewed changes

hikari-no-yume force-pushed the parallelised-run-tests branch from ef1fbb0 to 1b71858 Compare October 8, 2017 23:57

jhdxr reviewed Oct 9, 2017

View reviewed changes

hikari-no-yume force-pushed the parallelised-run-tests branch from 38b5452 to 9ba3362 Compare October 11, 2017 19:36

staabm reviewed Oct 17, 2017

View reviewed changes

jhdxr reviewed Oct 17, 2017

View reviewed changes

hikari-no-yume added 5 commits July 5, 2018 23:16

Use sockets for comms on all platforms

d06e513

Add experimental two- and four-worker parallel testing to AppVeyor ma…

c0e67d6

…trix

Swap -j4 Travis tests for non-ZTS non-debug -j2

dea5ecf

Swap -j4 AppVeyor tests for non-ZTS non-debug -j2

cabbfaa

Skip I/O capture tests in parallel mode

69b648a

hikari-no-yume force-pushed the parallelised-run-tests branch from 871febd to 69b648a Compare July 5, 2018 21:21

Don't choke on huge messages(?)

b3289c1

hikari-no-yume commented Jul 5, 2018

View reviewed changes

Don't choke on huge messages, approach 2

a15c666

hikari-no-yume force-pushed the parallelised-run-tests branch from 48d2dc6 to a15c666 Compare July 5, 2018 23:13

blank commit to force rebuild

ca17490

KalleZ mentioned this pull request Feb 12, 2019

[EXPERIMENTAL] Skip slow tests on Appveyor #3789

Closed

nikic changed the base branch from master to PHP-7.4 February 12, 2019 11:34

hikari-no-yume mentioned this pull request Feb 16, 2019

Experimental run-tests.php parallelisation (2019 rebase) #3838

Closed

hikari-no-yume closed this Feb 16, 2019

nikic mentioned this pull request Feb 18, 2019

Parallel test improvements #3851

Closed

TysonAndre mentioned this pull request Mar 23, 2019

Make run-tests.php compatible with php 7.0-7.2 #3981

Closed

TysonAndre mentioned this pull request Jul 14, 2019

Check for Windows utf-8 support in run-tests.php #4404

Closed

Experimental run-tests.php parallelisation #2822

Experimental run-tests.php parallelisation #2822

Uh oh!

Conversation

hikari-no-yume commented Oct 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Moved to #3838.

Uh oh!

hikari-no-yume commented Oct 8, 2017

Uh oh!

hikari-no-yume Oct 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kelunik Oct 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hikari-no-yume commented Oct 8, 2017

Uh oh!

kelunik commented Oct 8, 2017

Uh oh!

hikari-no-yume commented Oct 8, 2017

Uh oh!

hikari-no-yume commented Oct 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hikari-no-yume commented Oct 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hikari-no-yume Oct 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

staabm commented Oct 10, 2017

Uh oh!

hikari-no-yume commented Oct 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KalleZ commented Oct 17, 2017

Uh oh!

jhdxr commented Oct 17, 2017

Uh oh!

jhdxr commented Oct 17, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hikari-no-yume Oct 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

hikari-no-yume commented Oct 8, 2017 •

edited

Loading

hikari-no-yume Oct 8, 2017 •

edited

Loading

kelunik Oct 8, 2017 •

edited

Loading

hikari-no-yume commented Oct 8, 2017 •

edited

Loading

hikari-no-yume commented Oct 9, 2017 •

edited

Loading

hikari-no-yume Oct 9, 2017 •

edited

Loading

hikari-no-yume commented Oct 11, 2017 •

edited

Loading

hikari-no-yume Oct 17, 2017 •

edited

Loading