Move loading of test files to child process...

[bobjackman]

For context, please refer to my Stack Overflow question here: http://stackoverflow.com/questions/10077400/phpunit-not-continuing-with-tests-after-fatal-error-when-using-process-isolati

I am running into a problem where a test is extending a misnamed class. Since that test is not found, PHP throws a fatal error and PHPUnit dies (even when using --process-isolation). Now, fixing this misnamed class in the code is trivial to do, but in the event that a developer commits test code like this, it would be really nice for it not to kill the entire testing process.

A comment on that SO question states that PHPUnit loads all test files before running tests (which causes their top-level code to be executed). When using the --process-isolation flag, currently, this is done before any child processes are spawned, so the isolation doesn't help in this case.

Would it be possible to have PHPUnit (if using --process-isolation) do the loading of each test file in the child process rather than in the parent? The parent could find all files to use, then spawn child processes to both load the file (parse/exec top-level code) and run the tests. This would allow all tests to run even if there are fatal errors within some of them.

[edorian]

What PHPUnit does to scan a directory for test cases is to require every file that ends in the specified suffix (usually Test.php) and check if the file contains a class that extends PHPUnit_Framework_TestCase. If that is the case the file is added to the list of "testCases to execute".

After that it creates a new instance of the class for each test method in the Class and executes that test.

The numbers are gather to show the progress bar / status indicator and other things like filtering on @group tags.

If we would NOT INCLUDE the file we'd have to extract all of that information using some sort of static reflection which is slow, painful and error prone (try getting the inheritance hierarchy of something with only static analysis when you don't know where the source files are).

So I don't see any way for use to implement that.

Like my answer you linked showed it already works for stuff inside the test cases but to be honest I don't see how this could be done with only somewhat reasonable effort - Even after the planned test runner refactorings.

[bobjackman]

What if (only when using process isolation), you spawn a child process and perform an include. If it fails, then move on. If successful, then the parent can include it.

[bobjackman]

I guess that's a no?

[edorian]

Forking a child, making the include, collecting the method names and data provider data, communicating this back to $parent and then having the parent to the rest would be a possibility.

It would allow to get around parsing errors (not really worth it) and global state in the file that gets in includes (that changes every time the file gets included).

Given that process isolation is a way to test "strange/old" applications there would maybe be some value in doing it like this.

But it seems to be quite a lot of work to get right (as it requires test runner restructuring) just to be able to get the few benefits.

It's not a no but it's a "I'd rather not do it, if you can get it done nice and clean I don't mind having it"