Allow multiple runs

[Slamdunk]

Q	A
PHPUnit version	10.0.x-dev
PHP version	8.1.5
Installation Method	Composer

Summary

Hi, I'm the maintainer of ParaTest, a tool to run PHPUnit tests in parallel.

One way ParaTest can run is to spawn $(nproc) processes and run multiple tests within a single process.
Multiple tests are called each with a separate \PHPUnit\TextUI\Application::run call withing the same process.
This is a design choice that allow both:

1. Single bootstrap operation for all the tests within the same process
2. Dynamic (so non-preemptive) split of tests across $(nproc) processes

To maximise the overall speed.

This design collides with the new Facade singleton in c853041, of which seal is never released after the run ends.

So I'd like to ask:

1. either to reset the Singleton after the run
2. or (better) remove the Singleton design and leverage a proper dependency injection

[theseer]

While I agree the design with the Facade and how it is currently implemented leaves room for improvements, I so far fail to see how either of your proposed changes would address the issue - at least as how I currently understand it:

1. either to reset the Singleton after the run

As I understand your explanation above, you have multiple concurrent runs in parallel within the same application process - how ever that is achieved. How would resetting the state of the Facade help you after run? Each run would have to wait for the previous to complete - killing the very idea of paratest? What am I missing here?

2. or (better) remove the Singleton design and leverage a proper dependency injection

Nitpicking alert: Technically, there is no Singleton. The Facade keeps track of the instance it created, which is not the same as a Singleton but of course the point of the Facade. Using DI would not change that.

I'm afraid as of now I'm also not understanding (yet?) why that would be a major problem. The Application::run method and the application flow in general simply calls methods of the Facade, which dispatches it - as facades do ;) - to the instance of the dispatcher / emitter that it knows. Once created, the instance stays the same.

So where exactly is the problem when reaching the level of test execution? You should not add new listeners or tracers once testing started. For that reason, the sealing has been introduced. It only affects the registration of new listeners and tracers.

Maybe I'm missing something very obvious but so far I do not see a problem?

[Slamdunk]

As I understand your explanation above, you have multiple concurrent runs in parallel within the same application process - how ever that is achieved. How would resetting the state of the Facade help you after run? Each run would have to wait for the previous to complete - killing the very idea of paratest? What am I missing here?

Multiple consecutive application runs run in the same process. It is the equivalent of writing such script:

(new PHPUnit\TextUI\Application())->run(['phpunit', 'test/MyTest1.php'], false);
(new PHPUnit\TextUI\Application())->run(['phpunit', 'test/MyTest2.php'], false);
(new PHPUnit\TextUI\Application())->run(['phpunit', 'test/MyTest3.php'], false);

Nitpicking alert: Technically, there is no Singleton. The Facade keeps track of the instance it created, which is not the same as a Singleton but of course the point of the Facade. Using DI would not change that.

Sure it is not a Singleton strictly speaking, but it is a stateful static class, so it behaves like a Singleton

So where exactly is the problem when reaching the level of test execution? You should not add new listeners or tracers once testing started. For that reason, the sealing has been introduced. It only affects the registration of new listeners and tracers.

Maybe I'm missing something very obvious but so far I do not see a problem?

If you run the above example, the first test runs smoothly, but the second exits with the following error:

An error occurred inside PHPUnit.

Message:  (no message)
Location: ./phpunit/src/Event/Facade.php:32

#0 ./src/Runner/ResultCache/ResultCacheHandler.php(143): PHPUnit\Event\Facade::registerSubscriber()
#1 ./src/Runner/ResultCache/ResultCacheHandler.php(46): PHPUnit\Runner\ResultCache\ResultCacheHandler->registerSubscribers()
#2 ./src/TextUI/TestRunner.php(111): PHPUnit\Runner\ResultCache\ResultCacheHandler->__construct()
#3 ./src/TextUI/Application.php(110): PHPUnit\TextUI\TestRunner->run()
#4 ./src/TextUI/Application.php(78): PHPUnit\TextUI\Application->run()
#5 ./phpunit(91): PHPUnit\TextUI\Application::main()
#6 {main}

The optimal solution would be to remove every static keyword, forcing Facade object state to be bound to the Application that creates it, and not to the whole PHP process.

[sebastianbergmann]

Sorry to keep you waiting, Filippo.

In general, I am not a fan (quite the opposite, actually) of singletons, global state, etc. and agree with you that using dependency injection to make emitting events possible wherever needed. However, for this first iteration of the event system, which also has to deal with legacy test execution, I would prefer to stick with the current approach and revisit the situation when we have a new test runner (this is the next big improvement I want to work on after PHPUnit 10 has been shipped).

In the meanwhile, I have no problem at all with implementing your "reset the Singleton after the run" suggestion. Please let me know if you would like to send a pull request for that. Otherwise I will implement it ASAP.

[theseer]

Sorry, maybe I'm just blind or stupid - or it's still to early but I still fail to see the problem with the facade. I'm perfectly happy to admit the implementation is far from perfect and should probably get revisited.

But as @Slamdunk describes with the consecutive calls of Application::run, the problem is caused by calling that very method multiple times despite being the single application entry point. Instead of providing a "reset" feature to a Facade that PHPUnit doesn't need for itself and with that possibly causing inconsistencies, allowing others to call the reset mid-run and what not, we should provide a better way to 3rd parties to trigger test execution without rerunning things that do not require rerunning.

This all has nothing to do with the Facade. As I (currently?) see it we'd be fixing the wrong problem here..

[Slamdunk]

In the meanwhile, I have no problem at all with implementing your "reset the Singleton after the run" suggestion. Please let me know if you would like to send a pull request for that. Otherwise I will implement it ASAP.

It seems you've already in mind the best solution, so I'll leave it to you.

The only thing that matters to me is the following test to pass:

--TEST--
phpunit ../../_files/BankAccountTest.php
--FILE--
<?php declare(strict_types=1);
$_SERVER['argv'][] = '--do-not-cache-result';
$_SERVER['argv'][] = '--no-configuration';
$_SERVER['argv'][] = __DIR__ . '/../../_files/BankAccountTest.php';

require_once __DIR__ . '/../../bootstrap.php';
PHPUnit\TextUI\Application::main(false);
PHPUnit\TextUI\Application::main(false);
--EXPECTF--
PHPUnit %s by Sebastian Bergmann and contributors.

Runtime: %s

...                                                                 3 / 3 (100%)

Time: %s, Memory: %s

OK (3 tests, 3 assertions)
PHPUnit %s by Sebastian Bergmann and contributors.

Runtime: %s

...                                                                 3 / 3 (100%)

Time: %s, Memory: %s

OK (3 tests, 3 assertions)

@theseer I don't think 3rd parties should reset the state: it's PHPUnit's duty to leave a clean state after run

[theseer]

@theseer I don't think 3rd parties should reset the state: it's PHPUnit's duty to leave a clean state after run

Valid point. I was actually somehow expecting it to be a public method, but of course there could be varying degrees of public here if we hide that call somehow within the facade.

That leaves the question, what do we consider a "run"? And why would we want to re-setup everything of the 2nd+ calls? With the risk of those being somehow different?

[Slamdunk]

That leaves the question, what do we consider a "run"?

PHPUnit\TextUI\Application::main or PHPUnit\TextUI\Application::run?

And why would we want to re-setup everything of the 2nd+ calls? With the risk of those being somehow different?

I don't understand these two questions. At the end of run, PHPUnit should reset itself to be able to run again with the same result. What's the issue you are trying to pinpoint?

[theseer]

That leaves the question, what do we consider a "run"?

PHPUnit\TextUI\Application::main or PHPUnit\TextUI\Application::run?

See, that's where I'm not sure yet. And that would include a bunch of things done plus accompanying events that are useless to be redone and emitted again: Startup, Config handling, Facade Setup, Extension bootstrapping, .... as well as cleanup.

As far as I understand it, we're trying to allow 3rd parties to trigger test execution. We do not need the aforementioned steps to be run again for every consecutive call ("run").

And why would we want to re-setup everything of the 2nd+ calls? With the risk of those being somehow different?

I don't understand these two questions. At the end of run, PHPUnit should reset itself to be able to run again with the same result. What's the issue you are trying to pinpoint?

If we do as suggested, there is exactly no control from within PHPUnit to ensure it's exactly the same result. But maybe I'm still missing the point of all this..