PHPUnit warning cannot be ignored

[nikophil]

Q	A
PHPUnit version	12.2.2
PHP version	8.3.20
Installation Method	Composer

Summary

PHPUnit warnings are always failing the testsuite, diregarding of failOnWarning configuration.

The final outcome of the testsuite is:

OK, but there were issues!
Tests: 1, Assertions: 1, Warnings: 1.

but always return an exit code of 1

This wasn't a problem until #6213 which started to trigger PHPUnit warnings when the number of arguments returned by the data provider does not match the number of arguments of the test method.

This warning seems totally legit, but in some case, it could be useful to allow those numbers of arguments to not match, example:

final class SomeTest extends TestCase
{
    #[Before]
    public function initializationMethod(): void
    {
        // do some heavy initialization using `$this->providedData()`
        // which would be reused by all test methods in the class
    }

    #[Test]
    #[DataProvider('provider')]
    public function testMethod() void
    {
        self::assertTrue(true);
    }

    public static function provider(): iterable
    {
        yield ['foo'];
    }
}

I know this is a little bit hacky, but in some complex scenario, it is pretty handy.

How to reproduce

final class ReproducingTest extends TestCase
{
    #[Test]
    #[DataProvider('provider')]
    public function testMethod() void
    {
        self::assertTrue(true);
    }

    public static function provider(): iterable
    {
        yield ['foo'];
    }
}

Expected behavior

I'm not sure what's the best way to fix this... There could be some "global" solutions:

• make the config failOnWarning to act on PHPUnit warning as well
• add a new config failOnPHPUnitWarning

but both of them would hide other more legit warnings, I think they are not a really good solution.

Or maybe we could go for a more fine-grained solution:

• allow to ignore PHPUnit warning along with a baseline, as well as deprecations
• add a new parameter to DataProvider attribute? #[DataProvider('providerMethodName', allowExtraArguments: true)]
• introduce a new attribute #[IgnoreWarnings] / #[IgnorePHPUnitWarnings]

I'd be willing to provide a PR if we agree on a solution to implement.

thanks!

[sebastianbergmann]

PHPUnit warnings are always failing the testsuite, diregarding of failOnWarning configuration.

This is not a bug, this is by design. I am willing to discuss this design and the change the behaviour, of course.

[nikophil]

Another legit usecase (at least, in my opinion 😅 ) to have a different arguments number between the data provider and the test method, is when using the same data provider for several test methods, but one of the test methods does not need all the arguments:

final class ReproducingTest extends TestCase
{
    #[Test]
    #[DataProvider('provider')]
    public function testMethod1($arg1, $arg2) void
    {
        // ...
    }

    #[Test]
    #[DataProvider('provider')]
    public function testMethod2($arg1) void
    {
        // ...
    }

    public static function provider(): iterable
    {
        yield ['foo', 'bar'];
    }
}

I am willing to discuss this design and the change the behaviour, of course.

sweat! any thoughts about the solutions I proposed?

I think the one I prefer is definitely #[IgnoreWarnings] / #[IgnorePHPUnitWarnings].

Maybe, we could even go further and accept a pattern for a regex of warnings to ignore?