Setup mutation testing

[staabm]

requires

• failling CI build #688
• Regenerate baseline #693

[staabm]

atm this class replaces any call to a method named yes, or no.
for now it is not scoped to the TrinaryLogic class.

[staabm]

current challenge: figure out a way to kill mutants across github action jobs - I will think about that

[ondrejmirtes]

The test is failing sporadically after introducing the random order. It's probably because of:

phpstan-doctrine/tests/Rules/Doctrine/ORM/entity-manager.php

Lines 34 to 37 in 5eaf37b

	Type::overrideType(
	'date',
	DateTimeImmutableType::class,
	);

(which sometimes gets executed before this test and sometimes not).

I guess the test that executes this should clean up after itself.

[ondrejmirtes]

Also I don't see PHPStan configured as mutant killer here.

[ondrejmirtes]

Why turning off the logger-github?

[ondrejmirtes]

Also personally I wouldn't add this to Makefile. We'll need a more complex logic in the GHA job.

[staabm]

added it to makefile to easy running infection locally (to reproduce CI problems)

[staabm]

Why turning off the logger-github?

running infection in 3 php versions likely spams the 'files changed tab' with annotations.
I think we need a separate job which accumulates the separate job results and merges them togehter so we get a single final result with less false positives (mutation job on php 8.3 might error about a mutation which can only be killed in the php 8.1 test-suite)

[ondrejmirtes]

mutation job on php 8.3 might error about a mutation which can only be killed in the php 8.1 test-suite

I had this concern as well but I talked to Maks about this and given that Infection would only mutate covered lines then this shouldn't happen. If a test ran on PHP 8.3 covers a line and we mutate that line, it should fail the test on 8.3.

[staabm]

it should fail the test on 8.3.

but the very same mutation will happen on 8.1/8.2 and no test will kill the mutation (in case it is a PHP 8.3 only test)

[ondrejmirtes]

If this was really true Infection would be unusable for us. We'd have to be able to invoke each phase separately and modify the results before feeding them to the next phase. (Run tests on each PHP version, feed the data into Infection, let it mutate the sources, run tests on each PHP version again ourselves and so on.)

[staabm]

I think we could run infection on multiple php versions in parallel.. collect the mutations afterwards in a new followup github action and report only those as a problem, which were not killed in any of the previous jobs.
(as long as we do not do source transformations, the mutations should be the same on all 3 jobs)

alternatively we could run infection on a single php version only

[ondrejmirtes]

Some comments posted by me disappeared. I linked the --git-diff-filter / --git-diff-base / --git-diff-lines options

[ondrejmirtes]

The PHAR distribution is the recommended one for installing Infection. I think we could just add infection to tools in setup-php action.

[staabm]

Some comments posted by me disappeared. I linked the --git-diff-filter / --git-diff-base / --git-diff-lines options

yeah, I did not yet activate these options, because then there would be no results reported in this PR.

[staabm]

The PHAR distribution is the recommended one for installing Infection. I think we could just add infection to tools in setup-php action.

I think this only make sense in case we don't want run infection locally. the more we use GitHub Actions only stuff, the harder it is to run the very same setup on a local computer.

[staabm]

The test is failing sporadically after introducing the random order. It's probably because of:

I did not yet see a test, which started failling more often since random order was enabled.
which particular test do you mean?

the build on the master branch is broken long before random order with the same errors

[ondrejmirtes]

I think this only make sense in case we don't want run infection locally.

Yeah, I think we don't. Given the support for many PHP versions (and that we only typically have a single PHP version running locally), I think it's unlikely we'd run Infection locally.

Also I'm thinking we'll be passing GitHub Actions-specific env variables to the CLI options anyway.

[ondrejmirtes]

Why is the build green even when there are escaped mutants?

[staabm]

Why is the build green even when there are escaped mutants?

because we did not yet define a "min MSI", meaning there is no minimum threshold yet.
if we want the test to fail when at least a single escaped mutant occurs, we need min-msi=100

[ondrejmirtes]

Yeah, I want that, alongside the Git-filtering options.

We can demonstrate the failing by adding some new change here in this PR temporarily.

[staabm]

Also I don't see PHPStan configured as mutant killer here.

I think, as long as we only mutate with TrinaryLogicMutator its pretty unlikely that the PHPStan killer will help us.
in addition the process is already slow and adding the PHPStan killer to the mix will make it a even slower

[ondrejmirtes]

I really want it though. I don't want to write tests for things that would error in PHPStan. Also we'd have more incentives to make it faster 😊

[staabm]

Phpunit killer requires a green test run
Phpstan killer requires a green phpstan run

[ondrejmirtes]

Make sure to carry over PHPStan's result cache, probably with upload and download-artifact actions. It should speed up the build significantly.