CLI option to remove unmatched errors from baseline

[hkdobrev]

Feature request

Since PHPStan 0.12.18, there is a dedicated CLI option for regenerating the baseline file --generate-baseline.

When using it, it would do 2 things from user perspective:

1. Any new errors that are reported would be ignored
2. Any existing ignore rules which are no longer matching would be cleaned up

I'm suggesting another option which would only do the second. This option would just remove rules from the baseline that are not matching, but continue to report new errors.

This is useful in a very key scenario - you have a codebase where there are a lot of ignored errors, but you want to enforce stricter rules for any new code. When you use the baseline for that, you end up with a lot of ignored errors in the baseline. With refactoring, a lot of those would be reported as unmatched rules which is great and you'd clean those up. But if at the same time there are new errors, you would assimilate them in the baseline and ignore them instead of fixing them.

With this option a more automated process could be introduced where no longer matching rules are automatically or semi-automatically cleaned up from the baseline. It will also ensure a more stable usage of regenerating the baseline where people would more explicitly ignore actual errors.

[ondrejmirtes]

Hi, I think that the current best way to do this is:

• Set reportUnmatchedIgnoredErrors to false so you're not annoyed by unused entries in the baseline
• From time to time run --generate-baseline but first make sure that analyse returns 0 new errors.
• You'll end up with unused entries removed.

You can automatize this with a custom script. I don't think it's necessary to add something to PHPStan because of this.

[ondrejmirtes]

Alternatively, you can run --generate-baseline and make sure with a custom diff-scripting script that lines were only removed, not added.

[hkdobrev]

• Set reportUnmatchedIgnoredErrors to false so you're not annoyed by unused entries in the baseline
• From time to time run --generate-baseline but first make sure that analyse returns 0 new errors.
• You'll end up with unused entries removed.

This approach works, but to an extent. If you've already removed some code violating a rule, but you still keep the ignore pattern in the baseline, you can still introduce this error back in the same file before running the --generate-baseline.

Alternatively, you can run --generate-baseline and make sure with a custom diff-scripting script that lines were only removed, not added.

This could work, would look into it.

---

You can automate this with a custom script. I don't think it's necessary to add something to PHPStan because of this.

Biggest concern I want to tackle is people receiving some new errors and some unmatched ones and then just running a script and it's green again instead of properly looking into what's going on.

I agree if there's a way to properly address this without adding anything new to PHPStan that would be OK. We already have a Composer script to generate the baseline, so we can just modify that and have another one that would also ignore errors.

[ondrejmirtes]

This is really something very advanced and very-very-very nice to have which I'm not personally interested in. So closing :) Feel free to reach out if you need any help with the custom solution.

[hkdobrev]

Sure, I'll post back here when I figure it out for anyone who might stumble across the same thing.

[hkdobrev]

This script would suffice for simple cases:

phpstan analyse --generate-baseline \ 
&& git diff --numstat -- build/phpstan-baseline.neon \ 
| grep --quiet "^\s*0\s" \
|| echo "This would ignore new errors! Are you sure?" >&2 \ 
&& false

You could employ a function or another tool to do stashing and restoring to the
original content if necessary. A very simplistic lint-staged setup would like like this:

.lintstagedrc:

{
  "build/phpstan-baseline.neon": "git diff --numstat -- build/phpstan-baseline.neon | grep --quiet \"^\\s*0\\s\" || echo \"This would ignore new errors! Are you sure?\" >&2 && false"
}

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.