Provide bless_tests.patch for failing tests

[cmb69]

In case any tests are failing, we run bless_tests.php and provide the resulting git diff as artifact. This should be helpful for PR authors who don't have a Windows environment at hand.

The second commit introduces a test failure to be able to verify the results.

[cmb69]

Apparently, bless_test.php also fixes a few tests which didn't fail according to run-tests.php. Is this a caching issue? I'll have to check.

[mvorisek]

What is the best/easiest way to access the artifact?

For development, I started using Freebsd/Cirrus CI output as the output is accessible thru single click (like https://github.com/php/php-src/runs/2935436162), however it does not contain a diff. Can the output summary be updated so it contains the diff of FAILED TEST SUMMARY jobs?

[nikic]

ext/ffi/tests/bug80847.phpt has an "xfail" SKIPIF on Windows. bless_tests.php ignores XFAIL tests, but not that particular case.

[cmb69]

Right, the other test that is caught under ZTS also. I added a patch for bless_test.php, but I'm not particularly happy with it.

[iluuu1994]

@cmb69 A regex shows there's only 21 tests that use xfail in SKIPIF. Should we just convert them to skip to avoid the inaccurate regex?

[iluuu1994]

Alternatively we could not emit the .out file for xfail in CI via some flag. Not sure if that's better. bless shouldn't fix xfail tests locally either.

[cmb69]

Actually, I like the distinction between xfail (should work, but does not yet) and skip (can't work), but given that it's rather uncommon that xfail tests get adressed, and that they consume time for no good reason, switching dynamic xfail to skip (plus maybe a comment) might be sensible.

[iluuu1994]

Might be controversial: Maybe we should not run xfail at all? What's the point if they're gonna fail anyway. If you want to fix them removing the xfail seems sensible. This way we can:

• Keep the distinction of skip and xfail for developers
• Reduce unnecessary time running the tests
• Fix the bless problem

[nikic]

@iluuu1994 If we don't run xfail tests, then we'd no longer get the warning if an xfail test actually passes. Not sure how valuable that is though...

For the problem here, it might make sense to integrate a --bless option or so in run-tests.php, which can automatically bless failed tests (and has the necessary knowledge about xfailed tests).

[cmb69]

Yet another option might be to move dynamic xfail to the XFAIL section; empty XFAIL section would be treated like before; non empty XFAIL section would be handled like SKIPIF sections.

[nikic]

I've added a --bless option in bf40a93.

[cmb69]

Thank you, @nikic. I've updated the patch accordingly.