[EXPERIMENTAL] Skip slow tests on Appveyor

[cmb69]

Our Appveyor CI times out very often since quite a while. Let's see
whether we can trim down the build time considerably by skipping slow
tests.

[nikic]

Why is the --show-slow dropped here?

It might be interesting to do a run with --show-slow 0 to get a full list of test timings, including the "fast" ones (which are probably comparatively slow on AppVeyor).

[cmb69]

Oh, I've mistankenly assumed that --show-slow is only about slow tests.

Anyhow, it seems skipping the slow tests saves quite some time.

[cmb69]

Well, maybe we should also review some very slow tests (20+ secs for which purpose).

[nikic]

Based on the full log of test timings, it looks like our baseline execution time for tests on AppVeyor is about 0.05s. That seems very slow to me. This seems to indicate that PHP has a slow startup and/or shutdown on Windows. Maybe we can do something about that?

I remember an issue I had at some point on Linux, where OpenSSL decided to do some kind of FIPS verification on startup -- that took just a couple of ms, but was enough to really blow up test times, because we spawn so many processes. Maybe something similar could be the cause here...

[krakjoe]

@weltling any ideas with regard to this slow startup/shutdown thing ?

[staabm]

nikic deleted a lot of tests with 8bc663f

is this still required?

[cmb69]

This was mostly meant as an experiment. Let's wait if we find out more regarding the slow startup time.

[nikic]

@staabm I think all of those removed tests were pretty quick, so even on Windows this probably only shaved a minute or two off the build.

[weltling]

The most of the bottleneck on Windows is the filesystem. One might find, that some code, where the measurement is only done to the time when all the files/modules are loaded and the code that doesn't touch filesystem is not worse than on other platforms. Specific factors for that are security and the complex ACLs, other programs like antivirus software can inject themselves into some calls, etc. There is no good way to improve this except having a fast hardware and do some system tuning.

My guess on the AppVeyor performance is, that the jobs might run on a lower priority while the tests carve a lot of resources and are numerous. Also every version adds more and more tests, which increases the job runtime. In the todays run - 7.3.2 has 16344 tests, 7.2.15 has 15988 tests. Clear, that not all of them run, but they still need to be read in, processed for at least skip sections, etc. The slowdown seems improportional, but it's of course also about the tests themselves. It might make sense to talk to AppVeyor support first and go from there.

Thanks.

[weltling]

I've filed a ticket on the AppVeyor support https://help.appveyor.com/discussions/problems/20189-performance-issues-in-php-core-tests .

Thanks.

[nikic]

It looks like the timeout issue is fixed now. It would of course still be good to reduce the long build times, but I guess we won't have to skip slow tests.

@weltling Do you know whether parallel test running would help on Windows?

[KalleZ]

@nikic It does big time when I was testing @hikari-no-yume's PR here: #2822 it was insanely fast and really worth while for Windows

[nikic]

Okay, maybe I'll take a look at finalizing that PR then.

[hikari-no-yume]

That patch is hell to rebase and I basically need to redo it, but I could maybe have a crack at it today or tomorrow.

[nikic]

@hikari-no-yume Please feel free to directly commit the first commit (to PHP-7.4). That's probably a big and unnecessary source of conflicts, and makes reviewing harder as well.

[weltling]

@nikic Parallel runs would sure accelerate the test suite, likely worth while to pursue. Regarding the Azure Pipelines - if we have some volunteers, probably should take a look at it (especially as it also has Mac).

Thanks.