Look into running multiple Behat tests in parallel

[schlessera]

To speed up the Behat tests, we should look at splitting them up into independent groups that can run in parallel, and tweak this parallel running capacity to lower the combined total amount of time needed.

On machines with multiple processing cores, this can drastically speed up the tests.

[danielbachhuber]

Previously #1992, #3652

I actually looked at running tests in parallel for #1992, but wasn't able to get it to work well.

At this point, I think #3728 is the path forward, as it includes the added bonus of only running the tests specific to the command, and not the entire test suite.

[danielbachhuber]

as it includes the added bonus of only running the tests specific to the command, and not the entire test suite.

Another perceived pain point of running the entire test suite unnecessarily is that it blocks jobs on the other repositories.

[danielbachhuber]

@schlessera Given #3728, do you think this is still necessary?

[schlessera]

No, at least not as a priority.

The tests within one single package probably cannot be run concurrently anyway. That would only have made sense with several testing areas that don't overlap.

I'll close this, and if we still have major performance problems with the tests after having everything split up, we can reconsider this.

[swissspidy]

Just found this issue after stumbling upon https://github.com/Daniel-Marynicz/BehatParallelExtension. There is also https://github.com/liuggio/fastest

Tests for a single package can easily take over 10 minutes, even on an M1 Mac. If that extension works, I think it would be an interesting addition to the test suite, especially for local development.

[swissspidy]

Here is a proof of concept: wp-cli/wp-cli-tests#189