[FEATURE REQUEST] Codeception v5 support

[jasonbahl]

Is your feature request related to a problem? Please describe.
I'm interested in setting up "path coverage" for my tests, and it looks like this will be supported in Codeception v5 (see: https://github.com/Codeception/Codeception/releases/tag/5.0.0-alpha1) / (Codeception/Codeception#6158)

Describe the solution you'd like
I would like to use Codeception v5+ with wp-browser, specifically to take advantage of the path_coverage configuration

I'm not sure how much is changing in Coceception v5, so I understand this may not be a simple task.

Additional context

For some context, there's some discussion about the benefits of pathCoverage here: wp-graphql/wp-graphql#2298 (comment)

[lucatume]

HI @jasonbahl,

you've beaten me to the race: I was going to open a similar issue to track the progress.

I definitely want to support Codeception 5 and use the chance to break some back-compatibility and review/rethink a number of approaches.

Currently, I'm not able to provide a road-map as I still have to take the time to see how badly broken wp-browser would be using Codeception 5.

Since you've already set up this issue, I will use this to track my progress and updates.

[calvinalkan]

@lucatume codeception v5 will require PHP8 btw. IMO we should hold out a little bit more and then make a clean cut requiring 8.0 aswell here.

[lucatume]

@calvinalkan I know, I've started working on an exploratory branch v4 on Codeception 5 as well.

[calvinalkan]

@lucatume My suggestion is that we wait until 28 Nov 2022 and then release a new v4 that requires codeception 5 and PHP8.

7.4 loses security updates on that date.

[lucatume]

I do not have a due date for wp-browser v4 yet, if it's ready before then, I would release it, if not, it will be out when it's done.
Before that, my idea would be to have the v4 branch act as a beta for those daring to try it out.

[calvinalkan]

@lucatume fair enough, which PHP version are you aiming at tho?

[vyarmolenko]

@lucatume @calvinalkan waiting/deprioritizing work on this until the end of support of 7.4 doesn't sound like a very good option. Some hostings enforce user migration to PHP 8 before this date. Some users might need your library earlier than that date to verify their migration readiness.

[calvinalkan]

@lucatume See also Codeception/Codeception#6532

[calvinalkan]

@lucatume @calvinalkan waiting/deprioritizing work on this until the end of support of 7.4 doesn't sound like a very good option. Some hostings enforce user migration to PHP 8 before this date. Some users might need your library earlier than that date to verify their migration readiness.

You can use wp-browser on PHP8.1 without any problem.

The reverse is not true. Now that codeception5 is out with a minimum of 8.0 you cant use it anymore to test PHP7.X code.

[Veraxus]

What is the status of v4 w/ Codeception 5? PHP 7.x was fully EOL in December and even 8.0 will be EOL in less than 6 months... so we're pretty far behind the curve at this point.

[lucatume]

What is the status of v4 w/ Codeception 5?

The status is RC1.
You're welcome to test it out, documentation is still not there yet, and I'm working on it at this moment, so you will have to figure some things out on your own.
If you run into problems, I would ask you to report the issue with as much detail as you can.

PHP 7.x was fully EOL in December and even 8.0 will be EOL in less than 6 months... so we're pretty far behind the curve at this point.

WordPress requires a minimum PHP version of 5.6 and many plugins out there, including many using wp-browser, will require a minimum version of PHP that is EOL and will use v3 of wp-browser for some time to keep running tests on their minimum required PHP version.

[lucatume]

Version 4 of the project supporting version 5 of Codeception has been merged.

[calvinalkan]

Version 4 of the project supporting version 5 of Codeception has been merged.

Awesome. Is there a migration guide?

Or is that not needed?

[lucatume]

Awesome. Is there a migration guide?

Or is that not needed?

There is no migration guide yet; my idea is to clean up the current issues (stale, deprecated) and then start on that.
The configuration of the modules should not be that different.
In the meantime, you can look at the updated documentation and the project's configuration files.

[calvinalkan]

Awesome. Is there a migration guide?
Or is that not needed?

There is no migration guide yet; my idea is to clean up the current issues (stale, deprecated) and then start on that. The configuration of the modules should not be that different. In the meantime, you can look at the updated documentation and the project's configuration files.

I just checked the new branch.

I'm curious to know what happened to the docker based testing setup.

Why did you refactor it?

[lucatume]

I'm curious to know what happened to the docker based testing setup.

You mean the container-based setup for testing the wp-browser project itself, I presume.
One of the reworks of version 4 was revisiting the assumption that only a container-based stack would allow reliable and easy-to-set-up testing; the result of that is the default configuration is based on PHP built-in server (with multiple workers), the SQLite integration plugin and the use of ChromeDriver locally installed to run the tests.
I've updated the project workflow to use a similar stack, with an extension to control a really minimal docker-compose.yml file to test out its potential and reduce build times.
PHP Built-in servers with multiple workers offers performance comparable to a real web server in a testing context; detecting Chrome on the system and updating Chromedriver to the correct version is easy (e.g. with this command) and multi-site, both sub-domain and sub-folder works just fine on localhost, Chrome has no issues with blog1.localhost:9000.

In short, the refactoring of the project testing stack achieves two goals: remove complexity and "eat my own dog food", the controller extensions.