Travis CI: Support for Codeception tests

[JDGrimes]

See WordPoints/wordpoints#17.

[JDGrimes]

Codeception and Travis CI can be done, but "isn't for the faint of heart.".

I also found WP Browser, a codeception wrapper specifically for WordPress. It still will need work to function on Travis, though, I think.

[JDGrimes]

I've gotten the ball rolling with WP Browser and a custom Codeception module for testing with WordPoints in WordPoints/hooks-api#92.

However, I haven't tried to tackle running the tests on Travis CI yet.

What I've found out with an initial foray is that Codeception itself is currently using Semaphore to run its WebDriver tests. I couldn't find the story behind that, but basically what it is doing is using PHP's built-in webserver to make it possible to run the web app:

php -S 127.0.0.1:8000 -t tests/data/app >/dev/null 2>&1 &

[JDGrimes]

We may also need to exclude the Codeception test code from the PHP syntax checks on versions below 5.4.

[JDGrimes]

The closest thing that I've found to a working example of running a PHP webserver on Travis is this, for Drupal modules. There they are actually using Apache, not the built-in PHP webserver. (Related to that, I also found this ticket for Drupal itself, which notes that Drupal seems to still be using the built-in server, although there are times when it doesn't accurately mimic a real server.)

Travis CI even has some docs on using Apache with PHP (found through this question on SO). Maybe that is the best thing to do.

[JDGrimes]

Codeception/Codeception#2265 seems to indicate that there are issues running phantomjs to connect to the builtin PHP webserver on Travis CI. Selenium server and Firefox did work for them though, so perhaps that's what we should explore. Or we could try to figure out why phantom JS can't connect to localhost:8000 on Travis CI.

[JDGrimes]

I also asked about why Codeception doesn't use Travis CI on their Gitter channel, but no response yet.

[JDGrimes]

Last night I happened to think that maybe the issue we were running into with the request for a password isn't related to PhantomJS at all. I wonder if instead it is the mysql or mysqldump commands asking for a password. This might be happening even though we are passing a password to them because password for the database that we create is an empty string. I think maybe we need to use -p'' in our commands, and it would work.

[JDGrimes]

I just found this question on SO. It seems that the answer is to not use -p at all when the password is empty, or use the long --password='' flag.

[JDGrimes]

Tried passing the MySQL password differently, and got this:

[Codeception\Exception\ModuleException]                                      

  WordPoints\Tests\Codeception\Modules\WordPointsLoader:                       

  Failed to create database dump: mysqldump: Got error: 1045: Access denied f  

  or user 'root'@'localhost' (using password: YES) when trying to connect      

I'm specifying the password as --password="".

According to this SO thread, there could be many reasons for that error. My guess is that our best option to try next would be omitting the password options entirely.

[JDGrimes]

Now that I've got the MySQL password stuff figured out, I'm getting this:

1) Failed to save a points reaction in savePointsReactionCept (tests/codeception/acceptance/savePointsReactionCept.php)

 Step  I login as admin 

 Fail  Field by name, label, CSS or XPath element with '#user_login' was not found.

Scenario Steps:

 2. $I->loginAsAdmin() at tests/codeception/_support/AcceptanceTester.php:34

 1. $I->loadSessionSnapshot("admin") at tests/codeception/_support/AcceptanceTester.php:30

I'm not sure if it is an issue with the server or the browser.

[JDGrimes]

Through WP#34693, I've just found https://github.com/10up/wp-codeception. It looks like something that we should investigate in addition to WP Browser.

[JDGrimes]

I just found this, and thought it might be useful for reference: https://github.com/SochaDev/codeception-ci-generator

[JDGrimes]

If we can't get this to work on Travis CI, we might have to try Semaphore or CircleCI, or something else. The only reason that I shy away from doing that is that I'd like to text against all versions of WordPress and PHP, which I don't think will be possible with the free plans offered by those services. Although on the other hand we can't test against versions of PHP below 5.4 anyway. Still, I'd like to test against each version of WordPress that we support.

[JDGrimes]

I finally discovered the issue with Travis CI not loading up the WordPress site (which was the cause of the login step failing as above). The issue was that I was setting the WebDriver URL to 127.0.0.1:8080 instead of http://127.0.0.1:8080. The lack of http:// is what was causing the issue.

Now that that's settled though, it turns out that when accessed through the webserver this way, the WordPress site doesn't consider itself to be installed. The webserver redirects the request like so:

127.0.0.1:45837 [302]: /wp-login.php?redirect_to=wp-admin/admin.php?page=wordpoints_points_types
127.0.0.1:45838 [500]: /?redirect_to=wp-admin/admin.php?page=wordpoints_points_types/wp-admin/setup-config.php

We're being redirect to setup-config.php (but incorrectly, see trac ticket).

I think that this is just because we have a wp-tests-config.php but no wp-config.php. The solution is likely as simple as symlinking wp-tests-config.php to wp-config.php, or something of that sort.

[JDGrimes]

OK, that done, we are a step closer. wp-login.php is now loading, however, it is giving us a strange 404 page when the form is submitted. Actually, this is because the form is being submitted to example.org. This is because the wp-tests-config.php includes the line

define( 'WP_TESTS_DOMAIN', 'example.org' );

So this is what the site URL ends up getting defined as in the database. We can either update the value in the DB before running the tests, or just change that line in wp-tests-config.php before we run the PHPUnit tests.

[JDGrimes]

Hooray! It's alive! The only problem is that if we just modify the WP_TESTS_DOMAIN in wp-tests-config.php, the PHPUnit tests will break when running WordPress multisite. I'm not sure what's causing that, but for some reason it gives an error that it can't establish a database connection. Probably our Codeception plugin should be updating the value in the DB to match the given URL anyway. So perhaps all we need to do is update that.

[JDGrimes]

Actually, we use the tests bootstrap in our WordPoints Loader Codeception module. So we do need to use the constant, but I changed things so that we don't change the constant's value until after the PHPUnit tests have run. And now it is working great! We even discovered that one of our tests is failing because something in the Hooks API module is broken!