New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand tests to modern WordPress versions and PHP versions #46

Merged
merged 5 commits into from Apr 16, 2018

Conversation

Projects
None yet
2 participants
@benlk
Contributor

benlk commented Mar 20, 2018

Changes

In .travis.yml, and thus in the automated tests that run on Travis CI:

  • PHP versions now covered: 5.6 through 7.1 (not 7.2, for reasons explained later)
  • WordPress versions now covered: 3.8 through 4.9, at the level of the latest security patch

Why

This PR expands the versions tested because we want to make sure that the plugin is compatible with recent PHP and WordPress versions. The simple way to do that is to enumerate a list of WordPress versions and a list of PHP versions, and then run the tests at each intersection. Not only is this approach inefficient, it causes problems when old versions of WordPress include a test suite that assumes an old version of PHPUnit, which is not available when Travis CI only offers one version of PHPUnit per PHP version.

Based on research in npr/pmp-php-sdk#36,
Travis CI runs tests using the following PHPUnit versions:

  • PHP 5.6: PHPUnit 5.7.23
  • PHP 7: PHPUnit 6.4.3
  • PHP 7.1: PHPUnit 6.4.3
  • PHP 7.2: PHPUnit 7.0.2

Reasons that given versions of WordPress have tests incompatible with travis-ci.org:

  1. There are PHP warnings about deprecated PHP-4-style constructors.
  2. Requires PHPUnit 5 because PHPUnit 6 support was not included until WordPress 4.7.
  3. Unless there's been an update since this trac thread, WordPress tests still assume PHPUnit 6, and Travis runs PHP 7.2 tests using PHPUnit 7
phpunit 5.7 6.4 6.4 7.0
php 5.6 7.0 7.1 7.2
WP wp release year then-current php
3.8 1 1 1 2013
3.9 1 1 1 2014 5.5
4.0 1 1 1 2014 5.5
4.1 1 1 1 2014 5.5
4.2 2 2 2 2015 5.6
4.3 2 2 2 2015 5.6
4.4 2 2 2 2015 7.0
4.5 2 2 2 2016 7.0
4.6 2 2 2 2016 7.0
4.7 2 2 2 2016 7.1
4.8 3 2017 7.1
4.9 3 2017 7.2

benlk added some commits Mar 20, 2018

Drastically reduce the number of builds that are made with Travis
Based on research in npr/pmp-php-sdk#36,
Travis CI runs tests using the following PHPUnit versions:

- PHP 5.6: PHPUnit 5.7.23
- PHP 7: PHPUnit 6.4.3
- PHP 7.1: PHPUnit 6.4.3
- PHP 7.2: PHPUnit 7.0.2

This commit reduces the number of tests that are run by more than
two thirds. Before, we were running 4 different PHP versions and
11 diffferent WordPress versions. Many of those combinations were
incompatible, for different reasons:

1. There are PHP warnings about deprecated PHP-4-style
   constructors.
2. Requires PHPUnit 5 because PHPUnit 6 support was not
   included until WordPress 4.7.
3. Unless there's been an update since
   [this trac thread](https://core.trac.wordpress.org/ticket/39822),
   WordPress tests still assume PHPUnit 6, and Travis runs PHP 7.2
   tests with PHPUnit 7

| phpunit | 5.7 | 6.4 | 6.4 | 7.0 |                 |                  |
| php     | 5.6 | 7.0 | 7.1 | 7.2 |                 |                  |
| WP      |     |     |     |     | wp release year | then-current php |
|---------|-----|-----|-----|-----|-----------------|------------------|
| 3.8     |     | 1   | 1   | 1   | 2013            |                  |
| 3.9     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.0     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.1     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.2     |     | 2   | 2   | 2   | 2015            | 5.6              |
| 4.3     |     | 2   | 2   | 2   | 2015            | 5.6              |
| 4.4     |     | 2   | 2   | 2   | 2015            | 7.0              |
| 4.5     |     | 2   | 2   | 2   | 2016            | 7.0              |
| 4.6     |     | 2   | 2   | 2   | 2016            | 7.0              |
| 4.7     |     |     |     | 3   | 2016            | 7.1              |
| 4.8     |     |     |     | 3   | 2017            | 7.1              |
| 4.9     |     |     |     | 3   | 2017            | 7.2              |

Accordingly, this commit removes PHP 7.2 from tests, and replaces the
separate lists of PHP and WordPress versions with one list of selected
PHP and WordPress versions.
Reduce number of builds by removing WP 4.7 in PHP > 7
Updated version of the table in INN@09d72a8:

Based on research in npr/pmp-php-sdk#36,
Travis CI runs tests using the following PHPUnit versions:

- PHP 5.6: PHPUnit 5.7.23
- PHP 7: PHPUnit 6.4.3
- PHP 7.1: PHPUnit 6.4.3
- PHP 7.2: PHPUnit 7.0.2

Reasons that given versions of WordPress have tests incompatible
with travis-ci.org:

1. There are PHP warnings about deprecated PHP-4-style
   constructors.
2. Requires PHPUnit 5 because PHPUnit 6 support was not
   included until WordPress 4.7.
3. Unless there's been an update since
   [this trac thread](https://core.trac.wordpress.org/ticket/39822),
   WordPress tests still assume PHPUnit 6, and Travis runs PHP 7.2
   tests with PHPUnit 7

| phpunit | 5.7 | 6.4 | 6.4 | 7.0 |                 |                  |
| php     | 5.6 | 7.0 | 7.1 | 7.2 |                 |                  |
| WP      |     |     |     |     | wp release year | then-current php |
|---------|-----|-----|-----|-----|-----------------|------------------|
| 3.8     |     | 1   | 1   | 1   | 2013            |                  |
| 3.9     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.0     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.1     |     | 1   | 1   | 1   | 2014            | 5.5              |
| 4.2     |     | 2   | 2   | 2   | 2015            | 5.6              |
| 4.3     |     | 2   | 2   | 2   | 2015            | 5.6              |
| 4.4     |     | 2   | 2   | 2   | 2015            | 7.0              |
| 4.5     |     | 2   | 2   | 2   | 2016            | 7.0              |
| 4.6     |     | 2   | 2   | 2   | 2016            | 7.0              |
| 4.7     |     | 2   | 2   | 2   | 2016            | 7.1              |
| 4.8     |     |     |     | 3   | 2017            | 7.1              |
| 4.9     |     |     |     | 3   | 2017            | 7.2              |

@benlk benlk changed the title from Improve test coverage in all WordPress versions to Expand tests to modern WordPress versions and PHP versions Mar 20, 2018

@benlk benlk referenced this pull request Mar 28, 2018

Merged

Revisions to the NPR API editor meta box #47

1 of 3 tasks complete
@jgarlow

Looks good. Thanks for the thorough explanation.

@benlk benlk merged commit 053aaf7 into npr:master Apr 16, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment