Unable to run tests using composer, PHPUnit incompatible

[Lalee10]

Issue

Trying to run tests for a PR but they are failing due to old PHPUnit version.

Error: Looks like you're using PHPUnit 4.8.36. WordPress requires at least PHPUnit 5.4 and is currently only compatible with PHPUnit up to 7.x.
Please use the latest PHPUnit version from the 7.x branch.
Script phpunit --coverage-clover build/logs/clover.xml handling the test event returned with error code 1

OS Details

macOS Big Sur 11.5.1
PHP 7.4
composer 2.1.5

Full Output

> export $(cat phpunit.env | xargs) && bash bin/install-wp-tests.sh $WP_TEST_DB $WP_TEST_USER "$WP_TEST_PASS" $WP_TEST_HOST $WP_VERSION || true
+ install_wp
+ '[' -d /tmp/wordpress/ ']'
+ mkdir -p /tmp/wordpress/
+ [[ 5.7 == \n\i\g\h\t\l\y ]]
+ [[ 5.7 == \t\r\u\n\k ]]
+ '[' 5.7 == latest ']'
+ local ARCHIVE_NAME=wordpress-5.7
+ download https://wordpress.org/wordpress-5.7.tar.gz /tmp/wordpress.tar.gz
++ which curl
+ '[' /usr/bin/curl ']'
+ curl -s https://wordpress.org/wordpress-5.7.tar.gz
+ tar --strip-components=1 -zxmf /tmp/wordpress.tar.gz -C /tmp/wordpress/
+ download https://raw.github.com/markoheijnen/wp-mysqli/master/db.php /tmp/wordpress//wp-content/db.php
++ which curl
+ '[' /usr/bin/curl ']'
+ curl -s https://raw.github.com/markoheijnen/wp-mysqli/master/db.php
+ install_test_suite
++ uname -s
+ [[ Darwin == \D\a\r\w\i\n ]]
+ local 'ioption=-i .bak'
+ '[' '!' -d /tmp/wordpress-tests-lib ']'
+ mkdir -p /tmp/wordpress-tests-lib
+ svn co --quiet https://develop.svn.wordpress.org/tags/5.7/tests/phpunit/includes/ /tmp/wordpress-tests-lib/includes
+ svn co --quiet https://develop.svn.wordpress.org/tags/5.7/tests/phpunit/data/ /tmp/wordpress-tests-lib/data
+ '[' '!' -f wp-tests-config.php ']'
+ download https://develop.svn.wordpress.org/tags/5.7/wp-tests-config-sample.php /tmp/wordpress-tests-lib/wp-tests-config.php
++ which curl
+ '[' /usr/bin/curl ']'
+ curl -s https://develop.svn.wordpress.org/tags/5.7/wp-tests-config-sample.php
++ echo /tmp/wordpress/
++ sed 's:/\+$::'
+ WP_CORE_DIR=/tmp/wordpress/
+ sed -i .bak 's:dirname( __FILE__ ) . '\''/src/'\'':'\''/tmp/wordpress//'\'':' /tmp/wordpress-tests-lib/wp-tests-config.php
+ sed -i .bak s/youremptytestdbnamehere/rollbar_test/ /tmp/wordpress-tests-lib/wp-tests-config.php
+ sed -i .bak s/yourusernamehere/root/ /tmp/wordpress-tests-lib/wp-tests-config.php
+ sed -i .bak s/yourpasswordhere/1234/ /tmp/wordpress-tests-lib/wp-tests-config.php
+ sed -i .bak 's|localhost|localhost|' /tmp/wordpress-tests-lib/wp-tests-config.php
+ install_db
+ '[' false = true ']'
+ PARTS=(${DB_HOST//\:/ })
+ local PARTS
+ local DB_HOSTNAME=localhost
+ local DB_SOCK_OR_PORT=
+ local EXTRA=
+ '[' -z localhost ']'
++ echo
++ grep -e '^[0-9]\{1,\}$'
+ '[' ']'
+ '[' -z ']'
+ '[' -z localhost ']'
+ EXTRA=' --host=localhost --protocol=tcp'
+ mysqladmin create rollbar_test --user=root --password=1234 --host=localhost --protocol=tcp
mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: CREATE DATABASE failed; error: 'Can't create database 'rollbar_test'; database exists'
> phpunit --coverage-clover build/logs/clover.xml
PHP Deprecated:  The each() function is deprecated. This message will be suppressed on further calls in /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/Util/Getopt.php on line 38
PHP Stack trace:
PHP   1. {main}() /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/phpunit:0
PHP   2. PHPUnit_TextUI_Command::main($exit = *uninitialized*) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/phpunit:52
PHP   3. PHPUnit_TextUI_Command->run($argv = [0 => '/Users/lalee/Desktop/rollbar-php-wordpress/vendor/bin/phpunit', 1 => '--coverage-clover', 2 => 'build/logs/clover.xml'], $exit = TRUE) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/TextUI/Command.php:100
PHP   4. PHPUnit_TextUI_Command->handleArguments($argv = [0 => '/Users/lalee/Desktop/rollbar-php-wordpress/vendor/bin/phpunit', 1 => '--coverage-clover', 2 => 'build/logs/clover.xml']) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/TextUI/Command.php:111
PHP   5. PHPUnit_Util_Getopt::getopt($args = [0 => '/Users/lalee/Desktop/rollbar-php-wordpress/vendor/bin/phpunit', 1 => '--coverage-clover', 2 => 'build/logs/clover.xml'], $short_options = 'd:c:hv', $long_options = [0 => 'colors==', 1 => 'bootstrap=', 2 => 'columns=', 3 => 'configuration=', 4 => 'coverage-clover=', 5 => 'coverage-crap4j=', 6 => 'coverage-html=', 7 => 'coverage-php=', 8 => 'coverage-text==', 9 => 'coverage-xml=', 10 => 'debug', 11 => 'exclude-group=', 12 => 'filter=', 13 => 'testsuite=', 14 => 'group=', 15 => 'help', 16 => 'include-path=', 17 => 'list-groups', 18 => 'loader=', 19 => 'log-json=', 20 => 'log-junit=', 21 => 'log-tap=', 22 => 'process-isolation', 23 => 'repeat=', 24 => 'stderr', 25 => 'stop-on-error', 26 => 'stop-on-failure', 27 => 'stop-on-incomplete', 28 => 'stop-on-risky', 29 => 'stop-on-skipped', 30 => 'report-useless-tests', 31 => 'strict-coverage', 32 => 'disallow-test-output', 33 => 'enforce-time-limit', 34 => 'disallow-todo-tests', 35 => 'strict-global-state', 36 => 'strict', 37 => 'tap', 38 => 'testdox', 39 => 'testdox-html=', 40 => 'testdox-text=', 41 => 'test-suffix=', 42 => 'no-configuration', 43 => 'no-coverage', 44 => 'no-globals-backup', 45 => 'printer=', 46 => 'static-backup', 47 => 'verbose', 48 => 'version']) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/TextUI/Command.php:240
PHP   6. each($arr = [0 => '--coverage-clover', 1 => 'build/logs/clover.xml']) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/Util/Getopt.php:38

Deprecated: The each() function is deprecated. This message will be suppressed on further calls in /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/Util/Getopt.php on line 38

Call Stack:
    0.0004     399144   1. {main}() /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/phpunit:0
    0.0031     700016   2. PHPUnit_TextUI_Command::main($exit = ???) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/phpunit:52
    0.0031     700128   3. PHPUnit_TextUI_Command->run($argv = [0 => '/Users/lalee/Desktop/rollbar-php-wordpress/vendor/bin/phpunit', 1 => '--coverage-clover', 2 => 'build/logs/clover.xml'], $exit = TRUE) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/TextUI/Command.php:100
    0.0031     700128   4. PHPUnit_TextUI_Command->handleArguments($argv = [0 => '/Users/lalee/Desktop/rollbar-php-wordpress/vendor/bin/phpunit', 1 => '--coverage-clover', 2 => 'build/logs/clover.xml']) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/TextUI/Command.php:111
    0.0033     726320   5. PHPUnit_Util_Getopt::getopt($args = [0 => '/Users/lalee/Desktop/rollbar-php-wordpress/vendor/bin/phpunit', 1 => '--coverage-clover', 2 => 'build/logs/clover.xml'], $short_options = 'd:c:hv', $long_options = [0 => 'colors==', 1 => 'bootstrap=', 2 => 'columns=', 3 => 'configuration=', 4 => 'coverage-clover=', 5 => 'coverage-crap4j=', 6 => 'coverage-html=', 7 => 'coverage-php=', 8 => 'coverage-text==', 9 => 'coverage-xml=', 10 => 'debug', 11 => 'exclude-group=', 12 => 'filter=', 13 => 'testsuite=', 14 => 'group=', 15 => 'help', 16 => 'include-path=', 17 => 'list-groups', 18 => 'loader=', 19 => 'log-json=', 20 => 'log-junit=', 21 => 'log-tap=', 22 => 'process-isolation', 23 => 'repeat=', 24 => 'stderr', 25 => 'stop-on-error', 26 => 'stop-on-failure', 27 => 'stop-on-incomplete', 28 => 'stop-on-risky', 29 => 'stop-on-skipped', 30 => 'report-useless-tests', 31 => 'strict-coverage', 32 => 'disallow-test-output', 33 => 'enforce-time-limit', 34 => 'disallow-todo-tests', 35 => 'strict-global-state', 36 => 'strict', 37 => 'tap', 38 => 'testdox', 39 => 'testdox-html=', 40 => 'testdox-text=', 41 => 'test-suffix=', 42 => 'no-configuration', 43 => 'no-coverage', 44 => 'no-globals-backup', 45 => 'printer=', 46 => 'static-backup', 47 => 'verbose', 48 => 'version']) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/TextUI/Command.php:240
    0.0034     729376   6. each($arr = [0 => '--coverage-clover', 1 => 'build/logs/clover.xml']) /Users/lalee/Desktop/rollbar-php-wordpress/vendor/phpunit/phpunit/src/Util/Getopt.php:38

Error: Looks like you're using PHPUnit 4.8.36. WordPress requires at least PHPUnit 5.4 and is currently only compatible with PHPUnit up to 7.x.
Please use the latest PHPUnit version from the 7.x branch.
Script phpunit --coverage-clover build/logs/clover.xml handling the test event returned with error code 1

[mikemanger]

The old version of PHPUnit is also failing a PHP lint test (php -l) when using PHP 7.3 or above, which is breaking our CI:

PHP Parse error: Invalid body indentation level (expecting an indentation level of at least 1) in ./wp-content/plugins/rollbar/vendor/sebastian/diff/tests/ParserTest.php on line 78

I'd suggest using Yoast's PHPUnit-Polyfills, it downloads the correct PHPUnit version depending on the version of PHP installed. It would also require you to move the composer require to be in require-dev so it can be excluded from your production releases.

[aaronjorbin]

The tests can be run now, though it is now done via package.json and wp-env in order to allow for different PHP versions and different WordPress versions.