Missing Reference entries not detected by standard suite tests

[llaville]

While I'm begin to rewrite (adapt code) to PHP 7+ for a next major release 2.0, I found that some entries was missing in References.

Curiously, it was not detected by the standard GenericTest class, even when we run all tests.

Exemple with PHP 7.1.21 on my linux laptop

Time: 1.06 minutes, Memory: 96.00MB

There were 442 skipped tests:

OK, but incomplete, skipped, or risky tests!
Tests: 8040, Assertions: 6131, Skipped: 442.

At the same times, I wanted to test paratest to speed up time elapsed.
While standard test suite did not detect missing entries, paratest was able to do it !

I've then decided to rewrite GenericTest to provide extension entries as a new data provider, that will be able to show us what are missing !

[llaville]

Of course, the fix will also be added to the branch 1.x

[llaville]

Additionnal Information about GenericTest with standard phpunit.xml.dist config.

When I run full test suites, we got an OK (see above), but when we run individal tests like for DateExtension we got errors such as :

PHPUnit 7.3.2 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.1.21 with Xdebug 2.6.1
Configuration: /shared/httpd/php-compatinfo-db/phpunit.xml.dist

...............................................................  63 / 100 ( 63%)
.......F...........................F.                           100 / 100 (100%)

Time: 874 ms, Memory: 10.00MB

There were 2 failures:

1) Bartlett\Tests\CompatInfoDb\Reference\Extension\DateExtensionTest::testGetConstantsFromExtension
Defined constant 'DATE_RFC7231' not known in Reference.
Failed asserting that an array has the key 'DATE_RFC7231'.

/shared/httpd/php-compatinfo-db/tests/Reference/GenericTest.php:626

2) Bartlett\Tests\CompatInfoDb\Reference\Extension\DateExtensionTest::testGetClassConstantsFromExtension
Defined class constant 'DateTime::RFC7231' not known in Reference.
Failed asserting that an array has the key 'RFC7231'.

/shared/httpd/php-compatinfo-db/tests/Reference/GenericTest.php:828

FAILURES!
Tests: 100, Assertions: 184, Failures: 2.

[llaville]

Issue is not yet solved, but performance speed was improved by factor 6.
See commit a9536a9

PHPUnit 7.3.5 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.1.21 with Xdebug 2.6.1
Configuration: /shared/httpd/php-compatinfo-db/phpunit.xml.dist

Time: 9.54 seconds, Memory: 96.00MB

OK, but incomplete, skipped, or risky tests!
Tests: 8040, Assertions: 6131, Skipped: 442.

[llaville]

Good news ! I've found origin of issue and will publish soon a new release with fix (unit tests improved with new data providers to detect missing elements).
As there are a lot of elements missing in references definition, I need to fix it too, before adding support of latest PHP versions released on september 13rd.

[llaville]

In commit 90edebd, I've :

Added some new data provider to check real content of each extensions with provideExtensionValues

Added some new tests

• testGetIniEntriesFromExtension

• testGetFunctionsFromExtension

• _testGetConstantsFromExtension

• _testGetClassesFromExtension

• _testGetClassMethodsFromExtension

• _testGetClassConstantsFromExtension

• _testGetInterfacesFromExtension

Where tests prefixed with an underscore are not yet fully tested with all extensions (this is the reason why there are deactivated temporary)

[llaville]

Current tests status is :

PHPUnit 6.5.13 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.1.21 with Xdebug 2.6.1
Configuration: /shared/httpd/php-compatinfo-db/phpunit.xml.dist

Time: 4.58 seconds, Memory: 118.00MB


OK, but incomplete, skipped, or risky tests!
Tests: 9972, Assertions: 2700, Skipped: 559.

[llaville]

Fix tests below , with commit d8b12f4 :

• testGetClassConstantsFromExtension

• testGetInterfacesFromExtension

That give following results :

PHPUnit 6.5.13 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.1.21 with Xdebug 2.6.1
Configuration: /shared/httpd/php-compatinfo-db/phpunit.xml.dist
Time: 5.13 seconds, Memory: 142.00MB

OK, but incomplete, skipped, or risky tests!
Tests: 11558, Assertions: 4104, Skipped: 756.

[llaville]

Latest tests with PHP 5.6.38 and PHPUnit 5.7.27

Time: 8.55 seconds, Memory: 404.25MB

OK, but incomplete, skipped, or risky tests!
Tests: 11401, Assertions: 3753, Skipped: 731.

Latest tests with PHP 7.0.32 and PHPUnit 6.5.13

Time: 5.03 seconds, Memory: 146.00MB

OK, but incomplete, skipped, or risky tests!
Tests: 11547, Assertions: 4137, Skipped: 756.

Latest tests with PHP 7.1.22 and PHPUnit 6.5.13

Time: 5.23 seconds, Memory: 142.00MB

OK, but incomplete, skipped, or risky tests!
Tests: 11560, Assertions: 4104, Skipped: 756.

Reminding tests from extension will be fixed is next release 1.37
Will publish a version 1.36 in next minutes