More info on failure to parse dataProvider (#4439)

* Enable the "disabled" mode for symfony/phpunit-bridge

Deal with disabled mode for symfony/phpunit-bridge in the same way that symfony/phpunit-bridge bootstrap does as seen in symfony/symfony@1b21647#diff-81bfee6017752d99d3119f4ddb1a09edR27

* More info on failure to parse dataProvider

Provide exception file location, line number and message when an thrown during dataProvider execution.

* remove trailing whitespace

* Instead of catching all exceptions, only catch the exceptions that relate to execution of the reflected dataProvider method. The effect of this is to still provide useful information for when the dataProvider isn't specified correctly, but when there is an actual exception thrown in the execution of the dataProvider function it will not be caught and will allow the user to see the problem in the same way as an uncaught exception in an test would on stderr.

* Create new suite based on dummy suite to run these tests in. The dummy suite needs to be able to run without failure and these Cests deliberatley introduce failures so that we can test them.

* Comments to aid understanding of what is going on and removal of redundant tests from stdout.

* Disable the RunFailed extension so that these tests are not re-run later.

* Do not include the test suite that deliberately throws exceptions in the tests for re-running failed tests. Re-running a failed test with an exception results in the exception being thrown in the re-run.

* Make failing tests clean up after themselves so that there are no side effects in other tests. The failing tests seem to end up in the "failed" group and get re-run.

* app.wercker.com and appveyor behave differently in their output of the Application headers when stderr is redirected.

* Revert previous change, have the tests clean up after themselves rather than be ignored here.

* Make directory separators specific to the environment being tested.

* Remove incorrectly included file.

* Do not include the test suite that deliberately throws exceptions in the tests for re-running failed tests. Re-running a failed test with an exception results in the exception being thrown in the re-run.

* Revert "Comments to aid understanding of what is going on and removal of redundant tests from stdout."

This reverts commit ea4ce40.

* Revert "Comments to aid understanding of what is going on and removal of redundant tests from stdout."

This reverts commit ea4ce40.

* undo all the crap with the tests in the wrong place

* Instead of catching all exceptions during reflection and execution of dataProvider methods, only catch the exceptions that relate to execution of the reflected dataProvider method. The effect of this is to still provide useful information for when the dataProvider isn't specified correctly, but when there is an actual exception thrown in the execution of the dataProvider function it will not be caught and will allow the user to see the problem in the same way as an uncaught exception in an test would on stderr.

* PHPUnit version numbers can include alphas, dashes and possibly more. Regexp adjusted to accept any characters as version number.

src/Codeception/Test/Loader/Cest.php

@@ -65,7 +65,7 @@ function ($v) {
                         $data = ReflectionHelper::invokePrivateMethod($unit, $dataMethod);
                         // allow to mix example and dataprovider annotations
                         $examples = array_merge($examples, $data);
-                    } catch (\Exception $e) {
+                    } catch (\ReflectionException $e) {
                         throw new TestParseException(
                             $file,
                             "DataProvider '$dataMethod' for $testClass->$method is invalid or not callable.\n" .

tests/cli/DataProviderFailuresAndExceptionsCest.php

@@ -0,0 +1,176 @@
+<?php
+
+class DataProviderFailuresAndExceptionsCest
+{
+
+    /**
+     * @param CliGuy $I
+     */
+    protected function moveToPath(\CliGuy $I)
+    {
+        $I->amInPath('tests/data/dataprovider_failures_and_exceptions');
+    }
+
+    /**
+     * This looks at only the contents of stdout when there is a failure in parsing a dataProvider annotation.
+     * When there is a failure all the useful information should go to stderr, so stdout is left with
+     * only the version headers.
+     *
+     * @param CliGuy $I
+     * @before moveToPath
+     */
+    public function runTestWithDataProvidersFailureStdout(\CliGuy $I)
+    {
+        /**
+         * On windows /dev/null is NUL so detect running OS and return the appropriate string for redirection.
+         * As some systems have php_uname and co disabled, we use the DIRECTORY_SEPARATOR constant to
+         * figure out if we are running on windows or not.
+         */
+        $devNull = (DIRECTORY_SEPARATOR === '\\')?'NUL':'/dev/null';
+        $I->executeCommand('run -n -v unit DataProvidersFailureCest 2> '.$devNull,false);
+        // We should only see the version headers in stdout when there is this kind of failure.
+        $I->canSeeShellOutputMatches('/^Codeception PHP Testing Framework v[0-9\.]+\nPowered by PHPUnit .+ by Sebastian Bergmann and contributors\.$/');
+        $I->seeResultCodeIs(1);
+    }
+
+    /**
+     * This redirects stderr to stdout so that we can test the contents of stderr. Stderr is where all the interesting
+     * information should be when there is a failure.
+     *
+     * @param CliGuy $I
+     * @before moveToPath
+     */
+    public function runTestWithDataProvidersFailureStderr(\CliGuy $I)
+    {
+        $I->executeCommand('run -n unit DataProvidersFailureCest 2>&1',false);
+        $I->seeInShellOutput('[Codeception\Exception\TestParseException]');
+        $I->seeInShellOutput('Couldn\'t parse test');
+        $I->seeInShellOutput('DataProvider \'rectangle\' for DataProvidersFailureCest->testIsTriangle');
+        $I->seeInShellOutput('Make sure that the dataprovider exist within the test class.');
+        // For Unit tests PHPUnit throws the errors, this confirms that we haven't ended up running PHPUnit test Loader
+        $I->dontSeeInShellOutput('PHPUnit_Framework_Warning');
+        $I->dontSeeInShellOutput('The data provider specified for DataProvidersFailureCest::testIsTriangle');
+        $I->dontSeeInShellOutput('Method rectangle does not exist');
+        $I->dontSeeInShellOutput('FAILURES!');
+        $I->dontSeeInShellOutput('WARNINGS!');
+        $I->dontSeeInShellOutput('OK');
+        $I->dontSeeInShellOutput('Tests: 1, Assertions: 0, Warnings: 1.');
+        // In normal mode the Exception trace should not appear.
+        $I->dontSeeInShellOutput('Exception trace');
+        $I->dontSeeInShellOutput('Test'.DIRECTORY_SEPARATOR.'Loader'.DIRECTORY_SEPARATOR.'Cest.php:');
+        $I->seeResultCodeIs(1);
+    }
+
+
+    /**
+     * This adds the -v to the stderr test which should just add the Exception Trace to the output.
+     *
+     * @param CliGuy $I
+     * @before moveToPath
+     */
+    public function runTestWithDataProvidersFailureStderrVerbose(\CliGuy $I)
+    {
+        $I->executeCommand('run -n unit DataProvidersFailureCest -v 2>&1',false);
+        $I->seeInShellOutput('[Codeception\Exception\TestParseException]');
+        $I->seeInShellOutput('Couldn\'t parse test');
+        $I->seeInShellOutput('DataProvider \'rectangle\' for DataProvidersFailureCest->testIsTriangle');
+        $I->seeInShellOutput('Make sure that the dataprovider exist within the test class.');
+        // For Unit tests PHPUnit throws the errors, this confirms that we haven't ended up running PHPUnit test Loader
+        $I->dontSeeInShellOutput('PHPUnit_Framework_Warning');
+        $I->dontSeeInShellOutput('The data provider specified for DataProvidersFailureCest::testIsTriangle');
+        $I->dontSeeInShellOutput('Method rectangle does not exist');
+        $I->dontSeeInShellOutput('FAILURES!');
+        $I->dontSeeInShellOutput('WARNINGS!');
+        $I->dontSeeInShellOutput('OK');
+        $I->dontSeeInShellOutput('Tests: 1, Assertions: 0, Warnings: 1.');
+        // In verbose mode the Exception trace should be output.
+        $I->seeInShellOutput('Exception trace');
+        $I->seeInShellOutput('Test'.DIRECTORY_SEPARATOR.'Loader'.DIRECTORY_SEPARATOR.'Cest.php:');
+        $I->seeResultCodeIs(1);
+
+    }
+
+    /**
+     * This looks at only the contents of stdout when there is an exception thrown when executing a dataProvider
+     * function.
+     * When exception thrown all the useful information should go to stderr, so stdout is left with nothing.
+     *
+     * @param CliGuy $I
+     * @before moveToPath
+     */
+    public function runTestWithDataProvidersExceptionStdout(\CliGuy $I)
+    {
+        /**
+         * On windows /dev/null is NUL so detect running OS and return the appropriate string for redirection.
+         * As some systems have php_uname and co disabled, we use the DIRECTORY_SEPARATOR constant to
+         * figure out if we are running on windows or not.
+         */
+        $devNull = (DIRECTORY_SEPARATOR === '\\')?'NUL':'/dev/null';
+        $I->executeCommand('run -n unit DataProvidersExceptionCest -v 2> '.$devNull, false);
+        // Depending on the test environment, we either see nothing or just the headers here.
+        $I->canSeeShellOutputMatches('/^Codeception PHP Testing Framework v[0-9\.]+\nPowered by PHPUnit .+ by Sebastian Bergmann and contributors\.$/');
+        $I->seeResultCodeIs(1);
+    }
+
+    /**
+     * This redirects stderr to stdout so that we can test the contents of stderr. Stderr is where all the interesting
+     * information should be when there is a failure.
+     *
+     * @param CliGuy $I
+     * @before moveToPath
+     */
+    public function runTestWithDataProvidersExceptionStderr(\CliGuy $I)
+    {
+        $I->executeCommand('run -n unit DataProvidersExceptionCest 2>&1', false);
+        // For Unit tests PHPUnit throws the errors, this confirms that we haven't ended up running PHPUnit test Loader
+        $I->dontSeeInShellOutput('There was 1 warning');
+        $I->dontSeeInShellOutput('PHPUnit_Framework_Warning');
+        $I->dontSeeInShellOutput('The data provider specified for DataProvidersExceptionTest::testIsTriangle');
+        $I->dontSeeInShellOutput('FAILURES!');
+        $I->dontSeeInShellOutput('WARNINGS!');
+        $I->dontSeeInShellOutput('OK');
+        // We should not see the messages related to a failure to parse the dataProvider function
+        $I->dontSeeInShellOutput('[Codeception\Exception\TestParseException]');
+        $I->dontSeeInShellOutput('Couldn\'t parse test');
+        $I->dontSeeInShellOutput('DataProvider \'rectangle\' for DataProvidersFailureCest->testIsTriangle ');
+
+        // We should just see the exception and the message
+        $I->seeInShellOutput('[Exception]');
+        $I->seeInShellOutput('Something went wrong!!!');
+        // We don't have the verbose flag set, so there should be no trace.
+        $I->dontSeeInShellOutput('Exception trace:');
+        $I->dontSeeInShellOutput('DataProvidersExceptionCest');
+        $I->seeResultCodeIs(1);
+
+    }
+
+    /**
+     * This adds the -v to the stderr test which should just add the Exception Trace to the output of stderr.
+     *
+     * @param CliGuy $I
+     * @before moveToPath
+     */
+    public function runTestWithDataProvidersExceptionStderrVerbose(\CliGuy $I)
+    {
+        $I->executeCommand('run -n unit DataProvidersExceptionCest -v 2>&1', false);
+        // For Unit tests PHPUnit throws the errors, this confirms that we haven't ended up running PHPUnit test Loader
+        $I->dontSeeInShellOutput('There was 1 warning');
+        $I->dontSeeInShellOutput('PHPUnit_Framework_Warning');
+        $I->dontSeeInShellOutput('The data provider specified for DataProvidersExceptionTest::testIsTriangle');
+        $I->dontSeeInShellOutput('FAILURES!');
+        $I->dontSeeInShellOutput('WARNINGS!');
+        $I->dontSeeInShellOutput('OK');
+        // We should not see the messages related to a failure to parse the dataProvider function
+        $I->dontSeeInShellOutput('[Codeception\Exception\TestParseException]');
+        $I->dontSeeInShellOutput('Couldn\'t parse test');
+        $I->dontSeeInShellOutput('DataProvider \'rectangle\' for DataProvidersFailureCest->testIsTriangle is ');
+
+        // We should just see the exception and the message
+        $I->seeInShellOutput('[Exception]');
+        $I->seeInShellOutput('Something went wrong!!!');
+        // We have the verbose flag set, so there should be a trace.
+        $I->seeInShellOutput('Exception trace:');
+        $I->seeInShellOutput('DataProvidersExceptionCest');
+        $I->seeResultCodeIs(1);
+    }
+}

tests/data/dataprovider_failures_and_exceptions/codeception.yml

@@ -0,0 +1,10 @@
+paths:
+    tests: tests
+    output: tests/_output
+    data: tests/_data
+    support: tests/_support
+    envs: tests/_envs
+actor_suffix: Tester
+extensions:
+    enabled:
+        - Codeception\Extension\RunFailed

tests/data/dataprovider_failures_and_exceptions/tests/_support/UnitTester.php

@@ -0,0 +1,26 @@
+<?php
+
+
+/**
+ * Inherited Methods
+ * @method void wantToTest($text)
+ * @method void wantTo($text)
+ * @method void execute($callable)
+ * @method void expectTo($prediction)
+ * @method void expect($prediction)
+ * @method void amGoingTo($argumentation)
+ * @method void am($role)
+ * @method void lookForwardTo($achieveValue)
+ * @method void comment($description)
+ * @method \Codeception\Lib\Friend haveFriend($name, $actorClass = NULL)
+ *
+ * @SuppressWarnings(PHPMD)
+*/
+class UnitTester extends \Codeception\Actor
+{
+    use _generated\UnitTesterActions;
+
+   /**
+    * Define custom actions here
+    */
+}

tests/data/dataprovider_failures_and_exceptions/tests/_support/_generated/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

tests/data/dataprovider_failures_and_exceptions/tests/unit.suite.yml

@@ -0,0 +1 @@
+actor: UnitTester

tests/data/dataprovider_failures_and_exceptions/tests/unit/DataProvidersExceptionCest.php

@@ -0,0 +1,21 @@
+<?php
+
+class DataProvidersExceptionCest
+{
+
+    /**
+     * @dataProvider triangles
+     */
+    public function testIsTriangle(UnitTester $I)
+    {
+        $I->amGoingTo("Fail with an exception before I even get here");
+    }
+
+    // The test of this relies upon the line numbers being unchanged. If you do need to add lines
+    // please change the relevant test in tests/cli/RunCest:runTestWithDataProvidersExceptionStderrVerbose
+    public function triangles()
+    {
+        throw new \Exception("Something went wrong!!!");
+    }
+
+}

tests/data/dataprovider_failures_and_exceptions/tests/unit/DataProvidersFailureCest.php

@@ -0,0 +1,23 @@
+<?php
+
+class DataProvidersFailureCest
+{
+
+    /**
+     * @dataProvider rectangle
+     */
+    public function testIsTriangle(UnitTester $I)
+    {
+        $I->amGoingTo("Fail before I get here.");
+    }
+
+    public function triangles()
+    {
+        return array(
+            'real triangle' => array(3,4,5),
+            array(10,12,5),
+            array(7,10,15)
+        );
+    }
+
+}