sebastianbergmann / phpunit

I'm very excited to see this! We've had to increase our memory limit every so often to deal with test cases sticking around to the end. One thing you might not have considered yet is that by default any exceptions thrown during a test will have a reference to the test case itself in the backtrace.

As I've started working on a new Java project at my job I'm picking up the new JUnit features. One neat feature is the ability to add a TestDecorator to any test case using an annotation. This is how jMock checks its mocks at the end of each method, and a similar docblock annotation could be helpful for PHP tool integrators.

To make it clear: i can not notice the skipped test in the "Test Results" window of netbeans. In the command line output (not the XML) i can see the skipped test. Commandline output and xml should be semantic equal.

The actual issue is that PHPUnit_Util_Log_JUnit ignores PHPUnit_Framework_Warning errors (which is what you get when you specify a @dataProvider that doesn't exist).

For what it's worth, I fixed this in a local branch but it didn't solve the problem with netbeans.

Currently the JUnit result printer will not print the same count for test cases and failures or the same errors as the other result printers if any warnings are encountered. Even after fixing the JUnit result printer to match the other result printers, neatbeans still prints misleading results.

It seems that neatbeans is only looking at the individual test cases for failures and not the overall test suite results. This is problematic as some warnings refer to errors that occur outside of a specific test case. It's possible to create a specific class of warning that relates to warnings that are encountered while trying to execute a specific test case and then handle those in a way that netbeans can recognize but that doesn't solve the overall problem. I think this is something that will have to be fixed in neatbeans as well.

If your data provider returns an empty array your xml looks like this:
<?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="Webapp_Model_PhpUnitTest" file="S:\myproject\test\myproject\application\models\webapp\models\phpunitTest.php" fullPackage="Webapp.Model" tests="1" assertions="1" failures="0" errors="0" time="0.000676"> <testcase name="testSomething" class="Webapp_Model_PhpUnitTest" file="S:\myproject\test\myproject\application\models\webapp\models\phpunitTest.php" line="9" assertions="1" time="0.000676"/> <testsuite name="Webapp_Model_PhpUnitTest::testMyThing" tests="0" assertions="0" failures="0" errors="0" time="0.000000"/> </testsuite> </testsuites>

Netbeans shows test suite "Webapp_Model_PhpUnitTest::testMyThing" as skipped. Sorry i can not add a snapshot here.

When data provider method is missing the xml for the test suite is missing, too. Which makes netbeans not to show an entry for the skipped test (Webapp_Model_PhpUnitTest::testMyThing).

The attributes of the xml element (testsuite)) should be "tests="0" assertions="0" failures="0" errors="0" time="0.000000"" for both:
a) emty array
b) missing data provider method

On command line both should result in "FAILURES! Tests: 2, Assertions: 1, Failures: 1". An empty array makes no sense and is a failure.

Netbeans does not consider the command line output. It considers the xml only. Did you add the missing XML element? If your xml element is equal to <testsuite name="Webapp_Model_PhpUnitTest::testMyThing" tests="0" assertions="0" failures="0" errors="0" time="0.000000"/> then netbeans will recognize it as a skipped test suite.

Okay, I fixed this issue here: http://github.com/whatthejeff/phpunit/tree/topic_13. I made the following changes:

• Data providers that return an empty array now generate a warning that says, 'No tests found in suite.' This is more consistent with how normal test suites behave and will generate a warning in CLI or an empty <testsuite> element in the JUNIT format.
• All other errors that occur while setting up a data provider test suite now generate an empty <testsuite> element in the JUNIT format.

I still think that the JUNIT result printer could be more consistent. Judging from the netbeans source code, netbeans generates the "skipped" message because it knows that an empty test suite normally triggers a warning but it has no access to the actual warning in the JUNIT results. It seems like it would make more sense (and would be more consistent with the actual JUNIT DTD) if warnings were reported inside of the <tesetsuite> elements. For example:

<testsuite name="ATest" tests="0" assertions="0" failures="1" errors="0" time="0.000000">
  <failure type="PHPUnit_Framework_Warning">ATest::testA
No tests found in class "ATest".
  </failure>
</testsuite>

This would make the JUNIT results printer more consistent with the other result printers and it would allow applications like neatbeans the possibility of handling warnings more gracefully.

Would be nice if sebastian will merge this soon.

I hope to see this soon in 3.5.0RCx

Hi whatthejeff,

i can not get the result by using your code. I did a checkout on http://github.com/whatthejeff/phpunit.git and copy the "PHPUnit" folder to the PEAR folder of my XAMPP. But i did not get the <testsuite> element in the result xml (when i used an invalid @dataProvider method name when using the --log-junit). Issue not fixed?

Make sure you checkout the topic_13 branch as this bug is fixed in that branch and not the master branch. Something along the lines of:

git checkout topic_13