Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Risky test not counted on default console printer #1677

Closed
llaville opened this issue Apr 13, 2015 · 1 comment

Comments

@llaville
Copy link

commented Apr 13, 2015

I've run tests with PHPUnit 4.5.1 and 4.6.4 and get same issue on both versions.
Risky test not counted on final results.

Expected

FAILURES!
Tests: 4, Assertions: 2, Failures: 1, Incomplete: 1, Skipped: 1, Risky: 1.

Actual

FAILURES!
Tests: 4, Assertions: 2, Failures: 1, Incomplete: 1, Skipped: 1.

My simple demo test suite is testSuite.php

<?php

namespace Your\Name_Space;

class YourTestSuite extends \PHPUnit_Framework_TestCase
{
    public function testIncomplete()
    {
        // Optional: Test anything here, if you want.
        $this->assertTrue(TRUE, 'This should already work.');

        // Stop here and mark this test as incomplete.
        $this->markTestIncomplete(
          'This test has not been implemented yet.'
        );
    }

    public function testRisky()
    {
    }

    public function testSkipped()
    {
        $this->markTestSkipped('This test was skipped for any reason.');
    }

    public function testFailure()
    {
        $this->assertEmpty(array('foo'));
    }
}

My phpunit.xml config file is

<?xml version="1.0" encoding="UTF-8"?>
<phpunit 
    backupGlobals="true"
    backupStaticAttributes="false"
    bootstrap=""
    colors="false"
    stopOnError="false"
    stopOnFailure="false"
    stopOnIncomplete="false"
    stopOnRisky="false"
    stopOnSkipped="false"
    strict="true"
    verbose="false"
>
    <listeners>
        <listener class="Foo\SimpleTestListener">
        </listener>
    </listeners>
    <testsuites>
        <testsuite name="Demo Test Suite">
            <file>testSuite.php</file>
        </testsuite>
    </testsuites>    
</phpunit>

Invoking command

phpunit-4.5.1.phar  -c phpunit.xml

Give

PHPUnit 4.5.1 by Sebastian Bergmann and contributors.

Configuration read from C:\var\phpunit\issues\riskyTest\phpunit.xml

Deprecated configuration setting "strict" used

IRSF

Time: 82 ms, Memory: 4.00Mb

There was 1 failure:

1) Your\Name_Space\YourTestSuite::testFailure
Failed asserting that an array is empty.

C:\var\phpunit\issues\riskyTest\testSuite.php:29

FAILURES!
Tests: 4, Assertions: 2, Failures: 1, Incomplete: 1, Skipped: 1.

Invoking command

phpunit-4.6.4.phar  -c phpunit.xml --report-useless-tests

Give

PHPUnit 4.6.4 by Sebastian Bergmann and contributors.

Configuration read from C:\var\phpunit\issues\riskyTest\phpunit.xml

Deprecated configuration setting "strict" used

IRSF

Time: 344 ms, Memory: 8.25Mb

There was 1 failure:

1) Your\Name_Space\YourTestSuite::testFailure
Failed asserting that an array is empty.

C:\var\phpunit\issues\riskyTest\testSuite.php:29

FAILURES!
Tests: 4, Assertions: 2, Failures: 1, Incomplete: 1, Skipped: 1.

But, the test listener count all tests (including risky).

Add the simple Foo\SimpleTestListener class in a file named SimpleTestListener.php (already defined in phpunit.xml)

<?php

namespace Foo;

use PHPUnit_Util_Printer;
use PHPUnit_Framework_TestListener;
use PHPUnit_Framework_Test;
use PHPUnit_Framework_TestSuite;
use PHPUnit_Framework_AssertionFailedError;
use Exception;

class SimpleTestListener implements PHPUnit_Framework_TestListener
{
    public function addError(PHPUnit_Framework_Test $test, Exception $e, $time)
    {
        printf("Error while running test '%s'.\n", $test->getName());
    }
    public function addFailure(PHPUnit_Framework_Test $test, PHPUnit_Framework_AssertionFailedError $e, $time)
    {
        printf("Test '%s' failed.\n", $test->getName());
    }
    public function addIncompleteTest(PHPUnit_Framework_Test $test, Exception $e, $time)
    {
        printf("Test '%s' is incomplete.\n", $test->getName());
    }
    public function addRiskyTest(PHPUnit_Framework_Test $test, Exception $e, $time)
    {
        printf("Test '%s' is deemed risky.\n", $test->getName());
    }
    public function addSkippedTest(PHPUnit_Framework_Test $test, Exception $e, $time)
    {
        printf("Test '%s' has been skipped.\n", $test->getName());
    }
    public function startTest(PHPUnit_Framework_Test $test)
    {
        printf("Test '%s' started.\n", $test->getName());
    }
    public function endTest(PHPUnit_Framework_Test $test, $time)
    {
        printf("Test '%s' ended.\n", $test->getName());        
    }
    public function startTestSuite(PHPUnit_Framework_TestSuite $suite)
    {
        printf("TestSuite '%s' started with %d tests.\n", $suite->getName(), $suite->count());
    }
    public function endTestSuite(PHPUnit_Framework_TestSuite $suite)
    {
        printf("TestSuite '%s' ended.\n", $suite->getName());
    }
}

The bootstrap file

<?php
require_once __DIR__ . DIRECTORY_SEPARATOR . 'SimpleTestListener.php';

And invoke command with both versions (4.5.1 or 4.6.4)

phpunit.phar -c phpunit.xml --report-useless-tests --bootstrap bootstrap.php

Give such output

TestSuite 'Demo Test Suite' started with 4 tests.
TestSuite 'Your\Name_Space\YourTestSuite' started with 4 tests.
Test 'testIncomplete' started.
Test 'testIncomplete' is incomplete.
ITest 'testIncomplete' ended.
Test 'testRisky' started.
Test 'testRisky' is deemed risky.
RTest 'testRisky' ended.
Test 'testSkipped' started.
Test 'testSkipped' has been skipped.
STest 'testSkipped' ended.
Test 'testFailure' started.
Test 'testFailure' failed.
FTest 'testFailure' ended.
TestSuite 'Your\Name_Space\YourTestSuite' ended.
TestSuite 'Demo Test Suite' ended.

My Platform : Windows / PHP 5.6.7

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

commented Apr 28, 2015

This only seems to happen when there is at least one failing test:

With failing test

<?php
class Test extends PHPUnit_Framework_TestCase
{
    public function testSuccess()
    {
        $this->assertTrue(true);
    }

    public function testFailure()
    {
        $this->assertTrue(false);
    }

    public function testIncomplete()
    {
        $this->markTestIncomplete();
    }

    public function testSkipped()
    {
        $this->markTestSkipped();
    }

    public function testRisky()
    {
    }
}
phpunit --report-useless-tests Test
PHPUnit 4.6.4-2-gbc3ce9c by Sebastian Bergmann and contributors.

.FISR

Time: 93 ms, Memory: 4.50Mb

There was 1 failure:

1) Test::testFailure
Failed asserting that false is true.

/home/sb/Test.php:11

FAILURES!
Tests: 5, Assertions: 2, Failures: 1, Incomplete: 1, Skipped: 1.

Without failing test

<?php
class Test extends PHPUnit_Framework_TestCase
{
    public function testSuccess()
    {
        $this->assertTrue(true);
    }

    public function testIncomplete()
    {
        $this->markTestIncomplete();
    }

    public function testSkipped()
    {
        $this->markTestSkipped();
    }

    public function testRisky()
    {
    }
}
phpunit --report-useless-tests Test
PHPUnit 4.6.4-2-gbc3ce9c by Sebastian Bergmann and contributors.

.ISR

Time: 95 ms, Memory: 4.50Mb

OK, but incomplete, skipped, or risky tests!
Tests: 4, Assertions: 1, Incomplete: 1, Skipped: 1, Risky: 1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.