Lines not covered when more tests are run

[kubawerlos]

Hello,

I've spot an issue that some lines are not covered when tests are run together with other ones.

Easiest way to reproduce is to clone this repository: git clone git@github.com:kubawerlos/php-cs-fixer-custom-fixers.git.

When running vendor/bin/phpunit tests/Fixer all fixers get covered, which is expected.

But when running vendor/bin/phpunit tests some fixers don't get covered fully (but only some) and always the problematic is function isRisky, e.g. InternalClassCasingFixer::isRisky.

I've tested with PCOV v1.0.3 and phpunit/php-code-coverage 7.0.4.

[krakjoe]

You seem to have a non-standard directory setup ...

php -dpcov.directory=. -dpcov.exclude="~vendor~" vendor/bin/phpunit tests

That's likely what you are looking for ...

[kubawerlos]

What do you mean by "non-standard"? The src-dev directory? I have removed it and all references to it and still facing the issue. It was all fine a month ago -> https://coveralls.io/builds/23169688 (AFAIR it was PCOV v1.0.2 then).

However, your suggestion fixes it, thanks 👍

[krakjoe]

It looks like you're a victim of the last bug in the PHP part of the driver, which did not clear data appropriately, it just so happens you got expected results by accident. If you look carefully at those old results you will probably find anomolies like lines reported as covered by tests that did not execute those lines.

…

On Sun, 2 Jun 2019, 17:47 Kuba Werłos, ***@***.***> wrote: What do you mean by "non-standard"? The src-dev directory? I have removed it and all references to it and still facing the issue. It was all fine a month ago -> https://coveralls.io/builds/23169688 (AFAIR it was PCOV v1.0.2 then). However, your suggestion fixes it, thanks 👍 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#17?email_source=notifications&email_token=AARB52W6V7FGPAEDB36QPN3PYPTQFA5CNFSM4HSCZZMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWXYMKA#issuecomment-498042408>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARB52SO6TLI4HWGXH6CERDPYPTQFANCNFSM4HSCZZMA> .

[kubawerlos]

Doesn't seems so - there are tests that covers lines that are problematic right now - e.g. https://github.com/kubawerlos/php-cs-fixer-custom-fixers/blob/master/tests/Fixer/InternalClassCasingFixerTest.php#L19 - and as I mentioned, when running only tests from subdirectory tests/Fixer all is covered fine.

[krakjoe]

You're quite right, something doesn't look right ... (I'm at my keyboard properly now, looking) ...

[krakjoe]

Can you please try the tip of develop (currently 1.1.0-dev) and patch vendor/phpunit/php-code-coverage/src/Driver/PCOV.php with this stop method:

    /**
     * Stop collection of code coverage information.
     */
    public function stop(): array
    {
        \pcov\stop();

        $waiting  = \pcov\waiting();
        $collect  = [];

        if ($waiting) {
            $collect = \pcov\collect(\pcov\inclusive, $waiting);

            if ($collect) {
                \pcov\clear(\pcov\coverage | \pcov\waiting);
            } else {
                \pcov\clear(\pcov\waiting);
            }
        }

        return $collect;
    }

I believe this produces correct results ?

[kubawerlos]

Yes, it looks so, thanks for a help!

PR: https://github.com/kubawerlos/php-cs-fixer-custom-fixers/pull/98/files
Build: https://travis-ci.org/kubawerlos/php-cs-fixer-custom-fixers/jobs/540456641

[krakjoe]

I managed to go one better, you no longer need to patch the PHP driver, just update pcov ... 1.0.4-dev (managed to not break compat, so don't need to bump to 1.1) ...

[krakjoe]

Sorry about all the mess ...

[kubawerlos]

Sorry about all the mess ...

No need to, thanks for the very quick fix 👍

[krakjoe]

Very welcome.

[deyvisonrocha]

@krakjoe Can you get folders to coverage from the phpunit.xml?

[BHSPitMonkey]

I think something is still off here, and possibly related to #31 and #38 (my experience lines up with all of these issues). I have a <filter><whitelist> defined in phpunit.xml with some <directory> nodes and an <exclude> node, and those get processed and sent to the pcov driver by phpunit via sebastianbergmann/php-code-coverage. After executing a simple test (using either phpunit or paratest), the resulting test reports show everything as uncovered (including specific lines I know were executed).

If I phpdismod pcov && phpenmod xdebug, my simple test works exactly as expected. Surely php-code-coverage is sending the filter/whitelist to pcov properly, since I can see the source files/methods/lines listed in the coverage report output -- but they're all marked 0%.

[krakjoe]

To clarify, php-code-coverage doesn't "send" or pass anything from phpunit configuration to pcov. pcov must be configured properly with pcov.directory.

[BHSPitMonkey]

Right you are... I came to that realization after some additional digging. It really seems like there needs to be a runtime way to set the "directory" root instead of exclusively through ini, since the current scheme gives php-code-coverage no way of knowing that parts of its whitelist will not be getting tracked in pcov due to its "magic" approach of deciding on a scope automatically. To make things worse, the results come back confidently wrong instead of providing an error to the user, and out of the box will give a multi-directory scan incorrect results compared with the same PHPUnit config given to Xdebug instead.

…

On Wed, Dec 11, 2019, 19:57 Joe Watkins ***@***.***> wrote: To clarify, php-code-coverage doesn't "send" or pass anything from phpunit configuration to pcov. pcov must be configured properly with pcov.directory. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#17?email_source=notifications&email_token=AAAIHCAG5ZBNMMMDXIXXFRLQYGZEPA5CNFSM4HSCZZMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGVMHLQ#issuecomment-564839342>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAIHCDUQRFYW7FT7DRV2LDQYGZEPANCNFSM4HSCZZMA> .

[miceno]

@BHSPitMonkey I agree with you, I expend two days testing pcov with no coverage results due to this issue... I finally found the description here and figured it out to use -dpcov.directory.

[trip-somers]

Is it possible to run coverage on two folders? I basically have two /app folders -- one is legacy, and one is modern -- and it looks like the only alternative I have is to set directory to . and then exclude everything except those two folders. Is that really what I'd have to do?

EDIT: It looks like this works fine after just setting it to .. From there, it appears that the section of the phpunit.xml config is handling the rest of what I need.