phpunit 5 with php7 and xdebug 2.4.0RC3-dev coverage always 100%

[samsonasik]

in favor of sebastianbergmann/phpunit#1996 (comment).

[samsonasik]

This is show the code line is blank/white:

And the bar coverage show green:

I dug for this, found that @runInSeparateProcess in every test make works, but wondering why it need that as the tests covered only factory and using fake object, and in php 5.6, it just works without @runInSeparateProcess.

[samsonasik]

I've gotten reproduce-able case for it, I tried mock SplFileObject in one of my test:

$this->getMock(\SplFileObject::class, [], ['php://memory']);

Interestingly, this issue is only happen when using php7.

[Jean85]

The README.md as of now states that PHPDbg is the only source of raw code coverage for PHP 7. Maybe that's the issue? Or the readme is not updated?

[sebastianbergmann]

The documentation will be updated when Xdebug has a stable release that supports PHP 7.

[paolomainardi]

I confirm the very same problem, no coverage at all but stil see 100%, with php 7.0.2 and xdebug-2.4.0RC3, from what i can understand is used in place of that.

Anyway, find attached the Dockerfile to rebuild the environment, apache+php+xdebug.

FROM php:7.0.2-apache
MAINTAINER paolo.mainardi@sparkfabrik.com
ENV DEBIAN_FRONTEND noninteractive

# Enable apache rewrite.
RUN a2enmod rewrite proxy proxy_http

# Install php packages.
ENV MEMCACHE_PHP7_COMMIT 7ac4e83c6cf4aa8e16f8088534096293ee8e254d
RUN apt-get update \ 
  && apt-get install -y \
  curl \
  supervisor \
  mysql-client \
  libpng12-dev \
  libjpeg-dev \
  libmemcached-dev \
  libmemcached11 \
  libxml2-dev \
  ruby \
  ruby2.1 \
  ruby2.1-dev\
  git \
  unzip \
  && gem install bundler \
  && pecl install xdebug-2.4.0RC3 \
  && git clone https://github.com/php-memcached-dev/php-memcached /usr/src/php/ext/memcached \
  && cd /usr/src/php/ext/memcached && git checkout -b php7 $MEMCACHE_PHP7_COMMIT \
  && docker-php-ext-configure memcached \
  && docker-php-ext-configure gd --with-png-dir=/usr --with-jpeg-dir=/usr \
  && docker-php-ext-install soap gd mbstring pdo pdo_mysql zip memcached \
  && docker-php-ext-enable xdebug  \
  && apt-get autoremove -y \
  && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Copy application.
COPY . /var/www/html/

# Expose apache 
EXPOSE 80 

# Execute daemons.
CMD exec supervisord -n

[Engerim]

With xdebug-2.4.0RC4 seems the code coverage correct

[samsonasik]

@Engerim RC4 still not work in mine

[asgrim]

It's worth pointing out that this was also reported in https://bugs.xdebug.org/view.php?id=1224 (with steps to reproduce using @ramsey's uuid-doctrine library), but it was found not to be an issue in Xdebug.

[sebastianbergmann]

Give me a minimal, self-contained, reproducing test case and I will try to debug your issue. I can only say that I cannot reproduce this with any of my projects with PHP 7.0.3 and Xdebug 2.4.0RC4.

[asgrim]

I already linked to it, but here is the steps anyway (with the correct commit as mentioned in the Xdebug bug report):

I'm experiencing the same issue with PHP 7.0.0 and Xdebug v2.4.0RC3. Here are the steps to reproduce:

git clone https://github.com/ramsey/uuid-doctrine.git
cd uuid-doctrine/
git checkout abbcbe56b023319dea7a72177333089cf47f5a69
composer install
./vendor/bin/phpunit --verbose --coverage-html build/coverage

Open build/coverage/index.html. Click on UuidBinaryType.php and look at around lines 50-52, which should look like this:

return $platform->getBlobTypeDeclarationSQL([
    'length' => '16',
]);

[guenti]

Same issue here:
PHP 7.0.2 with Xdebug v2.4.0RC4

[sebastianbergmann]

@asgrim Sorry, but that is not really "minimal, self-contained". I'll try anyway but not before next week.

[asgrim]

@sebastianbergmann it's pretty minimal, the whole library consists of two src classes, and two test classes :)

[sebastianbergmann]

Okay, now I understand what this report is about.

I thought that you were not getting any code coverage at all. I remember a similar issue that was caused by weird characters in the source code that tripped up the HTML renderer. Will investigate, but no promises when I'll get to that, sorry.

[ramsey]

Thanks, @asgrim, for letting @sebastianbergmann know about this issue. I discussed it yesterday with @derickr, and it was clear I was going to need to open an issue over here about it. :-)

Sebastian, it's difficult to make a very minimal example, since it doesn't seem to be the actual code itself that is the problem. Derick's note here seems applicable: https://bugs.xdebug.org/view.php?id=1224#c3481

[kegi]

I'm running xdebug 2.4.0RC4 and I'm not able to run the code coverage at all : "No whitelist configured, no code coverage will be generated", the same behaviour as if xdebug was missing. Am I missing something ?

[sebastianbergmann]

Yes, you are missing the fact that configuring a whitelist is required.

[ramsey]

@kegi, see https://phpunit.de/manual/current/en/code-coverage-analysis.html#code-coverage-analysis.whitelisting-files

[raziel057]

I confirm the issue with the following configuration:
PHP_CodeCoverage 3.2.0 using PHP 7.0.0-5+deb.sury.org~trusty+1 and PHPUnit 5.2.5

PHP 7.0.0-5+deb.sury.org~trusty+1 (cli) ( NTS )
Copyright (c) 1997-2015 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.4.0RC4, Copyright (c) 2002-2016, by Derick Rethans

[sebastianbergmann]

@raziel057 What you report is something completely different from what @ramsey is reporting.

What is shown in your screenshot, @raziel057, could be correct in case that file is not loaded / executed at all while the tests are running, for instance. Anyway, this should be in a separate ticket with a minimal, self-contained, reproducing example.

[raziel057]

The code is not executed at all but the coverage is 100%, so for me it's a problem.
I don't understand why it's different to the original issue of this ticket.
I will try to provide a minimal example in another ticket...

[sebastianbergmann]

@raziel057 Again: it's a separate issue. And a cosmetic one. 0 executable lines are executed -> 0/0 -> 100%. If you have better idea of representing this information: please open a separate ticket. Thanks!

[paolomainardi]

@sebastianbergmann sorry but that 0/0 -> 100% coverage appear to me just like an error and not a cosmetic issue.

[sebastianbergmann]

@paolomainardi Cosmetic or not, it is something different than is discussed in this issue. Please open a separate issue for if you care about it so deeply.

[paolomainardi]

@sebastianbergmann i'm not really sure that is something different from the original issue, we have different behaviours running the same suite just switching php version from 5 to 7, for example showing 100% coverage.

[Jean85]

@paolomainardi this is not the same. If you look above, @sebastianbergmann posted a screenshot with covered lines highlighted as covered, where there should be none, and 0/0 should be a correct behaviour. The 0/0 -> 100% is, instead, the cosmetic issue.

[ramsey]

On PHP 5.6, @raziel057's code would show up as red and untested, even if it wasn't executed, and the coverage would not show 100%. I think it's the same (or related).

[ramsey]

@Jean85 In the screenshot @sebastianbergmann posted earlier, the green lines are correct. What are incorrect are the lines below the green lines. In PHP 5.6, this same coverage report shows the following, which is how I expect it to look:

[derickr]

It's not a renderer issue. If I remove some tests, then the lines would sometimes appear as they should be. I think some data actually gets thrown away somewhere.

[garex]

@samsonasik Pls create just one class and another file, that calls directly xdebug_* functions. Without any composer.json.

Then to prove it -- run through travis on matrix like php 5.4, 5.5, 5.6 and 7. To prove that it will bugs on 7.

[samsonasik]

@garex with just one class, it got expected result, the issue on highlight is when registering a custom autoloader as @mauryfs mentioned. I don't have idea about xdebug_* functions you meant.

btw, I've enable the travis and this is the travis result:

• for php 5.6 : https://travis-ci.org/samsonasik/highlight/jobs/157633998 ( expected )
• for php 7 : https://travis-ci.org/samsonasik/highlight/jobs/157633999 ( not expected )

[MatthiasKuehneEllerhold]

Just to clarify: the issue happens with the standard composer autoloader too.

[garex]

@samsonasik as you see here only @derickr can actually fix xdebug part. But also you see, that he is very busy man. So to help him maximally — please isolate the issue and show it in virgin state without any phpunit.

As we know it's a xdebug+php7+autoload stuff. So just show us this! Let it be some simple project with composer.json with autoload and test.php file, where you create some class, do nothing and then get code coverage info.

Docs are here: https://xdebug.org/docs/code_coverage

In travis tests with bad coverage should fail when good expected coverage should not fail.

[derickr]

I might have found an issue, where due to a bug the wrong function name was generated. Can you try the following branch: https://github.com/derickr/xdebug/tree/issue1331-code-coverage-segfault ?

[samsonasik]

@derickr I tried your branch and the detail of HighlightBad and HighlightGood still got 100% coverage:

Apples-MBP:highlight samsonasik$ php -v
PHP 7.0.10 (cli) (built: Aug 31 2016 10:25:51) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.10, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.5.0-dev, Copyright (c) 2002-2016, by Derick Rethans

Apples-MBP:highlight samsonasik$ vendor/bin/phpunit --coverage-text
PHPUnit 5.4.7 by Sebastian Bergmann and contributors.

..                                                                  2 / 2 (100%)

Time: 65 ms, Memory: 4.00MB

OK (2 tests, 2 assertions)

Generating code coverage report in HTML format ... done


Code Coverage Report:  
  2016-09-11 18:11:30  

 Summary:              
  Classes: 66.67% (2/3)
  Methods: 80.00% (4/5)
  Lines:   75.00% (6/8)

HighlightBad
  Methods: 100.00% ( 2/ 2)   Lines: 100.00% (  3/  3)
HighlightGood
  Methods: 100.00% ( 2/ 2)   Lines: 100.00% (  3/  3)

[romainneutron]

@derickr I Tried you branch without success

[romainneutron]

Hello,

I've tracked down the issue to Xdebug: in some condition the executable lines are not fully returned, resulting in the same amount of executed line and executable line.

It's pretty difficult to reproduce the issue in a very small amount of code, here's what I've seen using var_dump in code-coverage own code:

In some case (when it works as expected), here's the kind of code coverage data reported by Xdebug:

{
    "36": [
        "Bundle\\MyBundle\\Tests\\Controller\\Debug\\MyAccountControllerTest::testAccountIndex"
    ],
    "37": null,
    "49": [],
    "51": [],
    "52": [],
    "56": [],
    "58": null,
    "70": [],
    "71": null,
    "83": [],
    "84": null,
    "101": [],
    "102": [],
    "105": [],
    "107": [],
    "108": [],
    "110": [],
    "111": [],
    "112": [],
    "113": [],
    "114": [],
    "115": [],
    "116": [],
    "118": null
}

However, when the same test is run in the test suite, here's the code coverage data reported by xdebug:

array(2) {
  [36] =>
  array(1) {
    [0] =>
    string(87) "Bundle\MyBundle\Tests\Controller\Debug\MyAccountControllerTest::testAccountIndex"
  }
  [37] =>
  NULL
}

I hope it would help @derickr to solve the issue. In the meantime, I'm continuing to try to find a small reproducer

[garex]

@romainneutron

the same test is run in the test suite

What about some 'process isolation'? Could it be the issue?

[romainneutron]

No process isolation in my case

[romainneutron]

Reported to Xdebug tracker at https://bugs.xdebug.org/view.php?id=1355

[samsonasik]

its already reported before: https://bugs.xdebug.org/view.php?id=1224 , just need to re-open the issue

[derickr]

Please don't file issues or reopen issues until you have a short reproducible case with which I can reproduce it.

[romainneutron]

@derickr I'd love to get the 10lines reproducer you wish, however it's not trivial at all. I've spent my sunday afternoon to get it without success, just for you.
What I found is that the issue is not in PHP code coverage but in Xdebug. Where ? In Which condition? I still don't know. But I'd rather enjoy a cooperation here with you instead of "please don't report anything that I can not reproduce easily". Sounds a bit harsh and disrespectful for people trying to get the issue fixed.

We could find another way: Let's create a project that reproduce the issue, let's remove code until we get the smaller code that reproduce it.

Have a good monday !

[derickr]

Although I'd like to fix this, I don't know where the problem is either. If I can't reproduce it, I can't fix it. From my point of view, it mostly sounds like an "it doesn't work" bug report.

If you can provide a VM and a few lines that I need to execute to reproduce the problem (with the latest Xdebug's from GitHub), then I can have a play around.

Without a reproducible case, I can't do anything but guess. And guessing never gets things fixed.

[samsonasik]

sample reproducable case like my repo above: https://github.com/samsonasik/highlight detailed at comment: #411 (comment)

[romainneutron]

Sounds @samsonasik provided the reproducer you needed @derickr

[lordelph]

Nice work @samsonasik on that reproducable case. FWIW, I was able to reproduce the problem with that repo using php-code-coverage 4.0.0 / PHP 7.0.12 / Xdebug 2.4.1 / PHPUnit 5.4.7

[o5]

Same issue here :/

PHP 7.0.12-1+deb.sury.org~xenial+1 (cli) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.12-1+deb.sury.org~xenial+1, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.4.1, Copyright (c) 2002-2016, by Derick Rethans
PHPUnit 5.6.2 by Sebastian Bergmann and contributors.

Used processIsolation=true as workaround, but very slow from now.

48 tests, 98 assertions

• with processIsolation=false Time: 3.16 seconds, Memory: 22.00MB
• with processIsolation=true Time: 14.64 seconds, Memory: 6.00MB

[deviantintegral]

In some spot testing, I'm getting different results on a linux installation compared to OS X installed via homebrew.

Homebrew (I just forced it down to 7.0.12 to match). The HTML shows unexecuted code as executed:

PHP 7.0.12 (cli) (built: Nov 28 2016 16:34:11) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.12, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.4.1, Copyright (c) 2002-2016, by Derick Rethans

PHP in the official Drupal docker container, with the same code, shows the report properly:

PHP 7.0.12 (cli) (built: Oct 21 2016 23:27:05) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.12, Copyright (c) 1999-2016, by Zend Technologies
    with Xdebug v2.4.1, Copyright (c) 2002-2016, by Derick Rethans

[Jean85]

Everything seems the same, except for the build... Maybe the PHP build on OS X is somehow botched?

[MatthiasKuehneEllerhold]

@Jean85 Yeah maybe, but we've seen plenty of reports (in this issue!) with native linux PHP environments that are failing.

[sebastianbergmann]

Sorry, but this ticket is a nightmare as multiple issues are discussed.

I have moved the problem that @ramsey reported to a #488.

I close this ticket and lock its conversation now. If any of the other issues discussed here still exist with the current stable version of PHPUnit and php-code-coverage then please open a new ticket. Sorry for any inconvenience this may cause.

[eostis]

Same issue of 100% code coverage on partially tested files solved by upgrading xdebug.

See results from https://xdebug.org/wizard.php below.

Before:

Xdebug installed: 2.4.0
Server API: Command Line Interface
Windows: no
Zend Server: no
PHP Version: 7.0.12-1
Zend API nr: 320151012
PHP API nr: 20151012
Debug Build: no
Thread Safe Build: no
Configuration File Path: /etc/php/7.0/cli
Configuration File: /etc/php/7.0/cli/php.ini
Extensions directory: /usr/lib/php/20151012

After:

Xdebug installed: 2.5.4
Server API: Command Line Interface
Windows: no
Zend Server: no
PHP Version: 7.0.12-1
Zend API nr: 320151012
PHP API nr: 20151012
Debug Build: no
Thread Safe Build: no
Configuration File Path: /etc/php/7.0/cli
Configuration File: /etc/php/7.0/cli/php.ini
Extensions directory: /usr/lib/php/20151012