Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect coverage reported #526

Closed
vpassapera opened this Issue May 11, 2017 · 27 comments

Comments

Projects
None yet
8 participants
@vpassapera
Copy link

vpassapera commented May 11, 2017

Q A
php-code-coverage version 5.2.1
PHP version 7.0.19
Driver Xdebug
Xdebug version (if used) v2.5.3
Installation Method Composer
Usage Method PHPUnit
PHPUnit version 6.1.3

Today, after doing a composer update, my coverage guards broke, when phpunit started reporting that my coverage had dropped to 88% from 100%. (I started with phpunit version 5.6.x, and upgraded to phpunit 6.1 after I saw the bug, in hopes of it being fixed, seems to be present in all recent versions including the 5.x line)

Since no code had changed this seemed incorrect (only had done a composer update, no code changes).

Further investigation shows that PHPUnit is incorrectly reporting random arguments in methods and functions as not covered (even though they are).

Example:

example

In the example above, you can see that 4 tests actually cover that method, but it is saying that 1 argument is not covered. That seems impossible, since I have to call that method with those arguments. Especially since these tests are actually functional tests. Previously, these tests showed full coverage accross all metrics (Line, Functions and Methods, Classes and Traits), now, Lines show 100% but not Functions and Methods or Classes and Traits.

Example of one of the tests:

    public function testPostManifestNotification()
    {
        $payload = ['emails' => [
            'mysupercoolemail@mysupercooladdress.com',
        ]];
        /** @var Branch $branch */
        $branch = $this->referenceRepository->getReference('release/COM170401.0');
        $path = sprintf(
            '/manifests/%s/actions/notify.json',
            StringType::create($branch->getFullName())->replace('/', '-')->toString()
        );
        $this->client->request('POST', $path, $payload);
        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_ACCEPTED, $response->getStatusCode());
    }

As you can see, this is fully a functional test that should cover all the lines. PHPUnit reports they are not covered?

This is causing all my reports and CI to break, since it uses phpunit to determine coverage thresholds.

My phpunit config

<?xml version="1.0" encoding="UTF-8"?>

<!-- https://phpunit.de/manual/current/en/appendixes.configuration.html -->
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/5.7/phpunit.xsd"
     backupGlobals="false"
     colors="true"
     convertWarningsToExceptions="true"
     convertErrorsToExceptions="true"
     convertNoticesToExceptions="true"
     bootstrap="app/autoload.php">
    <php>
        <ini name="error_reporting" value="-1" />
        <server name="KERNEL_DIR" value="app/" />
    </php>

    <testsuites>
        <testsuite name="Isengard Tests">
            <directory suffix="Test.php">tests</directory>
        </testsuite>
    </testsuites>

    <filter>
        <whitelist>
            <directory>.</directory>
            <exclude>
                <directory>./.hooks</directory>
                <directory>./app</directory>
                <directory>./bin</directory>
                <directory>./var</directory>
                <directory>./src/*Bundle/Resources</directory>
                <directory>./src/*Bundle/DataFixtures</directory>
                <directory>./src/AppBundle/Entity</directory>
                <directory>./src/*/*Bundle/Resources</directory>
                <directory>./src/*/Bundle/*Bundle/Resources</directory>
                <directory>./vendor</directory>
                <directory>./web</directory>
                <directory>./tests</directory>
                <file>RoboFile.php</file>
            </exclude>
        </whitelist>
    </filter>
    <logging>
        <log type="coverage-clover" target="build/phpunit/logs/clover.xml" />
        <log type="coverage-crap4j" target="build/phpunit/logs/crap4j.xml" />
        <log type="coverage-html" target="build/phpunit/html"/>
        <log type="junit" target="build/phpunit/logs/junit.xml" logIncompleteSkipped="false"/>
    </logging>
</phpunit>

Relevant composer parts:

    "require": {
        "php": ">=7.0",
        "ext-redis": "*",
        "symfony/symfony": "~3.2",
        "doctrine/orm": "^2.5",
        "doctrine/doctrine-bundle": "^1.6",
        "doctrine/doctrine-cache-bundle": "^1.2",
        "doctrine/doctrine-migrations-bundle": "^1.2",
        "stof/doctrine-extensions-bundle": "^1.2",
        "league/tactician-doctrine": "^1.0",
        "ramsey/uuid-doctrine": "^1.2",
        "symfony/swiftmailer-bundle": "^2.3",
        "symfony/monolog-bundle": "~2.11.3",
        "symfony/polyfill-apcu": "^1.0",
        "sensio/distribution-bundle": "^5.0",
        "sensio/framework-extra-bundle": "^3.0.2",
        "incenteev/composer-parameter-handler": "^2.0",
        "league/tactician-bundle": "^0.4",
        "friendsofsymfony/rest-bundle": "~2.2",
        "nelmio/api-doc-bundle": "^2.11",
        "jms/serializer-bundle": "^1.1",
        "jms/di-extra-bundle": "^1.8",
        "guzzlehttp/guzzle": "6.2.2",
        "mopa/bootstrap-bundle": "~3.0",
        "twbs/bootstrap": "~3.3.0",
        "symfony/assetic-bundle": "^2.8",
        "friendsofsymfony/jsrouting-bundle": "^1.6",
        "bmatzner/fontawesome-bundle": "~4.7",
        "tdn/php-types": "dev-develop as 3.0.0",
        "kzykhys/git": "v0.1.2",
        "cache/apcu-adapter": "~0.2",
        "cache/redis-adapter": "~0.4",
        "cache/cache-bundle": "~0.5",
        "cache/adapter-bundle": "~0.4",
        "cache/psr-6-doctrine-bridge": "^3.0",
        "kachkaev/assets-version-bundle": "~2.0",
        "components/jquery": "~3.1",
        "components/jqueryui": "~1.12",
        "datatables/datatables": "~1.10",
        "e-moe/guzzle6-bundle": "~1.1",
        "composer/semver": "^1.4",
        "paquettg/php-html-parser": "~1.7",
        "verbalexpressions/php-verbal-expressions": "dev-master",
        "typo3/class-alias-loader": "dev-master"
    },
    "require-dev": {
        "phploc/phploc": "^3.0",
        "sensio/generator-bundle": "^3.1",
        "dephpend/dephpend": "dev-develop",
        "henrikbjorn/lurker": "@stable",
        "doctrine/doctrine-fixtures-bundle": "^2.3",
        "symfony/phpunit-bridge": "dev-master",
        "friendsofphp/php-cs-fixer": "~2.0",
        "squizlabs/php_codesniffer": "~2.7",
        "phpunit/phpunit": "~6.1",
        "sebastian/phpcpd": "~3.0",
        "mockery/mockery": "~0.9",
        "liip/functional-test-bundle": "~1.7",
        "jakub-onderka/php-parallel-lint": "0.*",
         "phpmd/phpmd" : "@stable",
        "consolidation/robo": "~1.0",
        "seld/jsonlint": "~1.6",
        "kevinlebrun/colors.php": "0.*",
        "rskuipers/php-assumptions": "dev-feature/update-parser",
        "povils/phpmnd": "~1.0",
        "escapestudios/symfony2-coding-standard": "^2.10",
        "tm/tooly-composer-script": "^1.2"
    },
    "scripts": {
        "symfony-scripts": [
            "Incenteev\\ParameterHandler\\ScriptHandler::buildParameters",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::buildBootstrap",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::clearCache",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installAssets",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installRequirementsFile",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::prepareDeploymentTarget",
            "Mopa\\Bundle\\BootstrapBundle\\Composer\\ScriptHandler::postInstallSymlinkTwitterBootstrap",
            "Tooly\\ScriptHandler::installPharTools"
        ],
        "post-install-cmd": [
            "@symfony-scripts"
        ],
        "post-update-cmd": [
            "@symfony-scripts"
        ]
    },
    "config": {
        "process-timeout": 10000,
        "preferred-install": "dist",
        "github-protocols": ["https"],
        "bin-dir": "bin",
        "platform": {
            "php": "7.0"
        }
    },
    "extra": {
        "symfony-app-dir": "app",
        "symfony-bin-dir": "bin",
        "symfony-var-dir": "var",
        "symfony-web-dir": "web",
        "symfony-tests-dir": "tests",
        "symfony-assets-install": "relative",
        "incenteev-parameters": {
            "file": "app/config/parameters.yml"
        },
        "tools": {
            "phpmd-extension": {
                "url": "https://github.com/mi-schi/phpmd-extension/releases/download/stable/phpmd-extension.phar",
                "only-dev": true,
                "force-replace": true
            },
            "phpmetrics": {
                "url": "https://github.com/phpmetrics/PhpMetrics/releases/download/v2.2.0/phpmetrics.phar",
                "only-dev": true,
                "force-replace": true
            }

        },
        "typo3/class-alias-loader": {
            "class-alias-maps": [
                "src/AppBundle/AliasMap.php"
            ],
            "always-add-alias-loader": true
        }
    },
    "repositories": [
        {
            "type": "git",
            "url": "https://github.com/vpassapera/php-assumptions.git"
        }
    ]

Xdebug Config

; Managed by ansible
zend_extension=/opt/remi/php70/root/usr/lib64/php/modules/xdebug.so
xdebug.remote_enable=1
xdebug.remote_autostart=1
xdebug.remote_host=10.10.10.1
xdebug.max_nesting_level=250
xdebug.remote_port=9000
xdebug.idekey=PHPSTORM
[09:47:59] [vagrant@isengard] /vagrant [][develop ✓] →  php -i | grep -i "xdebug"                   
/etc/php.d/15-xdebug.ini,
    with Xdebug v2.5.3, Copyright (c) 2002-2017, by Derick Rethans
xdebug
xdebug support => enabled
xdebug.auto_trace => Off => Off
xdebug.cli_color => 0 => 0
xdebug.collect_assignments => Off => Off
xdebug.collect_includes => On => On
xdebug.collect_params => 0 => 0
xdebug.collect_return => Off => Off
xdebug.collect_vars => Off => Off
xdebug.coverage_enable => On => On
xdebug.default_enable => On => On
xdebug.dump.COOKIE => no value => no value
xdebug.dump.ENV => no value => no value
xdebug.dump.FILES => no value => no value
xdebug.dump.GET => no value => no value
xdebug.dump.POST => no value => no value
xdebug.dump.REQUEST => no value => no value
xdebug.dump.SERVER => no value => no value
xdebug.dump.SESSION => no value => no value
xdebug.dump_globals => On => On
xdebug.dump_once => On => On
xdebug.dump_undefined => Off => Off
xdebug.extended_info => On => On
xdebug.file_link_format => no value => no value
xdebug.force_display_errors => Off => Off
xdebug.force_error_reporting => 0 => 0
xdebug.halt_level => 0 => 0
xdebug.idekey => PHPSTORM => PHPSTORM
xdebug.max_nesting_level => 250 => 250
xdebug.max_stack_frames => -1 => -1
xdebug.overload_var_dump => 2 => 2
xdebug.profiler_aggregate => Off => Off
xdebug.profiler_append => Off => Off
xdebug.profiler_enable => Off => Off
xdebug.profiler_enable_trigger => Off => Off
xdebug.profiler_enable_trigger_value => no value => no value
xdebug.profiler_output_dir => /tmp => /tmp
xdebug.profiler_output_name => cachegrind.out.%p => cachegrind.out.%p
xdebug.remote_addr_header => no value => no value
xdebug.remote_autostart => On => On
xdebug.remote_connect_back => Off => Off
xdebug.remote_cookie_expire_time => 3600 => 3600
xdebug.remote_enable => On => On
xdebug.remote_handler => dbgp => dbgp
xdebug.remote_host => 10.10.10.1 => 10.10.10.1
xdebug.remote_log => no value => no value
xdebug.remote_mode => req => req
xdebug.remote_port => 9000 => 9000
xdebug.scream => Off => Off
xdebug.show_error_trace => Off => Off
xdebug.show_exception_trace => Off => Off
xdebug.show_local_vars => Off => Off
xdebug.show_mem_delta => Off => Off
xdebug.trace_enable_trigger => Off => Off
xdebug.trace_enable_trigger_value => no value => no value
xdebug.trace_format => 0 => 0
xdebug.trace_options => 0 => 0
xdebug.trace_output_dir => /tmp => /tmp
xdebug.trace_output_name => trace.%c => trace.%c
xdebug.var_display_max_children => 128 => 128
xdebug.var_display_max_data => 512 => 512
xdebug.var_display_max_depth => 3 => 3

Seems that anywhere I use a mock (i use mockery), or symfony functional tests, phpunit is not honoring that coverage under the Methods & Functions | Classes metrics...it did not that long ago.


MOVED FROM sebastianbergmann/phpunit#2676

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

sebastianbergmann commented May 11, 2017

Can you please use PNG insteaf of WebP for the screenshot? GitHub is not able to display that inline for some reason.

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 11, 2017

Are you able to see this one?
screen

The otherone was PNG also. Not sure why it didn't display.

They all display properly on my chrome brower under linux and osx.

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

sebastianbergmann commented May 11, 2017

@derickr Xdebug issue?

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 11, 2017

I did also do an install with ansible on my build box where i first saw the issue (it's a continuous build so it could have updated xdebug), then i started seeing it on my vagrant box after a destroy and up and destroying compoer dir to try to replicate locally. Just adding a bit more info. The behavior is extremely bizarre.

@derickr

This comment has been minimized.

Copy link

derickr commented May 11, 2017

@sebastianbergmann — Can't say without having the full source file.

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 11, 2017

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 11, 2017

Code:

<?php

namespace AppBundle\Controller;

use AppBundle\CommandBus\GetManifestYamlQuery;
use AppBundle\CommandBus\SendManifestNotificationCommand;
use AppBundle\Form\Type\ManifestNotificationType;
use AppBundle\Model\ManifestEmail;
use Doctrine\ORM\NoResultException;
use FOS\RestBundle\Controller\FOSRestController;
use JMS\Serializer\SerializationContext;
use League\Tactician\CommandBus;
use Nelmio\ApiDocBundle\Annotation as Nelmio;
use FOS\RestBundle\Controller\Annotations as FOS;
use JMS\DiExtraBundle\Annotation as DI;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Method;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpFoundation\ResponseHeaderBag;
use Tdn\PhpTypes\Type\StringType;

/**
 * Class ManifestController.
 */
class ManifestController extends FOSRestController
{
    /**
     * @var CommandBus
     *
     * @DI\Inject("tactician.commandbus")
     */
    private $commandBus;

    /**
     * @param string $branch
     *
     * @FOS\Route("/manifests/{branch}.yml", name="get_manifest", requirements={"branch"="(.+)[^.yml]"})
     *
     * @Method({"GET"})
     * @Nelmio\ApiDoc(
     *     resource=true,
     *     section="Manifests",
     *     description="Download a manifest for a specific branch in yaml format.",
     *     output="\Symfony\Component\HttpFoundation\Response",
     *     https=true
     * )
     *
     * @return Response
     */
    public function getManifestAction(string $branch)
    {
        try {
            $yaml = $this->commandBus->handle(
                new GetManifestYamlQuery(
                    $branch,
                    SerializationContext::create()->setGroups(['Manifest'])
                )
            );

            $response = new Response($yaml);
            $disposition = $response->headers->makeDisposition(
                ResponseHeaderBag::DISPOSITION_ATTACHMENT,
                sprintf(
                    'manifest-%s.yml',
                    StringType::create($branch)->replace('/', '-')->toString()
                )
            );

            $response->headers->set('Content-Type', 'text/yaml');
            $response->headers->set('Content-Disposition', $disposition);

            return $response;
        } catch (NoResultException $e) {
            // Throw below.
        }

        throw $this->createNotFoundException();
    }

    /**
     * @Method({"POST"})
     * @FOS\Route("/manifests/{branch}/actions/notify", defaults={"_format":"json"})
     *
     * @Nelmio\ApiDoc(
     *     resource=true,
     *     section="Manifests",
     *     description="Send release information to specified email addresses.",
     *     input="\AppBundle\Form\Type\ManifestNotificationType",
     *     output="\Symfony\Component\HttpFoundation\Response",
     *     https=true
     * )
     *
     * @param Request $request
     * @param string  $branch
     *
     * @return Response
     */
    public function postManifestNotificationAction(Request $request, string $branch)
    {
        try {
            $result = $this->commandBus->handle(
                new GetManifestYamlQuery(
                    $branch,
                    SerializationContext::create()->setGroups(ManifestEmail::SERIALIZATION_CONTEXT)
                )
            );

            $command = new SendManifestNotificationCommand($result, $branch);
            $form = $this->createForm(
                ManifestNotificationType::class,
                $command,
                ['method' => 'POST']
            );

            $form->handleRequest($request);
            if ($form->isSubmitted() && $form->isValid()) {
                $this->commandBus->handle($command);

                return new Response('', Response::HTTP_ACCEPTED);
            }

            return $this->handleView($this->view($form, Response::HTTP_BAD_REQUEST));
        } catch (NoResultException $e) {
            // Throw below.
        }

        throw $this->createNotFoundException();
    }
}

Tests:

<?php

namespace Tests\AppBundle\Controller;

use AppBundle\Entity\Branch;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Yaml\Yaml;
use Tdn\PhpTypes\Type\StringType;
use Tests\AppBundle\FunctionalTestCase;

class ManifestControllerTest extends FunctionalTestCase
{
    public function testGetManifestYamlNotFound()
    {
        $this->client->request('GET', '/manifests/noop.yml');
        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_NOT_FOUND, $response->getStatusCode());
    }

    public function testGetManifestYaml()
    {
        $keys = [
            'appName',
            'version',
            'buildNum',
            'sourceRef',
        ];

        /** @var Branch $branch */
        $branch = $this->referenceRepository->getReference('release/COM170401.0');
        $path = sprintf('/manifests/%s.yml', $branch->getFullName());
        $this->client->request('GET', $path);
        $response = $this->client->getResponse();
        $this->isSuccessful($response);
        $content = Yaml::parse($response->getContent());

        $this->assertEquals(
            'attachment; filename="manifest-release-COM170401.0.yml"',
            $response->headers->get('Content-Disposition')
        );

        $this->assertEquals(
            'text/yaml; charset=UTF-8',
            $response->headers->get('Content-Type')
        );

        $this->assertArrayHasKey('artifacts', $content);
        foreach ($content['artifacts'] as $artifact) {
            $this->assertEquals($keys, array_keys($artifact));
        }
    }

    public function testGetManifestYamlAlternateConvention()
    {
        $keys = [
            'appName',
            'version',
            'buildNum',
            'sourceRef',
        ];

        /** @var Branch $branch */
        $branch = $this->referenceRepository->getReference('release/COM170401.0');
        $path = sprintf(
            '/manifests/%s.yml',
            StringType::create($branch->getFullName())->replace('/', '-')->toString()
        );
        $this->client->request('GET', $path);
        $response = $this->client->getResponse();
        $this->isSuccessful($response);
        $content = Yaml::parse($response->getContent());

        $this->assertEquals(
            'attachment; filename="manifest-release-COM170401.0.yml"',
            $response->headers->get('Content-Disposition')
        );

        $this->assertEquals(
            'text/yaml; charset=UTF-8',
            $response->headers->get('Content-Type')
        );

        $this->assertArrayHasKey('artifacts', $content);
        foreach ($content['artifacts'] as $artifact) {
            $this->assertEquals($keys, array_keys($artifact));
        }
    }

    public function testPostManifestNotificationNotFound()
    {
        $this->client->request('POST', '/manifests/noop/actions/notify.json', ['emails' => []]);
        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_NOT_FOUND, $response->getStatusCode());
    }

    public function testPostManifestNotificationBadRequest()
    {
        $payload = ['emails' => []];
        /** @var Branch $branch */
        $branch = $this->referenceRepository->getReference('release/COM170401.0');
        $path = sprintf(
            '/manifests/%s/actions/notify.json',
            StringType::create($branch->getFullName())->replace('/', '-')->toString()
        );
        $this->client->request('POST', $path, $payload);
        $response = $this->client->getResponse();
        $content = json_decode($response->getContent(), true);
        $this->assertEquals(Response::HTTP_BAD_REQUEST, $response->getStatusCode());
        $this->assertCount(1, $content['errors']);
        $this->assertEquals('You must specify at least one email address.', $content['errors'][0]);
    }

    public function testPostManifestNotificationBadRequestInvalidEmail()
    {
        $payload = ['emails' => [
            'thisisnotanemail',
        ]];
        /** @var Branch $branch */
        $branch = $this->referenceRepository->getReference('release/COM170401.0');
        $path = sprintf(
            '/manifests/%s/actions/notify.json',
            StringType::create($branch->getFullName())->replace('/', '-')->toString()
        );
        $this->client->request('POST', $path, $payload);
        $response = $this->client->getResponse();
        $content = json_decode($response->getContent(), true);
        $this->assertEquals(Response::HTTP_BAD_REQUEST, $response->getStatusCode());
        $this->assertCount(
            1,
            $content['children']['emails']['children'][0]['errors']
        );
        $this->assertEquals(
            'Invalid email found in collection.',
            $content['children']['emails']['children'][0]['errors'][0]
        );
    }

    public function testPostManifestNotification()
    {
        $payload = ['emails' => [
            'foo@bar.com',
        ]];
        /** @var Branch $branch */
        $branch = $this->referenceRepository->getReference('release/COM170401.0');
        $path = sprintf(
            '/manifests/%s/actions/notify.json',
            StringType::create($branch->getFullName())->replace('/', '-')->toString()
        );
        $this->client->request('POST', $path, $payload);
        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_ACCEPTED, $response->getStatusCode());
    }
}

Coverage report:

coverage-header
coverage-first
coverage-second

These are functional tests, and before were reporting 100% coverage across all metrics (as they should).

What makes less sense is that they are missing coverage on things like the second argument of a method. That's just odd.

@derickr

This comment has been minimized.

Copy link

derickr commented May 12, 2017

PHP doesn't generate code on lines 55, 103 and 111, so I don't know what this can be:

filename:       /tmp/foo.php
function name:  getManifestAction
number of ops:  88
compiled vars:  !0 = $branch, !1 = $yaml, !2 = $response, !3 = $disposition, !4 = $e
line     #* E I O op                           fetch          ext  return  operands
-------------------------------------------------------------------------------------
  50     0  E >   EXT_NOP                                                  
         1        RECV                                             !0      
  53     2        EXT_STMT                                                 
         3        FETCH_OBJ_R                                      $5      'commandBus'
         4        INIT_METHOD_CALL                                         $5, 'handle'
         5        EXT_FCALL_BEGIN                                          
  54     6        NEW                                              $5      :-3
         7        EXT_FCALL_BEGIN                                          
         8        SEND_VAR_EX                                              !0
  56     9        INIT_STATIC_METHOD_CALL                                  'JMS%5CSerializer%5CSerializationContext', 'create'
        10        EXT_FCALL_BEGIN                                          
        11        DO_FCALL                                      0  $6      
        12        EXT_FCALL_END                                            
        13        INIT_METHOD_CALL                                         $6, 'setGroups'
        14        EXT_FCALL_BEGIN                                          
…
function name:  postManifestNotificationAction
number of ops:  98
compiled vars:  !0 = $request, !1 = $branch, !2 = $result, !3 = $command, !4 = $form, !5 = $e
line     #* E I O op                           fetch          ext  return  operands
-------------------------------------------------------------------------------------
  98     0  E >   EXT_NOP                                                  
         1        RECV                                             !0      
         2        RECV                                             !1      
 101     3        EXT_STMT                                                 
         4        FETCH_OBJ_R                                      $6      'commandBus'
         5        INIT_METHOD_CALL                                         $6, 'handle'
         6        EXT_FCALL_BEGIN                                          
 102     7        NEW                                              $6      :-3
         8        EXT_FCALL_BEGIN                                          
         9        SEND_VAR_EX                                              !1
 104    10        INIT_STATIC_METHOD_CALL                                  'JMS%5CSerializer%5CSerializationContext', 'create'
        11        EXT_FCALL_BEGIN                                          
        12        DO_FCALL                                      0  $7      
        13        EXT_FCALL_END                                            
        14        INIT_METHOD_CALL                                         $7, 'setGroups'
        15        EXT_FCALL_BEGIN                                          
        16        FETCH_CONSTANT                                   ~7      'AppBundle%5CModel%5CManifestEmail', 'SERIALIZATION_CONTEXT'
        17        SEND_VAL_EX                                              ~7
        18        DO_FCALL                                      0  $7      
        19        EXT_FCALL_END                                            
        20        SEND_VAR_NO_REF                               4          $7
        21        DO_FCALL                                      0          
        22        EXT_FCALL_END                                            
        23        SEND_VAR_NO_REF                               0          $6
        24        DO_FCALL                                      0  $6      
        25        EXT_FCALL_END                                            
        26        ASSIGN                                                   !2, $6
 108    27        EXT_STMT                                                 
        28        NEW                                              $6      :8
        29        EXT_FCALL_BEGIN                                          
        30        SEND_VAR_EX                                              !2
        31        SEND_VAR_EX                                              !1
        32        DO_FCALL                                      0          
        33        EXT_FCALL_END                                            
        34        ASSIGN                                                   !3, $6
 109    35        EXT_STMT                                                 
        36        INIT_METHOD_CALL                                         'createForm'
        37        EXT_FCALL_BEGIN                                          
 110    38        SEND_VAL_EX                                              'AppBundle%5CForm%5CType%5CManifestNotificationType'
        39        SEND_VAR_EX                                              !3
 112    40        SEND_VAL_EX                                              <array>
        41        DO_FCALL                                      0  $6      
        42        EXT_FCALL_END                                            
        43        ASSIGN                                                   !4, $6
 115    44        EXT_STMT                                                 
…
@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 12, 2017

This is happening on a few tests.... always on parameters. The order seems random. I do see them listed above referenced though (e.g. !0 on line 12)

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 14, 2017

Any ideas?

@aviator-ua

This comment has been minimized.

Copy link

aviator-ua commented May 15, 2017

The same issue. After update to Xdebug v2.5.3

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

sebastianbergmann commented May 15, 2017

Looks like this is not an issue with this component (or PHPUnit). Either the PHP compiler does not generated opcodes for the sourcecode lines in question or Xdebug does not recognize them.

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 24, 2017

This should be re-opened as per https://bugs.xdebug.org/view.php?id=1440

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 25, 2017

Not sure if it's possible for someone to test this on a version of PHP before 7.0.19 and see if something in that release broke it?

@vpassapera vpassapera changed the title Incorrect coverage reported, invocation not being honored? Incorrect coverage reported May 25, 2017

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 25, 2017

Alright, this is getting to be a major pain, almost detrimental in nature, due to the bad reporting coming in....

    /**
     * @FOS\Route("/releases/{release}", defaults={"_format":"json"}, requirements={"release"="(.+)[^[.json|.html]"})
     *
     * @Nelmio\ApiDoc(
     *     resource=true,
     *     section="Releases",
     *     description="Get a release.",
     *     https=true,
     *     output="\AppBundle\Response\ReleaseResponse"
     * )
     *
     * @Method("GET")
     * @FOS\View(
     *     templateVar="release",
     *     serializerGroups={"Default", "Detailed"}
     * )
     *
     * @param string $release
     *
     * @return ReleaseResponse
     */
    public function getReleaseAction(string $release): ReleaseResponse
    {
        try {
            return $this->commandBus->handle(new GetReleaseQuery($release));
        } catch (NoResultException $e) {
            // Throw exception below.
        }

        throw $this->createNotFoundException();
    }

With the following tests

<?php

namespace Tests\AppBundle\Controller;

use AppBundle\Entity\Release;
use Tests\AppBundle\FunctionalTestCase;
use Symfony\Component\HttpFoundation\Response;
use Tdn\PhpTypes\Type\StringType;

/**
 * Class ReleaseControllerTest.
 */
class ReleaseControllerTest extends FunctionalTestCase
{
    public function testGetReleaseNotFound()
    {
        $this->client->request('GET', '/releases/foo.json');
        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_NOT_FOUND, $response->getStatusCode());
    }

    public function testGetAllReleasesJson()
    {
        /** @var Release[] $expected */
        $expected = [
            $this->referenceRepository->getReference('release-release/COM170701.0'),
            $this->referenceRepository->getReference('release-hotfix/COM170301.2'),
            $this->referenceRepository->getReference('release-release/COM170401.0'),
            $this->referenceRepository->getReference('release-release/COM170501.0'),
        ];

        $this->client->request('GET', '/releases.json');
        $response = $this->client->getResponse();
        $this->isSuccessful($response);
        $response = json_decode($response->getContent(), true);

        $this->assertEquals(count($expected), count($response['releases']));
        foreach ($response['releases'] as $actualBranch) {
            $currentExpected = array_shift($expected);
            $this->assertEquals($currentExpected->getName(), $actualBranch['name']);
            $this->assertEquals($currentExpected->getIaspec(), $actualBranch['iaSpec']);
            $this->assertEquals($currentExpected->getType(), $actualBranch['type']);
            $this->assertEquals($currentExpected->getCreatedAt()->format(DATE_ATOM), $actualBranch['createdAt']);
            $this->assertEquals($currentExpected->getUpdatedAt()->format(DATE_ATOM), $actualBranch['updatedAt']);
        }
    }

    public function testGetReleaseJson()
    {
        /** @var Release $expected */
        $expected = $this->referenceRepository->getReference('release-hotfix/COM170301.2');
        $path = sprintf(
            '/releases/%s.json',
            StringType::create($expected->getFullName())->replace('/', '-')->toString(),
            $expected->getName()
        );
        $this->client->request('GET', $path);
        $response = $this->client->getResponse();
        $this->isSuccessful($response);
        $response = json_decode($response->getContent(), true);
        $this->assertEquals($expected->getName(), $response['name']);
        $this->assertEquals($expected->getType(), $response['type']);
        $this->assertEquals($expected->getIaspec(), $response['iaSpec']);
        $this->assertEquals($expected->getStatus(), $response['status']);
        $this->assertEquals($expected->getCreatedAt()->format(\DateTime::ATOM), $response['createdAt']);
        $this->assertEquals($expected->getUpdatedAt()->format(\DateTime::ATOM), $response['updatedAt']);
        $this->assertArrayHasKey('artifacts', $response);
        $this->assertArrayHasKey('removedArtifacts', $response);
    }
}

Says:

Imgur

I literally cannot have code coverage guard turned on anywhere, cause it says it is not covered.... What can we do to get this fixed please?

Also please notice how

    public function testGetReleaseNotFound()
    {
        $this->client->request('GET', '/releases/foo.json');
        $response = $this->client->getResponse();
        $this->assertEquals(Response::HTTP_NOT_FOUND, $response->getStatusCode());
    }

Covers explicitly those lines. PHPUnit runs that tests.

[23:26:00] [vagrant@isengard] /vagrant [][feature/releases-view S:16 U:2 ✗] →  ./bin/phpunit tests/AppBundle/Controller/ReleaseControllerTest.php
PHPUnit 6.1.4 by Sebastian Bergmann and contributors.

...                                                                 3 / 3 (100%)

Time: 40.82 seconds, Memory: 34.00MB

OK (3 tests, 32 assertions)

Generating code coverage report in Clover XML format ... done

Generating Crap4J report XML file ... done

Generating code coverage report in HTML format ... done

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 25, 2017

:(

@sebastianbergmann

This comment has been minimized.

Copy link
Owner

sebastianbergmann commented May 26, 2017

Thank you for your report.

Please provide a minimal, self-contained, reproducing test case that shows the problem you are reporting. In your specific case this means that no framework code etc. is used/required that is affected by annotations.

Without such a minimal, self-contained, reproducing test case I will not be able to investigate this issue.

@neomerx

This comment has been minimized.

Copy link

neomerx commented May 29, 2017

I think since 7.0 I started to see it very often. I found phpdbg gives accurate results.

phpdbg -qrr ./vendor/bin/phpunit --coverage-text
@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 29, 2017

It increased my coverage, but it still didn't fix the last outlined item.

Thanks @neomerx, this is better than nothing for now. (Coverage back up to 99.5%)

Basically it started reporting coverage for the item listed in the OP (not reporting in multi-line statements, and functions/methods and classes/traits reports 100%) that @derickr @sebastianbergmann said was fixed here:

https://bugs.xdebug.org/view.php?id=1440#c4349

It still however lacks coverage in the symfony framework functional test case. That may yet be a bug with this.

@derickr

This comment has been minimized.

Copy link

derickr commented May 30, 2017

@vpassapera — There can be many reasons why a specific line is not covered. Each of them are likely a different issue. In order to be able to reproduce anything and not get bogged down with details, you need to provide the following in ONE new issue per case.

That means:

  • The exact versions of (this is what the template requires too, but more):
    • PHP
    • PHP Code Coverage
    • Xdebug
    • PHPUnit
  • php -v output
  • Either a .tar.gz archive or a GitHub repository with a reproduce case. With one file that shows the problem and one test case file (with the exception that sometimes, you might need two source files to show a problem.

This information should be file as one specific case per issue.

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented May 30, 2017

Understood.

I'll try to reproduce the symfony controller gaps in coverage in a public repo and link it here (I'll add some ansible provisioning and a vagrant file to ensure the same version runs on that example repo).

thanks for your time @derickr

While I switched the driver to phpdbg for coverage, I still use xdebug for actual debugging (e.g. phpstorm, etc) :)

@vpassapera

This comment has been minimized.

Copy link
Author

vpassapera commented Jun 23, 2017

Seems like this was fixed as my tests are now reporting 100% coverage.

Huzzah!

@vpassapera vpassapera closed this Jun 23, 2017

@wired00

This comment has been minimized.

Copy link

wired00 commented Sep 7, 2017

Glad I found this, I have the exact issue. What did you do exactly to fix the issue? I'm on PHP 7.1, PHP unit 5.x X. Not sure which version xdebug.

Should I upgrade PHP unit 6.x X and xdebug?

@VasekPurchart

This comment has been minimized.

Copy link

VasekPurchart commented Sep 7, 2017

@wired00 it was fixed in Xdebug 2.5.5, don't know if you need to update PhpUnit too or not.

@wired00

This comment has been minimized.

Copy link

wired00 commented Sep 7, 2017

Thanks @VasekPurchart I'll upgrade now. I'm currently on:

PHP 7.1.5
Xdebug 2.5.4
phpunit/php-code-coverage 4.0.8

@wired00

This comment has been minimized.

Copy link

wired00 commented Sep 7, 2017

Ok Great, for the benefit of others. I'm on OSX and fixed by going from xdebug 2.5.4 > 2.5.5.

I simply ran brew update > brew upgrade

now running 7.1.8 with Xdebug 2.5.5 and coverage is working perfectly.

@filips123

This comment has been minimized.

Copy link

filips123 commented Oct 30, 2018

@sebastianbergmann @filips123

I have same problem with PHP 7.2 and XDebug 2.6.

Q A
php-code-coverage version 6.1.3
PHP version 7.2.11-3+ubuntu16.04.1+deb.sury.org+1
Driver Xdebug
Xdebug version 2.6.1
Installation Method Composer and PHAR
Usage Method PHPUnit
PHPUnit version 7.4.3

Source file:

<?php

namespace SunshineCMS\Services;

use Illuminate\Database\Capsule\Manager as IlluminateCapsule;
use Pimple\Container;
use Pimple\ServiceProviderInterface;

class DatabaseService implements ServiceProviderInterface
{
    public function register(Container $container)
    {
        if (!isset($container['database'])) {
            $container['database'] = function ($container) {
                $settings = $container->get('settings')['database'];
                $capsule = new IlluminateCapsule;

                $capsule->addConnection($settings);

                $capsule->setAsGlobal();
                $capsule->bootEloquent();

                return clone $capsule;
            };
        }
    }
}

Test file:

<?php

namespace SunshineCMS\Tests\Unit\Services;

use PHPUnit\Framework\TestCase;
use SunshineCMS\Container;
use SunshineCMS\Services\DatabaseService;

class DatabaseServiceTest extends TestCase
{
    /**
     * @var array
     */
    protected $settings;

    public function setUp()
    {
        $this->settings = [
            'siteURL' => 'https://sunshinecms.local/',
            'siteDir' => dirname(dirname(__DIR__)),
            'scriptName' => '/index.php',
        ];
    }

    /**
     * @covers SunshineCMS\Services\DatabaseService::register()
     */
    public function testServicesRegistration()
    {
        $container = new Container(['settings' => $this->settings]);

        $service = new DatabaseService;
        $service->register($container);

        $this->assertTrue(isset($container->database));
        $this->assertInstanceOf('\Illuminate\Database\Capsule\Manager', $container->database);
    }
}

Coverage details:

coverage1

coverage2

If I run PHPUnit with phpdbg -qrr ./vendor/bin/phpunit, coverage is normal (100%).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.