Using environment does not work with code coverage #2634

Closed
drodil opened this Issue Dec 15, 2015 · 4 comments

Comments

Projects
None yet
4 participants
@drodil

drodil commented Dec 15, 2015

Hi,

I am trying to run codecoverage for my project with codeception. Here is my build/codeception.yml file:

actor: Tester
paths:
    tests: ../tests
    log: ../tests/_output
    data: ../tests/_data
    helpers: ../tests/_support
settings:
    bootstrap: _bootstrap.php
    colors: false
    memory_limit: 2048M
coverage:
    enabled: true
    remote: false
    remote_context_options:
        http:
            timeout: 60
        ssl:
            verify_peer: false
    blacklist:
        include:
            - ../db/*
            - ../data/*
            - ../config/*
modules:
    config:
        Db:
            dsn: ''
            user: ''
            password: ''
            dump: tests/_data/dump.sql

I have defined the following environments in tests/acceptance.yml:

class_name: AcceptanceTester
modules:
    enabled: [WebDriver, AcceptanceHelper]
    config:
        WebDriver:
            url: 'http://localhost/project/'
            browser: firefox
env:
    ci:
        modules:
            config:
                WebDriver:
                    url: 'https://server1.example.com/project/'
                    host: server1.example.com
                    browser: firefox
                    window_size: 1920x1080
    ci-mbv:
        modules:
            config:
                WebDriver:
                    url: 'https://server1.example.com/project-mbv/'
                    host: server1.example.com
                    browser: firefox
                    window_size: 1920x1080

When I run the following command:

php codecept.phar run -f -vvv -c build/codeception.yml --xml build/logs/junit.xml --html build/logs/junit.html --coverage-html build/coverage/html --coverage-xml build/coverage/converage.xml --env ci-mbv

I will get the error:

The "/var/lib/jenkins/workspace/project/build/../tests/acceptance.ci-mbv/" directory does not exist.  

The tests are created in subfolders under tests folder (acceptance/, unit/ and api/). Why does the codeception add the environment to end of the test folder? How can I prevent that or am I doing something wrong? This does not work even with remote option set as true - the error in that case is "Suite acceptance-ci-mbv was not loaded".

Here is the codeception exception backtrace:

Exception trace:
 () at /var/lib/jenkins/workspace/project/vendor/symfony/finder/Finder.php:656
 Symfony\Component\Finder\Finder->in() at /var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Lib/TestLoader.php:112
 Codeception\Lib\TestLoader->loadTests() at /var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/SuiteManager.php:95
 Codeception\SuiteManager->loadTests() at /var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Codecept.php:199
 Codeception\Codecept->runSuite() at /var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Codecept.php:191
 Codeception\Codecept->run() at /var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Command/Run.php:262
 Codeception\Command\Run->runSuites() at /var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Command/Run.php:189
 Codeception\Command\Run->execute() at /var/lib/jenkins/workspace/project/vendor/symfony/console/Command/Command.php:259
 Symfony\Component\Console\Command\Command->run() at /var/lib/jenkins/workspace/project/vendor/symfony/console/Application.php:841
 Symfony\Component\Console\Application->doRunCommand() at /var/lib/jenkins/workspace/project/vendor/symfony/console/Application.php:192
 Symfony\Component\Console\Application->doRun() at /var/lib/jenkins/workspace/project/vendor/symfony/console/Application.php:123
 Symfony\Component\Console\Application->run() at /var/lib/jenkins/workspace/project/vendor/codeception/codeception/codecept:28

Thanks for help!

@drodil

This comment has been minimized.

Show comment
Hide comment
@drodil

drodil Dec 15, 2015

Seems like there is some magic handling of the environment name in SuiteManager.php:136:

$suite->setBaseName($this->env ? substr($name, 0, strpos($name, '-' . $this->env)) : $name);

After changing the environment name to be "cimbv" instead "ci-mbv" something at least happened in the running of the test but still got error:

  [RemoteException] Remote Application Error:
Suite++was+not+loaded  

Exception trace:

/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Coverage/Subscriber/LocalServer.php:98
/var/lib/jenkins/workspace/project/vendor/symfony/event-dispatcher/EventDispatcher.php:181
/var/lib/jenkins/workspace/project/vendor/symfony/event-dispatcher/EventDispatcher.php:46
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Shared/Actor.php:73
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Shared/Actor.php:106
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Scenario.php:101
/var/lib/jenkins/workspace/project/tests/_support/_generated/AcceptanceTesterActions.php:268
/var/lib/jenkins/workspace/project/tests/acceptance/BaseAcceptanceTest.php:78
/var/lib/jenkins/workspace/project/tests/acceptance/BaseAcceptanceTest.php:100
/var/lib/jenkins/workspace/project/tests/acceptance/AdminCest.php:32
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Lib/Di.php:112
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Cest.php:127
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Cest.php:211
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Cest.php:64
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestCase.php:909
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestCase.php:768
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestResult.php:612
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestCase.php:724
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestSuite.php:747
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/PHPUnit/Runner.php:89
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/SuiteManager.php:153
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Codecept.php:203
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Codecept.php:194
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Command/Run.php:262
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Command/Run.php:189
/var/lib/jenkins/workspace/project/vendor/symfony/console/Command/Command.php:259
/var/lib/jenkins/workspace/project/vendor/symfony/console/Application.php:841
/var/lib/jenkins/workspace/project/vendor/symfony/console/Application.php:192
/var/lib/jenkins/workspace/project/vendor/symfony/console/Application.php:123
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/codecept:28

drodil commented Dec 15, 2015

Seems like there is some magic handling of the environment name in SuiteManager.php:136:

$suite->setBaseName($this->env ? substr($name, 0, strpos($name, '-' . $this->env)) : $name);

After changing the environment name to be "cimbv" instead "ci-mbv" something at least happened in the running of the test but still got error:

  [RemoteException] Remote Application Error:
Suite++was+not+loaded  

Exception trace:

/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Coverage/Subscriber/LocalServer.php:98
/var/lib/jenkins/workspace/project/vendor/symfony/event-dispatcher/EventDispatcher.php:181
/var/lib/jenkins/workspace/project/vendor/symfony/event-dispatcher/EventDispatcher.php:46
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Shared/Actor.php:73
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Shared/Actor.php:106
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Scenario.php:101
/var/lib/jenkins/workspace/project/tests/_support/_generated/AcceptanceTesterActions.php:268
/var/lib/jenkins/workspace/project/tests/acceptance/BaseAcceptanceTest.php:78
/var/lib/jenkins/workspace/project/tests/acceptance/BaseAcceptanceTest.php:100
/var/lib/jenkins/workspace/project/tests/acceptance/AdminCest.php:32
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Lib/Di.php:112
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Cest.php:127
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Cest.php:211
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/TestCase/Cest.php:64
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestCase.php:909
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestCase.php:768
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestResult.php:612
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestCase.php:724
/var/lib/jenkins/workspace/project/vendor/phpunit/phpunit/src/Framework/TestSuite.php:747
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/PHPUnit/Runner.php:89
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/SuiteManager.php:153
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Codecept.php:203
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Codecept.php:194
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Command/Run.php:262
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/src/Codeception/Command/Run.php:189
/var/lib/jenkins/workspace/project/vendor/symfony/console/Command/Command.php:259
/var/lib/jenkins/workspace/project/vendor/symfony/console/Application.php:841
/var/lib/jenkins/workspace/project/vendor/symfony/console/Application.php:192
/var/lib/jenkins/workspace/project/vendor/symfony/console/Application.php:123
/var/lib/jenkins/workspace/project/vendor/codeception/codeception/codecept:28
@drodil

This comment has been minimized.

Show comment
Hide comment
@drodil

drodil Dec 15, 2015

Seems like the acceptance cookie does not have CodeCoverage_Suite set correctly:

[Cookies] [{"name":"PHP_SESSID","path":"/","value":"6121sul17iolebiqn4ve4354aka3cvf1nn3u6f57g2g70u50rmekgp9d5c084uegi1cu1c0khcn24l55gr3r9j10ihj6e8f5rm0p8r3","secure":true,"domain":"server1.example.com","class":"org.openqa.selenium.Cookie","httpOnly":false,"hCode":1755402541},{"name":"CODECEPTION_CODECOVERAGE","path":"/","value":"{\"CodeCoverage\":\"adminModeTest\",\"CodeCoverage_Suite\":\"\",\"CodeCoverage_Config\":null}","secure":false,"domain":"server1.example.com","class":"org.openqa.selenium.Cookie","httpOnly":false,"hCode":960895263}]

Any idea why this would occur? If I set code coverage disabled, the acceptance tests run ok.

drodil commented Dec 15, 2015

Seems like the acceptance cookie does not have CodeCoverage_Suite set correctly:

[Cookies] [{"name":"PHP_SESSID","path":"/","value":"6121sul17iolebiqn4ve4354aka3cvf1nn3u6f57g2g70u50rmekgp9d5c084uegi1cu1c0khcn24l55gr3r9j10ihj6e8f5rm0p8r3","secure":true,"domain":"server1.example.com","class":"org.openqa.selenium.Cookie","httpOnly":false,"hCode":1755402541},{"name":"CODECEPTION_CODECOVERAGE","path":"/","value":"{\"CodeCoverage\":\"adminModeTest\",\"CodeCoverage_Suite\":\"\",\"CodeCoverage_Config\":null}","secure":false,"domain":"server1.example.com","class":"org.openqa.selenium.Cookie","httpOnly":false,"hCode":960895263}]

Any idea why this would occur? If I set code coverage disabled, the acceptance tests run ok.

@Naktibalda Naktibalda added the Coverage label Jan 14, 2016

@neronmoon

This comment has been minimized.

Show comment
Hide comment
@neronmoon

neronmoon Jan 21, 2016

Contributor

+1

Contributor

neronmoon commented Jan 21, 2016

+1

@DavertMik

This comment has been minimized.

Show comment
Hide comment
@DavertMik

DavertMik Jan 21, 2016

Member

Ok, thanks for the investigation. I will take a detailed look

Member

DavertMik commented Jan 21, 2016

Ok, thanks for the investigation. I will take a detailed look

@DavertMik DavertMik added the BUG label Jan 21, 2016

@DavertMik DavertMik closed this in 25c23b2 Jan 21, 2016

DavertMik added a commit that referenced this issue Jan 21, 2016

DavertMik added a commit that referenced this issue Jan 21, 2016

DavertMik added a commit that referenced this issue Jan 25, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment