loadSessionSnapshot does not work with coverage enabled #2923

Closed
shlainn opened this Issue Mar 25, 2016 · 9 comments

Comments

Projects
None yet
6 participants
@shlainn

shlainn commented Mar 25, 2016

Hiya,

I have some working acceptance tests and am now trying to get code coverage for them. As recommended by the Guides, I use an @before directive to handle login before each test, and use saveSessionSnapshot to avoid having to go through login for each test.

This works nicely until I run codeception with --coverage. What happens then is that the first test in the file goes through nicely, but all subsequent tests, which rely on loadSessionSnapshot to skip login, do not work any more.

Is this behavior known?

@Naktibalda Naktibalda added the Coverage label Mar 25, 2016

@shlainn

This comment has been minimized.

Show comment
Hide comment
@shlainn

shlainn Mar 27, 2016

I did some more digging, maybe someone with more experience with codeception can make sense of this. I am running
Codeception PHP Testing Framework v2.1.5 Powered by PHPUnit 4.8.21 by Sebastian Bergmann and contributors.
I ran one of my Cests, once with the --coverage flag, and once without and compared the -vvv outputs. Without --coverage each subsequent test in the Cest continues where the previous test had left off - same page, same cookies, same everything.
With --coverage it resets to the base url for each test. This difference breaks those of my tests which assume that they are still on the same page - because without --coverage they still would be.

EDIT: Confirmed workaround for my issue is to explicitly set $I->amOnPage("whatever.php"); at the beginning of each test.

EDIT2: The culprit is in Codeception\Coverage\Subscriber\LocalServer (protected function startCoverageCollection) Line 151 which explicitly does $this->module->amOnPage('/');. Why?

EDIT3: Ah, because we can only set cookies for the current domain. But why does it not return to the previous page then?

EDIT4: After some messing around, I think that moving $this->module->amOnPage('/'); from startCoverageCollection to beforeSuite should do the trick

shlainn commented Mar 27, 2016

I did some more digging, maybe someone with more experience with codeception can make sense of this. I am running
Codeception PHP Testing Framework v2.1.5 Powered by PHPUnit 4.8.21 by Sebastian Bergmann and contributors.
I ran one of my Cests, once with the --coverage flag, and once without and compared the -vvv outputs. Without --coverage each subsequent test in the Cest continues where the previous test had left off - same page, same cookies, same everything.
With --coverage it resets to the base url for each test. This difference breaks those of my tests which assume that they are still on the same page - because without --coverage they still would be.

EDIT: Confirmed workaround for my issue is to explicitly set $I->amOnPage("whatever.php"); at the beginning of each test.

EDIT2: The culprit is in Codeception\Coverage\Subscriber\LocalServer (protected function startCoverageCollection) Line 151 which explicitly does $this->module->amOnPage('/');. Why?

EDIT3: Ah, because we can only set cookies for the current domain. But why does it not return to the previous page then?

EDIT4: After some messing around, I think that moving $this->module->amOnPage('/'); from startCoverageCollection to beforeSuite should do the trick

@DavertMik DavertMik added the BUG label Apr 15, 2016

@DavertMik DavertMik closed this in bc79819 Apr 19, 2016

@boboldehampsink

This comment has been minimized.

Show comment
Hide comment
@boboldehampsink

boboldehampsink Feb 17, 2017

Contributor

This is still not working. I am on Codeception 2.2.9. Commenting lines 161-162 in Codeception\Coverage\Subscriber\LocalServer makes it work again:

$this->module->amOnPage('/');
$this->module->setCookie(self::COVERAGE_COOKIE, $value);
Contributor

boboldehampsink commented Feb 17, 2017

This is still not working. I am on Codeception 2.2.9. Commenting lines 161-162 in Codeception\Coverage\Subscriber\LocalServer makes it work again:

$this->module->amOnPage('/');
$this->module->setCookie(self::COVERAGE_COOKIE, $value);

@Naktibalda Naktibalda reopened this Feb 17, 2017

@boboldehampsink

This comment has been minimized.

Show comment
Hide comment
@boboldehampsink

boboldehampsink Feb 17, 2017

Contributor

The fix mentioned above fixes bc79819 but does not fix the problem.

Contributor

boboldehampsink commented Feb 17, 2017

The fix mentioned above fixes bc79819 but does not fix the problem.

@boboldehampsink

This comment has been minimized.

Show comment
Hide comment
@boboldehampsink

boboldehampsink Feb 18, 2017

Contributor

I have found this is because calling $this->module->amOnPage('/') before every suite resets the session on the server. We should somehow restore the session snapshot before calling this (perhaps by emitting a START_COVERAGE event?)

Contributor

boboldehampsink commented Feb 18, 2017

I have found this is because calling $this->module->amOnPage('/') before every suite resets the session on the server. We should somehow restore the session snapshot before calling this (perhaps by emitting a START_COVERAGE event?)

DavertMik added a commit that referenced this issue Mar 8, 2017

@lowerends

This comment has been minimized.

Show comment
Hide comment
@lowerends

lowerends Apr 27, 2017

I can confirm this is still happening in version 2.2.10. This version contains #4020.

I can confirm this is still happening in version 2.2.10. This version contains #4020.

@boboldehampsink

This comment has been minimized.

Show comment
Hide comment
@boboldehampsink

boboldehampsink Apr 27, 2017

Contributor

#4020 is different fix

Contributor

boboldehampsink commented Apr 27, 2017

#4020 is different fix

chris1312 added a commit to chris1312/Codeception that referenced this issue Jun 16, 2017

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Jul 10, 2017

Contributor

Thanks for the tips in this. I recently upgraded to 2.3.3 and implemented code coverage and saw my tests failing due to loadSessionSnapshot. I think it was my fault at bad test design, but enforcing all my tests to do $I->amOnPage('/whatever') before execution fixed this.

Prior to this, I was depending on the order of tests of where they ended for the next one to begin. (Yes, I know that sounds horrible, that has been resolved).

Contributor

iBotPeaches commented Jul 10, 2017

Thanks for the tips in this. I recently upgraded to 2.3.3 and implemented code coverage and saw my tests failing due to loadSessionSnapshot. I think it was my fault at bad test design, but enforcing all my tests to do $I->amOnPage('/whatever') before execution fixed this.

Prior to this, I was depending on the order of tests of where they ended for the next one to begin. (Yes, I know that sounds horrible, that has been resolved).

@DavertMik

This comment has been minimized.

Show comment
Hide comment
@DavertMik

DavertMik Jul 10, 2017

Member

@iBotPeaches strange, coverage cookies are ignored during when the snapshot is saved. See
https://github.com/Codeception/Codeception/blob/2.3/src/Codeception/Module/WebDriver.php#L2919

Could you debug, is that so? If accidentally some coverage cookies are getting saved - that's the issue.

Member

DavertMik commented Jul 10, 2017

@iBotPeaches strange, coverage cookies are ignored during when the snapshot is saved. See
https://github.com/Codeception/Codeception/blob/2.3/src/Codeception/Module/WebDriver.php#L2919

Could you debug, is that so? If accidentally some coverage cookies are getting saved - that's the issue.

@iBotPeaches

This comment has been minimized.

Show comment
Hide comment
@iBotPeaches

iBotPeaches Jul 10, 2017

Contributor

@DavertMik Maybe I don't understand this enough, but at my first request (ie login), I had no session so saveSessionSnapshot was called. It only looped the following cookies

  • _identity
  • _csrf
  • XDEBUG_SESSION
  • PHPSESSID

Of course none of those were the COVERAGE cookies array so it saved them all, and thus for remainder of the test suite it loaded that session point and never had a need to save again. If additional cookies were created throughout the test suite (possibly from coverage), they were never saved since saveSessionSnapshot is only called on first test to log in.

My lines didn't match up with the above link. I see 2.3.4 came out, so I might need to report back after I update.

Contributor

iBotPeaches commented Jul 10, 2017

@DavertMik Maybe I don't understand this enough, but at my first request (ie login), I had no session so saveSessionSnapshot was called. It only looped the following cookies

  • _identity
  • _csrf
  • XDEBUG_SESSION
  • PHPSESSID

Of course none of those were the COVERAGE cookies array so it saved them all, and thus for remainder of the test suite it loaded that session point and never had a need to save again. If additional cookies were created throughout the test suite (possibly from coverage), they were never saved since saveSessionSnapshot is only called on first test to log in.

My lines didn't match up with the above link. I see 2.3.4 came out, so I might need to report back after I update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment