[Testing] Wrappers over Session, Headers, Cookies

[DavertMik]

In order to make application accept multiple requests in testing environment PHP session_*, header, cookies functions must not be called. Instead of creating real headers, sessions, and cookies, a fake inmemory storage should be used.

Example:

<?php
class TestSession {

  public static $session;

  public function set($key,$val)
  {
      self::$session[$key] = $val;
  }

}

This in memory storage should be easy to clean up.
I'm not aware of how implementation of this may look in Yii2. Probably there should be components for this. But the idea is to create wrappers and not allow this PHP functions to be called in testing env.

[qiangxue]

We basically need to create Session and Cookie components specifically for testing purpose. For headers, most likely we need to provide a Response component for testing (also need to clean up existing code that directly uses header() calls).

[Ragazzo]

Yes, @qiangxue i will write in this issue tomorrow some common pitfalls that i faced with when developing integration for Yii 1.1.

[qiangxue]

That will be cool. Thank you!

[Ragazzo]

Ok, so what basically need to be done. In the Codeception (similar to Behat+Mink) there is a 2 type of tests for web-interface:

• acceptance (also known as "inbrowser" tests);
• functional (also know as "headless" tests, uses curl).

There are two types of functional tests in Codeception: one can be run with web-server ("PhpBrowser" module), other can be preformed through symfony browser-kit component. As you see some basic support for Codeception in Yii2 to be done is to implement connector (Yii2 client) (symfony browser-kit client) and module (Yii2 codeception module) that inherits his behavior with common methods from Codeception\Framework (it has methods to interact with symfony browser-kit).
Integration for Yii1 you can see here :

• connector (symfony browser-kit client) https://github.com/Codeception/Codeception/blob/master/src/Codeception/Util/Connector/Yii1.php
• module (codeception module) https://github.com/Codeception/Codeception/blob/master/src/Codeception/Module/Yii1.php

Now you also can use Yii1 CTestCase and CDbTestCase in Codeception, it is done by modifying bootstrap file from framework core, you can see that in plugins directory of the Codeception here - https://github.com/Codeception/Codeception/tree/master/plugins/frameworks/yii in yiit.php file. So it was modified because of Codeception already use require for PhphUnit classes mentioned in yiit.php core file. I think this problem can be easy solved by just moving phpunits-includes and autoload registereing from core yiit.php file to application tests folder bootstrap.php file, i think it is even better)

So overall, all that need to be done for second type of functional Codeception tests is implementing client-connector and codeception module. Yii applications can be tested via Codeception even with "PhpBrowser" module (through curl) that requires server, but i think that integration through browser-kit would be very useful too.

P.S. in following messages i will write some common pitfalls for sessions, headers, cookies.

[Ragazzo]

Session.

Main thing for session was to avoid session_start() call because it was already started by Codeception (in php 5.3 even @ will not help). So first it was decided to leave it as-is and just exclude warning error-level. Later it was added defer-flush into Codeception to avoid any buffer flush until all tests will run, and with defer-flush it works correctly for headers too.
So about session, i think that we do not need any stub for session because the only "painful" moment is second session_start(), also it was decided to force session close after all global arrays were cleaned (https://github.com/Codeception/Codeception/blob/master/src/Codeception/Module/Yii1.php#L141). Also if session use DB or Redis or some other storage and 3d-party components may use that storage directly to process some session data, then stub will make whole application ruined, because session_* functions will be stubbed.
Summary: dont need (at my opinion) session-stub.

[Ragazzo]

Request and Cookies.

Main work to be done is here. Unfortunately Yii1 does not provide some response-object and dont mock header function call, so new component called CodeceptionHttpRequest was created for those purposes, it is in plugins Codeception directory too - https://github.com/Codeception/Codeception/blob/master/plugins/frameworks/yii/web/CodeceptionHttpRequest.php. Things that was done are very easy as you can see.
But there is one "pitfall", if user have already custom http-request (i do, and many developers also i think), then he need to modify this component(codeception httprequest component) and extend it from his custom http-request component.
Summary: some work here definitely need to be done and discussed.

[Ragazzo]

Summary:
Other things to be done:

• correct determining of some paths and pathinfo, for example Yii1 CHttpRequest. That two rows https://github.com/Codeception/Codeception/blob/master/src/Codeception/Module/Yii1.php#L108 so http-request component can determine all paths correctly.
• setting correct url for domain. For now in Yii1 integration it is only localhost but it must be a Yii2 codeception module param i guess, so url can be like my.super.domain and other.
• some more debug info. Different modules in Codeception provide more info when you run tests with --debug option, you can see it for example in PhpBrowser module (https://github.com/Codeception/Codeception/blob/master/src/Codeception/Module/PhpBrowser.php#L172). So for Yii it would be enough i think to provide overall sql-queries time i think, and maybe some other timings, not so much because Yii has great WebLogRoute, but just some common of them i guess.
  So i think thats all.

[qiangxue]

@Ragazzo Thank you very much for these writeup. I have been reading Codeception tutorials these days when having time. So far it looks great to me (the tutorials are also very well written BTW).

From my understanding, we mainly need to provide integration support for functional testing, right? Do we need to do anything for unit testing? And as you wrote, for functional testing, we should provide at least provide connector and module for Codeception. We also need to provide additional application components specifically for testing purpose, including Request, CookieCollection, Response, and other things that help to reset the testing environments.

[Ragazzo]

From my understanding, we mainly need to provide integration support for functional testing, right?

yes, for second kind of functional codeception tests that run with symfony-browser-kit. (first "PhpBrowser" module is run with curl and web-server, but ofcourse without web-server it would be faster, maybe difference will be not so big of course but anyway).
To do this as i said we only need to implement: connector (Yii2 client, like for Yii1), and module for codeception (like Yii1 module), also some stub for HttpRequest and CookieCollection needed here, i think thats enough. Not sure about Response object, because simple output catch can be made like in current Yii1 connector with ob_start and ob_get_clean (https://github.com/Codeception/Codeception/blob/master/src/Codeception/Util/Connector/Yii1.php#L80).

Do we need to do anything for unit testing?

Well, as i said Yii1 have good fixture manager and test-cases like CTestCase and CDbTestCase so the only thing that need to be done here is just move phpunit-includes from core yiit.php file to the application tests\bootstrap.php to avoid including phpunit-classes 2 times because codeception include them already, and to avoid error cannot redeclare class PHPUnit_Framework_TestCase.....
So as i use for unit-testing phpunit, and Codeception can run phpunit tests out-of-the-box too i think simple modifying core yiit.php class will be enough. Just need to avoid that error with cannot redeclare class. I am talking about this phpunit includes https://github.com/yiisoft/yii/blob/master/framework/test/CTestCase.php#L11

Maybe also we can ask other developers like @schmunk42 or @jacmoe if they need some extra thing, but i think all extra must go in Yii unit-tests integration with phpunit, and for Codeception we need simple bridge to run Yii unit-tests.

Also, it would be nice i think to have 2 different modules, for those reasons:

• module for browser-kit integration will have all public methods from \Codeception\Util\Framework, and they serve for page iterations, so they dont need in unit-tests. By default all public methods are put in Guy-classes. So it would be 2 modules i think, Yii2Unit (include some yii needed files for phpunit tests) and Yii2Functional (implementes WebInterface and inherits from \Codeception\Util\Framework).

Also maybe @DavertMik has some more ideas for unit-tets, dont know ;) Things that must be taken into considerations:

• Codeception does not provide full support of all phpunit features, as @DavertMik pointed there is no way to use dataproviders. I think that dataproviders may be added to codeception to support, because they are commonly used, but dont know about other phpunit features that may not be supported.

[philippfrenzel]

A Quick and dirty Fixtures Generator... Based upon the one from yii 1 TEST

---

long comment removed. content is at #347

[qiangxue]

The Response class is done, which provides wrapping of of headers and cookies.

@DavertMik For the in-memory session, is it used by functional tests or acceptance tests? Could file-based storage be used? We just need to clean up the session files after each test is done, and this would support multiple requests.

[DavertMik]

@DavertMik For the in-memory session, is it used by functional tests or acceptance tests? Could file-based storage be used?

For acceptance tests, file based storage should be used. With no alternatives.
In memory session might speed up execution a bit...
Still there should not be any problem with file based sessions for functional tests.

[qiangxue]

@DavertMik Thank you for your answer. Is there any requirement to clean up all session data? Since we have CacheSession which can make use of memory cache as storage, I think it should be fine for users to use it if speed is an issue.

BTW, do you have any suggestion how we should redistribute codeception? I'm thinking to have deep integration with codeception, meaning we will develop tools (such as Gii) to generate test skeletons for codeception. Should we include codeception.phar in the Yii release, or make codeception a dependency of Yii?

[Ragazzo]

I think that there is no need to include phar file in release, all can be done with composer, or user can download file by himself. I think for deep integration as i've described we at least will need 2 modules, 1 for functional tests (based on symfony browser-kit), another for simple integration for unit-testing and other usage. Need this 2 modules just to avoid same methods names that will be mapped to Guy class for example when user use Selenium2 and Yii functional module(they have same interface so just need 1 of them).
Hm, dont know about gii, but what kind of integration for gii do u suggest?

[schmunk42]

Just for the record, I also experimented with the phar file and installation via composer.
With the phar file I had some path issues I couldn't resolve, so I switched to composer installation which just worked almost out-of-the-box.

But I'd prefer using a global codeception installation, because installation via composer takes quite a while (without --prefer-dist) - or even better support both!

[Ragazzo]

@schmunk42 yes, i think those issues (with phar) were solved, can u check them again and if not - create issue on codeception repo? Also feel free to add some other suggestions if you want to ;)

[schmunk42]

@Ragazzo Thanks for the info, works like a charm now!

[DavertMik]

@qiangxue @schmunk42 I think Composer's version should be preferable to ship with Yii2. Currently we include too much dependencies within Codeception. If Yii2 will be installable via Composer it's better to try to include it as dependency (maybe a recommended to install). If Yii2 will be packaged as zip, a phar version can be included.

Phar version requires manual upgrades, and has limitations in viewing and editing source code. So I think it is better for testers but developers will prefer composer's package instead.

[schmunk42]

You're right, it couldn't be me who was arguing against composer ;)
Maybe a thing to note is, that if you install with --prefer-dist and you have the packages in the cache, it runs blazingly fast.

I'd say require-dev would be the right place for Codeception. So we could have a pretty fast project bootstrap with --nodev and --prefer-dist.

[qiangxue]

I have developed the Yii2 connector and module: Codeception/Codeception#387
The "basic" app is now covered by both the acceptance and functional tests.

I met some problems while developing the Yii2 connector and module:

• How should cookies be handled? While from Yii Response we can get the cookies to be sent, I don't know how to pass them to Codeception.
• The Yii User component will call session_regenerate_id() when a user logs in. This will cause the "headers already sent" problem during testing. Currently I simply disable this line when in testing mode.
• If a page has a direction (via Location header), I had to set $_POST to be empty (https://github.com/Codeception/Codeception/pull/387/files#L2R73). Otherwise, an infinite redirection will occur. Is this the right way to go, or did I miss anything?
• If I only have *.suite.dist.yml without *.suite.yml, the tests will fail. Do you think we can make the tests run with *.suite.dist.yml only? I expect *.suite.yml contains local configurations which should not be put under source control.
• By checking the code, I found Module::_initialize() and Module::_before() are called at the same stage during a test work flow. Perhaps _initialize() can be removed to avoid confusion. I thought _initialize() would be called only once before reading the code.

@DavertMik and @Ragazzo: it would be great if you could shed some lights on my above questions. Thank you!

[Ragazzo]

Well, cool, i will do some brief look shortly) So about problems:

The Yii User component will call session_regenerate_id()

I think this can be solved by some workaround with @ in session class, and with just forcing session close in _after like in Yii1 module it made now - https://github.com/Codeception/Codeception/blob/master/src/Codeception/Module/Yii1.php#L142

How should cookies be handled? While from Yii Response we can get the cookies to be sent, I don't know how to pass them to Codeception.

just pass them in connector like in Yii1, assigning to global $_COOKIE, they will be cleared in _after so thats ok.

If a page has a direction (via Location header), I had to set $_POST to be empty

i think headers can be hold in the same way as in Yii1 connector, just pass them in 3 param, location header in Yii1 connector works fine.
https://github.com/Codeception/Codeception/blob/master/src/Codeception/Util/Connector/Yii1.php#L88

If I only have *.suite.dist.yml without *.suite.yml, the tests will fail

well, need to think about this, @DavertMik ?

By checking the code, I found Module::_initialize() and Module::_before()

_before is for before each Cept/Cest-public method, and _initialize is for per 1 suite start.

I also think that we still need 2 different modules: 1 - for functional tests (you did it now), 2 - for unit-tests and basic integration with Yii2, just to have ability to use Yii in Codeception for example to run unit-tests Yii with Codeception. For example, simple Yii1Basic helper that i use is very similar to Yii1 module - Codeception/Codeception#383 (comment). But need to consider that we do not need to include PhpUnit files twice, i wrote about that above, also will pay attention to this later in comments)

Will also definitely make some review today-tomorrow and maybe say something more. Overall thanks for Yii2 support)) woot-woot :D

Also would be cool to hear other Yii developers opinions.

[Ragazzo]

Things to be noticed, $_SERVER and $_COOKIE better to make like in Yii1 connector to avoid CSRF problems (missing CSRF cookie) and other:
https://github.com/Codeception/Codeception/blob/master/src/Codeception/Util/Connector/Yii1.php#L49.
$_COOKIE and $_SERVER will be cleared in _after - after each Cept, or each Cest-public-method -
https://github.com/Codeception/Codeception/pull/387/files#L1R56

[Ragazzo]

I also think, that to show users ability of UI-mapping and other we can make those Cept's in new Cest formats, will do this (there will be Cepts and Cest's just to show users the ability of the choice). Also i think it is better to put codeception tets under separated folder, i use it in Yii1 in subfolder codeception in main Yii tests folder.

[schmunk42]

If I only have *.suite.dist.yml without *.suite.yml, the tests will fail. Do you think we can make the tests run with *.suite.dist.yml only? I expect *.suite.yml contains local configurations which should not be put under source control.

@qiangxue What's about copying the *.suite.dist.yml on create-project to *.suite.yml? See also: #510

[DavertMik]

The Yii User component will call session_regenerate_id() when a user logs in. This will cause the "headers already sent" problem during testing. Currently I simply disable this line when in testing mode.

Yep, a well known issue in Yii1. Even If you didn't use Codeception for testing, you will get this errors while testing session/auth things.

http://www.yiiframework.com/forum/index.php/topic/22802-session-regenerate-id-cannot-regenerate-session-id-headers-already-sent/

The best workaround I've seen is:

$mockSession = $this->getMock('CHttpSession', array('regenerateID'));
Yii::app()->;setComponent('session', $mockSession);

But disabling session_regenerate_id call in testing mode is much better solution then mocking.

If I only have *.suite.dist.yml without *.suite.yml, the tests will fail. Do you think we can make the tests run with *.suite.dist.yml only? I expect *.suite.yml contains local configurations which should not be put under source control.

Yep, that's a good point. .suite.dist was not documented yet, so I didn't pay proper attention to this configuration files. Sure, this will be fixed.

If a page has a direction (via Location header), I had to set $_POST to be empty
i think headers can be hold in the same way as in Yii1 connector, just pass them in 3 param, location header in Yii1 connector works fine.
https://github.com/Codeception/Codeception/blob/master/src/Codeception/Util/Connector/Yii1.php#L88

No need to set $_POST empty, but to provide proper status code, for BrowserKit to handle this redirection.
Yes, as you see, currently there is a rough hook to get redirection working :(
If only we can get status code from Yii, that will help to manage redirection easily.

By checking the code, I found Module::_initialize() and Module::_before() are called at the same stage during a test work flow. Perhaps _initialize() can be removed to avoid confusion. I thought _initialize() would be called only once before reading the code.

Yep, _initialize is called only once, at the very beginning of loading suite: https://github.com/Codeception/Codeception/blob/master/src/Codeception/SuiteManager.php#L55
Yep, probably it may be confused with _beforeSuite, but _beforeSuite is executed after all tests are loaded ad prepared for execution.

I have planned a post on Codeception internals. In 1.6.3 I refactored there a lot. So probably a time came to show how some things work and why.

[qiangxue]

How should cookies be handled? While from Yii Response we can get the cookies to be sent, I don't know how to pass them to Codeception.

just pass them in connector like in Yii1, assigning to global $_COOKIE, they will be cleared in _after so thats ok.

What if we want to remove some cookies?

If a page has a redirection (via Location header), I had to set $_POST to be empty

i think headers can be hold in the same way as in Yii1 connector, just pass them in 3 param, location header in Yii1 connector works fine.

That's how Yii2 is doing now. It will cause infinite redirections if $_POST is not set empty. I verified the status code was correct (302). Perhaps you can help review the code to see if I did something wrong.

_before is for before each Cept/Cest-public method, and _initialize is for per 1 suite start.

I also think that we still need 2 different modules: 1 - for functional tests (you did it now), 2 - for unit-tests and basic

Sorry, I read the code wrong. Actually I wanted to describe a different problem: if I modify $this->config in _initialize(), it seems the change will not be carried over to _before(). As a result, I have to move all code to _before().

I also think, that to show users ability of UI-mapping and other we can make those Cept's in new Cest formats, will do this

Could you explain a bit more about this?

[Ragazzo]

@qiangxue about cests (very similar to phpunit selenium testcase), will be good if in demo apps and boilerplates cepts and cests will be used just to show the ability of different formats.
http://codeception.com/docs/07-AdvancedUsage#Cest-Classes
http://codeception.com/05-11-2013/playing-with-cests.html

[qiangxue]

Yes, I read that before. Which format is the recommended one? I don't think we should provide both for demo purpose. We should mainly provide the best practice.

[Ragazzo]

Well, the answer is like always "It depends". If it is a very simple page and not so many actions to do on it, then simple plain Cept is enough, if it is a some CRUD or other actions that can be structured in separated blocks, then it is better to use Cest, because it allows to split some actions in protected methods, and also has _before/_after/_fail.
P.S. will have some extra time to get into problems that you've described above.

[DavertMik]

There is no best practice if we speak about formats. They are both good depending on cases you need to implement.

Actually, I think that Cest is better for cases where you need several similarly written test. For example, you want to check one form with different values. You create a Cest:

<?php

protected function fillForm($I, $credit, $email)
{
$I->amOnPage('/purchase');
$I->fillField('Credit Card', $credit);
$I->fillField('Email', $credit);
$I->click('Order');
}

public function purchaseWithValidCard($I)
{
$this->fillForm($I, '1233 3453 2342 2343', 'jon@doe.com');
$I->see('Thank you for your order');
}

public function purchaseWithInvalidCard($I)
{
$this->fillForm($I, 'xxxxxx', 'jon@doe.com');
$I->see('Sorry, your crdit card is invalid');
}

Cepts are better for more complex scenarios. If some parts scenario are common it's better to move them out to a PageObject or Controller class.

Perhaps you can help review the code to see if I did something wrong.

Sure. Do we have a demo app with some tests there?
Also I think it may be related to changes in Symfony2 BrowserKit component. I just upgraded to 2.3 few days before the Codeception 1.6.3. release.

[schmunk42]

Do we have a demo app with some tests there?

Not yet I think, but you could just install the advanced app and add some tests there.

You may have a look at my posting in #389 for an app setup with additional extensions.

Btw: I'd like to see examples and/or skeletons for all formats ... if possible gii templates would be great.

[samdark]

Basic app has some.

[qiangxue]

Thank you for explaining the cepts.:)

@DavertMik You can try the basic app template: https://github.com/yiisoft/yii2/tree/master/apps/basic
Just follow the installation instructions in the README file.
I have created both acceptance and functional test cases for every page in this app.
You just need to copy tests/acceptance.suite.dist.xml to tests/acceptance.suite.xml and modify the url option according to your installation. Do the same for tests/functional.suite.dist.xml file.

[schmunk42]

I know I've asked this before, but I still don't get the difference between functional and acceptance tests, why are they doing the same thing?!

[samdark]

@qiangxue anything to be done on this one except documentation?

[qiangxue]

I kept this open because we need to track down the last few comments in this thread. Will readdress these issues upon beta.

[qiangxue]

Closed as we have addressed all issues in this ticket.