Behat 3 Support

[chillu]

WIP under https://github.com/silverstripe-labs/silverstripe-behat-extension/tree/experiments/behat-3-upgrade

Requires CMS and Framework branches as well through composer.json:

{
    "name": "silverstripe/installer",
    "description": "The SilverStripe Framework Installer",
    "require": {
        "php": ">=5.3.2",
        "silverstripe/cms": "dev-experiments/behat-3-upgrade",
        "silverstripe/framework": "dev-experiments/behat-3-upgrade",
        "silverstripe-themes/simple": "*"
    },
    "minimum-stability": "dev",
    "require-dev": {
        "silverstripe/behat-extension": "dev-experiments/behat-3-upgrade"
    }
}

Since the extension was heavily based on the Symfony2Extension, tracking changes required to get that extension working will provide at least an indication what needs to change (https://github.com/Behat/Symfony2Extension). Specifically this commit: Behat/Symfony2Extension@65c80f2

Also useful are the Magento Extension upgrade and the Drupal Extension upgrade (1, 2, 3)

Tasks (incomplete):

• Change ModuleContextClassGuesser to implement SpecificationLocator (and use in Extension->load())
• Change InitProcessor to implement ClassGenerator (and use in Extension->load())
• Update actual PHP generated by InitProcessor
• Refactor core.xml to PHP in Extension->configure() (unclear benefits?)
• Replace Step\Given namespaces with Call\Given throughout all tests (cms, framework, modules)
• Update extension README
• Write upgrading guide for projects using Behat+SilverStripe
• Upgrade PHPUnit to 4.1 (see Updating composer to pull in PHPUnit 4.1 #52)

[chillu]

@jeffreyguo @srizzling Made a start on this, but its quite complicated - they've changed pretty much everything, without upgrading docs.

[srizzling]

http://docs.behat.org/en/v3.0/ These are less technical docs to follow as well. Would need a task to migrate the regex definitions to use the new twig. Although, from what I know its BC.

[srizzling]

Another important document is the changelog https://github.com/Behat/Behat/blob/master/CHANGES.md

[chillu]

@srizzling I've found those guides are way too highlevel to describe the actual changes required.

[srizzling]

So I have tried to start this but my composer complains

Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - behat/mink-selenium2-driver v1.1.1 requires behat/mink ~1.5.0 -> no matching package found.
    - behat/mink-selenium2-driver v1.1.0 requires behat/mink ~1.5.0 -> no matching package found.
    - behat/mink-selenium2-driver v1.0.6 requires behat/mink ~1.4.3 -> no matching package found.
    - behat/mink-selenium2-driver v1.0.5 requires behat/mink ~1.4.3 -> no matching package found.
    - behat/mink-selenium2-driver v1.0.4 requires behat/mink >=1.4,<1.5-dev -> no matching package found.
    - behat/mink-selenium2-driver v1.0.3 requires behat/mink >=1.4,<1.5-dev -> no matching package found.
    - behat/mink-selenium2-driver v1.0.2 requires behat/mink >=1.4.0,<1.5.0 -> no matching package found.
    - behat/mink-selenium2-driver v1.0.1 requires behat/mink >=1.4.0,<1.5.0-dev -> no matching package found.
    - behat/mink-selenium2-driver v1.0.0 requires behat/mink >=1.4.0,<1.5.0-dev -> no matching package found.
    - Installation request for behat/mink-selenium2-driver *@stable -> satisfiable by behat/mink-selenium2-driver[v1.0.0, v1.0.1, v1.0.2, v1.0.3, v1.0.4, v1.0.5, v1.0.6, v1.1.0, v1.1.1].

Potential causes:
 - A typo in the package name
 - The package is not available in a stable-enough version according to your minimum-stability setting
   see <https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion> for more details.

Read <http://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.

This seems to me that @stable doesn't work as it should on behat/mink-selenium-driver so I havve vhanged it to "~1.1". Not sure its what we wanted to do :)

[chillu]

Should be fine, wanna commit that to the composer.json?

[srizzling]

I will till I ensure its what its required composer took a long time install everything :/

[srizzling]

Brain Dumping for my sake :P:

ModuleContextClassGuesser is how we are able to write tests for separate modules by specifying via @cms for example.

https://github.com/Behat/Behat/blob/master/src/Behat/Testwork/Specification/Locator/SpecificationLocator.php

SpecficationLocator requires two methods to be implemented:

• public function getLocatorExamples() - This method returns documentation/examples for how to specify different locators. Example provided here: https://github.com/Behat/Symfony2Extension/blob/master/src/Behat/Symfony2Extension/Specification/BundleFeatureLocator.php#L42

Subtasks Required to make this extension work:

• Folder Structure needs to change, Need to move ModuleContextClassGuesser into Specfication folder and Create a ModuleContextClassLocator.php in the Specfication namespace
• Depreciated ModuleContextClassGuesser and moved to depreciated folder (for now)
• Make ModuleContextClassLocator implement SpecificationLocator
• Need to make Suite Class that encapsulates the SilverStripe Modules
• Load locator in the Extenstion file

[srizzling]

Not much of a licence but given this module is based off Symfony quite heavily what are requirements?
As long as we use reference this file?

https://github.com/Behat/Symfony2Extension/blob/master/LICENSE

[srizzling]

Sorry for the spam :P.

@chillu original issue comment points to changes made to Symfonys behat upgrade. It seems that commit is quite old and a few changes have been made since then. The changes described are enough to get a working copy of behat 3 tho.

Renaming Extension.php:
Behat/Symfony2Extension@a9fea2e

This is an interesting change the reason says "to benefit from the behat shortcuts". Don't really know what that means... unclear benefits?

[tractorcow]

@chillu change your minkselenium2driver to 1.2.*-dev

[srizzling]

Is it 1.2.*`dev it worked as is currently but I guess locking it down is easier :P

[tractorcow]

There's no stable release for 1.2 I'm afraid, but 1.1.1 (latest stable) is against the old mink 1.5 API.

[markguinn]

Are there any plans for this task to be completed any time soon or are you planning to stay with Behat 2 for the foreseeable future?

[tractorcow]

I think for 4.0 we should probably upgrade behat. But, it's not currently on the official roadmap, so it'll have to wait until someone finds time to finish it. ;)

[markguinn]

Do you have any estimate how big of a task this is for a reasonably good programmer? Are we talking 10 hours or 100 hours or more than that?

[chillu]

I've spent a few hours on it, and it still looked pretty complicated - the whole Behat internals changed. Gut feel would be 2-3 days of work.

[dhensby]

+1 for codeception instead :P

[markguinn]

Codeception does make a lot of sense if developers are writing the tests (which is almost always the case with a SilverStripe module)...

[stevie-mayhew]

Codeception makes more sense, especially if SS 4.0 is going to be using PSR-4. Not sure if thats the case, but it should be ;)

[dhensby]

It will be using PSR4 I believe

[stevie-mayhew]

@dhensby as we talked about on Gitter I'm pretty interested in codeception right now and would be willing to put time in getting it up and running for SS. @chillu thoughts / feelings for that against behat?

I realise there is a huge amount of work that has gone into the behat framework and testing for SS and don't want to step in and change something that you have internal knowledge or resource for

[dhensby]

I am 100% for codeception over separated unit tests/behat tests.

It provides a more consistent and simple framework for testing - it's just a big time investment at the moment and I couldn't see a nice way to splitting tests across modules like we do at the moment. Though I only looked into it for half-an-hour about a year ago.

[chillu]

I'd be interested to hear success stories of Codeception first before making that call. It sounds a bit like throwing out the baby with the bathwater: Behat is slow and complex, so Codeception must automatically be better because it looks more like PHP ;)

• Developer friendlyness: Writing Cucumber through regexes and remembering step definitions is a pain. Codeception makes composition easier by staying within the PHP language, e.g. using method hinting in IDEs like other PHP code.
• Client friendlyness: Behat tests in Cucumber are more likely to be written in a "business readable" fashion, although you can bend your Codeception PHP code that way as well. In practice, we haven't been able to meaningfully engage a client in this process here at SilverStripe Ltd., and don't write our acceptance criterias in Cucumber. So the value is a bit limited. I think this is mainly due to other aspects like CI and performance though.
• Performance: I don't think Behat/Mink on their own are particularly slow, it's just the level of SilverStripe overhead we're adding through flush and recreating the database state on each scenario. I don't see an easy way around this in Codeception either. Both Mink and Codeception support headless browsers and PHP emulated browsers. So I'd rather invest the energy in getting that working in the current Behat structure.
• Stability: End-to-end testing tends to be fragile and hard to maintain because of all the moving parts. In my opinion, most of these complexities come from the layers which Behat and Codeception share: Selenium and Browsers. That's the case even in other languages like Ruby. Plus, multi-process debugging is hard.
• Same tools for unit, integration and end-to-end tests: That's a point for Codeception (same tooling everywhere). It's possible to write BDD-style unit tests with Behat as well, but becomes less attractive due to the Cucumber language. Unit tests are still just PHPUnit, with a preferred mocking framework - so not a strong driver to switch end-to-end testing tools. Moving all core unit tests to BDD-style Codeception is a massive investment. And unless we're prepared to do that, there's not really much substance to the "same tools everywhere" claim.
• Testing component reuse: We already use the same bootstrap, FixtureFactory and database scaffolding in Behat and unit tests. Is there other cases where Behat use is causing duplication? The testsession module is independent of Behat already.
• API tests: Neat feature available in Codeception, but doesn't strike me as essential. There'll be other libraries to solve this alongside PHPUnit.
• Functional tests: End-to-end tests (both in Behat and Codeception) are a very costly approach that should be used in moderation. We're generally not using functional tests enough: Testing multiple components, but not necessarily end-to-end. FunctionalTest in SilverStripe is underpowered and using an outdated framework (SimpleTest), so Codeception could help here. Either way, I'd expect more frontend testing to move into JavaScript/Node-based layers as we adopt a more frontend-heavy approach in SilverStripe CMS. One example could be testing a React+API-powered GridField via Selenium, but with API mock data rather than a slow SilverStripe backend. Here's an interesting take from Google about the testing pyramid and end-to-end testing: http://googletesting.blogspot.co.nz/2015/04/just-say-no-to-more-end-to-end-tests.html.

That being said, I'm not "married" to Behat ;) The missing upgrade path and its overarchitected internal structure has annoyed me quite a bit already. I'm just quite aware that we have many other challenges in core in the next months, and this seems like a distraction. Most of the code in the SS-specific Behat feature files would be quite easy to transfer to Codeception though (e.g. dealing with Select2, CMS ajax behaviour), since it's all just Selenium interactions anyway. So happy to be convinced if somebody feels like throwing together a proof of concept.

[sminnee]

My view is that if someone out there wanted to do a complete port of the CMS or Framework Behat test suite to Codeception, I'd support its merge, but I wouldn't want to see the tests of a single module sit in a state of being half Behat and half Codeception. I'm not sure what @chillu and the other core committers think of that? Last time I spoke to @dhensby he was keen on Codeception.

Of course, as PoC it would probably be best to start with 1 or 2 tests to see how it looks.

I tend to agree with Ingo that Behat/Codeception isn't the biggest pain point, but this is open-source: people can scratch whichever itch frustrates them the most.

[anselmdk]

Just a little support for Behat here.
Beyond using Behat to use frontend functionality, I've now also started to use it to test CMS UX on my projects, and really enjoy how smooth the implementation is. You can actually do quite a lot without having to write any PHP.
I of course haven't tried Codeception, and I agree Behat can be cumbersome to deal with, but I feel the benefits outweight the issues I'm having with it.

[chillu]

Laravel has just brought out a nice lightweight browser testing library as well: https://laravel.com/docs/master/dusk. It runs without a Java/Selenium dependency by default, but you can opt-in to that for additional browser support. Doesn't support Cucumber, but I don't think that's necessarily a blocker.

We'll likely need to get some movement here soon, since the Symfony 2.x dependencies in Behat 2.x are holding us back in other areas where dependencies overlap (e.g. Symfony YAML)

[tractorcow]

If the amount of work is 2-3 days to upgrade I'll look at taking it on in a side project.

[tractorcow]

I've rebased and updated the prior branch. WIP at:

https://github.com/tractorcow/silverstripe-framework-behat3
https://github.com/tractorcow/silverstripe-behat3
https://github.com/open-sausages/silverstripe-testsession/tree/pulls/4.0/upgrade-4
https://github.com/open-sausages/silverstripe-config/tree/experiments/behat-3-upgrade
https://github.com/open-sausages/silverstripe-installer/tree/pulls/4.0/behat-3-upgrade
https://github.com/open-sausages/silverstripe-cms/tree/pulls/4.0/behat-3-upgrade
https://github.com/open-sausages/silverstripe-asset-admin/tree/pulls/4.0/behat-3-upgrade

Still lots and lots of work to do.

[tractorcow]

I've made some progress!

[tractorcow]

@chillu this is the new part of the loader; Instead of having a single behat.yml config file, we inspect the root for a configured and named suite, and if not found we dynamically load this from the @named module.

https://github.com/tractorcow/silverstripe-behat3/blob/experiments/behat-3-upgrade/src/Controllers/ModuleSuiteLocator.php

This means we can use composable configs, and the root config doesn't need to have all per-module configs setup.

[tractorcow]

Can you please review this loader to see if this is heading in the right track?

[tractorcow]

Updated: All framework tests are now passing in behat 3 :) moving onto cms

[tractorcow]

most cms tests green, a few failed.

[tractorcow]

--- Failed scenarios:

    cms/tests/behat/features/duplicate-a-page.feature:14
    cms/tests/behat/features/manage-page-permissions.feature:46
    cms/tests/behat/features/manage-page-permissions.feature:52
    cms/tests/behat/features/preview-a-page.feature:10
    cms/tests/behat/features/preview-a-page.feature:25

Not bad.

[tractorcow]

--- Failed scenarios:

    cms/tests/behat/features/manage-page-permissions.feature:46
    cms/tests/behat/features/manage-page-permissions.feature:52

getting closer.

[tractorcow]

CMS tests green, :D

[tractorcow]

framework, asset-admin, and cms tests are all done.

Just need to finish siteconfig tests and ready to PR.

[tractorcow]

PR at #152

[tractorcow]

All merged and green.