Behat uses a lot of third-party libraries. These third-party libraries are sometimes (or in a Symfony2 context, almost always) loaded in the same PHP runtime environment. This could or actually causes conflicts between different versions quite easily.
One way to make Behat more robust would be to reduce dependencies, i.e. not using Symfony2 components, but that would mean more work to replicate these components.
Another alternative that I can think of is to write an automated script which prefixes the Symfony2 namespaces with for example Behat\ when the PHAR file is compiled. This should allow to use different versions of Symfony2 components inside Behat than the ones that you want to test.
Yeah, i thought about something like that myself. But this solution is actually great as it could work. Also, in the process of doing so, we could also optimize PHAR package to dump most of the classes into single file, which will also improve speed.
If you want to update PHAR compiler to do so, i'll be glad to participate :)
I'm not sure how it is done at the moment, but one problem is that all extensions would have to be compiled with the same compiler otherwise they might refer to classes which do not actually exist anymore.
Yeah, there is common compiler script, that most extension devs use (as far as i know) - build.php. We could just provide more robust one, with what we need. Even in 3rd party repo (ExtensionCompiler).
But the problem is - those extensions (*.phar) wouldn't be compatible with normal (composer) Behat and normal (composer) extensions would not be compatible with behat.phar, which could cause frustration. Especially if we're talking bout custom user extensions (user tries to create his own extension with behat.phar). Hmmm.
Are third-party classes usually used in extensions?
Well, yeah. Especially configuration loaders. We could normalize that, of course, hiding those dependencies inside Behat core, but it still could cause problems for someone, who'll want to extend/reuse something from the Symfony2 core :/
I have made an initial implementation for this. We should probably add some checks to ensure that only compiled extensions can be used with a compiled behat.phar file.
Also, we probably have to make it easier to compile an extension to a phar file. I don't know what the current process is here. Would you mind taking a look?
Overall, this should allow Behat to be used for testing any Symfony version, and make it less fragile, and more independent. Also, the maintenance burden should go down considerably as changes in Symfony2 will not immediately break Behat anymore.