Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upAdd support for Codeception Test Framework #800
Conversation
src/TestFramework/Codeception/Config/AbstractYamlConfiguration.php
Outdated
Show resolved
Hide resolved
This comment has been minimized.
This comment has been minimized.
|
Following. Unfortunately was never able to get back to this. Spending 60+ hours in your day job tends to have that effect. |
|
Great work @maks-rafalko, i got some minor nit picks right now, but i'll try and really check it out sometime soon |
...t/TestFramework/Codeception/Config/Builder/MutationConfigBuilderTest.php
Outdated
Show resolved
Hide resolved
tests/phpunit/TestFramework/Codeception/Adapter/CodeceptionAdapterTest.php
Outdated
Show resolved
Hide resolved
tests/phpunit/TestFramework/Codeception/Adapter/CodeceptionAdapterTest.php
Outdated
Show resolved
Hide resolved
src/TestFramework/Codeception/Config/InitialYamlConfiguration.php
Outdated
Show resolved
Hide resolved
| final class CodeceptionAdapterTest extends TestCase | ||
| { | ||
| /** | ||
| * @var CodeceptionAdapter|MockObject |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
maks-rafalko
Oct 8, 2019
•
Author
Member
I get
PHPDoc tag @var for property Infection\Tests\TestFramework\Codeception\Adapter\CodeceptionAdapterTest::$adapter contains unresolvable type.we don't use https://github.com/phpstan/phpstan-phpunit because we use PHPStan PHAR at the moment
This comment has been minimized.
This comment has been minimized.
BackEndTea
Oct 8, 2019
Member
Have you tried reseting cache in between that? I've been having some issues with phpstan and cache of doc annotations lately
This comment has been minimized.
This comment has been minimized.
maks-rafalko
Oct 9, 2019
Author
Member
Where is the phpstan cache located? I don't see anything related to it in our command lines and in the --help output.
I tried to run phpstan from the different folder - same result
This comment has been minimized.
This comment has been minimized.
BackEndTea
Oct 10, 2019
Member
I dont' really know where it stores it by default, my best guess would be somewhere in /tmp
This comment has been minimized.
This comment has been minimized.
dirx
commented
Oct 17, 2019
|
Looks very promising! I tested the current state and run into relative path issue with DB module and the dump path(s) configuration - reading sql files is based on the codeception project dir. |
This comment has been minimized.
This comment has been minimized.
|
Ok, could you please create a minimum reproducable repository I can look at? or is it a public repo? |
This comment has been minimized.
This comment has been minimized.
dirx
commented
Oct 17, 2019
|
No public repo - maybe this helps to dig into this: Codeception sets its project dir based on the config yml file: In DB Module project dir is then used as base path to read sql files: Running codecept with initial / mutation yml configurations in infection tmp path then sets "wrong" relative project dir. |
This comment has been minimized.
This comment has been minimized.
|
Not enough information unfortunately. Could you please at least show your folder structure and |
This comment has been minimized.
This comment has been minimized.
dirx
commented
Oct 17, 2019
|
@maks-rafalko Please take a look at https://github.com/dirx/infection-codeception-test |
This comment has been minimized.
This comment has been minimized.
|
Thank you very much for creating a repo, that helped a lot. Really appreciate it. Unfortunately, you've discovered a serious problem. Let me explain in details (would also be useful for those who don't know how Codeception works with config files). So, we have the following nested configs in your repo: ├── codeception.yml
└── tests
├── _data
├── _output
├── _support
├── integration
└── integration.suite.ymlIt means that Codeception merges content of Infection replaces only main codecept run --config /app/./infection/codeception.initial.infection.yml [...]But That's why all paths from I think we can't do anything with it, without seriously patching Codeception itself. But, I found something interesting. We can declare all suites in the main This is what I did in your repository, and with the latest commit it works!
# codeception.yaml
[..]
suites:
integration:
actor: IntegrationTester
modules:
enabled:
- \Stuff\Helper\Integration
- Db:
dsn: 'sqlite:stuff.sq3'
user: 'test'
password: 'test'
dump:
- 'tests/_data/stuff.sql'
cleanup: false
populate: true
step_decorators: ~I would say that moving suites's configuration to the main file can be a good solution without any changes in the code of Codeception and Infection itself. Otherwise, we need to think about what we can do with nested configuration files that are merged by Codeception itself, which can be a very difficult task. Please, pull the latest commit and let me know if it works for you! Thank you. |
This comment has been minimized.
This comment has been minimized.
dirx
commented
Oct 20, 2019
|
@maks-rafalko thank you for taking a deeper look into this. Looks like this workaround might work, but probably not for other codeception config scenarios. For example "extends:" would not work without rewriting paths again. Rewriting paths seems to bring a lot of trouble. Why not staying with the original codeception config (and path) and overriding config values needed for the initial and all the mutation runs? Initial command line might look like: vendor/codeception/codeception/codecept run --no-colors --fail-fast \
--config ./codeception.yml \
--coverage-phpunit ./infection/codeception-coverage-xml \
--xml ./infection/junit.xml \
--override 'extends: ./infection/codeception.initial.infection.yml'Mutation command line might look like: vendor/codeception/codeception/codecept run --no-colors --fail-fast \
--config ./codeception.yml \
--coverage-phpunit ./infection/codeception-coverage-xml \
--xml ./infection/junit.xml \
--override 'extends: ./infection/codeceptionConfiguration.03a5c2544152d2cbf1e679057880c806.infection.yaml'Unfortunately Codeception does currently not support Until then config values have to be explicitly overridden: Initial command line: vendor/codeception/codeception/codecept run --no-colors --fail-fast \
--config ./codeception.yml \
--coverage-phpunit ./infection/codeception-coverage-xml \
--xml ./infection/junit.xml \
--override 'paths: output: ./infection' \
--override 'coverage: enabled: true' \
--override 'coverage: include: ["src/*.php"]'Mutation command line: vendor/codeception/codeception/codecept run --no-colors --fail-fast \
--config ./codeception.yml \
--group infection \
--override 'paths: output: ./infection/63ddd408b7cbd80f5c4ce426fcf9eeb4' \
--override 'coverage: enabled: false' \
--override 'bootstrap: ./infection/interceptor.codeception.63ddd408b7cbd80f5c4ce426fcf9eeb4.php' \
--override 'groups: infection: [/app/tests/integration/DatabaseTest.php]'This approach would allow the current suite based config structure (or any other) and simplify path handling a lot (you may not even need it). Preparing mutation command line will be challenging but it looks doable - maybe a generic transformation of yaml to cli config values... What do you think? |
This comment has been minimized.
This comment has been minimized.
dirx
commented
Oct 21, 2019
•
|
@maks-rafalko |
This comment has been minimized.
This comment has been minimized.
|
Interesting idea ;) Didn't even think about it. Unfortunately, after digging into existing issues of Codeception, I don't think it's possible.
As I understand, we are able to Other things to note:
Don't get me wrong. I'm not against this solution, just throwing my thoughts to discuss. If I'm wrong and it can work, I will be happy to implement it.
Exactly. But this is not our fault, but how Codeception works:
If we had only 1 config with absolute paths, we would not have all these issues. Personally, I would better require people, who want to use Infection+Codeception, to use only one @DavertMik I would really appreciate your input. If you have a chance, please read starting from this comment #800 (comment) |
This comment has been minimized.
This comment has been minimized.
|
I would imagine it'd be pretty simple to add a new CLI option of One thing I am noticing is that when I was working on it, I added logic to recursively update the paths on all keys that start with |
This comment has been minimized.
This comment has been minimized.
|
Better yet, I think i did |
This comment has been minimized.
This comment has been minimized.
dirx
commented
Oct 22, 2019
Is there any need to override suite based config values? I only see global config values (when there is no need to rewrite paths). According to current configuration builder those are:
As I understand suite configs can then only override suite based config values in global config. This won´t be an issue either.
The limited set of config values that need to be overriden plus using the param short form
This might need extra handling -
True: it would work differently then for the other testframe adapters, but would be a slim solution with little (path rewriting) magic. |
This comment has been minimized.
This comment has been minimized.
What about for example Also, Anyway, let me give your idea a try in a separate branch since we all are not 100% sure on how Codeception works internally with merging configs and what issues we can face with. I will report back when I have news in the following couple of days. |
This comment has been minimized.
This comment has been minimized.
Unfortunately, suite's config can override Example# unit.suite.yml
...
# note that this value overrides *global* coverage setting even when it is located in the suite's config
coverage:
example: truecommand line: vendor/bin/codecept run -vvv --override 'coverage: enabled: false'Actual result inside Codeception code after merging / overriding: [coverage] => Array
(
[enabled] => true
)Conclusion:
This is not true. For example POC with
|
This comment has been minimized.
This comment has been minimized.
dirx
commented
Nov 2, 2019
|
@maks-rafalko Thank you for providing the override implementation. I did a quick test run with that feature branch on a smaller project with a simple unit & integration suite setup - in fact i did 3 runs with variations on |
This comment has been minimized.
This comment has been minimized.
|
I completely rewrote implementation with the @dirx's idea and already merged it to this PR.
It is ready for review. I'm planning to merge it soon and release, don't want to freeze it anymore. So please, anyone interested - review.
|
This comment has been minimized.
This comment has been minimized.
alfredbez
commented
Nov 20, 2019
Looks like the Deploy Step is not green, but canceled. Maybe it helps if you rerun the travis build |
|
I don't feel qualified to do give an approval since I'm no Codeception user. However the code looks good to me overall aside from a minor points. It looks though that this could be split in two here: one PR which tweaks the framework adapter approach and another one taking advantage of the first to introduce the Codeception support. |
| @@ -5,7 +5,7 @@ sudo: false | |||
| env: | |||
| global: | |||
| - INFECTION_FLAGS='--threads=4 --min-msi=67 --min-covered-msi=75 --coverage=coverage --log-verbosity=none' | |||
| - PHPUNIT_BIN='vendor/bin/phpunit --prepend=devTools/xdebug-filter.php --coverage-clover=clover.xml --coverage-xml=coverage/coverage-xml --log-junit=coverage/phpunit.junit.xml' | |||
| - PHPUNIT_BIN='vendor/bin/phpunit --prepend=devTools/xdebug-filter.php --coverage-clover=clover.xml --coverage-xml=coverage/coverage-xml --log-junit=coverage/junit.xml' | |||
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
maks-rafalko
Nov 23, 2019
Author
Member
well, I extracted a couple of PRs from here. Others: benefits at this point of time, when it is already here and coupled to the changes in this PR
| @@ -48,7 +48,7 @@ install: | |||
|
|
|||
| test_script: | |||
| - cd c:\projects\workspace | |||
| - vendor\bin\phpunit --prepend=devTools/xdebug-filter.php --coverage-clover=clover.xml --coverage-xml=coverage/coverage-xml --log-junit=coverage/phpunit.junit.xml | |||
| - vendor\bin\phpunit --prepend=devTools/xdebug-filter.php --coverage-clover=clover.xml --coverage-xml=coverage/coverage-xml --log-junit=coverage/junit.xml | |||
This comment has been minimized.
This comment has been minimized.
src/TestFramework/PhpUnit/Config/Builder/MutationConfigBuilder.php
Outdated
Show resolved
Hide resolved
…ling it each time we need to run test framework
… 3.1.0
… is provided by user
…d by speed
…implementation
… parameters required
…` file
…nd Mutation test runs
This comment has been minimized.
This comment has been minimized.
|
I think everyone who wanted to review this PR - did it, since it was opened 2 months ago. Feel free though to comment, test and suggest anything regarding this PR. Since I'm not a Codeception user and don't have real projects - users who use Codeception can help a lot testing it. I'm merging and planning to release it soon with Symfony 5 & PHP 7.4 support. Thank you @dirx, @BackEndTea, @theofidry, @Fenikkusu |
d803a76
into
master
maks-rafalko commentedOct 2, 2019
•
edited
This PR adds support for Codeception Testing Framework as this is the most requested feature in Infection issues.
Before reviewing, I would like to describe a couple of things so please make sure read them.
Why another PR?
There were 2 PRs trying to add support for Codeception:
Unfortunately, none of them could work (not even close). I will explain below why.
However, @tobiasstadler & @Fenikkusu did a great job moving the whole process forward and patching Codeception itself. Thank you guys!
Explanation
Previous PRs:
So, instead of fixing all those issues I decided to start from scratch, borrowing some ideas from that 2 PRs.
What was done
During the integration, there was 2 types of issues: blockers and things that are very annoying in Codeception that led me write strange code.
Blockers:
3.1.0] It was not possible to have one bootstrap file in Codeception. Fortunately, we fixed the issue in Codeception. Without this change it's impossible (or extremely hard to) using Codeception with Infection. (we are using bootstrap file to add Interceptor)3.1.1] Ability to use absolute and relative paths ingroups Codeception/Codeception#5674. This was needed to run particular tests that cover mutated line of the code. Without this, running all the tests for each Mutant is practically useless.Annoying issues:
These issues are workarounded in the code, but fixing them in the Codeception will make the Infection code a lot easier & understandable. These are not blockers though.
coverage-phpunitin the config file instead of as CLI option Codeception/Codeception#5643Implementation details
This PR adds integration with Codeception
^3.1.1. Working with the previous versions potentially possible, but requires extremely huge amount of time, dirty workarounds to avoid Codeception bugs. I'm not sure we need it, at least for now.Codeception adapter supports:
JUnitreport. It means that InfectionBC Break
The name of the JUnit report file has been changed from
phpunit.junit.xmltojunit.xml. This is done to have common approach for PHPUnit / Codeception and any other future Test Framework.Now, providing code coverage for PHPUnit and Codeception requires exaclty the same files.
Future plans
I'm really going to extract Test Framework Adapters to the separate packages. And Codeception is a good evidence that this must be done. Why? One of the many reasons: because we can't say in Infection's
composer.jsonthat werequire: codeception/codeception:^3.1.1because not all people use codeception as a test framework. Instead, we should introduceinfection/coreandinfection/codeception,infection/phpunit(names are just to understand the goal). More on it in a separate issue.Last words: I'm not going to support every project's codeception test suite because there are tons of settings and cases in Codeception where Infection can fail. IMO we should stabilize this basic implementation, merge it, and keep improving it step by step to not have this PR open for ages. Unfortunately, this takes too much time (debugging Codeception itself and patching it).
This change is