Combining coverage of behave and pytest

[AndreLouisCaron]

Hi there!

I've recently started using behave and I hit an issue while combining the code coverage from behave and pytest in the same project.

I initially tried the simplest possible recipe:

coverage run -m pytest tests/
coverage run -a -m behave

This seems to work fine on OSX, but I'm running into issues on Windows. In short, it doesn't seem to recognize that src/mylib and .tox\py35\lib\site-packages\mylib are the same path.

I wrote a complete recipe to reproduce the issue, see AndreLouisCaron/coverage-behave-pytest.

I can temporarily work around this issue using seperate test environments for pytest and behave runs and then combining the coverage (see AndreLouisCaron/coverage-behave-pytest#2). However, this recipe is quite verbose and makes tox.ini really painful when trying to cover multiple Python versions.

Is this something you've encountered before? Any idea what's wrong?

Thanks!

P.S.: I'm willing to put some time into a fix if you can provide some basic guidance into what's causing the issue :-)

[nedbat]

You need to run coverage combine. You have the [paths] section in your configuration file, and you've noted in your README that it only applies to the combine command.

I think it could be a good idea to use the [paths] information when reporting even without combine, but that isn't how it works today.

[AndreLouisCaron]

Hi Ned, thanks for replying :-)

Indeed, I'm aware that running coveage combine solves the problem. However, I cannot directly run coverage combine: that requires running pytest and behave in separate Tox environments to get separate .coverage.* files (see my demo PR). This slows down my CI builds considerably (more venvs to install) and makes the tox.ini unwiedly when running against multiple Python versions. I would really like to avoid this...

I'd really like to understand why coverage run -m pytest and coverage run -m behave in the same virtualenv on the same machine don't produce the same paths (only on Windows). Can you provide any insight as to why that is?

Thanks!

[AndreLouisCaron]

Actually, maybe we can rephrase this in another manner: why is coverage combine not required when running on OSX?

[nedbat]

I don't know why the behavior is different on Windows and Mac. For your tox problem, use parallel=true so the files are named differently, then you don't need to use different tox environments.

[AndreLouisCaron]

Oh neat, I really like that workaround better!

[AndreLouisCaron]

Since I have a simple workaround, I'll close this issue. Thanks for the help!