New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Your tox / coverage setup should not work, but it does #1

Closed
rfleschenberg opened this Issue Oct 7, 2015 · 6 comments

Comments

Projects
None yet
4 participants
@rfleschenberg
Copy link

rfleschenberg commented Oct 7, 2015

tox installs the package into the virtualenv it creates, then runs the tests inside that virtualenv. However, the source option from .coveragerc points at the source tree, not at the installation inside the tox virtualenv. As a result, coverage should not collect any data, and you should get an empty coverage report. But you don't.

Apparently, the tests run at least partly not in the tox virtualenv, but in the source tree.

This can be confirmed when modifying the test command to show some debugging information (thanks to @nedbat for suggesting this):

commands =
    coverage run --debug=sys,config,trace -a {envbindir}/py.test -s tests   

Sample output:

tests/test_tx.py Tracing '/tmp/prometheus_async/prometheus_async/aio/__init__.py'

Right now, I have no idea what causes this. I will try to investigate this further. However, you probably want to fix this, since it may cause unintended effects further down the road. It may be worthwhile to investigate the changedir option of tox (thanks to @brechtm for the suggestion).

@brechtm

This comment has been minimized.

@rfleschenberg

This comment has been minimized.

Copy link
Author

rfleschenberg commented Oct 7, 2015

One possible workaround for the issue you should be seeing (but are not) is to use usedevelop=True in tox.ini (again, thanks @nedbat).

As an aside, py.test recommends not having an __init__.py in your tests dir. I don't think this is related, though.

@hynek

This comment has been minimized.

Copy link
Owner

hynek commented Oct 9, 2015

I’m traveling so I can’t look into it deeper but it seems like a mistake most people do. ;) It probably works, because the source code within the tree and under tox are the same?

I think this is also the reason some people put their packages into a separate src directory, so they avoid accidental imports.

I know about the __init__.py in tests thing but sometimes I just like to import things in my tests… :)

@brechtm

This comment has been minimized.

Copy link

brechtm commented Oct 16, 2015

For my project, I'm now making use of the changedir option, setting it to tox's {envtmpdir}. This directory is emptied before the test commands are executed, so no installed modules can be shadowed.

This had the annoying side-effect that the coverage report output contained the absolute path to my package's modules. To obtain relative paths, it is necessary to run coverage report from the site-packages directory where tox installs my package. See my tox.ini and coveralls.py (which should really be named coverage.py) how this can be done.

This approach does make it more difficult to combine --parallel runs though, I suppose. I'm currently only testing coverage in a single tox environment.

@nedbat

This comment has been minimized.

Copy link

nedbat commented Oct 16, 2015

@brechtm have you looked at the [paths] option in the .coveragerc? http://coverage.readthedocs.org/en/coverage-4.0.1/config.html#paths

@brechtm

This comment has been minimized.

Copy link

brechtm commented Oct 16, 2015

@nedbat Thanks. I was vaguely aware of the [paths] section :). That should indeed help combining the coverage output from several tox enviroments. Now if only we could coverage combine after all others when using detox...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment