New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mark/fixture to turn coverage recording off #78

Closed
The-Compiler opened this Issue Aug 12, 2015 · 12 comments

Comments

Projects
None yet
3 participants
@The-Compiler
Member

The-Compiler commented Aug 12, 2015

I don't know how feasible this is, but it'd be nice to have a mark and/or contextmanager-fixture to turn coverage collection off:

@pytest.mark.coverage_disabled
def test_foo():
    # do some integration test or whatever
    pass

def test_bar(coverage):
    with coverage.disabled():
        pass
@ionelmc

This comment has been minimized.

Member

ionelmc commented Aug 12, 2015

Definitely doable but just to understand your use case: does pytest-cov make your integration much slower?

Also, you can always have a no_coverage fixture:

@pytest.yield_fixture
def no_coverage():
    tracer = sys.gettrace()
    sys.settrace(None)
    yield
    sys.settrace(tracer)
@The-Compiler

This comment has been minimized.

Member

The-Compiler commented Aug 12, 2015

My use case is that I use coverage as a "todo-list" for the things I still need to write tests for (a lot...) - but when writing a broader integration test, I'd like that test to not "distort" my coverage.

@ionelmc

This comment has been minimized.

Member

ionelmc commented Aug 12, 2015

In my projects integration tests tend to be the best kind of tests to measure feature coverage.

Unit/component tests have the nasty tendency to rely on internals so even if they do cover code they don't really cover the feature. Their only advantage is speed.

Would the fixture solve your usecase?

PS. The pytest_runtest_call hook can be used to get the same effect, eg:

def pytest_runtest_call(item, __multicall__): # TODO: use hookwrapper
    if node.get_marker("no_coverage"):
        tracer = sys.gettrace()
        sys.settrace(None)
        __multicall__.execute()
        sys.settrace(tracer)
    else:
        __multicall__.execute()
@ionelmc

This comment has been minimized.

Member

ionelmc commented Aug 12, 2015

Alternatively, you can put integration tests in a different directory (that's what I do for one large internal project).

Here's another example: https://github.com/ionelmc/virtualenv/tree/develop/tests (but there Donald did the layout and automarker thing in conftest).

@ionelmc

This comment has been minimized.

Member

ionelmc commented Aug 12, 2015

Do you think other people would make use of such a marker? How would you name it? no_coverage? uncovered?

@The-Compiler

This comment has been minimized.

Member

The-Compiler commented Aug 12, 2015

Let me try to explain one of my usecases in more detail - I have a module configtypes which define types which values in a config file can have. Those have a transform and a validate method, and could look somewhat like this:

class Bool(BaseType):

    def validate(self, value):
        if value not in ['true', 'false']:
            raise ValidationError

    def transform(self, value):
        ...

Now I have a default config, and I want to make sure the default config is valid, so I validate it which runs all the config type's validate functions - yet those are not actually tested. Here for example I might want to test True and discover I should probably use value.lower() instead. But the default config test marks it as covered.

I have no idea if people would use it in general - I asked in #pylib, let's see if/how people respond.

@ionelmc

This comment has been minimized.

Member

ionelmc commented Aug 13, 2015

Just a note here: if we reinstall the tracer it's going to be way slower on reinstall. More details: https://bitbucket.org/ned/coveragepy/issues/397/stopping-and-resuming-coverage-with

@nedbat won't release a fix for this very soon so best to wait on this for a while?

@The-Compiler

This comment has been minimized.

Member

The-Compiler commented Oct 14, 2015

This was fixed in coverage 4.0.1 - thanks @nedbat!

@ionelmc

This comment has been minimized.

Member

ionelmc commented Oct 14, 2015

Ok lets find a good name for this marker.

Which one?

  • pytest.mark.nocover
  • pytest.mark.nocoverage
  • pytest.mark.skipcover
  • pytest.mark.skipcoverage
  • pytest.mark.no_covererage
  • pytest.mark.skip_coverage
@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Oct 14, 2015

Since the special comments to exclude lines is # pragma: no cover, I would vote for pytest.mark.no_cover, and mention it in the documentation to help people remember by association. 😁

@The-Compiler

This comment has been minimized.

Member

The-Compiler commented Oct 15, 2015

I agree with @nicoddemus - pytest.mark.no_cover makes the most sense IMHO.

@ionelmc ionelmc added this to the v2.6 milestone Oct 28, 2017

@ionelmc ionelmc closed this in a02e1c8 Nov 24, 2017

@ionelmc

This comment has been minimized.

Member

ionelmc commented Nov 24, 2017

So this is it: http://pytest-cov.readthedocs.io/en/latest/markers-fixtures.html (not released yet tho)

terencehonles added a commit to terencehonles/pytest-cov that referenced this issue Apr 14, 2018

ionelmc added a commit that referenced this issue Sep 3, 2018

bors bot added a commit to rehandalal/therapist that referenced this issue Sep 6, 2018

Merge #31
31: Update pytest-cov to 2.6.0 r=rehandalal a=pyup-bot


This PR updates [pytest-cov](https://pypi.org/project/pytest-cov) from **2.5.1** to **2.6.0**.



<details>
  <summary>Changelog</summary>
  
  
   ### 2.6.0
   ```
   ------------------

* Dropped support for Python &lt; 3.4, Pytest &lt; 3.5 and Coverage &lt; 4.4.
* Fixed some documentation formatting. Contributed by Jean Jordaan and Julian.
* Added an example with ``addopts`` in documentation. Contributed by Samuel Giffard in
  `195 &lt;https://github.com/pytest-dev/pytest-cov/pull/195&gt;`_.
* Fixed ``TypeError: &#39;NoneType&#39; object is not iterable`` in certain xdist configurations. Contributed by Jeremy Bowman in
  `213 &lt;https://github.com/pytest-dev/pytest-cov/pull/213&gt;`_.
* Added a ``no_cover`` marker and fixture. Fixes
  `78 &lt;https://github.com/pytest-dev/pytest-cov/issues/78&gt;`_.
* Fixed broken ``no_cover`` check when running doctests. Contributed by Terence Honles in
  `200 &lt;https://github.com/pytest-dev/pytest-cov/pull/200&gt;`_.
* Fixed various issues with path normalization in reports (when combining coverage data from parallel mode). Fixes
  `130 &lt;https://github.com/pytest-dev/pytest-cov/issues/161&gt;`_.
  Contributed by Ryan Hiebert &amp; Ionel Cristian Mărieș in
  `178 &lt;https://github.com/pytest-dev/pytest-cov/pull/178&gt;`_.
* Report generation failures don&#39;t raise exceptions anymore. A warning will be logged instead. Fixes
  `161 &lt;https://github.com/pytest-dev/pytest-cov/issues/161&gt;`_.
* Fixed multiprocessing issue on Windows (empty env vars are not passed). Fixes
  `165 &lt;https://github.com/pytest-dev/pytest-cov/issues/165&gt;`_.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest-cov
  - Changelog: https://pyup.io/changelogs/pytest-cov/
  - Repo: https://github.com/pytest-dev/pytest-cov
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment