New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More tolerant in certain circumstances where report generation was failed #161

Closed
huyphan opened this Issue Jun 21, 2017 · 17 comments

Comments

Projects
None yet
4 participants
@huyphan

huyphan commented Jun 21, 2017

Back in version 2.2.1 we were more tolerant when something went wrong with coverage report generation:

        if not (self.failed and self.options.no_cov_on_fail):
            try:
                total = self.cov_controller.summary(terminalreporter.writer)
            except CoverageException as exc:
                terminalreporter.writer.write('Failed to generate report: %s\n' % exc)
                total = 0

With 2.5.1, we raise exception instead:

        if not self._is_slave(session) and self._should_report():
            try:
                self.cov_total = self.cov_controller.summary(self.cov_report)
            except CoverageException as exc:
                raise pytest.UsageError(
                    'Failed to generate report: %s\n' % exc
                )
            assert self.cov_total is not None, 'Test coverage should never be `None`'

Do you think it would make sense if we make it to be tolerant like the old days? I'm having some projects that are built on multiple Python implementations with the same code base. One of the implementation (Jython) doesn't support code coverage and will cause CoverageException. If we re-raise the exception here, the whole test will fail.

I can send a PR for this, but I would like to get some opinions first.

@nicoddemus

This comment has been minimized.

Member

nicoddemus commented Jun 21, 2017

Optionally a new ini option:

[pytest]
cov_fail_no_coverage = false
@ionelmc

This comment has been minimized.

Member

ionelmc commented Jun 21, 2017

Not saying no but I'm not a huge fan of yet another option.

If coverage is broken on jython why are you running it there? what's the point?

And the old behavior could be brought back, sure, but then coverage failure suddenly become a silent problem. How will people notice and act upon it if it's just one line in a ocean of lines of the output?

@huyphan

This comment has been minimized.

huyphan commented Jun 21, 2017

In my use case, I use the same code base (including setup.py and pytest configurations) to build the package under multiple Python implementations. I still want to have code coverage report in Python2, Python3 etc. I wish there were an easy way to change pytest parameters under different Python implementations.

I agree that bringing back the old behavior is not a good idea for the reason you mentioned. However making it a new option is an appropriate practice I guess. We can default that option to true to maintain the current behavior and still allow people to have a choice to ignore coverage issue.

@huyphan

This comment has been minimized.

huyphan commented Jul 18, 2017

@ionelmc: Do you think that's fine to move forward with cov_fail_no_coverage ini option then?

@ionelmc

This comment has been minimized.

Member

ionelmc commented Jul 18, 2017

Lemme dig a bit in the history. Maybe we can have the old behavior back again (without option).

@ionelmc

This comment has been minimized.

Member

ionelmc commented Jul 18, 2017

Looks like this new behavior was introduced in 3656515 (#116).

I think it's safe-ish to revert. @davidszotten, you have anything against? It's about making this sort of change:

diff --git a/src/pytest_cov/plugin.py b/src/pytest_cov/plugin.py
index 05ac039..e02f6cd 100644
--- a/src/pytest_cov/plugin.py
+++ b/src/pytest_cov/plugin.py
@@ -238,9 +238,8 @@ class CovPlugin(object):
             try:
                 self.cov_total = self.cov_controller.summary(self.cov_report)
             except CoverageException as exc:
-                raise pytest.UsageError(
-                    'Failed to generate report: %s\n' % exc
-                )
+                terminalreporter.writer.write('Failed to generate report: %s\n' % exc)
+                self.cov_total = 0
             assert self.cov_total is not None, 'Test coverage should never be `None`'
             if self._failed_cov_total():
                 # make sure we get the EXIT_TESTSFAILED exit code
@davidszotten

This comment has been minimized.

Contributor

davidszotten commented Jul 19, 2017

hi. thanks for pinging me. i'd like to reply in more depth than i have time for right now, but wanted to at least respond.

to me "failed to generate a report" feels like a failure condition, not something we should just warn about and silently continue

having said that i don't recall the exact reason i cared about this, which might have been mainly around not hiding the coverage output when fail-under triggered. presumably even with the patch above, given setting total to 0, fail-under would still result in an error if report generation failed

alternatively, maybe with more context we could help op pass arguments conditionally based on environment to not request options they expect to fail

@ionelmc

This comment has been minimized.

Member

ionelmc commented Jul 19, 2017

Well there is PYTEST_ADDOPTS, which @huyphan could set for non-jython envs. Eg: PYTEST_ADDOPTS=--cov

@ionelmc

This comment has been minimized.

Member

ionelmc commented Jul 19, 2017

Or alternatively use PYTEST_ADDOPTS=--no-cov just in the jython envs.

@huyphan

This comment has been minimized.

huyphan commented Jul 19, 2017

Thanks @ionelmc. Yea I'm aware of those options, unfortunately the CI system that we are using is not flexible enough to allow setting different environment variables based on Python implementation.

I understand that this is something the build system should fix rather than pytest-cov. I went to pytest-cov first because it's more lightweight to fix it here, and because it used to work fine with our edge case.

I think that's totally fair if we don't want to follow this path, it wouldn't make much sense to change a community library just because of a single/rare use case.

@ionelmc

This comment has been minimized.

Member

ionelmc commented Jul 20, 2017

It's not that I don't care about your usecase, but I want to know for sure you got no other way to deal with this. Take this example: #164

Most of the problems can be solved with some wrappers - lets talk about those first. I don't suppose your CI has hardcoded the pytest binary name as well? :)

@ionelmc

This comment has been minimized.

Member

ionelmc commented Jul 20, 2017

@davidszotten another way, maybe too clever for its own good, could be enabling strict summary handling only if --cov-fail-under is used. Then again, one could argue that incorrect coverage configuration (resulting in no coverage) is a failure condition as well but we don't do or can do anything about that. That's why I'm suggesting dropping the strict .summary(...) handling

@huyphan

This comment has been minimized.

huyphan commented Jul 20, 2017

@ionelmc: I totally got you :) It's just that I do see my use case a bit strange and narrow after thinking about it for a while.

My CI doesn't hardcode the pytest binary name, I will have to dig into it a bit to see how hard it really is to have some wrappers.

@ionelmc ionelmc closed this in 5a79a88 Jul 30, 2017

@ionelmc

This comment has been minimized.

Member

ionelmc commented Aug 1, 2017

@huyphan @davidszotten so this change is going into a release soon, last chance to voice your concerns/disagreements.

@davidszotten

This comment has been minimized.

Contributor

davidszotten commented Aug 1, 2017

i guess

to me "failed to generate a report" feels like a failure condition, not something we should just warn about and silently continue

summarises my main thoughts, but happy to be overruled if the consensus is elsewhere.

can you confirm the new behaviour of using --cov-fail-under whilst hitting this?

@ionelmc

This comment has been minimized.

Member

ionelmc commented Aug 1, 2017

@davidszotten new behavior still allows fail-under to work (eg, make suite fail)

ionel:xenial ~/osp/pytest-cov master#> tox -e py34-31-44 -- pytest tests/helper.py --cov-fail-under=1 --cov                                                 3.12s => 1 13:41
GLOB sdist-make: /home/ionel/osp/pytest-cov/setup.py
py34-31-44 inst-nodeps: /home/ionel/osp/pytest-cov/.tox/dist/pytest-cov-2.5.1.zip
py34-31-44 installed: apipkg==1.4,coverage==4.4.1,execnet==1.4.1,fields==5.0.0,process-tests==1.2.1,py==1.4.34,pytest==3.1.3,pytest-cov==2.5.1,pytest-xdist==1.18.2,six==1.10.0,virtualenv==15.1.0
py34-31-44 runtests: PYTHONHASHSEED='2258415925'
py34-31-44 runtests: commands[0] | pytest tests/helper.py --cov-fail-under=1 --cov
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.4.5, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
rootdir: /home/ionel/osp/pytest-cov, inifile: setup.cfg
plugins: xdist-1.18.2, cov-2.5.1
collected 0 items
Coverage.py warning: No data was collected. (no-data-collected)
WARNING: Failed to generate report: i just patched coverage to fail



----------- coverage: platform linux, python 3.4.5-final-0 -----------

FAIL Required test coverage of 1% not reached. Total coverage: 0.00%
============================================================================= warnings summary =============================================================================
None
  [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.
  Failed to generate report: i just patched coverage to fail

-- Docs: http://doc.pytest.org/en/latest/warnings.html
======================================================================== 2 warnings in 0.03 seconds ========================================================================
ERROR: InvocationError: '/home/ionel/osp/pytest-cov/.tox/py34-31-44/bin/pytest tests/helper.py --cov-fail-under=1 --cov'
_________________________________________________________________________________ summary __________________________________________________________________________________
ERROR:   py34-31-44: commands failed
@huyphan

This comment has been minimized.

huyphan commented Aug 1, 2017

@ionelmc: no concern at all. Thanks for making the code change.

bors bot added a commit to rehandalal/therapist that referenced this issue Sep 6, 2018

Merge #31
31: Update pytest-cov to 2.6.0 r=rehandalal a=pyup-bot


This PR updates [pytest-cov](https://pypi.org/project/pytest-cov) from **2.5.1** to **2.6.0**.



<details>
  <summary>Changelog</summary>
  
  
   ### 2.6.0
   ```
   ------------------

* Dropped support for Python &lt; 3.4, Pytest &lt; 3.5 and Coverage &lt; 4.4.
* Fixed some documentation formatting. Contributed by Jean Jordaan and Julian.
* Added an example with ``addopts`` in documentation. Contributed by Samuel Giffard in
  `195 &lt;https://github.com/pytest-dev/pytest-cov/pull/195&gt;`_.
* Fixed ``TypeError: &#39;NoneType&#39; object is not iterable`` in certain xdist configurations. Contributed by Jeremy Bowman in
  `213 &lt;https://github.com/pytest-dev/pytest-cov/pull/213&gt;`_.
* Added a ``no_cover`` marker and fixture. Fixes
  `78 &lt;https://github.com/pytest-dev/pytest-cov/issues/78&gt;`_.
* Fixed broken ``no_cover`` check when running doctests. Contributed by Terence Honles in
  `200 &lt;https://github.com/pytest-dev/pytest-cov/pull/200&gt;`_.
* Fixed various issues with path normalization in reports (when combining coverage data from parallel mode). Fixes
  `130 &lt;https://github.com/pytest-dev/pytest-cov/issues/161&gt;`_.
  Contributed by Ryan Hiebert &amp; Ionel Cristian Mărieș in
  `178 &lt;https://github.com/pytest-dev/pytest-cov/pull/178&gt;`_.
* Report generation failures don&#39;t raise exceptions anymore. A warning will be logged instead. Fixes
  `161 &lt;https://github.com/pytest-dev/pytest-cov/issues/161&gt;`_.
* Fixed multiprocessing issue on Windows (empty env vars are not passed). Fixes
  `165 &lt;https://github.com/pytest-dev/pytest-cov/issues/165&gt;`_.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest-cov
  - Changelog: https://pyup.io/changelogs/pytest-cov/
  - Repo: https://github.com/pytest-dev/pytest-cov
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment