Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage suddenly missing everywhere but unit test modules #1580

Open
KelSolaar opened this issue Mar 10, 2023 · 4 comments
Open

Coverage suddenly missing everywhere but unit test modules #1580

KelSolaar opened this issue Mar 10, 2023 · 4 comments
Labels
bug Something isn't working question Further information is requested

Comments

@KelSolaar
Copy link

KelSolaar commented Mar 10, 2023

Describe the bug
Hello,

We have had a dramatic reduction in coverage for about a month and I'm not sure what is the issue. We basically lost coverage everywhere but in the unit test modules. It is very similar to this issue: #1520. It started on Github Actions where we use the Linux build to report coverage.

There is no real difference between old "successful" builds and new "broken" ones. I managed to reproduce the behaviour locally on macOs and fixed it by using sigterm = True and I can confirm that it is now fixed on CI for macOs but Windows and Linux are still not showing the expected coverage.

To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

  1. What version of Python are you using? Python 3.9, 3.10, 3.11
  2. What version of coverage.py shows the problem? The output of coverage debug sys is helpful. 6.5.0
  3. What versions of what packages do you have installed? The output of pip freeze is helpful.
alabaster (0.7.13)
anyio (3.6.2)
argon2-cffi (21.3.0)
argon2-cffi-bindings (21.2.0)
arrow (1.2.3)
astor (0.8.1)
asttokens (2.2.1)
attrs (22.2.0)
babel (2.11.0)
backcall (0.2.0)
beautifulsoup4 (4.11.2)
biblib-simple (0.1.2)
black (23.1.0)
blackdoc (0.3.8)
bleach (6.0.0)
certifi (2022.12.7)
cffi (1.15.1)
cfgv (3.3.1)
charset-normalizer (3.0.1)
click (8.1.3)
comm (0.1.2)
contourpy (1.0.7)
coverage (6.5.0)
coveralls (3.3.1)
cryptography (39.0.1)
cycler (0.11.0)
debugpy (1.6.6)
decorator (5.1.1)
defusedxml (0.7.1)
distlib (0.3.6)
docopt (0.6.2)
docutils (0.17.1)
exceptiongroup (1.1.0)
execnet (1.9.0)
executing (1.2.0)
fastjsonschema (2.16.2)
filelock (3.9.0)
flynt (0.77)
fonttools (4.38.0)
fqdn (1.5.1)
identify (2.5.18)
idna (3.4)
imageio (2.25.1)
imagesize (1.4.1)
importlib-metadata (6.0.0)
iniconfig (2.0.0)
invoke (2.0.0)
ipykernel (6.21.2)
ipython (8.10.0)
ipython-genutils (0.2.0)
ipywidgets (8.0.4)
isoduration (20.11.0)
jaraco-classes (3.2.3)
jedi (0.18.2)
jeepney (0.8.0)
jinja2 (3.1.2)
jsonpointer (2.3)
jsonschema (4.17.3)
jupyter (1.0.0)
jupyter-client (8.0.3)
jupyter-console (6.5.1)
jupyter-core (5.2.0)
jupyter-events (0.6.3)
jupyter-server (2.3.0)
jupyter-server-terminals (0.4.4)
jupyterlab-pygments (0.2.2)
jupyterlab-widgets (3.0.5)
keyring (23.13.1)
kiwisolver (1.4.4)
latexcodec (2.0.1)
markdown-it-py (2.1.0)
markupsafe (2.1.2)
matplotlib (3.7.0)
matplotlib-inline (0.1.6)
mdurl (0.1.2)
mistune (2.0.5)
more-itertools (9.0.0)
mypy-extensions (1.0.0)
nbclassic (0.5.1)
nbclient (0.7.2)
nbconvert (7.2.9)
nbformat (5.7.3)
nest-asyncio (1.5.6)
networkx (2.8.8)
nodeenv (1.7.0)
notebook (6.5.2)
notebook-shim (0.2.2)
numpy (1.24.2)
packaging (23.0)
pandas (1.5.3)
pandocfilters (1.5.0)
parso (0.8.3)
pathspec (0.11.0)
pexpect (4.8.0)
pickleshare (0.7.5)
pillow (9.4.0)
pkginfo (1.9.6)
platformdirs (3.0.0)
pluggy (1.0.0)
pre-commit (3.0.4)
prometheus-client (0.16.0)
prompt-toolkit (3.0.36)
psutil (5.9.4)
ptyprocess (0.7.0)
pure-eval (0.2.2)
pybtex (0.24.0)
pybtex-docutils (1.0.2)
pycparser (2.21)
pydata-sphinx-theme (0.12.0)
pygments (2.14.0)
pygraphviz (1.10)
pyparsing (3.0.9)
pyright (1.1.294)
pyrsistent (0.19.3)
pytest (7.2.1)
pytest-cov (4.0.0)
pytest-xdist (3.2.0)
python-dateutil (2.8.2)
python-json-logger (2.0.6)
pytz (2022.7.1)
pyyaml (6.0)
pyzmq (25.0.0)
qtconsole (5.4.0)
qtpy (2.3.0)
readme-renderer (37.3)
requests (2.28.2)
requests-toolbelt (0.10.1)
restructuredtext-lint (1.4.0)
rfc3339-validator (0.1.4)
rfc3986 (2.0.0)
rfc3986-validator (0.1.1)
rich (13.3.1)
ruff (0.0.247)
scipy (1.10.0)
secretstorage (3.3.3)
send2trash (1.8.0)
six (1.16.0)
sniffio (1.3.0)
snowballstemmer (2.2.0)
soupsieve (2.4)
sphinx (4.5.0)
sphinxcontrib-applehelp (1.0.4)
sphinxcontrib-bibtex (2.5.0)
sphinxcontrib-devhelp (1.0.2)
sphinxcontrib-htmlhelp (2.0.1)
sphinxcontrib-jsmath (1.0.1)
sphinxcontrib-qthelp (1.0.3)
sphinxcontrib-serializinghtml (1.1.5)
stack-data (0.6.2)
terminado (0.17.1)
tinycss2 (1.2.1)
toml (0.10.2)
tomli (2.0.1)
tornado (6.2)
tqdm (4.64.1)
traitlets (5.9.0)
trimesh (3.20.0)
twine (4.0.2)
typing-extensions (4.5.0)
uri-template (1.2.0)
urllib3 (1.26.14)
virtualenv (20.19.0)
wcwidth (0.2.6)
webcolors (1.12)
webencodings (0.5.1)
websocket-client (1.5.1)
widgetsnbextension (4.0.5)
zipp (3.13.0)

Note that "successful" and "broken" jobs use for practical purposes the same packages.

  1. What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix.
    I have unfortunately been unable to fix the issue anywhere but macOs by using sigterm = True
  2. What commands did you run?

pytest --doctest-modules --ignore=colour/examples --cov=colour colour

My .coveragerc file:

[run]
source = colour
concurrency = multiprocessing
sigterm = True
[report]
exclude_lines =
    pragma: no cover
    if __name__ == .__main__.:
    pass

Expected behavior
The coverage should have stayed the same and not changed.

Additional context
None at this point but happy to try as many things as required to understand what is going on.

@KelSolaar KelSolaar added bug Something isn't working needs triage labels Mar 10, 2023
@nedbat
Copy link
Owner

nedbat commented Mar 10, 2023

I notice that your pip freeze output doesn't mention "coverage"? Are you sure you are using the same version? You mention 6.5.0, but perhaps it is different in the working and non-working scenarios?

I'm not sure how to help without a reproducible case.

@KelSolaar
Copy link
Author

Hi @nedbat,

Sorry, when I copied from Github Actions, it seems like the selection from the Log Viewer missed most a chunk of the packages. I have updated the OP using the raw logs and I can confirm that they are the same in both scenarios:

The packages differences are as follows:

identify (2.5.17) --> (2.5.18)
imageio (2.25.0) --> (2.25.1)
ipykernel (6.21.1) --> (6.21.2)
jupyter-client (8.0.2) --> (8.0.3)
jupyter-console (6.5.0) --> (6.5.1)
jupyter-server (2.2.1) --> (2.3.0)
matplotlib (3.6.3) --> (3.7.0)
pyright (1.1.293) --> (1.1.294)
python-json-logger (2.0.4) --> (2.0.6)
ruff (0.0.245) --> (0.0.247)
soupsieve (2.3.2.post1) --> (2.4)
trimesh (3.19.4) --> (3.20.0)
typing-extensions (4.4.0) --> (4.5.0)

@nedbat
Copy link
Owner

nedbat commented Apr 24, 2023

Do you have any more information about how I might be able to reproduce this? Can you link us to the repo with the code?

@nedbat nedbat added the question Further information is requested label Apr 24, 2023
@KelSolaar
Copy link
Author

Hi @nedbat,

The code is here:

We for now only upload coverage from macOs and Python 3.11 as Windows and Linux seemed to be affected.

Cheers,

Thomas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants