Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coverage regression with Python 3.7 ? #679

Closed
0LL13 opened this issue Jul 12, 2018 · 30 comments
Closed

coverage regression with Python 3.7 ? #679

0LL13 opened this issue Jul 12, 2018 · 30 comments

Comments

@0LL13
Copy link

@0LL13 0LL13 commented Jul 12, 2018

I posted a question on Stackoverflow because the time for my tests running with coverage and Python 3.7 were about 8 times longer than with Python 3.5 and 3.6. Hoefling suggested I run the tests without coverage and times for testing were about the same for the different versions. Hoefling suggested that it might be a coverage regression and that I post an issue.

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 12, 2018

@0LL13 Can you verify for me the versions of coverage you have running in each Python version? "coverage --version" should do it. In particular, that will tell me whether you are using the C tracer (fast) or the Python tracer (slow).

@0LL13
Copy link
Author

@0LL13 0LL13 commented Jul 13, 2018

(vEnv)$ coverage --version
Coverage.py, version 4.0.3.

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 13, 2018

@0LL13 Can you upgrade to the latest version? 4.0.3 is over 2.5 years old. The latest version is 4.5.1. Once you've done that, if there is still a difference in performance, show the coverage version for each Python environment. Thanks.

@0LL13
Copy link
Author

@0LL13 0LL13 commented Jul 14, 2018

(vEnv)$ pip install -U coverage
(vEnv)$ coverage --version
Coverage.py, version 4.5.1 with C extension

#tox.ini
...
[testenv]
basepython =
    py35 = python3.5
    py36 = python3.6
    py37 = python3.7
deps = 
    pytest
    pytest-cov
commands =
    py.test --cov=pdf2textbox --duration=0 tests

(vEnv)$ tox
GLOB sdist-make: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/setup.py
py35 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py35
py35 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py35 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py35 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py35 runtests: PYTHONHASHSEED='2082557123'
py35 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py35 runtests: commands[1] | python --version
Python 3.5.5
py35 runtests: commands[2] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.5.5, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items                                                                    

tests/test_cli_slice_option.py ...                                              [ 20%]
tests/test_convert_pdf.py ......                                                [ 60%]
tests/test_get_pdf.py ....                                                      [ 86%]
tests/test_types.py ..                                                          [100%]

----------- coverage: platform linux, python 3.5.5-final-0 -----------
Name                                   Stmts   Miss  Cover
pdf2textbox/pdf2textbox.py     506    201    60%


=============================== slowest test durations ================================
54.32s call     tests/test_cli_slice_option.py::test_pdf_to_text_slice
1.26s call     tests/test_types.py::test_textbox_structure
1.25s call     tests/test_types.py::test_pdf_to_text_all
============================ 15 passed in 59.82 seconds ==============================

py36 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py36
py36 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py36 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py36 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py36 runtests: PYTHONHASHSEED='2082557123'
py36 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py36 runtests: commands[1] | python --version
Python 3.6.3
py36 runtests: commands[2] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.6.3, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items                                                                    

tests/test_cli_slice_option.py ...                                              [ 20%]
tests/test_convert_pdf.py ......                                                [ 60%]
tests/test_get_pdf.py ....                                                      [ 86%]
tests/test_types.py ..                                                          [100%]

----------- coverage: platform linux, python 3.6.3-final-0 -----------
Name                                   Stmts   Miss  Cover
pdf2textbox/pdf2textbox.py     506    201    60%


=============================== slowest test durations ================================
53.42s call     tests/test_cli_slice_option.py::test_pdf_to_text_slice
1.23s call     tests/test_types.py::test_pdf_to_text_all
1.22s call     tests/test_types.py::test_textbox_structure
0.46s call     tests/test_convert_pdf.py::test_convert_three_cols_and_header
============================ 15 passed in 58.94 seconds ==============================

py37 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py37
py37 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py37 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py37 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py37 runtests: PYTHONHASHSEED='2082557123'
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items                                                                    

tests/test_cli_slice_option.py ...                                              [ 20%]
tests/test_convert_pdf.py ......                                                [ 60%]
tests/test_get_pdf.py ....                                                      [ 86%]
tests/test_types.py ..                                                          [100%]

----------- coverage: platform linux, python 3.7.0-final-0 -----------
Name                                   Stmts   Miss  Cover
pdf2textbox/pdf2textbox.py     506    201    60%


=============================== slowest test durations ================================
426.08s call     tests/test_cli_slice_option.py::test_pdf_to_text_slice
9.55s call     tests/test_types.py::test_pdf_to_text_all
9.50s call     tests/test_types.py::test_textbox_structure
3.48s call     tests/test_convert_pdf.py::test_convert_three_cols_and_header
3.42s call     tests/test_convert_pdf.py::test_convert_three_cols
3.15s call     tests/test_convert_pdf.py::test_convert_two_cols_and_header
3.09s call     tests/test_convert_pdf.py::test_convert_two_cols
2.81s call     tests/test_convert_pdf.py::test_convert_text_only
2.74s call     tests/test_convert_pdf.py::test_convert_one_col_and_header
0.38s call     tests/test_get_pdf.py::test_get_pdf_file_url
...
============================ 15 passed in 465.17 seconds =============================

__________________________________ summary _______________________________________
  py35: commands succeeded
  py36: commands succeeded
  py37: commands succeeded
  congratulations :)

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 14, 2018

Thanks. Now can you add "coverage --version" to the tox.ini [commands] section, so that we can see the coverage version for each of the different Python versions?

@0LL13
Copy link
Author

@0LL13 0LL13 commented Jul 14, 2018

I did that and now as command[2] there is:

py36 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io

Otherwise no change.

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 14, 2018

@0LL13 I don't know how else to say this: I need to see the version of coverage for each Python version. I suspect that Python 3.7 is running with the Python extension, not the C extension.

@0LL13
Copy link
Author

@0LL13 0LL13 commented Jul 16, 2018

Sorry for the delay. You were absolutely right:

ppy35 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py35 runtests: commands[1] | python --version
Python 3.5.5
py35 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
py35 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests

py36 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py36 runtests: commands[1] | python --version
Python 3.6.3
py36 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
py36 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests

py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 without C extension
Documentation at https://coverage.readthedocs.io
py37 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests

Otherwise the same output as above.
I will be offline for a couple of days.

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 16, 2018

Thanks. 4.5.1 doesn't have a wheel for 3.7, so it tries to install from source. Perhaps you don't have the needed python-dev packages installed? http://coverage.readthedocs.io/en/coverage-4.5.1a/install.html#c-extension

@0LL13
Copy link
Author

@0LL13 0LL13 commented Jul 19, 2018

frodo@theshire:~$ sudo apt-get install python3-dev gcc
[sudo] Passwort für frodo:
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.
Statusinformationen werden eingelesen.... Fertig
gcc ist schon die neueste Version (4:7.3.0-3ubuntu2).
gcc wurde als manuell installiert festgelegt.
python3-dev ist schon die neueste Version (3.6.5-3ubuntu1).
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 6 nicht aktualisiert.

(vEnv)frodo@theshire:pdf2textbox$ python -m coverage --version
Coverage.py, version 4.5.1 with C extension

Coverage is only within the virtualenv for my project.

@vst
Copy link

@vst vst commented Jul 19, 2018

Confirming that pip install coverage comes without C extension under Python 3.7 which I have installed using pyenv. As a result, my pytest --cov took ~20 mins instead of usual ~5mins.

Solution for my case:

pip uninstall coverage
pip install --global-option=build_ext --global-option="$(python-config --includes)" coverage

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 19, 2018

OK, I've uploaded wheels for v4.5.1 for Python 3.7 on mac and manylinux to pypi. Let me know if that solves the problem. Thanks for all the details.

@cardoe
Copy link

@cardoe cardoe commented Jul 19, 2018

I believe this change has broken folks using pipenv and a Pipfile/Pipfile.lock as the hash for 4.5.1 has now changed.

Installing dependencies from Pipfile.lock (a03fec)...
An error occurred while installing coverage==4.5.1! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 25/25 — 00:00:01
Installing initially failed dependencies...
Looking in indexes: https://pypi.python.org/simple00
Collecting coverage==4.5.1
  Using cached https://files.pythonhosted.org/packages/c7/d0/337673c08f5b0cc7ada3dfe2a998ae8a97d482722045644be3d79bbcbe05/coverage-4.5.1-cp37-cp37m-macosx_10_13_x86_64.whl

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.```

@0LL13
Copy link
Author

@0LL13 0LL13 commented Jul 20, 2018

(vEnv)$ pip uninstall coverage
(vEnv)$ pip install --no-cache-dir coverage
Collecting coverage
Downloading https://files.pythonhosted.org/packages/3d/a0/b12090c40e0b8196b973962ec71c1c541a6c04af58ba5ad85683b3de251a/coverage-4.5.1-cp36-cp36m-manylinux1_x86_64.whl (202kB)
100% |████████████████████████████████| 204kB 6.9MB/s
Installing collected packages: coverage
Successfully installed coverage-4.5.1

No change in py.test --cov time for Python 3.7

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 20, 2018

@cardoe I don't understand how uploading a new file that never existed before could break an installation. Can you provide more details? What was the old hash computed from, and what is the new hash computed from?

@0LL13 Looks like that wheel you installed was for Python 3.6. Can you double-check that you are installing for 3.7?

@cardoe
Copy link

@cardoe cardoe commented Jul 20, 2018

@nedbat It came from running pipenv install --dev coverage a few months back sometime after 4.5.1 was released. pipenv automatically generates a Pipfile.lock which contains all the hashes of that version on pypi. According to pypa folks its a feature. https://docs.pipenv.org/basics/#pipfile-lock-security-features From the rough looks of it they're just using pip hash which appears to have changed for a pip cache containing coverage from about 10 hours ago to a cache that was recently created. Ultimately our tests started failing because pipenv reported that coverage may have been tampered with. I came to the GitHub project and searched and landed on this issue. Our last successful test was 42 minutes before your comment about new wheels being uploaded. Our first failure was 23 minutes after that comment. I very infrequently write Python now days so I'm just trying to go with the tools I've got. 🤷‍♂️

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 20, 2018

@cardoe I guess I understand what is happening. I'm not sure what I can do about it. Uploading new wheels for new Python versions seems like a net positive to me. Perhaps you should re-create your Pipfile.lock file? I'm not sure what the pipenv authors would recommend.

@vst
Copy link

@vst vst commented Jul 20, 2018

@nedbat It seems that wheel works fine now for py37. Thanks a lot!

coverage comes with C extension via tox --recreate, too, as expected:

python runtests: commands[0] | coverage --version
Coverage.py, version 4.5.1 with C extension
Documentation at https://coverage.readthedocs.io
python runtests: commands[1] | pytest --cov
...

As for pipenv: I am not using it right now but I guess that the only solution is to recreate Pipfile.lock. Yet, some friends may need to make new releases for packaged applications if they are shipping Pipfile.lock (such as Docker etc...).

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 20, 2018

I asked about pipenv in the pipenv repo: pypa/pipenv#2618

@abk-code
Copy link

@abk-code abk-code commented Jul 20, 2018

I can confirm that recreation of Pipfile.lock followed by pipenv install --dev will solve. Thanks @cardoe for noticing this issue. Thank you!

@0LL13
Copy link
Author

@0LL13 0LL13 commented Jul 21, 2018

I guess you need these lines, if not I need advice how to do it:

(vEnv)$ tox
...
py37 inst-nodeps: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py37 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py37 runtests: PYTHONHASHSEED='513262920'
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 without C extension

Documentation at https://coverage.readthedocs.io
py37 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items

Times are as before.

@vst
Copy link

@vst vst commented Jul 21, 2018

@0LL13 Can you try tox --recreate instead of just tox?

@0LL13
Copy link
Author

@0LL13 0LL13 commented Jul 21, 2018

Cool.

(vEnv)$ tox --recreate
...
py37 recreate: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/py37
py37 installdeps: check-manifest, pytest, requests, pdfminer2, coverage, pytest-cov
py37 inst: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox/.tox/dist/pdf2textbox-0.3.1.zip
py37 installed: atomicwrites==1.1.5,attrs==18.1.0,certifi==2018.4.16,chardet==3.0.4,check-manifest==0.37,coverage==4.5.1,idna==2.7,more-itertools==4.2.0,pdf2textbox==0.3.1,pdfminer2==20151206,pluggy==0.6.0,py==1.5.4,pytest==3.6.3,pytest-cov==2.5.1,requests==2.19.1,six==1.11.0,urllib3==1.23
py37 runtests: PYTHONHASHSEED='3853648192'
py37 runtests: commands[0] | check-manifest
lists of files in version control and sdist match
py37 runtests: commands[1] | python --version
Python 3.7.0
py37 runtests: commands[2] | coverage --version
Coverage.py, version 4.5.1 with C extension

Documentation at https://coverage.readthedocs.io
py37 runtests: commands[3] | py.test --cov=pdf2textbox --duration=0 tests
================================= test session starts =================================
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /home/frodo/Python-Projekte/.virtualenvs/pdf2textbox, inifile:
plugins: cov-2.5.1
collected 15 items

tests/test_cli_slice_option.py ... [ 20%]
tests/test_convert_pdf.py ...... [ 60%]
tests/test_get_pdf.py .... [ 86%]
tests/test_types.py .. [100%]

----------- coverage: platform linux, python 3.7.0-final-0 -----------
Name Stmts Miss Cover
.....................................................................................
pdf2textbox/pdf2textbox.py 506 201 60%

=============================== slowest test durations ================================
52.36s call tests/test_cli_slice_option.py::test_pdf_to_text_slice
2.64s call tests/test_get_pdf.py::test_get_pdf_file_url
1.14s call tests/test_types.py::test_textbox_structure
1.14s call tests/test_types.py::test_pdf_to_text_all
...
========================= 15 passed in 59.84 seconds ==============================
__________________________________ summary _______________________________________
py35: commands succeeded
py36: commands succeeded
py37: commands succeeded
congratulations :)

@nedbat
Copy link
Owner

@nedbat nedbat commented Jul 21, 2018

@0LL13 looks like it's all fixed. :)

@nedbat nedbat closed this Jul 21, 2018
@0LL13
Copy link
Author

@0LL13 0LL13 commented Jul 21, 2018

Yep :)

@coffeebenzene
Copy link

@coffeebenzene coffeebenzene commented Oct 17, 2018

Hi, sorry for reviving this issue, but are there C extensions for Python 3.7 on windows?
I have:

python --version
Python 3.7.0
coverage --version
Coverage.py, version 4.5.1 without C extension

@nedbat
Copy link
Owner

@nedbat nedbat commented Oct 21, 2018

@coffeebenzene Right, there are no windows 3.7 files at the moment. I can try to build them, but something is wrong at AppVeyor, and it is difficult to diagnose.

@nedbat
Copy link
Owner

@nedbat nedbat commented Oct 21, 2018

@coffeebenzene ah, but they are here: https://www.lfd.uci.edu/~gohlke/pythonlibs/#coverage . Perhaps I will upload those.

@nedbat
Copy link
Owner

@nedbat nedbat commented Oct 21, 2018

@coffeebenzene I uploaded those .whl to pypi. You should be all set.

@coffeebenzene
Copy link

@coffeebenzene coffeebenzene commented Oct 26, 2018

Thanks!
I just reinstalled with pip and I got the C extensions now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants