3.6.0 breaking test: internal error loading conftest.py #3503

sbouchut · 2018-05-24T15:12:48Z

No error with 3.5.1. Since 3.6.0 I get an unexpected error.

pytest --basetemp=/builds/company/project/.tox/py27/tmp --cov=project --no-cov-on-fail --html reports/test-reports/pytest-py27.html --self-contained-html --junitxml reports/test-reports/pytest-py27.xml tests --dburl postgresql://login:password@postgres:5432/tests

Traceback (most recent call last):
  File "/builds/forcity/ewan/.tox/py27/local/lib/python2.7/site-packages/_pytest/config.py", line 376, in _importconftest
    mod = conftestpath.pyimport()
  File "/builds/forcity/ewan/.tox/py27/local/lib/python2.7/site-packages/py/_path/local.py", line 668, in pyimport
    __import__(modname)
  File "/builds/forcity/ewan/.tox/py27/local/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 145, in find_module
    _write_pyc(state, co, source_stat, pyc)
  File "/builds/forcity/ewan/.tox/py27/local/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 269, in _write_pyc
    marshal.dump(co, fp.file)
TypeError: marshal.dump() 2nd arg must be file
ERROR: could not load /builds/forcity/ewan/tests/conftest.py

pip freeze

amqp==1.4.9
anyjson==0.3.3
arrow==0.12.1
asn1crypto==0.24.0
atomicwrites==1.1.5
attrs==18.1.0
backports.functools-lru-cache==1.5
bcrypt==3.1.4
bottle==0.12.13
bulb==0.1.4
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
cimkit==3.6.1
click==6.7
click-plugins==1.0.3
cligj==0.4.0
configparser==3.5.0
coverage==4.5.1
cryptography==2.2.2
cycler==0.10.0
descartes==1.1.0
enum-compat==0.0.2
enum34==1.1.6
eventlet==0.20.1
project==0.4.1rc1
Fiona==1.7.11.post2
flake8==3.5.0
flake8-html==0.4.0
company-classes==0.20.0rc0
company-modelingtoolkit==1.6.0
companyalchemy==0.10.0
funcsigs==1.0.2
functools32==3.2.3.post2
GeoAlchemy2==0.2.6
geojson==2.4.0
geopandas==0.3.0
greenlet==0.4.13
hypothesis==3.57.0
idna==2.6
invoke==1.0.0
ipaddress==1.0.22
Jinja2==2.10
jsonschema==2.6.0
kiwisolver==1.0.1
kombu==3.0.37
lumis-commons==0.0.1
MarkupSafe==1.0
matplotlib==2.2.2
mccabe==0.6.1
mock==2.0.0
more-itertools==4.2.0
munch==2.3.2
nameko==2.8.5
Nameko-Logger==1.6.1
numexpr==2.6.5
numpy==1.14.3
orca==1.4.0
pandas==0.23.0
paramiko==2.4.1
path.py==11.0.1
patsy==0.5.0
pbr==4.0.3
pluggy==0.6.0
prettytable==0.7.2
psycopg2==2.7.4
py==1.5.3
pyasn1==0.4.3
pycodestyle==2.3.1
pycparser==2.18
pyflakes==1.6.0
Pygments==2.2.0
PyNaCl==1.2.1
pyparsing==2.2.0
pyproj==1.9.5.1
pytest==3.6.0
pytest-cov==2.5.1
pytest-company-db==0.3.1
pytest-html==1.18.0
pytest-metadata==1.7.0
python-dateutil==2.7.3
pytz==2018.4
PyYAML==3.12
requests==2.18.4
retrying==1.3.3
Rtree==0.8.3
scipy==1.1.0
Shapely==1.6.4.post1
simplejson==3.15.0
six==1.11.0
SQLAlchemy==1.2.7
SQLAlchemy-Utils==0.33.3
statsmodels==0.8.0
structlog==18.1.0
subprocess32==3.5.1
tables==3.4.3
tblib==1.3.2
toolz==0.9.0
tox==3.0.0
ujson==1.35
urbansim==3.1.1
urllib3==1.22
virtualenv==16.0.0
Werkzeug==0.14.1
wrapt==1.10.11
yapf==0.22.0
zbox==1.2.0

The text was updated successfully, but these errors were encountered:

pytestbot · 2018-05-24T15:19:01Z

GitMate.io thinks possibly related issues are #913 (py.test exits with code 0 on errors), #3181 (3.4.0 logging changes break devpi tests), #60 (py.test 2.1.0 causes new errors), #3498 (3.6.0 breaks usefixtures when using unittest/unittest2), and #1404 (conftest.py loaded in wrong order.).

sbouchut · 2018-05-24T15:40:20Z

It seems this error may be related to missing __init__.py file in test directory (which was not raising error with previous version?). I will perform some additional checks and close the issue if so.

Edit: Unfortunately it was not enough to solve the issue in my case.

RonnyPfannschmidt · 2018-05-24T16:32:33Z

@nicoddemus this seems to be a bug in the bytecode writing of the assertion rewriter - based on the diff/blame it correlates with the switch to use atomic-files to avoid race conditions

…HIS COMMIT

nicoddemus · 2018-05-24T17:58:37Z

Hi @sbouchut, thanks for the report.

@RonnyPfannschmidt is right, this is related to #3390: we are now using atomicwrites to write rewritten .pyc files because they might be corrupted on Windows otherwise.

The snippet showing the problem is:

pytest/_pytest/assertion/rewrite.py

Lines 263 to 271 in 48215fd

    
           with atomicwrites.atomic_write(pyc, mode="wb", overwrite=True) as fp: 
        
               fp.write(imp.get_magic()) 
        
               mtime = int(source_stat.mtime) 
        
               size = source_stat.size & 0xFFFFFFFF 
        
               fp.write(struct.pack("<ll", mtime, size)) 
        
               if six.PY2: 
        
                   marshal.dump(co, fp.file) 
        
               else: 
        
                   marshal.dump(co, fp)

I'm surprised fp.file somehow is not a file, given that all our py27 suite works with that code.

@sbouchut could you please run your code with pytest from nicoddemus/temp-check-atomicwrite-file? I added a temporary check and better error message to that branch.

I see you are using tox, so this would be a matter of changing this in your tox.ini:

deps = 
    pytest

To this:

deps = 
    git+https://github.com/nicoddemus/pytest@temp-check-atomicwrite-file

Could you then post the output? Thanks!

sbouchut · 2018-05-25T08:09:07Z

Thank you for your help.
The new traceback below.

Traceback (most recent call last):
  File "/home/sbouchut/dev/project/.tox/py27/local/lib/python2.7/site-packages/_pytest/config.py", line 376, in _importconftest
    mod = conftestpath.pyimport()
  File "/home/sbouchut/dev/project/.tox/py27/local/lib/python2.7/site-packages/py/_path/local.py", line 668, in pyimport
    __import__(modname)
  File "/home/sbouchut/dev/project/.tox/py27/local/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 145, in find_module
    _write_pyc(state, co, source_stat, pyc)
  File "/home/sbouchut/dev/project/.tox/py27/local/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 270, in _write_pyc
    assert isinstance(fp.file, types.FileType), "unexpected file type: {!r} ({!r})".format(fp.file, type(fp.file))
AssertionError: unexpected file type: <open GreenPipe '<fd:10>', mode 'wb' at 0x7f5cc29c4440> (<class 'eventlet.greenio.py2.GreenPipe'>)
ERROR: could not load /home/sbouchut/dev/project/tests/conftest.py

ERROR: InvocationError for command '/home/sbouchut/dev/project/.tox/py27/bin/pytest --basetemp=/home/sbouchut/dev/project/.tox/py27/tmp --cov=project --no-cov-on-fail --html reports/test-reports/pytest-py27.html --self-contained-html --junitxml reports/test-reports/pytest-py27.xml tests --dburl=postgresql://postgres@localhost/test' (exited with code 4)

RonnyPfannschmidt · 2018-05-25T08:16:26Z

@sbouchut thanks, this looks like gevent/eventlet breaking the world with a monkeypatch

@nicoddemus i'm wondering - would it work to switch to marshal.dumps and manually write the data out (as it would allow to use the file object directly and avoid hitting the marshal check)

nicoddemus · 2018-05-25T10:30:49Z

@sbouchut thanks, now the problem is clear: like @RonnyPfannschmidt said greenlet monkeypatches things and marshal.dump doesn't know about the greenlet's file object.

@RonnyPfannschmidt: yep, that will work and be simpler even. I'm on it.

This works around the fact that some libraries might monkey patch the file object, so the previous approach of marshal.dump(co, file) breaks because file is not a built-in file object anymore. Fix pytest-dev#3503

nicoddemus · 2018-05-25T20:30:48Z

PR is up #3509

This PR updates [pytest](https://pypi.org/project/pytest) from **3.5.1** to **3.6.3**. <details> <summary>Changelog</summary> ### 3.6.2 ``` ========================= Bug Fixes --------- - Fix regression in ``Node.add_marker`` by extracting the mark object of a ``MarkDecorator``. (`3555 <https://github.com/pytest-dev/pytest/issues/3555>`_) - Warnings without ``location`` were reported as ``None``. This is corrected to now report ``<undetermined location>``. (`3563 <https://github.com/pytest-dev/pytest/issues/3563>`_) - Continue to call finalizers in the stack when a finalizer in a former scope raises an exception. (`3569 <https://github.com/pytest-dev/pytest/issues/3569>`_) - Fix encoding error with `print` statements in doctests (`3583 <https://github.com/pytest-dev/pytest/issues/3583>`_) Improved Documentation ---------------------- - Add documentation for the ``--strict`` flag. (`3549 <https://github.com/pytest-dev/pytest/issues/3549>`_) Trivial/Internal Changes ------------------------ - Update old quotation style to parens in fixture.rst documentation. (`3525 <https://github.com/pytest-dev/pytest/issues/3525>`_) - Improve display of hint about ``--fulltrace`` with ``KeyboardInterrupt``. (`3545 <https://github.com/pytest-dev/pytest/issues/3545>`_) - pytest's testsuite is no longer runnable through ``python setup.py test`` -- instead invoke ``pytest`` or ``tox`` directly. (`3552 <https://github.com/pytest-dev/pytest/issues/3552>`_) - Fix typo in documentation (`3567 <https://github.com/pytest-dev/pytest/issues/3567>`_) ``` ### 3.6.1 ``` ========================= Bug Fixes --------- - Fixed a bug where stdout and stderr were logged twice by junitxml when a test was marked xfail. (`3491 <https://github.com/pytest-dev/pytest/issues/3491>`_) - Fix ``usefixtures`` mark applyed to unittest tests by correctly instantiating ``FixtureInfo``. (`3498 <https://github.com/pytest-dev/pytest/issues/3498>`_) - Fix assertion rewriter compatibility with libraries that monkey patch ``file`` objects. (`3503 <https://github.com/pytest-dev/pytest/issues/3503>`_) Improved Documentation ---------------------- - Added a section on how to use fixtures as factories to the fixture documentation. (`3461 <https://github.com/pytest-dev/pytest/issues/3461>`_) Trivial/Internal Changes ------------------------ - Enable caching for pip/pre-commit in order to reduce build time on travis/appveyor. (`3502 <https://github.com/pytest-dev/pytest/issues/3502>`_) - Switch pytest to the src/ layout as we already suggested it for good practice - now we implement it as well. (`3513 <https://github.com/pytest-dev/pytest/issues/3513>`_) - Fix if in tests to support 3.7.0b5, where a docstring handling in AST got reverted. (`3530 <https://github.com/pytest-dev/pytest/issues/3530>`_) - Remove some python2.5 compatibility code. (`3529 <https://github.com/pytest-dev/pytest/issues/3529>`_) ``` ### 3.6.0 ``` ========================= Features -------- - Revamp the internals of the ``pytest.mark`` implementation with correct per node handling which fixes a number of long standing bugs caused by the old design. This introduces new ``Node.iter_markers(name)`` and ``Node.get_closest_mark(name)`` APIs. Users are **strongly encouraged** to read the `reasons for the revamp in the docs <https://docs.pytest.org/en/latest/mark.htmlmarker-revamp-and-iteration>`_, or jump over to details about `updating existing code to use the new APIs <https://docs.pytest.org/en/latest/mark.htmlupdating-code>`_. (`3317 <https://github.com/pytest-dev/pytest/issues/3317>`_) - Now when ``pytest.fixture`` is applied more than once to the same function a ``ValueError`` is raised. This buggy behavior would cause surprising problems and if was working for a test suite it was mostly by accident. (`2334 <https://github.com/pytest-dev/pytest/issues/2334>`_) - Support for Python 3.7's builtin ``breakpoint()`` method, see `Using the builtin breakpoint function <https://docs.pytest.org/en/latest/usage.htmlbreakpoint-builtin>`_ for details. (`3180 <https://github.com/pytest-dev/pytest/issues/3180>`_) - ``monkeypatch`` now supports a ``context()`` function which acts as a context manager which undoes all patching done within the ``with`` block. (`3290 <https://github.com/pytest-dev/pytest/issues/3290>`_) - The ``--pdb`` option now causes KeyboardInterrupt to enter the debugger, instead of stopping the test session. On python 2.7, hitting CTRL+C again exits the debugger. On python 3.2 and higher, use CTRL+D. (`3299 <https://github.com/pytest-dev/pytest/issues/3299>`_) - pytest not longer changes the log level of the root logger when the ``log-level`` parameter has greater numeric value than that of the level of the root logger, which makes it play better with custom logging configuration in user code. (`3307 <https://github.com/pytest-dev/pytest/issues/3307>`_) Bug Fixes --------- - A rare race-condition which might result in corrupted ``.pyc`` files on Windows has been hopefully solved. (`3008 <https://github.com/pytest-dev/pytest/issues/3008>`_) - Also use iter_marker for discovering the marks applying for marker expressions from the cli to avoid the bad data from the legacy mark storage. (`3441 <https://github.com/pytest-dev/pytest/issues/3441>`_) - When showing diffs of failed assertions where the contents contain only whitespace, escape them using ``repr()`` first to make it easy to spot the differences. (`3443 <https://github.com/pytest-dev/pytest/issues/3443>`_) Improved Documentation ---------------------- - Change documentation copyright year to a range which auto-updates itself each time it is published. (`3303 <https://github.com/pytest-dev/pytest/issues/3303>`_) Trivial/Internal Changes ------------------------ - ``pytest`` now depends on the `python-atomicwrites <https://github.com/untitaker/python-atomicwrites>`_ library. (`3008 <https://github.com/pytest-dev/pytest/issues/3008>`_) - Update all pypi.python.org URLs to pypi.org. (`3431 <https://github.com/pytest-dev/pytest/issues/3431>`_) - Detect `pytest_` prefixed hooks using the internal plugin manager since ``pluggy`` is deprecating the ``implprefix`` argument to ``PluginManager``. (`3487 <https://github.com/pytest-dev/pytest/issues/3487>`_) - Import ``Mapping`` and ``Sequence`` from ``_pytest.compat`` instead of directly from ``collections`` in ``python_api.py::approx``. Add ``Mapping`` to ``_pytest.compat``, import it from ``collections`` on python 2, but from ``collections.abc`` on Python 3 to avoid a ``DeprecationWarning`` on Python 3.7 or newer. (`3497 <https://github.com/pytest-dev/pytest/issues/3497>`_) ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest - Changelog: https://pyup.io/changelogs/pytest/ - Repo: https://github.com/pytest-dev/pytest/issues - Homepage: http://pytest.org </details>

pytestbot added the type: bug problem that needs to be addressed label May 24, 2018

RonnyPfannschmidt added the topic: rewrite related to the assertion rewrite mechanism label May 24, 2018

nicoddemus added the type: regression indicates a problem that was introduced in a release which was working previously label May 24, 2018

nicoddemus added a commit to nicoddemus/pytest that referenced this issue May 24, 2018

TEMPORARY COMMIT: type check fp.file (pytest-dev#3503) DO NOT MERGE T…

2cf8350

…HIS COMMIT

rgommers mentioned this issue May 24, 2018

CI failing after new pytest release scipy/scipy#8866

Closed

pytestbot mentioned this issue May 25, 2018

3.6.0 can't find fixtures for cheroot.test.webtest.WebCase based test classes #3507

Closed

nicoddemus self-assigned this May 25, 2018

larsoner mentioned this issue May 25, 2018

FIX: Pin version of pytest != 3.6.0 scipy/scipy#8869

Merged

nicoddemus mentioned this issue May 25, 2018

Write rewritten code using file.write(marshal.dumps()) #3509

Merged

RonnyPfannschmidt mentioned this issue May 25, 2018

run ci against gevent/greenlet monkeypatching #3510

Closed

nicoddemus closed this as completed in #3509 May 26, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3.6.0 breaking test: internal error loading conftest.py #3503

3.6.0 breaking test: internal error loading conftest.py #3503

sbouchut commented May 24, 2018 •

edited

Loading

pytestbot commented May 24, 2018

sbouchut commented May 24, 2018 •

edited

Loading

RonnyPfannschmidt commented May 24, 2018

nicoddemus commented May 24, 2018

sbouchut commented May 25, 2018

RonnyPfannschmidt commented May 25, 2018

nicoddemus commented May 25, 2018 •

edited

Loading

nicoddemus commented May 25, 2018

3.6.0 breaking test: internal error loading conftest.py #3503

3.6.0 breaking test: internal error loading conftest.py #3503

Comments

sbouchut commented May 24, 2018 • edited Loading

pytestbot commented May 24, 2018

sbouchut commented May 24, 2018 • edited Loading

RonnyPfannschmidt commented May 24, 2018

nicoddemus commented May 24, 2018

sbouchut commented May 25, 2018

RonnyPfannschmidt commented May 25, 2018

nicoddemus commented May 25, 2018 • edited Loading

nicoddemus commented May 25, 2018

sbouchut commented May 24, 2018 •

edited

Loading

sbouchut commented May 24, 2018 •

edited

Loading

nicoddemus commented May 25, 2018 •

edited

Loading