New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UnicodeEncodeError on source files containing non-ascii characters #4074

Closed
wants to merge 2 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@spg

spg commented Oct 3, 2018

This PR fixes an issue I had when running tests that:

  1. Fail, and
  2. Contain non-ascii characters (e.g. accented characters)

Here's an example of such a test (test_bug.py):

# coding=utf-8
def test__bug():
    a = 'é'
    assert False

Running this test outputs the following:

test_env ❯ pytest tests/test_bug.py
===================================================================================================================================================================== test session starts ======================================================================================================================================================================
platform darwin -- Python 2.7.14, pytest-3.8.2, py-1.5.2, pluggy-0.7.1
Django settings: settings.test (from ini file)
rootdir: /Users/spgingras/test_sample, inifile: pytest.ini
plugins: teamcity-messages-1.21, xdist-1.23.0, random-0.2, mock-1.10.0, forked-0.2, env-0.6.2, django-3.4.2, cov-2.5.1, pyfakefs-3.4.3, celery-4.1.1
collected 1 item                                                                                                                                                                                                                                                                                                                                               

tests/test_bug.py 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/main.py", line 178, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/main.py", line 215, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/hooks.py", line 258, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/manager.py", line 61, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/main.py", line 236, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/hooks.py", line 258, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/manager.py", line 61, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/callers.py", line 201, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/runner.py", line 66, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/runner.py", line 81, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/runner.py", line 163, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/hooks.py", line 258, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/manager.py", line 67, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/manager.py", line 61, in <lambda>
INTERNALERROR>     firstresult=hook.spec_opts.get('firstresult'),
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/callers.py", line 196, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/skipping.py", line 123, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/callers.py", line 77, in get_result
INTERNALERROR>     _reraise(*ex)  # noqa
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/pluggy/callers.py", line 180, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/runner.py", line 240, in pytest_runtest_makereport
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/python.py", line 774, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/python.py", line 767, in _repr_failure_py
INTERNALERROR>     return super(FunctionMixin, self)._repr_failure_py(excinfo, style=style)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/nodes.py", line 388, in _repr_failure_py
INTERNALERROR>     truncate_locals=truncate_locals,
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/_code/code.py", line 480, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/_code/code.py", line 715, in repr_excinfo
INTERNALERROR>     reprtraceback = self.repr_traceback(excinfo)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/_code/code.py", line 672, in repr_traceback
INTERNALERROR>     reprentry = self.repr_traceback_entry(entry, einfo)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/_code/code.py", line 617, in repr_traceback_entry
INTERNALERROR>     source = self._getentrysource(entry)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/_code/code.py", line 537, in _getentrysource
INTERNALERROR>     source = entry.getsource(self.astcache)
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/_code/code.py", line 222, in getsource
INTERNALERROR>     self.lineno, source, astnode=astnode
INTERNALERROR>   File "/Users/spgingras/.virtualenvs/test_env/lib/python2.7/site-packages/_pytest/_code/source.py", line 340, in getstatementrange_ast
INTERNALERROR>     content = str(source)
INTERNALERROR> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 41: ordinal not in range(128)

================================================================================================================================================================= no tests ran in 0.82 seconds =================================================================================================================================================================

Aside from the obvious stacktrace, you can see that pytest reported that 0 tests ran.

With my fix:

test_env ❯ pytest tests/test_bug.py
===================================================================================================================================================================== test session starts ======================================================================================================================================================================
platform darwin -- Python 2.7.14, pytest-3.8.2, py-1.5.2, pluggy-0.7.1
Django settings: settings.test (from ini file)
rootdir: /Users/spgingras/test_sample, inifile: pytest.ini
plugins: teamcity-messages-1.21, xdist-1.23.0, random-0.2, mock-1.10.0, forked-0.2, env-0.6.2, django-3.4.2, cov-2.5.1, pyfakefs-3.4.3, celery-4.1.1
collected 1 item                                                                                                                                                                                                                                                                                                                                               

tests/test_bug.py F                                                                                                                                                                                                                                                                                                                                 [100%]

=========================================================================================================================================================================== FAILURES ===========================================================================================================================================================================
__________________________________________________________________________________________________________________________________________________________________________ test__bug ___________________________________________________________________________________________________________________________________________________________________________

    def test__bug():
        a = 'é'
>       assert False
E       assert False

tests/test_bug.py:4: AssertionError
=================================================================================================================================================================== 1 failed in 0.80 seconds ===================================================================================================================================================================

I'm not sure if or how I should write a test for this particular case. Any help is appreciated.

spg added some commits Oct 3, 2018

@@ -337,7 +337,10 @@ def get_statement_startend2(lineno, node):
def getstatementrange_ast(lineno, source, assertion=False, astnode=None):
if astnode is None:
content = str(source)
if six.PY2:
content = unicode(source)

This comment has been minimized.

@nicoddemus

nicoddemus Oct 3, 2018

Member

Unfortunately this still might fail, because unicode will try to use the ascii codec to decode the bytestream (see py27 failure). Not sure if by this point getstatementrange_ast shouldn't already be a unicode stream actually. 🤔

@nicoddemus

nicoddemus Oct 3, 2018

Member

Unfortunately this still might fail, because unicode will try to use the ascii codec to decode the bytestream (see py27 failure). Not sure if by this point getstatementrange_ast shouldn't already be a unicode stream actually. 🤔

@asottile

This comment has been minimized.

Show comment
Hide comment
@asottile

asottile Oct 3, 2018

Member

Your original reproduction does not reproduce for me:

$ pytest t.py
============================= test session starts ==============================
platform darwin -- Python 2.7.10, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
rootdir: /private/tmp/t, inifile:
collected 1 item                                                               

t.py F                                                                   [100%]

=================================== FAILURES ===================================
__________________________________ test__bug ___________________________________

    def test__bug():
        a = 'é'
>       assert False
E       assert False

t.py:4: AssertionError
=========================== 1 failed in 0.08 seconds ===========================

Can you list all of your currently installed dependencies? I suspect something suspicious is afoot

Member

asottile commented Oct 3, 2018

Your original reproduction does not reproduce for me:

$ pytest t.py
============================= test session starts ==============================
platform darwin -- Python 2.7.10, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
rootdir: /private/tmp/t, inifile:
collected 1 item                                                               

t.py F                                                                   [100%]

=================================== FAILURES ===================================
__________________________________ test__bug ___________________________________

    def test__bug():
        a = 'é'
>       assert False
E       assert False

t.py:4: AssertionError
=========================== 1 failed in 0.08 seconds ===========================

Can you list all of your currently installed dependencies? I suspect something suspicious is afoot

@spg

This comment has been minimized.

Show comment
Hide comment
@spg

spg Oct 10, 2018

@asottile Yes you are right, I created a fresh virtualenv (Python 2.7.14), and like you said, the problem is not happening.

spg commented Oct 10, 2018

@asottile Yes you are right, I created a fresh virtualenv (Python 2.7.14), and like you said, the problem is not happening.

@asottile

This comment has been minimized.

Show comment
Hide comment
@asottile

asottile Oct 10, 2018

Member

@spg could you list the deps in your (broken) venv so we can triage what package is doing nefarious things?

Member

asottile commented Oct 10, 2018

@spg could you list the deps in your (broken) venv so we can triage what package is doing nefarious things?

@spg

This comment has been minimized.

Show comment
Hide comment
@spg

spg Oct 10, 2018

Here is the output from pip freeze in the broken venv:

amqp==2.3.2
apipkg==1.4
arnparse==0.0.1
asn1crypto==0.24.0
atomicwrites==1.2.1
attrs==17.4.0
aws-requests-auth==0.4.2
aws-xray-sdk==0.95
backports.ssl-match-hostname==3.5.0.1
backports.tempfile==1.0
backports.weakref==1.0.post1
billiard==3.5.0.4
boto==2.48.0
boto3==1.7.84
botocore==1.10.84
celery==4.1.1
celery-haystack==0.10
certifi==2018.8.13
cffi==1.11.4
chardet==3.0.4
checksumdir==1.1.4
click==6.7
concurrent-log-handler==0.9.12
ConcurrentLogHandler==0.9.1
contextlib2==0.5.5
contexttimer==0.3.3
cookies==2.2.1
coreapi==2.3.3
coreschema==0.0.4
coverage==4.5.1
cryptography==2.1.4
cssselect==1.0.3
Django==1.11.15
django-appconf==1.0.2
django-bower==5.2.0
django-cache-friendly-timestamp-signer==1.0.1
django-common-helpers==0.9.1
django-compressor==2.2
django-contrib-comments==1.8.0
django-cors-headers==2.4.0
django-countries==5.3.2
django-cron==0.5.0
django-datetime-widget==0.9.3
django-debreach==1.5.2
django-dirtyfields==1.3.1
django-embed-template==0.2.0
django-filter==1.1.0
django-formtools==2.1
django-haystack==2.8.1
django-hstore==1.4.2
django-imagekit==4.0.2
django-macros==0.4.0
django-redis-cache==1.7.1
django-redis-sessions==0.6.1
django-rest-swagger==2.2.0
django-ses==0.8.5
-e git+https://github.com/PokaInc/django-ses-boto3.git@c08f23b5e8be2188eff1dbdd3a7410878625e741#egg=django_ses_boto3
django-sortedm2m==1.5.0
django-storages==1.7.1
django-timezone-field==2.1
django-widget-tweaks==1.4.2
djangorestframework==3.8.1
djangorestframework-csv==2.1.0
djangorestframework-filters==0.10.2
djangorestframework-hstore==1.3
docker==3.5.0
docker-pycreds==0.3.0
docutils==0.14
drf-extensions==0.4.0
drf-haystack==1.8.4
ecdsa==0.13
elasticsearch==6.3.1
enum34==1.1.6
et-xmlfile==1.0.1
execnet==1.5.0
factory-boy==2.8.1
Faker==0.9.0
filelock==3.0.6
freezegun==0.3.10
funcsigs==1.0.2
functools32==3.2.3.post2
future==0.16.0
futures==3.1.1
geoip2==2.9.0
gevent==1.3.6
greenlet==0.4.14
gunicorn==19.9.0
hiredis==0.2.0
httpretty==0.9.5
huey==1.10.2
idna==2.7
inflection==0.3.1
ipaddress==1.0.22
isodate==0.6.0
itypes==1.1.0
jdcal==1.4
Jinja2==2.10
jmespath==0.9.3
jsondiff==1.1.1
jsonpickle==1.0
jsonschema==2.6.0
kombu==4.2.1
logstash-formatter==0.5.17
lxml==4.2.4
markdown2==2.3.5
MarkupSafe==1.0
maxminddb==1.4.1
mock==2.0.0
mock-django==0.6.10
modernize==0.6.1
more-itertools==4.3.0
moto==1.3.5
netaddr==0.7.19
numpy==1.15.1
openapi-codec==1.3.2
openpyxl==2.5.6
pathlib2==2.3.2
pbr==3.1.1
pilkit==2.0
Pillow==5.2.0
pluggy==0.7.1
progressbar==2.3
ProxyTypes==0.9
psycopg2==2.7.5
psycopg2-binary==2.7.5
py==1.5.2
pyaml==17.12.1
pycparser==2.18
pycryptodome==3.6.6
pyelasticsearch==1.4
pyfakefs==3.4.3
PyJWT==1.6.4
pyquery==1.4.0
pytest==3.8.2
pytest-cov==2.5.1
pytest-django==3.4.2
pytest-env==0.6.2
pytest-forked==0.2
pytest-mock==1.10.0
pytest-random==0.2
pytest-xdist==1.23.0
python-dateutil==2.7.3
python-intercom==3.1.0
python-jose==2.0.2
python-magic==0.4.15
python-memcached==1.59
python-mimeparse==1.6.0
python-redis-lock==3.2.0
pytz==2018.5
PyYAML==3.13
qrcode==6.0
raven==6.9.0
rcssmin==1.0.6
redis==2.10.6
reportlab==3.5.6
requests==2.19.1
responses==0.9.0
retrying==1.3.3
rjsmin==1.0.12
s3transfer==0.1.13
scandir==1.9.0
scipy==1.1.0
semver==2.8.1
setuptools-scm==3.1.0
simplejson==3.16.0
six==1.11.0
swagger-spec-validator==2.4.0
tabulate==0.8.2
teamcity-messages==1.21
text-unidecode==1.2
unicodecsv==0.14.1
Unidecode==1.0.22
uritemplate==3.0.0
urllib3==1.23
vine==1.1.4
websocket-client==0.53.0
Werkzeug==0.14.1
wrapt==1.10.11
xmltodict==0.11.0

spg commented Oct 10, 2018

Here is the output from pip freeze in the broken venv:

amqp==2.3.2
apipkg==1.4
arnparse==0.0.1
asn1crypto==0.24.0
atomicwrites==1.2.1
attrs==17.4.0
aws-requests-auth==0.4.2
aws-xray-sdk==0.95
backports.ssl-match-hostname==3.5.0.1
backports.tempfile==1.0
backports.weakref==1.0.post1
billiard==3.5.0.4
boto==2.48.0
boto3==1.7.84
botocore==1.10.84
celery==4.1.1
celery-haystack==0.10
certifi==2018.8.13
cffi==1.11.4
chardet==3.0.4
checksumdir==1.1.4
click==6.7
concurrent-log-handler==0.9.12
ConcurrentLogHandler==0.9.1
contextlib2==0.5.5
contexttimer==0.3.3
cookies==2.2.1
coreapi==2.3.3
coreschema==0.0.4
coverage==4.5.1
cryptography==2.1.4
cssselect==1.0.3
Django==1.11.15
django-appconf==1.0.2
django-bower==5.2.0
django-cache-friendly-timestamp-signer==1.0.1
django-common-helpers==0.9.1
django-compressor==2.2
django-contrib-comments==1.8.0
django-cors-headers==2.4.0
django-countries==5.3.2
django-cron==0.5.0
django-datetime-widget==0.9.3
django-debreach==1.5.2
django-dirtyfields==1.3.1
django-embed-template==0.2.0
django-filter==1.1.0
django-formtools==2.1
django-haystack==2.8.1
django-hstore==1.4.2
django-imagekit==4.0.2
django-macros==0.4.0
django-redis-cache==1.7.1
django-redis-sessions==0.6.1
django-rest-swagger==2.2.0
django-ses==0.8.5
-e git+https://github.com/PokaInc/django-ses-boto3.git@c08f23b5e8be2188eff1dbdd3a7410878625e741#egg=django_ses_boto3
django-sortedm2m==1.5.0
django-storages==1.7.1
django-timezone-field==2.1
django-widget-tweaks==1.4.2
djangorestframework==3.8.1
djangorestframework-csv==2.1.0
djangorestframework-filters==0.10.2
djangorestframework-hstore==1.3
docker==3.5.0
docker-pycreds==0.3.0
docutils==0.14
drf-extensions==0.4.0
drf-haystack==1.8.4
ecdsa==0.13
elasticsearch==6.3.1
enum34==1.1.6
et-xmlfile==1.0.1
execnet==1.5.0
factory-boy==2.8.1
Faker==0.9.0
filelock==3.0.6
freezegun==0.3.10
funcsigs==1.0.2
functools32==3.2.3.post2
future==0.16.0
futures==3.1.1
geoip2==2.9.0
gevent==1.3.6
greenlet==0.4.14
gunicorn==19.9.0
hiredis==0.2.0
httpretty==0.9.5
huey==1.10.2
idna==2.7
inflection==0.3.1
ipaddress==1.0.22
isodate==0.6.0
itypes==1.1.0
jdcal==1.4
Jinja2==2.10
jmespath==0.9.3
jsondiff==1.1.1
jsonpickle==1.0
jsonschema==2.6.0
kombu==4.2.1
logstash-formatter==0.5.17
lxml==4.2.4
markdown2==2.3.5
MarkupSafe==1.0
maxminddb==1.4.1
mock==2.0.0
mock-django==0.6.10
modernize==0.6.1
more-itertools==4.3.0
moto==1.3.5
netaddr==0.7.19
numpy==1.15.1
openapi-codec==1.3.2
openpyxl==2.5.6
pathlib2==2.3.2
pbr==3.1.1
pilkit==2.0
Pillow==5.2.0
pluggy==0.7.1
progressbar==2.3
ProxyTypes==0.9
psycopg2==2.7.5
psycopg2-binary==2.7.5
py==1.5.2
pyaml==17.12.1
pycparser==2.18
pycryptodome==3.6.6
pyelasticsearch==1.4
pyfakefs==3.4.3
PyJWT==1.6.4
pyquery==1.4.0
pytest==3.8.2
pytest-cov==2.5.1
pytest-django==3.4.2
pytest-env==0.6.2
pytest-forked==0.2
pytest-mock==1.10.0
pytest-random==0.2
pytest-xdist==1.23.0
python-dateutil==2.7.3
python-intercom==3.1.0
python-jose==2.0.2
python-magic==0.4.15
python-memcached==1.59
python-mimeparse==1.6.0
python-redis-lock==3.2.0
pytz==2018.5
PyYAML==3.13
qrcode==6.0
raven==6.9.0
rcssmin==1.0.6
redis==2.10.6
reportlab==3.5.6
requests==2.19.1
responses==0.9.0
retrying==1.3.3
rjsmin==1.0.12
s3transfer==0.1.13
scandir==1.9.0
scipy==1.1.0
semver==2.8.1
setuptools-scm==3.1.0
simplejson==3.16.0
six==1.11.0
swagger-spec-validator==2.4.0
tabulate==0.8.2
teamcity-messages==1.21
text-unidecode==1.2
unicodecsv==0.14.1
Unidecode==1.0.22
uritemplate==3.0.0
urllib3==1.23
vine==1.1.4
websocket-client==0.53.0
Werkzeug==0.14.1
wrapt==1.10.11
xmltodict==0.11.0
@spg

This comment has been minimized.

Show comment
Hide comment
@spg

spg Oct 10, 2018

I have been able to reproduce the issue with a fresh venv and by installing the following packages: pytest (latest), moto==1.3.6 and pyfakefs==3.4.3. The resulting output from pip freeze is:

asn1crypto==0.24.0
atomicwrites==1.2.1
attrs==18.2.0
aws-xray-sdk==0.95
backports.ssl-match-hostname==3.5.0.1
backports.tempfile==1.0
backports.weakref==1.0.post1
boto==2.49.0
boto3==1.7.84
botocore==1.10.84
certifi==2018.8.24
cffi==1.11.5
chardet==3.0.4
cookies==2.2.1
cryptography==2.3.1
docker==3.5.0
docker-pycreds==0.3.0
docutils==0.14
ecdsa==0.13
enum34==1.1.6
funcsigs==1.0.2
future==0.16.0
futures==3.2.0
idna==2.7
ipaddress==1.0.22
Jinja2==2.10
jmespath==0.9.3
jsondiff==1.1.1
jsonpickle==1.0
MarkupSafe==1.0
mock==2.0.0
more-itertools==4.3.0
moto==1.3.6
pathlib2==2.3.2
pbr==4.3.0
pluggy==0.7.1
py==1.6.0
pyaml==17.12.1
pycparser==2.19
pycryptodome==3.6.6
pyfakefs==3.4.3
pytest==3.8.2
python-dateutil==2.7.3
python-jose==2.0.2
pytz==2018.5
PyYAML==3.13
requests==2.19.1
responses==0.9.0
s3transfer==0.1.13
scandir==1.9.0
six==1.11.0
urllib3==1.23
websocket-client==0.53.0
Werkzeug==0.14.1
wrapt==1.10.11
xmltodict==0.11.0

spg commented Oct 10, 2018

I have been able to reproduce the issue with a fresh venv and by installing the following packages: pytest (latest), moto==1.3.6 and pyfakefs==3.4.3. The resulting output from pip freeze is:

asn1crypto==0.24.0
atomicwrites==1.2.1
attrs==18.2.0
aws-xray-sdk==0.95
backports.ssl-match-hostname==3.5.0.1
backports.tempfile==1.0
backports.weakref==1.0.post1
boto==2.49.0
boto3==1.7.84
botocore==1.10.84
certifi==2018.8.24
cffi==1.11.5
chardet==3.0.4
cookies==2.2.1
cryptography==2.3.1
docker==3.5.0
docker-pycreds==0.3.0
docutils==0.14
ecdsa==0.13
enum34==1.1.6
funcsigs==1.0.2
future==0.16.0
futures==3.2.0
idna==2.7
ipaddress==1.0.22
Jinja2==2.10
jmespath==0.9.3
jsondiff==1.1.1
jsonpickle==1.0
MarkupSafe==1.0
mock==2.0.0
more-itertools==4.3.0
moto==1.3.6
pathlib2==2.3.2
pbr==4.3.0
pluggy==0.7.1
py==1.6.0
pyaml==17.12.1
pycparser==2.19
pycryptodome==3.6.6
pyfakefs==3.4.3
pytest==3.8.2
python-dateutil==2.7.3
python-jose==2.0.2
pytz==2018.5
PyYAML==3.13
requests==2.19.1
responses==0.9.0
s3transfer==0.1.13
scandir==1.9.0
six==1.11.0
urllib3==1.23
websocket-client==0.53.0
Werkzeug==0.14.1
wrapt==1.10.11
xmltodict==0.11.0
@spg

This comment has been minimized.

Show comment
Hide comment
@spg

spg Oct 10, 2018

I have narrowed down the venv even further. I can reproduce the issue by installing only: pip install pytest future pyfakefs
Now the venv looks like this:

atomicwrites==1.2.1
attrs==18.2.0
funcsigs==1.0.2
future==0.16.0
more-itertools==4.3.0
pathlib2==2.3.2
pluggy==0.7.1
py==1.6.0
pyfakefs==3.4.3
pytest==3.8.2
scandir==1.9.0
six==1.11.0

spg commented Oct 10, 2018

I have narrowed down the venv even further. I can reproduce the issue by installing only: pip install pytest future pyfakefs
Now the venv looks like this:

atomicwrites==1.2.1
attrs==18.2.0
funcsigs==1.0.2
future==0.16.0
more-itertools==4.3.0
pathlib2==2.3.2
pluggy==0.7.1
py==1.6.0
pyfakefs==3.4.3
pytest==3.8.2
scandir==1.9.0
six==1.11.0
@asottile

This comment has been minimized.

Show comment
Hide comment
@asottile

asottile Oct 10, 2018

Member

I swear, it's going to be python-future again 😡 😡 😡

the number of week-hours of developer time I've lost to that thing.

Member

asottile commented Oct 10, 2018

I swear, it's going to be python-future again 😡 😡 😡

the number of week-hours of developer time I've lost to that thing.

@asottile

This comment has been minimized.

Show comment
Hide comment
@asottile

asottile Oct 10, 2018

Member

ok yep, it's a bug in pyfakefs -- I'll look on making a PR there

Member

asottile commented Oct 10, 2018

ok yep, it's a bug in pyfakefs -- I'll look on making a PR there

asottile added a commit to asottile/pyfakefs that referenced this pull request Oct 10, 2018

Fix pytest when pyfakefs + future is installed
`python-future` is notorious for breaking modules which use `try:` / `except:`
to import modules based on version.  In this case, `pyfakefs` imported the
backported `builtins` module which changes the semantics of the `open()`
function.  `pyfakefs` then monkeypatches `linecache` which breaks any module
which attempts to use `linecache` (in this case `pytest`).

The downstream issue is pytest-dev/pytest#4074
@asottile

This comment has been minimized.

Show comment
Hide comment
@asottile

asottile Oct 10, 2018

Member

Here's that PR, with that applied the example works again: jmcgeheeiv/pyfakefs#441

$ pytest t.py 
============================= test session starts ==============================
platform linux2 -- Python 2.7.15rc1, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
rootdir: /tmp/t, inifile:
plugins: pyfakefs-3.5
collected 1 item                                                               

t.py F                                                                   [100%]

=================================== FAILURES ===================================
__________________________________ test__bug ___________________________________

    def test__bug():
        a = 'é'
>       assert False
E       assert False

t.py:4: AssertionError
=========================== 1 failed in 0.02 seconds ===========================


$ pytest --version
This is pytest version 3.8.2, imported from /tmp/t/venv/local/lib/python2.7/site-packages/pytest.pyc
setuptools registered plugins:
  pyfakefs-3.5 at /tmp/pyfakefs/pyfakefs/pytest_plugin.py

Thanks again for the issue and reproduction steps, this was paramount to finding the root cause! 🎉

Member

asottile commented Oct 10, 2018

Here's that PR, with that applied the example works again: jmcgeheeiv/pyfakefs#441

$ pytest t.py 
============================= test session starts ==============================
platform linux2 -- Python 2.7.15rc1, pytest-3.8.2, py-1.6.0, pluggy-0.7.1
rootdir: /tmp/t, inifile:
plugins: pyfakefs-3.5
collected 1 item                                                               

t.py F                                                                   [100%]

=================================== FAILURES ===================================
__________________________________ test__bug ___________________________________

    def test__bug():
        a = 'é'
>       assert False
E       assert False

t.py:4: AssertionError
=========================== 1 failed in 0.02 seconds ===========================


$ pytest --version
This is pytest version 3.8.2, imported from /tmp/t/venv/local/lib/python2.7/site-packages/pytest.pyc
setuptools registered plugins:
  pyfakefs-3.5 at /tmp/pyfakefs/pyfakefs/pytest_plugin.py

Thanks again for the issue and reproduction steps, this was paramount to finding the root cause! 🎉

@spg

This comment has been minimized.

Show comment
Hide comment
@spg

spg Oct 10, 2018

Thanks for the fix! I applied your patch locally, and it all works now! 🎉

spg commented Oct 10, 2018

Thanks for the fix! I applied your patch locally, and it all works now! 🎉

@spg spg closed this Oct 10, 2018

mrbean-bremen added a commit to jmcgeheeiv/pyfakefs that referenced this pull request Oct 11, 2018

Fix pytest when pyfakefs + future is installed
`python-future` is notorious for breaking modules which use `try:` / `except:`
to import modules based on version.  In this case, `pyfakefs` imported the
backported `builtins` module which changes the semantics of the `open()`
function.  `pyfakefs` then monkeypatches `linecache` which breaks any module
which attempts to use `linecache` (in this case `pytest`).

The downstream issue is pytest-dev/pytest#4074
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment