pytest fails with UnicodeEncodeError on doctest containing "print(u'Ä')"

[lsaffre]

I am stuck... please help to find a workaround!
In an otherwise empty directory I have a file named "test.txt" with two lines of text:

>>> print(u'Ä')
Ä

I verified whether my file is indeed UTF-8 encoded:

$ file test.txt
test.txt: UTF-8 Unicode text

When I run pytest on this, it fails with a UnicodeEncodeError.
Here is a full session log: https://bpaste.net/show/b0adfb13e3e3

When I remove the "u" (i.e. the output of the command being tested is itself encoded), then pytest works well and says that the test passed. The problem occurs only under Python 2 and only when I want to test unicode output which contains non-ascii text.

Reproduced on several machines with
pytest version 3.2.1 and 3.2.2.dev17+g539523c (current clone)
Python 2.7.9 and 2.7.12

Thanks for any suggesteions.
Luc

[RonnyPfannschmidt]

this looks like even more of a problem - pytest is already working around some of the rotten unicode related bugs in doctest

@nicoddemus @hpk42 im starting to think we should move doctest in a extra plugin that can have bugfixes independent of pytest

[nicoddemus]

@lsaffre thanks for the report. I managed to reproduce and workaround the problem by changing the input text to:

>>> print('Ä')  # doctest: +ALLOW_UNICODE
Ä

============================= test session starts =============================
platform win32 -- Python 2.7.13, pytest-3.2.2.dev8+g16b248f.d20170817, py-1.4.34, pluggy-0.4.0
rootdir: C:\pytest\.tmp, inifile: pytest.ini
plugins: ast-back-to-python-0.1.0, hypothesis-3.13.1
collected 1 item

.tmp\test.txt .

========================== 1 passed in 0.01 seconds ===========================

But this still feels like a bug that should be investigated.

im starting to think we should move doctest in a extra plugin that can have bugfixes independent of pytest

Not sure having doctest as a separate plugin from pytest would bring much benefit, we have been having fairly regular bug-fix releases lately.

[lsaffre]

Hm, I cannot confirm that workaround. Problem persists on my machine even with +ALLOW_UNICODE. I tried with the newest development version and with the last released version. Luc

…

On 30/08/17 02:50, Bruno Oliveira wrote: @lsaffre <https://github.com/lsaffre> thanks for the report. I managed to reproduce and workaround the problem by changing the input text to: |>>> print('Ä') # doctest: +ALLOW_UNICODE Ä | |============================= test session starts ============================= platform win32 -- Python 2.7.13, pytest-3.2.2.dev8+g16b248f.d20170817, py-1.4.34, pluggy-0.4.0 rootdir: C:\pytest\.tmp, inifile: pytest.ini plugins: ast-back-to-python-0.1.0, hypothesis-3.13.1 collected 1 item .tmp\test.txt . ========================== 1 passed in 0.01 seconds =========================== | But this still feels like a bug that should be investigated. im starting to think we should move doctest in a extra plugin that can have bugfixes independent of pytest Not sure having doctest as a separate plugin from |pytest| would bring much benefit, we have been having fairly regular bug-fix releases lately. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2728 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEM-J4bED79-xsyXN_6KkM-X33KfK2Ubks5sdKO3gaJpZM4PFP9G>.

[nicoddemus]

@lsaffre hmm that's strange. Which OS are you using?

[asottile]

🎉 this got fixed with #3584 -- upgrade to pytest>=3.6.2