testdir.makepyfile fails to write contents with utf-8 on Python 2.7

[fgmacedo]

Versions:

Python 2.7.13
pytest==3.2.2 (master)
Linux L251 4.4.0-92-generic #115-Ubuntu x86_64 x86_64 x86_64 GNU/Linux

The class _pytest.pytester.Testdir._makefile crashes with UnicodeDecodeError when trying to write Python with utf-8 string.

There's a test that reproduces the bug:

    @pytest.mark.xfail("sys.version_info[0] < 3")
    def test_setup_fixture_error_with_mixed_encoding(self, testdir):
        testdir.makepyfile("""
            # coding: utf-8
            def mixed_encoding(as_unicode, as_utf8):
                assert 0
            def setup_function(function):
                mixed_encoding(u"'São Paulo'", "'S\xc3\xa3o Paulo'")
            def test_nada():
                pass
        """)
        result = testdir.runpytest()
        result.stdout.fnmatch_lines([
            "*ERROR at setup of test_nada*",
            "*setup_function(function):*",
            "*mixed_encoding(as_unicode, as_utf8):*",
            "*assert 0*",
            "*1 error*",
        ])
        assert result.ret != 0
        if sys.version_info[0] >= 3:
            result.stdout.fnmatch_lines([
                "*as_unicode = \"'São Paulo'\", as_utf8 = \"'São Paulo'\"*",
            ])
        else:
            result.stdout.fnmatch_lines([
                "*as_unicode = \"'São Paulo'\", as_utf8 = \"'São Paulo'\"*",
            ])

Resulting in:

/opt/loggi/pytest/.tox/py27/lib/python2.7/site-packages/_pytest/pytester.py:550: in makepyfile
    return self._makefile('.py', args, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Testdir local('/tmp/pytest-of-fernando/pytest-4/testdir/test_setup_fixture_error_with_mixed_encoding0')>, ext = '.py'
args = ('\n            # coding: utf-8\n            def mixed_encoding(as_unicode, as_utf8):\n                assert 0\n     ...oding(u"\'S\xc3\xa3o Paulo\'", "\'S\xc3\xa3o Paulo\'")\n            def test_nada():\n                pass\n        ',)
kwargs = {}, encoding = 'utf-8'

    def _makefile(self, ext, args, kwargs, encoding="utf-8"):
        items = list(kwargs.items())
        if args:
            source = py.builtin._totext("\n").join(
>               map(py.builtin._totext, args)) + py.builtin._totext("\n")
E           UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 184: ordinal not in range(128)

/opt/loggi/pytest/.tox/py27/lib/python2.7/site-packages/_pytest/pytester.py:493: UnicodeDecodeError

[nicoddemus]

Thanks for the report. I experienced this myself on occasion and also noticed you had the same problem on your PR yesterday.

[fgmacedo]

Yeah. I had to find a way to test without mixing the scopes :)

Do you have a fix already? Should I do a PR with the test in 4111b66?

[nicoddemus]

I don't think it will be needed, I will add a specific test for makepyfile instead but thanks for the offer. 👍

[bilderbuchi]

@nicoddemus @RonnyPfannschmidt as the PR is merged this can be closed I think.

[RonnyPfannschmidt]

yup, thanks 👍