UnicodeDecodeError content.py text_content()

[elopio]

I've just switched my language to Esperanto, and started getting a testtools exception when a test failed. The exception was hiding the error, so I had to debug to understand what was going on.

This is the exception:
Traceback (most recent call last):
File "/usr/bin/autopilot", line 9, in
load_entry_point('autopilot==1.4.0', 'console_scripts', 'autopilot')()
File "/usr/lib/python2.7/dist-packages/autopilot/run.py", line 414, in main
test_app.run()
File "/usr/lib/python2.7/dist-packages/autopilot/run.py", line 254, in run
self.run_tests()
File "/usr/lib/python2.7/dist-packages/autopilot/run.py", line 367, in run_tests
test_result = test_suite.run(result)
File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
test(result)
File "/usr/lib/python2.7/unittest/case.py", line 395, in call
return self.run(_args, *_kwds)
File "/usr/lib/python2.7/dist-packages/testscenarios/testcase.py", line 65, in run
return super(WithScenarios, self).run(result)
File "/usr/lib/python2.7/dist-packages/testtools/testcase.py", line 568, in run
return self.RunTest(self, self.exception_handlers).run(result)
File "/usr/lib/python2.7/dist-packages/testtools/runtest.py", line 74, in run
return self._run_one(actual_result)
File "/usr/lib/python2.7/dist-packages/testtools/runtest.py", line 88, in _run_one
return self._run_prepared_result(ExtendedToOriginalDecorator(result))
File "/usr/lib/python2.7/dist-packages/testtools/runtest.py", line 107, in _run_prepared_result
handler(self.case, self.result, e)
File "/usr/lib/python2.7/dist-packages/testtools/testcase.py", line 533, in _report_error
result.addError(self, details=self.getDetails())
File "/usr/lib/python2.7/dist-packages/testtools/testresult/real.py", line 1120, in addError
return self.decorated.addError(test, details=details)
File "/usr/lib/python2.7/dist-packages/autopilot/testresult.py", line 61, in addError
self._log_details(logging.ERROR, test.getDetails())
File "/usr/lib/python2.7/dist-packages/testtools/testcase.py", line 234, in getDetails
return self._details
File "/usr/lib/python2.7/bdb.py", line 53, in trace_dispatch
return self.dispatch_return(frame, arg)
File "/usr/lib/python2.7/bdb.py", line 88, in dispatch_return
self.user_return(frame, arg)
File "/usr/lib/python2.7/pdb.py", line 190, in user_return
self.interaction(frame, None)
File "/usr/lib/python2.7/pdb.py", line 209, in interaction
self.print_stack_entry(self.stack[self.curindex])
File "/usr/lib/python2.7/pdb.py", line 900, in print_stack_entry
prompt_prefix)
File "/usr/lib/python2.7/bdb.py", line 381, in format_stack_entry
s = s + repr.repr(rv)
File "/usr/lib/python2.7/repr.py", line 24, in repr
return self.repr1(x, self.maxlevel)
File "/usr/lib/python2.7/repr.py", line 32, in repr1
return getattr(self, 'repr' + typename)(x, level)
File "/usr/lib/python2.7/repr.py", line 85, in repr_dict
valrepr = repr1(x[key], newlevel)
File "/usr/lib/python2.7/repr.py", line 34, in repr1
s = __builtin.repr(x)
File "/usr/lib/python2.7/dist-packages/testtools/content.py", line 124, in repr
self.content_type, _join_b(self.iter_bytes()))
File "/usr/lib/python2.7/dist-packages/testtools/content.py", line 97, in iter_bytes
return self._get_bytes()
File "/usr/lib/python2.7/dist-packages/testtools/content.py", line 266, in
return Content(UTF8_TEXT, lambda: [text.encode('utf8')])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 135: ordinal not in range(128)

This is the error message the test is trying to add as detail:

/usr/lib/python2.7/dist-packages/testtools/content.py(267)text_content()
-> return Content(UTF8_TEXT, lambda: [text.encode('utf8')])
(Pdb) text
'Testability driver loaded. Wire protocol version is "1.4".\nQInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: Ne havi\xc4\x9das plu da spaco sur aparato\nfile:///home/elopio/workspace/ubuntu/ubuntu-clock-app/reviews/nik90/transition-worldclock-u1db/ubuntu-clock-app.qml:24 module "U1db" is not installed\n\n'

[thomir]

Leo,

We changed testtools to error when text_content was passed text, rather than bytes, which I think is what's happening here. Does this still happen with the newest testtools?

Cheers,

[rbtcollins]

ping?

[elopio]

Hello, sorry for the late reply.

With this test I can reproduce it on python2, on ubuntu vivid. It works on python3. I will try it later in a virtualenv with the version from pypi.

# -*- coding: utf-8 -*-                                                         

from testtools import TestCase
from testtools.content import text_content


class Test(TestCase):

    def test_detail(self):
        self.addDetail('arbitrary-text', text_content('ĝ'))
        self.fail()

elopio@tangamandapio76:~$ apt-cache policy python-testtools
python-testtools:
  Instalita: 0.9.39-1
  Kandidato: 0.9.39-1
  Version table:
 *** 0.9.39-1 0
        500 http://cr.archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages
        100 /var/lib/dpkg/status
elopio@tangamandapio76:~$ python -m unittest discover /tmp/
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
  File "/usr/lib/python2.7/unittest/main.py", line 95, in __init__
    self.runTests()
  File "/usr/lib/python2.7/unittest/main.py", line 232, in runTests
    self.result = testRunner.run(self.test)
  File "/usr/lib/python2.7/unittest/runner.py", line 151, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/suite.py", line 70, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/suite.py", line 108, in run
    test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python2.7/dist-packages/testtools/testcase.py", line 585, in run
    return self.__RunTest(self, self.exception_handlers).run(result)
  File "/usr/lib/python2.7/dist-packages/testtools/runtest.py", line 74, in run
    return self._run_one(actual_result)
  File "/usr/lib/python2.7/dist-packages/testtools/runtest.py", line 88, in _run_one
    return self._run_prepared_result(ExtendedToOriginalDecorator(result))
  File "/usr/lib/python2.7/dist-packages/testtools/runtest.py", line 107, in _run_prepared_result
    handler(self.case, self.result, e)
  File "/usr/lib/python2.7/dist-packages/testtools/testcase.py", line 558, in _report_failure
    result.addFailure(self, details=self.getDetails())
  File "/usr/lib/python2.7/dist-packages/testtools/testresult/real.py", line 1148, in addFailure
    err = self._details_to_exc_info(details)
  File "/usr/lib/python2.7/dist-packages/testtools/testresult/real.py", line 1210, in _details_to_exc_info
    _StringException(_details_to_str(details, special='traceback')),
  File "/usr/lib/python2.7/dist-packages/testtools/testresult/real.py", line 1752, in _details_to_str
    text = content.as_text().strip()
  File "/usr/lib/python2.7/dist-packages/testtools/content.py", line 94, in as_text
    return _u('').join(self.iter_text())
  File "/usr/lib/python2.7/dist-packages/testtools/content.py", line 117, in _iter_text
    for bytes in self.iter_bytes():
  File "/usr/lib/python2.7/dist-packages/testtools/content.py", line 98, in iter_bytes
    return self._get_bytes()
  File "/usr/lib/python2.7/dist-packages/testtools/content.py", line 269, in <lambda>
    return Content(UTF8_TEXT, lambda: [text.encode('utf8')])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in range(128)

elopio@tangamandapio76:~$ apt-cache policy python3-testtools
python3-testtools:
  Instalita: 0.9.39-1
  Kandidato: 0.9.39-1
  Version table:
 *** 0.9.39-1 0
        500 http://cr.archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages
        100 /var/lib/dpkg/status
elopio@tangamandapio76:~$ python3 -m unittest discover /tmp/
F
======================================================================
FAIL: test_detail (test.Test)
test.Test.test_detail
----------------------------------------------------------------------
testtools.testresult.real._StringException: arbitrary-text: {{{ĝ}}}

Traceback (most recent call last):
  File "/tmp/test.py", line 12, in test_detail
    self.fail()
  File "/usr/lib/python3.4/unittest/case.py", line 642, in fail
    raise self.failureException(msg)
AssertionError: None


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

[elopio]

Reproduced using the version from pypi, that's 1.8.0: http://paste.ubuntu.com/11794827/
I'm having problems running this from source: http://paste.ubuntu.com/11794827/
It'll have to wait until I have some more time to figure out what am I doing wrong in there.

[thomir]

@elopio - your last comment has the same pastebin twice - I guess that's not what you meant to do :D

I have some time to try and reproduce and fix this, but I'm unable to reproduce. I suspect it requires some environment variables to be set: LANG etc.

Are you able to take another look and see if this still happens?

Cheers.

[elopio]

Yeah, I'm not sure what I was trying to say in that repeated paste.

Did you try with this, from my previous comment, on py2?
self.addDetail('arbitrary-text', text_content('ĝ'))

[thomir]

@elopio I did, and it works for me in both 2.7.9 and 3.4.3. Like I say, I imagine it depends on certain environment variables - you mention this only started happening when you updated some language settings?

[rbtcollins]

I'm going to close this - another 6 months and no reproducer. please do reopen if you can provide updated instructions on reproducing.