Strange output with non-ascii string

[tkamenoko]

Platform: Windows 10 64bit
Python: 3.7
Shell: Powershell 6.1.1

Test code:

#test_green.py
from unittest import TestCase


class OutputTests(TestCase):
    def test_success(self):
        self.assertEqual("lang: 日本語", "lang: 日本語")

    def test_fail(self):
        self.assertEqual("lang: 日本語", "lang: English")

Output by Green:

test_green
OutputTests
F test_fail
. test_success

Failure in test_green.OutputTests.test_fail
// Traceback here
AssertionError: 'lang: Ri Ben Yu ' != 'lang: English'
- lang: Ri Ben Yu
+ lang: English

Ran 2 tests in 0.740s

FAILED (failures=1, passes=1)

Output by builtin unittest:

F.

======================================================================
FAIL: test_fail (test_green.OutputTests)
----------------------------------------------------------------------
//Traceback here
AssertionError: 'lang: 日本語' != 'lang: English'
- lang: 日本語
+ lang: English

----------------------------------------------------------------------
Ran 2 tests in 0.002s

FAILED (failures=1)

Test finished as expected, but that is wrong output(日本語->Ri Ben Yu ).

[CleanCut]

I can't reproduce that on my Mac. 🤷‍♂️

What happens if you try setting an encoding with a line like this at the top of your file? (If you are not using utf8, then replace that with your actual encoding):

# -*- coding: utf8 -*-

[MinchinWeb]

Also, what is your codepage set to?

[tkamenoko]

@CleanCut Setting encoding comment has no effect.

@MinchinWeb default 932(shift-jis), but same result on 65001(utf-8).

In my opinion, this is caused by unidecode package.

green/green/output.py

Lines 145 to 148 in ce9383b

	if self._ascii_only_output:
	# Windows doesn't actually want unicode, so we get
	# the closest ASCII equivalent
	text = text_type(unidecode(text))

To reproduce, run this code.

from unidecode import unidecode
print("日本語") # -> 'Ri Ben Yu'

[MinchinWeb]

I think you figured out what's going on, and I can fill in the why: unidecode was brought in as Unicode output was causing a number of hard to debug and reproduce issues on Windows. With unidecode, green would at least output something, rather than crash.

So green is working as it was designed to. But do we need a command line or environmental variable to turn this off?

[CleanCut]

I agree. This is expected, but not necessarily ideal.

If we added an option to disable this behavior, would anyone use it? @tkamenoko @MinchinWeb

[tkamenoko]

Yes, I think so. Current behavior is not documented and may lead to confusion. The same output as builtin unittest is required.

[CleanCut]

Fix is in 2.14.0, just released.