idle test fails at least on the 3.6 branch

[doko42]

BPO	32826
Nosy	@terryjreedy, @doko42, @serhiy-storchaka, @csabella, @miss-islington
PRs	bpo-32826: Add "encoding=utf-8" to open() in idle_test/test_help_about. #5639 [3.7] bpo-32826: Add "encoding=utf-8" to open() in idle_test/test_help_about. (GH-5639) #5642 [3.6] bpo-32826: Add "encoding=utf-8" to open() in idle_test/test_help_about. (GH-5639) #5643

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2018-02-12.22:28:44.194>
created_at = <Date 2018-02-12.10:43:22.992>
labels = ['easy', '3.8', 'expert-IDLE', 'type-bug', '3.7']
title = 'idle test fails at least on the 3.6 branch'
updated_at = <Date 2018-02-12.22:28:44.192>
user = 'https://github.com/doko42'

bugs.python.org fields:

activity = <Date 2018-02-12.22:28:44.192>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2018-02-12.22:28:44.194>
closer = 'terry.reedy'
components = ['IDLE']
creation = <Date 2018-02-12.10:43:22.992>
creator = 'doko'
dependencies = []
files = []
hgrepos = []
issue_num = 32826
keywords = ['patch', 'easy']
message_count = 10.0
messages = ['312037', '312059', '312060', '312064', '312065', '312066', '312073', '312075', '312077', '312086']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'doko', 'serhiy.storchaka', 'cheryl.sabella', 'miss-islington']
pr_nums = ['5639', '5642', '5643']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32826'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[doko42]

seen with the 3.6 branch 20180212, last known succeeding test is the 3.6.4 release.

test test_idle failed -- Traceback (most recent call last):
  File "/home/packages/python/3.6/python3.6-3.6.4/Lib/idlelib/idle_test/test_help_about.py", line 78, in test_file_buttons
    self.assertEqual(f.readline().strip(), get('1.0', '1.end'))
  File "/home/packages/python/3.6/python3.6-3.6.4/build-debug/../Lib/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1540: ordinal not in range(128)

[terryjreedy]

What system? test_idle passes on Windows. Unfortunately, none of the *nix or Mac buildbots run tkinter and hence no tests that require a live tk window, and this one does. Cheryl, what do you see on your linux?

Byte 0xc3 in position 1540 is the first byte of the utf-8 encoding of the 'ö' in Löwis on line 27 of CREDITS.txt. So on your system, readline must be reading *and decoding* more of the file.

I am puzzled that the test fails now if it passed with the 3.6.4 release, as it and the textfile have not been touched since last summer.
The files are the same on all 3 python versions.

In any case, I will add encoding='utf-8'to open. I will also use subtest to identify failing cases. I will need you to verify the fix on your system.

[serhiy-storchaka]

The failure is reproduced on Posix locale. Files are open with bare open() which uses the locale encoding by default. Which is ASCII in this case.

[terryjreedy]

I just discovered that the default for me is some version of latin1, not ascii, so that utf-8 'ö' is read as 'ö'. Hence, no UnicodeDecodeError even if 'ö' is being read and decoded by the initial readline call. I presume the same is true on current Windows buildbots. Hence, it was a bug to not include an encoding in the test file open calls. (The widget itself *does* use explicit 'ascii' or 'utf-8' encodings for open().)

[csabella]

I've been able to run the IDLE test suite on Linux.

As far as this issue, there's also a readline in test_textview that you may want to add the encoding to, even though it's only reading __file__.

[terryjreedy]

I reproduced the exact failure by temporarily replacing 'utf-8' with 'ascii' in my clone. So I don't need anyone else to verify the fix.

[terryjreedy]

New changeset f34e03e by Terry Jan Reedy in branch 'master':
bpo-32826: Add "encoding=utf-8" to open() in idle_test/test_help_about. (GH-5639)
f34e03e

[miss-islington]

New changeset 38b4dd7 by Miss Islington (bot) in branch '3.7':
bpo-32826: Add "encoding=utf-8" to open() in idle_test/test_help_about. (GH-5639)
38b4dd7

[miss-islington]

New changeset 46daf39 by Miss Islington (bot) in branch '3.6':
bpo-32826: Add "encoding=utf-8" to open() in idle_test/test_help_about. (GH-5639)
46daf39

[terryjreedy]

Serhiy, thanks for comment and review.
Cheryl, bpo-32837 expands upon your comment.