Shell restart when error message contains non-BMP characters

[TheMathsGod]

BPO	36698
Nosy	@terryjreedy, @serhiy-storchaka, @miss-islington
PRs	bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. #16583 [3.8] bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583) #16649 [3.7] bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583) #16650
Superseder	bpo-22742: IDLE shows traceback when printing non-BMP character

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2019-10-08.12:03:42.416>
created_at = <Date 2019-04-22.17:15:57.723>
labels = ['3.8', 'expert-IDLE', 'type-bug', '3.7', '3.9']
title = 'Shell restart when error message contains non-BMP characters'
updated_at = <Date 2019-10-08.12:03:42.411>
user = 'https://bugs.python.org/TheMathsGod'

bugs.python.org fields:

activity = <Date 2019-10-08.12:03:42.411>
actor = 'serhiy.storchaka'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2019-10-08.12:03:42.416>
closer = 'serhiy.storchaka'
components = ['IDLE']
creation = <Date 2019-04-22.17:15:57.723>
creator = 'TheMathsGod'
dependencies = []
files = []
hgrepos = []
issue_num = 36698
keywords = ['patch']
message_count = 7.0
messages = ['340662', '340677', '353932', '353968', '354187', '354189', '354191']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'serhiy.storchaka', 'miss-islington', 'TheMathsGod']
pr_nums = ['16583', '16649', '16650']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = '22742'
type = 'behavior'
url = 'https://bugs.python.org/issue36698'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[TheMathsGod]

When attempting to raise an error with a message containing non-BMP characters (Unicode ordinals above U+0xFFFF), the python shell displays a long traceback including several UnicodeEncodeErrors and then restarts. Example:

>>> raise Exception('\U0001f603')
Traceback (most recent call last):
  File "<pyshell#84>", line 1, in <module>
    raise Exception('\U0001f603')
Traceback (most recent call last):
  File "<pyshell#84>", line 1, in <module>
    raise Exception('\U0001f603')
Traceback (most recent call last):
  File "D:\Python37\lib\idlelib\run.py", line 474, in runcode
    exec(code, self.locals)
  File "<pyshell#84>", line 1, in <module>
Traceback (most recent call last):
  File "D:\Python37\lib\idlelib\run.py", line 474, in runcode
    exec(code, self.locals)
  File "<pyshell#84>", line 1, in <module>
Exception: 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Python37\lib\idlelib\run.py", line 144, in main
    ret = method(*args, **kwargs)
  File "D:\Python37\lib\idlelib\run.py", line 486, in runcode
    print_exception()
  File "D:\Python37\lib\idlelib\run.py", line 234, in print_exception
    print_exc(typ, val, tb)
  File "D:\Python37\lib\idlelib\run.py", line 232, in print_exc
    print(line, end='', file=efile)
  File "D:\Python37\lib\idlelib\run.py", line 362, in write
    return self.shell.write(s, self.tags)
  File "D:\Python37\lib\idlelib\rpc.py", line 608, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "D:\Python37\lib\idlelib\rpc.py", line 220, in remotecall
    return self.asyncreturn(seq)
  File "D:\Python37\lib\idlelib\rpc.py", line 251, in asyncreturn
    return self.decoderesponse(response)
  File "D:\Python37\lib\idlelib\rpc.py", line 271, in decoderesponse
    raise what
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 11-11: Non-BMP character not supported in Tk

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Python37\lib\idlelib\run.py", line 158, in main
    print_exception()
  File "D:\Python37\lib\idlelib\run.py", line 234, in print_exception
    print_exc(typ, val, tb)
  File "D:\Python37\lib\idlelib\run.py", line 220, in print_exc
    print_exc(type(context), context, context.__traceback__)
  File "D:\Python37\lib\idlelib\run.py", line 232, in print_exc
    print(line, end='', file=efile)
  File "D:\Python37\lib\idlelib\run.py", line 362, in write
    return self.shell.write(s, self.tags)
  File "D:\Python37\lib\idlelib\rpc.py", line 608, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "D:\Python37\lib\idlelib\rpc.py", line 220, in remotecall
    return self.asyncreturn(seq)
  File "D:\Python37\lib\idlelib\rpc.py", line 251, in asyncreturn
    return self.decoderesponse(response)
  File "D:\Python37\lib\idlelib\rpc.py", line 271, in decoderesponse
    raise what
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 11-11: Non-BMP character not supported in Tk

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Python37\lib\idlelib\run.py", line 162, in main
    traceback.print_exception(type, value, tb, file=sys.__stderr__)
  File "D:\Python37\lib\traceback.py", line 105, in print_exception
    print(line, file=file, end="")
  File "D:\Python37\lib\idlelib\run.py", line 362, in write
    return self.shell.write(s, self.tags)
  File "D:\Python37\lib\idlelib\rpc.py", line 608, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "D:\Python37\lib\idlelib\rpc.py", line 220, in remotecall
    return self.asyncreturn(seq)
  File "D:\Python37\lib\idlelib\rpc.py", line 251, in asyncreturn
    return self.decoderesponse(response)
  File "D:\Python37\lib\idlelib\rpc.py", line 271, in decoderesponse
    raise what
UnicodeEncodeError: 'UCS-2' codec can't encode characters in position 11-11: Non-BMP character not supported in Tk

=============================== RESTART: Shell ===============================

>>

I presume the error is caused by Tk being unable to display the characters in the error message, but being forced to anyway by the traceback, causing a series of UnicodeEncodeErrors. Perhaps the error handler should use repr() or similar methods to convert the message into a displayable form?

[terryjreedy]

You are correct about reason. This is essentially a duplicate of bpo-22742, but I am leaving it open for now because the behavior is worse and there is additional information in the tracebacks. The lines for IDLE internals are normally removed, but the exception during the exception prevented that.

[serhiy-storchaka]

Mostly fixed by PR 16545 (see bpo-13153). The original example is now passed.

>>> raise Exception('\U0001f603')
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    raise Exception('\U0001f603')
Exception: 😃
>>> 

But you can still cause a restart of the shell by using surrogate characters:

>>> raise Exception('\ud800')
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    raise Exception('\ud800')
Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    raise Exception('\ud800')

================================ RESTART: Shell ================================

>>

[terryjreedy]

As with bpo-22742, the problem, without PR 16583, only occurs with *nix. On Windows, the traceback ends with "Exception: �" and a prompt is printed.

[serhiy-storchaka]

New changeset b690a27 by Serhiy Storchaka in branch 'master':
bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)
b690a27

[miss-islington]

New changeset a1f4500 by Miss Islington (bot) in branch '3.8':
bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)
a1f4500

[miss-islington]

New changeset 4f962ec by Miss Islington (bot) in branch '3.7':
bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)
4f962ec