readline interferes with characters beginning with byte \xe9

[takluyver]

BPO	11679
Nosy	@vstinner, @merwok, @takluyver, @akheron, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-11-19.10:45:14.564>
created_at = <Date 2011-03-26.00:28:38.643>
labels = ['type-bug', 'invalid', 'expert-unicode']
title = 'readline interferes with characters beginning with byte \\xe9'
updated_at = <Date 2012-11-19.10:45:14.562>
user = 'https://github.com/takluyver'

bugs.python.org fields:

activity = <Date 2012-11-19.10:45:14.562>
actor = 'takluyver'
assignee = 'none'
closed = True
closed_date = <Date 2012-11-19.10:45:14.564>
closer = 'takluyver'
components = ['Unicode']
creation = <Date 2011-03-26.00:28:38.643>
creator = 'takluyver'
dependencies = []
files = []
hgrepos = []
issue_num = 11679
keywords = []
message_count = 6.0
messages = ['132192', '141435', '175836', '175837', '175854', '175954']
nosy_count = 5.0
nosy_names = ['vstinner', 'eric.araujo', 'takluyver', 'petri.lehtinen', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11679'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']

[takluyver]

To replicate, in Python 3.1 on Linux (utf-8 console):

>>> print(chr(0x9000))
退

Copy and paste this character into the prompt. It appears correctly (as a Chinese character). Then:

>> import readline
>> readline.parse_and_bind('"\M-i":" "')

Now try to paste the character again: it appears as " ��"
(four spaces, two unknown character symbols), and if you press return, you get a SyntaxError.

This happens with all characters beginning with \xe9: In UTF-8, that's 0x9000-0x9fff. If the terminal encoding is changed to cp1252, I'm told that the same thing can be achieved with é, which is \xe9 there.

[akheron]

You're binding the M-i keyboard sequence. Could it be that the \xe9 byte is translated by the terminal to M-i, and that causes the interference? In this case, it's not really a bug.

[merwok]

Original bug report: ipython/ipython#58

[vstinner]

I confirm that the issue exists, but I don't think that it comes from Python. I bet that the readline library uses *byte* string, not *character* string, and so is unable to handle correctly multibyte characters like the chinese character U+9000.

[serhiy-storchaka]

Yes, this is a readline issue.

Add '"\M-i":" "' line to ~/.inputrc, run 'rlwrap cat' command, paste this multibyte character and you got the same result.

This is not a Python bug.

[takluyver]

OK, thanks, and sorry for the noise. I've closed this issue.

Looking at the readline manual, it looks like this is tied up with the options input-meta, output-meta and convert-meta. Fiddling around with .inputrc hasn't clarified exactly what they do, but it seems that the terminal can either handle unicode, or shortcuts involving meta (alt), but not both.