Patch: new method get_wch for ncurses bindings: accept wide characters (unicode)

[inigoserna]

BPO	6755
Nosy	@akuchling, @jcea, @cben, @pitrou, @vstinner, @bitdancer
Files	curses.get_wch.patch: Patch for the documentation test_get_wch.py: Test example _cursesmodule.get_wch.patch: Patch against Python 2.6.2 _cursesmodule.c _cursesmodule.311.get_wch.patch: Patch against Python 3.1.1 _cursesmodule.c test_ucs2w.py: Several implementations of wcwidth() and wcswidth() ucs2w.c: C extension implementation of wcwidth() and wcswidth()

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-03-08.01:18:55.099>
created_at = <Date 2009-08-21.11:43:30.490>
labels = ['extension-modules', 'type-feature']
title = 'Patch: new method get_wch for ncurses bindings: accept wide characters (unicode)'
updated_at = <Date 2012-03-08.01:18:55.098>
user = 'https://bugs.python.org/inigoserna'

bugs.python.org fields:

activity = <Date 2012-03-08.01:18:55.098>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2012-03-08.01:18:55.099>
closer = 'vstinner'
components = ['Extension Modules']
creation = <Date 2009-08-21.11:43:30.490>
creator = 'inigoserna'
dependencies = []
files = ['14753', '14754', '14755', '14756', '14782', '14783']
hgrepos = []
issue_num = 6755
keywords = ['patch']
message_count = 30.0
messages = ['91816', '91817', '91818', '91819', '91821', '91867', '91868', '91956', '91973', '91974', '91975', '91982', '108249', '139333', '139335', '140371', '140374', '140378', '140403', '140404', '140408', '140409', '140410', '140412', '140640', '143798', '155052', '155054', '155119', '155146']
nosy_count = 13.0
nosy_names = ['akuchling', 'jcea', 'cben', 'pitrou', 'vstinner', 'gpolo', 'r.david.murray', 'inigoserna', 'phep', 'zeha', 'schodet', 'python-dev', 'Nicholas.Cole']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue6755'
versions = ['Python 3.3']

[inigoserna]

Currently,there is no a simple way in curses bindings to get the code
associated with a key press of non ascii keystroke (f.e. ç) in terminals
configured with UTF-8 encoding.

getch returns the code for a wide character byte a byte.
But ncurses library has a proper function to do it: get_wch.

Patch against Python v2.6.2 to provide this missing get_wch method
in the ncurses bindings.

Include a test example and a patch to the documentation as well.

More info and a partial solution without patching python curses module
on this thread:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/67dce30f0a2742a6?fwc=2

[inigoserna]

Added patch for the documentation

[inigoserna]

Added test example

[inigoserna]

Added missing file: patch against Python v2.6.2

[inigoserna]

Added patch against Python v3.1.1.
NOT TESTED!

[gpolo]

Have you looked into bpo-700921 already ? It seems a lot of discussion
was generated there, but no patches.

[inigoserna]

Thanks for the pointer, haven't seen anything when I searched for get_wch.

The patch provided here only adds this get_wch function, because as A.M.
Kuchling explained in bpo-700921, it's possible to use wide chars now,
the only feature missing is get_wch.

[cben]

Nice. 2 questions:

1. Why not change getch() to always use get_wch()?
2. I think you also want fix getkey() / introduce get_wkey().

[inigoserna]

Q. Why not change getch() to always use get_wch()?

This could break backwards compatibility.
There are some code out there that may use getch() to get the bytes
stream one by one and build the wide char.
In fact I'm using this trick to get unicode chars by now.
Look the thread link in first comment to find the implementation I've
developed for my app. Other people are using similar approaches too.

Q. I think you also want fix getkey() / introduce get_wkey().

In my own experience get_wkey isn't be as useful when dealing with wide
chars. But, of course, that's only my use cases.

[inigoserna]

Btw, I don't know if this is the best place to comment it but as it is
somehow related with ncurses...

Other functions I miss a lot are wcwidth() and wcswidth().

These functions return the real width (read, cells length in screen) for
unicode strings.

An example to clarify the issue: one simple Chinese character could need
2 cells on screen, thus len(chinese_unicode_string) won't return the
real screen width needed to show the string.

i.e., len(chinese_unicode_string) != wcswidth(chinese_unicode_string)

Those functions are included into not so old glibc versions (2.2+?), at
least on my Linux systems.

Sadly enough, python doesn't bind them, afaik.
I've tried ctypes but don't work for me (don't know the reason), so I've
written some replacements.

Please look at these files:

• test_ucs2w.py: benchmarks to different implementations. Most of them,
  pure python. Please consider only ucs2w_1x, other are only experiments.

• ucs2w.c: C extension implementation

I think Python could benefit from having these functions in the standard
library. Surely, most simple way should be to bind glibc functions, but
don't know if they exist on other platforms such as MacOS X or Windows.

Neither do I know where they fit... perhaps in unicodedata module.

What do you think? who is the person to convince? (please, don't ask me
to write a PEP, my English is not good enough).

[bitdancer]

For the title concern of this patch I'm adding akuchling as nosy.

Judging by your post your English probably is good enough to write a PEP
(the PEP editors should help with fine tuning it, at least in theory).
However, I doubt a PEP would be necessary.

As for where to raise the (new) issue...given that these functions are
independent of curses, I'd say open a new issue. If you do that, I'd
suggest making lemburg nosy on the issue, as he is the original author
of the unicodedata module and many other unicode things in python.

I'm setting the stage to test needed since your test case isn't a unit
test. I have no idea (never having worked with curses) how hard a unit
test would be to write. Nor have I reviewed the patch, for the same reason.

[vstinner]

inigoserna>> Other functions I miss a lot are wcwidth() and wcswidth()

I wrote a patch to implement unicode.width() method:
http://bugs.python.org/file13357/unicode_width.patch

It's part of the issue bpo-2382 (SyntaxError cursor shifted if multibyte
character is in line)

[zeha]

Will this be part of 3.2 and possibly 2.7?

Without these patches wide character input using curses is basically impossible (on at least some platforms).