Curses Unicode Support

[atagar1]

BPO	4787
Nosy	@vstinner, @ashemedai

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-05-08.11:28:27.333>
created_at = <Date 2008-12-30.22:57:42.353>
labels = ['type-bug', 'invalid', 'expert-unicode']
title = 'Curses Unicode Support'
updated_at = <Date 2009-05-08.11:28:27.331>
user = 'https://bugs.python.org/atagar1'

bugs.python.org fields:

activity = <Date 2009-05-08.11:28:27.331>
actor = 'asmodai'
assignee = 'none'
closed = True
closed_date = <Date 2009-05-08.11:28:27.333>
closer = 'asmodai'
components = ['Unicode']
creation = <Date 2008-12-30.22:57:42.353>
creator = 'atagar1'
dependencies = []
files = []
hgrepos = []
issue_num = 4787
keywords = []
message_count = 14.0
messages = ['78563', '78564', '78566', '78567', '78568', '78569', '78571', '78572', '78573', '78574', '78576', '78588', '78600', '87436']
nosy_count = 3.0
nosy_names = ['vstinner', 'asmodai', 'atagar1']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4787'
versions = ['Python 3.0']

[atagar1]

Hi, in switching to Python 3.0 I've run into an issue with displaying
Unicode characters via curses. In Python 2.x a simple hello-world looks
like:

#!/usr/bin/python
# coding=UTF-8

import curses
import locale

locale.setlocale(locale.LC_ALL,"")

def doStuff(stdscr):
  message = u"hello わたし!"
  stdscr.addstr(0, 0, message.encode("utf-8"), curses.A_BLINK)
  stdscr.getch() # pauses until a key's hit

curses.wrapper(doStuff)

This works. However, when I try to come up with an equivalent for Python
3.0:

#!/usr/bin/python

import curses
import locale

locale.setlocale(locale.LC_ALL,"")

def doStuff(stdscr):
  message = "hello わたし!"
  stdscr.addstr(0, 0, message, curses.A_BLINK)
  stdscr.getch() # pauses until a key's hit

curses.wrapper(doStuff)

It fails (printing gibberish to the console). It seems that the problem
is that the curses module isn't respecting the system's preferred
encoding (utf-8) which was set via the setlocale function (as per
instructions at
http://docs.python.org/dev/3.0/library/curses.html#module-curses).

My apologies in advance if this is my mistake. Cheers! -Damian

[vstinner]

I tested your example on Linux (Ubuntu Gutsy) and it works correctly.
What is your:

• OS (name, version)
• locale (charset?)

[atagar1]

My OS is Ubuntu 8.04 (Hardy) and the locale is utf-8:

>>> locale.setlocale(locale.LC_ALL,"")
'en_US.UTF-8'

You do mean the Python 3.0 example didn't work, right? The Python3.0
header is:
Python 3.0 (r30:67503, Dec 21 2008, 02:16:52)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)] on linux2

in case that makes a difference.

[atagar1]

Ack - sorry, typo. I meant "You do mean the Python 3.0 example did work,
right?"

[vstinner]

You do mean the Python 3.0 example didn't work, right?

I only tested yje Python3 example and it works correctly on my
computer. I'm using Python3 trunk but I don't think that the curses
module changed after the 3.0 release.

$ ./python
Python 3.1a0 (py3k:67973M, Dec 28 2008, 17:53:52)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
>>> import locale; locale.setlocale(locale.LC_ALL, '')
'fr_FR.UTF-8'

[atagar1]

Doing a checkout of the trunk - I'll let you know if it works. Thanks!

[vstinner]

I think that I catched the problem:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=379447

_curses.so should be linked to libncursesw.so.5 and not
libncurses.so.5. I tested on Hardy and I doesn't work because
_curses.so was linked to libncurses.so.5.

Few commands...
sudo apt-get install libncursesw5-dev
make clean
make

and it works ;-) Possible workaround: reject non-unicode ncurses
library in setup.py. The ugly patch:

Index: setup.py
===================================================================

--- setup.py    (révision 67973)
+++ setup.py    (copie de travail)
@@ -848,23 +848,6 @@
             panel_library = 'panelw'
             exts.append( Extension('_curses', ['_cursesmodule.c'],
                                    libraries = curses_libs) )
-        elif (self.compiler.find_library_file(lib_dirs, 'ncurses')):
-            curses_libs = ['ncurses']
-            exts.append( Extension('_curses', ['_cursesmodule.c'],
-                                   libraries = curses_libs) )
-        elif (self.compiler.find_library_file(lib_dirs, 'curses')
-              and platform != 'darwin'):
-                # OSX has an old Berkeley curses, not good enough for
-                # the _curses module.
-            if 
(self.compiler.find_library_file(lib_dirs, 'terminfo')):
-                curses_libs = ['curses', 'terminfo']
-            elif 
(self.compiler.find_library_file(lib_dirs, 'termcap')):
-                curses_libs = ['curses', 'termcap']
-            else:
-                curses_libs = ['curses']
-
-            exts.append( Extension('_curses', ['_cursesmodule.c'],
-                                   libraries = curses_libs) )
         else:
             missing.append('_curses')

Ugly because it may break the curses module compilation on many OS
(BSD and Solaris?).

[vstinner]

See also the old (and closed) issue bpo-1428494: Prefer linking against
ncursesw over ncurses library.

[atagar1]

Just finished recompiling and works perfectly. My hat's off to you -
many thanks! -Damian

[vstinner]

ncursesw looks to be available for:

• Linux: eg. packaged in Ubunbut
• NetBSD:
  ftp://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/devel/ncursesw/README.html
• FreeBSD: ncurses package with USE="unicode"
• Mac OS X: http://ncursesw.darwinports.com/
• Windows: by Cygwin (libncurses5 package)

I think that libncursesw is working well on Linux and BSD, but Windows
port may be buggy.

[vstinner]

For Solaris, see this bug report of nano editor:
https://savannah.gnu.org/bugs/?24028

libcurses has been added to OpenSolaris in Septembre 2008. But it
don't see the unicode version (libncusesw) :-/

[atagar1]

Looks like this was my mistake, not a bug. According to:
http://mail.python.org/pipermail/python-list/2007-July/450133.html

Python 2.5 also requires the addition of libcursesw but it was working
for the Ubuntu release because they specifically added it. There's no
missing functionality - just my mistake. Cheers! -Damian

[vstinner]

Python 2.5 also requires the addition of libcursesw
but it was working for the Ubuntu release because
they specifically added it.

What do you mean by "the addition of libcursesw"? _curses.so of Python
2.5 is linked to libncursesw.so on Ubuntu Gutsy and Hardy.

If _curses.so is linked to libncurses.so, you can not used multibyte
charset like UTF-8.