print() fails on latin1 characters on OSX

[javahaxxor]

BPO	14986
Nosy	@ronaldoussoren, @ned-deily, @bitdancer, @hynek

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-06-02.21:30:07.040>
created_at = <Date 2012-06-02.13:18:23.735>
labels = ['OS-mac', 'invalid']
title = 'print() fails on latin1 characters on OSX'
updated_at = <Date 2012-06-03.09:25:26.631>
user = 'https://bugs.python.org/javahaxxor'

bugs.python.org fields:

activity = <Date 2012-06-03.09:25:26.631>
actor = 'javahaxxor'
assignee = 'none'
closed = True
closed_date = <Date 2012-06-02.21:30:07.040>
closer = 'hynek'
components = ['macOS']
creation = <Date 2012-06-02.13:18:23.735>
creator = 'javahaxxor'
dependencies = []
files = []
hgrepos = []
issue_num = 14986
keywords = []
message_count = 12.0
messages = ['162134', '162135', '162143', '162150', '162156', '162158', '162164', '162173', '162174', '162177', '162178', '162202']
nosy_count = 5.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'r.david.murray', 'hynek', 'javahaxxor']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue14986'
versions = ['Python 3.2']

[javahaxxor]

print(listentry) fails on folder name with swedish (latin1) characters
Error:

File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/encodings/mac_roman.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u030a' in position 33: character maps to <undefined>

[bitdancer]

A mac expert can confirm, but I think that just means that the default mac_roman encoding (which is made the default by the OS, if I understand correctly) can't handle that character. I believe it will work if you use utf-8. And no, I don't know how to do that, not be being a Mac person.

[hynek]

'\u030a' can’t be latin1 as 0x030a = 778 which is waaay beyond 255. :) That's gonna be utf-8 and indeed that maps to " ̊".

My best guess is that your LC_CTYPE is set to Mac Roman. You can check it using "import os;os.environ.get('LC_CTYPE')".

Try running python as "LC_CTYPE=sv_SE.UTF-8 python3" and do a "print('\u030a')" to try if it helps.

Otherwise a more complete (but minimal) example demonstrating the problem would be helpful.

[ned-deily]

mac_roman is an obsolete encoding from Mac OS 9 days; it is seldom seen on modern OS X systems. But it is often the fallback encoding set in ~/.CFUserTextEncoding if the LANG or a LC_* environment variable is not set (see, for example, http://superuser.com/questions/82123/mac-whats-cfusertextencoding-for). If you run a terminal session using Terminal.app, the LANG environment variable is usually set for you to an appropriate modern value, like 'en_US.UTF-8' in the US locale; this is controlled by a Terminal.app preference; other terminal apps like iTerm2 have something similar. But if you are using xterm with X11, xterm does not inject a LANG env variable. So, something like:

python3.2 -c 'print("\u030a")'

may fail running under xterm with UnicodeEncodeError but will print the expected character when run under Terminal.app. I avoid those kinds of issues by explicitly setting LANG in my shell profile.

Let us know if that helps or, if not, how to reproduce your issue.

[javahaxxor]

The char in question: 'å'. It is a folder with this character in the name. My encoding is UTF-8. Running print("\u030a") gives a blank line

U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE
General Character Properties
In Unicode since: 1.1
Unicode category: Letter, Uppercase
Canonical decomposition: U+0041 LATIN CAPITAL LETTER A + U+030A COMBINING RING ABOVE
Various Useful Representations
UTF-8: 0xC3 0x85
UTF-16: 0x00C5
C octal escaped UTF-8: \303\205
XML decimal entity: Å
Annotations and Cross References
See also:
• U+212B ANGSTROM SIGN
Equivalents:
• U+0041 LATIN CAPITAL LETTER A U+030A COMBINING RING ABOVE

The code:

def traverse (targetDir):
    currentDir = targetDir
    dirs = os.listdir(targetDir)
    for entry in dirs:
        if os.path.isdir(entry):
            print("Traversing " + entry)
            traverse(entry)
        else:
            print("Not dir: " + entry)
            if os.path.isfile(entry):
                print("Processing " + " " + currentDir + " " + entry)
            else:
                print("Not file: " + entry)
    print("\n")

[javahaxxor]

The last post is the CAPITAL Å. The following is the small letter "å"

U+00E5 LATIN SMALL LETTER A WITH RING ABOVE
General Character Properties
In Unicode since: 1.1
Unicode category: Letter, Lowercase
Canonical decomposition: U+0061 LATIN SMALL LETTER A + U+030A COMBINING RING ABOVE
Various Useful Representations
UTF-8: 0xC3 0xA5
UTF-16: 0x00E5
C octal escaped UTF-8: \303\245
XML decimal entity: å
Annotations and Cross References
Notes:
• Danish, Norwegian, Swedish, Walloon
Equivalents:
• U+0061 LATIN SMALL LETTER A U+030A COMBINING RING ABOVE

[ned-deily]

The character in question is not the problem and the code snippet you provide looks fine. The problem is almost certainly that you are running the code in an execution environment where the LANG environment variable is either not set or is set to an encoding that doesn't support higher-order Unicode characters. The fallback 'mac_roman' is such an encoding. The default encodings used by the Python 3 interpreter are influenced by the value of these environment variables. So the questions are: how are you running your code and what are the values of the environment variables that your Python program inherits, and, by any chance, is your program using the 'locale' module, and if so, exactly what functions from it?

Please try adding the following in the environment you are seeing the problem:

import sys
print(sys.stdout)
import os
print([(k, os.environ[k]) for k in os.environ if k.startswith('LC')])
print([(k, os.environ[k]) for k in os.environ if k.startswith('LANG')])
import locale
print(locale.getlocale())
print('\u00e5')
print('\u0061\u030a')

If I paste the above into a Python3.2 interactive terminal session using the python.org 64-/32-bit Python 3.2.3, I see the following:

$ python3.2
Python 3.2.3 (v3.2.3:3d0686d90f55, Apr 10 2012, 11:25:50) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.stdout)
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
>>> import os
>>> print([(k, os.environ[k]) for k in os.environ if k.startswith('LC')])
[]
>>> print([(k, os.environ[k]) for k in os.environ if k.startswith('LANG')])
[('LANG', 'en_US.UTF-8')]
>>> import locale
>>> print(locale.getlocale())
('en_US', 'UTF-8')
>>> print('\u00e5')
å
>>> print('\u0061\u030a')
å

But, if I explicitly remove the LANG environment variable:

$ unset LANG
$ python3.2
Python 3.2.3 (v3.2.3:3d0686d90f55, Apr 10 2012, 11:25:50) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.stdout)
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='US-ASCII'>
>>> import os
>>> print([(k, os.environ[k]) for k in os.environ if k.startswith('LC')])
[]
>>> print([(k, os.environ[k]) for k in os.environ if k.startswith('LANG')])
[]
>>> import locale
>>> print(locale.getlocale())
(None, None)
>>> print('\u00e5')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xe5' in position 0: ordinal not in range(128)
>>> print('\u0061\u030a')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\u030a' in position 1: ordinal not in range(128)
>>>

[javahaxxor]

Output in console:

Python 3.2.3 (v3.2.3:3d0686d90f55, Apr 10 2012, 11:25:50) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.stdout)
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
>>> import os
>>> print([(k, os.environ[k]) for k in os.environ if k.startswith('LC')])
[('LC_CTYPE', 'UTF-8')]
>>> print([(k, os.environ[k]) for k in os.environ if k.startswith('LANG')])
[]
>>> import locale
>>> print(locale.getlocale())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/locale.py", line 524, in getlocale
    return _parse_localename(localename)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/locale.py", line 433, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: UTF-8
>>> print('\u00e5')
å
>>> print('\u0061\u030a')
å
**********************

Output from Eclipse:

<_io.TextIOWrapper name='<stdout>' mode='w' encoding='MacRoman'>
[]
[]
(None, None)
å
Traceback (most recent call last):
  File "/Users/adyhasch/Documents/PythonWorkspace/PatternRenamer/src/prenamer.py", line 70, in <module>
    print('\u0061\u030a')
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/encodings/mac_roman.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u030a' in position 1: character maps to <undefined>
************************************

I'm running PyDev ..

[javahaxxor]

my code runs fine in a console window, so it's some kind of configuration error. Sorry for wasting your time guys .. It would be nice to know why PyDev is not setting the right environment vars though ..

>>> traverse(".")
Processing ./.DS_Store
Traversing ./2011-10-03--Sebi_o_costi_ny_frisyr
Processing ./2011-10-03--Sebi_o_costi_ny_frisyr/.DS_Store
Processing ./2011-10-03--Sebi_o_costi_ny_frisyr/.picasa.ini
Traversing ./2011-10-03--Sebi_o_costi_ny_frisyr/2011-10-04--Sebastian_2år
Processing ./2011-10-03--Sebi_o_costi_ny_frisyr/2011-10-04--Sebastian_2år/.DS_Store
Processing ./2011-10-03--Sebi_o_costi_ny_frisyr/2011-10-04--Sebastian_2år/.picasa.ini
Processing ./2011-10-03--Sebi_o_costi_ny_frisyr/2011-10-04--Sebastian_2år/DSC_5467.JPG
Processing ./2011-10-03--Sebi_o_costi_ny_frisyr/2011-10-04--Sebastian_2år/DSC_5468.JPG
Processing ./2011-10-03--Sebi_o_costi_ny_frisyr/2011-10-04--Sebastian_2år/DSC_5472.JPG

Processing ./2011-10-03--Sebi_o_costi_ny_frisyr/DSC_5440.JPG
Processing ./2011-10-03--Sebi_o_costi_ny_frisyr/DSC_5441.JPG

Processing ./init.py
Processing ./DSC_5440.JPG
Processing ./DSC_5453.JPG
Processing ./prenamer.py

[hynek]

Glad we could help. I suspected it was running under "special circumstances".