Regression: import fails (MacOS X 10.10.1, Python 3.4.2, matplotlib ca10a34e06645c35a98e9ed3db35f6a8efee70f2 )

[HubertHolin]

Using MacOS X 10.10.1, Python 3.4.2, matplotlib ca10a34 and the simple input "import matplotlib" results in failure with the following message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py", line 1100, in <module>
    rcParams = rc_params()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py", line 956, in rc_params
    return rc_params_from_file(fname, fail_on_error)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py", line 1069, in rc_params_from_file
    config_from_file = _rc_params_in_file(fname, fail_on_error)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py", line 995, in _rc_params_in_file
    with _open_file_or_url(fname) as fd:
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py", line 980, in _open_file_or_url
    with io.open(fname, encoding=locale.getdefaultlocale()[1]) as f:
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py", line 556, in getdefaultlocale
    return _parse_localename(localename)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py", line 484, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: UTF-8

The import works without problem, however, using MacOS X 10.10.1, Python 3.4.2, matplotlib-1.4.2 .

Hubert Holin

[mdboom]

It looks like locale.getdefaultlocale() is failing on your system, which is usually the result of a configuration problem on the system. Can you do the following at a Python prompt (in the same Python executable you are using above):

>>> import locale
>>> locale.getdefaultlocale()

[HubertHolin]

Yes locale.getdefaultlocale() is failing on my system (as shown below). I'll try and see how to fix that (any suggestion welcome!). However, this prevents me (until I fix the locale problem) from using (actually, trying to use… I have another blocking issue that I will report separately) the latest commit, but not the 1.4.2 release, all things being equal beyond the version of matplotlib used. Presumably therefore locale.getdefaultlocale() either is not called in the earlier release or its failure was dealt with transparently there.

Python 3.4.2 (default, Nov 23 2014, 00:54:57) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.getdefaultlocale()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py", line 556, in getdefaultlocale
    return _parse_localename(localename)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py", line 484, in _parse_localename
    raise ValueError('unknown locale: %s' % localename)
ValueError: unknown locale: UTF-8
>>> 

[tacaswell]

The change that introduced this was probably 4374d8e in #3575. Unfortunately, the github foo has a typo and it isn't clear what the original bug number was.

[tacaswell]

That said, I think it is fair to assume that core library functions are working correctly.

Unless we are using locale.getdefaultlocale() wrong, this is not a mpl bug.

[HubertHolin]

Fair enough. There definitely seems to be a locale problem with my setup. Debugging will not be fun (different behaviours under bash and under IDLE…).

[tacaswell]

I am going to close this for now, please ping to have this re-opened if we are doing something wrong and it is not a configuration issue on your end.

[HubertHolin]

In light of our exchange, it is clear that what I see is not a matplolib bug (so I guess this issue should not be re-opened).

However, it appears passing environment variables has become quite problematic under MacOS X 10.10 (this is what I glean from trying to find answers on the net; the following gives an idea of the scope of the problem: http://stackoverflow.com/questions/25385934/setting-environment-variables-via-launchd-conf-no-longer-works-in-os-x-yosemite). At any rate, even though LANGUAGE is set under bash, it is obviously not passed on to Python, which leads to the problem originally described.

Could it be possible then to implement a workaround, i.e. if the locale is not fount (or is C or UTF-8… depending on the precise circumstances of the query…), then fall back to the previous matplotlib (1.4.2) behaviour?

[tacaswell]

Am I reading that SO question right that Apple decided to break the OS?

It looks like you are using the system python? Does this problem exist if you use one of the other python distributions (anaconda, canopy, homebrew)?

Reading a bit more of the commit that added this change the issue it was closing is #1713 which is a rather major feature.

I don't fully understand the exception you are getting as my system happily returns 'UTF-8' as the encoding part of the locale:

(py3) [tcaswell@arya ~]$ ipython
Python 3.4.2 (default, Oct  8 2014, 13:44:52) 
Type "copyright", "credits" or "license" for more information.                                                                                                                                                                        

IPython 3.0.0-dev -- An enhanced Interactive Python.                                                                                                                                                                                  
?         -> Introduction and overview of IPython's features.                                                                                                                                                                         
%quickref -> Quick reference.                                                                                                                                                                                                         
help      -> Python's own help system.                                                                                                                                                                                                
object?   -> Details about 'object', use 'object??' for extra details.                                                                                                                                                                

In [1]: import locale                                                                                                                                                                                                                 

In [2]: locale.getdefaultlocale()
Out[2]: ('en_US', 'UTF-8')                                                                                                                                                                         

Can you please tell us what envs you think are set? It is really hard to sort out what is going on here if no one else can reproduce it.

[HubertHolin]

Sorry for the rambling bellow and the inconvenience, but typing it led me to find the problem! The env which was the root of the difficulties was LC_CTYPE=UTF-8. I do not known were it was set, and I had not set it explicitly. Setting explicitly LC_CTYPE=en_us in .bash_profile solved the problem. I believe it would help if this were documented in the newt release.

Now on to the next problem (later!)…

==== text below superseded by the above ====

I have the same reading of that SO question…

I am not using the system python. I use a compiled-from-(unmodified)-sources python, framework build.

I have the following env:

hubert-holins-mac-pro:~ hubertholin$ env
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/bash
TERM=xterm-256color
TMPDIR=/var/folders/l5/c3kp63k510b0h9qwnpqtj7mr0000gn/T/
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.DaBDuBfah1/Render
TERM_PROGRAM_VERSION=343
TERM_SESSION_ID=14B139E0-CBB1-461D-8B62-EC09476FB9F6
SGML_CATALOG_FILES=/usr/share/xml/docbook/xsl-ns-1.75/catalog
USER=hubertholin
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.xM52rkQjv5/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
PATH=/usr/share/xml/docbook/xsl-ns-1.75/tools/bin:/opt/gcc:/usr/local/pgsql/bin:/opt/subversion/bin:/Library/Frameworks/Python.framework/Versions/Current/bin:/Library/TeX/Root/bin/powerpc-apple-darwin-current:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin
XML_CATALOG_FILES=/usr/share/xml/docbook/xsl-ns-1.75/catalog.xml
PWD=/Users/hubertholin
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=1
HOME=/Users/hubertholin
LANGUAGE=en_us
LOGNAME=hubertholin
CVS_RSH=ssh
CLASSPATH=/Users/hubertholin/.resolver
LC_CTYPE=UTF-8
DISPLAY=/private/tmp/com.apple.launchd.dYbofvc2Yu/org.macosforge.xquartz:0
PYTHON=/Library/Frameworks/Python.framework/Versions/Current/bin/python
_=/usr/bin/env

and replicating your code I get the following:

hubert-holins-mac-pro:~ hubertholin$ ipython3
Python 3.4.2 (default, Nov 23 2014, 00:54:57) 
Type "copyright", "credits" or "license" for more information.

IPython 3.0.0-dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import locale

In [2]: locale.getlocale()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-917b878b39ad> in <module>()
----> 1 locale.getlocale()

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py in getlocale(category)
    573     if category == LC_ALL and ';' in localename:
    574         raise TypeError('category LC_ALL is not supported')
--> 575     return _parse_localename(localename)
    576 
    577 def setlocale(category, locale=None):

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py in _parse_localename(localename)
    482     elif code == 'C':
    483         return None, None
--> 484     raise ValueError('unknown locale: %s' % localename)
    485 
    486 def _build_localename(localetuple):

ValueError: unknown locale: UTF-8

(same with variants: LANGUAGE=en-us or LANGUAGE=en-us.UTF-8; I may have the syntax wrong…). Could LC_CTYPE=UTF-8 be the culprit? YES!

To add to the confusion, within IDLE I get the following:

Python 3.4.2 (default, Nov 23 2014, 00:54:57) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)] on darwin
Type "copyright", "credits" or "license()" for more information.
>>> import locale
>>> locale.getlocale()
(None, None)
>>> 

hence, it is not possible to debug that way.

Explicitly setting the locale before importing matplotlib fails as well:

hubert-holins-mac-pro:~ hubertholin$ ipython3
Python 3.4.2 (default, Nov 23 2014, 00:54:57) 
Type "copyright", "credits" or "license" for more information.

IPython 3.0.0-dev -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import locale

In [2]: locale.getlocale()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-917b878b39ad> in <module>()
----> 1 locale.getlocale()

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py in getlocale(category)
    573     if category == LC_ALL and ';' in localename:
    574         raise TypeError('category LC_ALL is not supported')
--> 575     return _parse_localename(localename)
    576 
    577 def setlocale(category, locale=None):

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py in _parse_localename(localename)
    482     elif code == 'C':
    483         return None, None
--> 484     raise ValueError('unknown locale: %s' % localename)
    485 
    486 def _build_localename(localetuple):

ValueError: unknown locale: UTF-8

In [3]: locale.setlocale(locale.LC_ALL, 'en_us')
Out[3]: 'en_us'

In [4]: locale.getlocale()
Out[4]: ('en_US', 'ISO8859-1')

In [5]: import matplotlib
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-82be63b7783c> in <module>()
----> 1 import matplotlib

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py in <module>()
   1098 
   1099 # this is the instance used by the matplotlib classes
-> 1100 rcParams = rc_params()
   1101 
   1102 if rcParams['examples.directory']:

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py in rc_params(fail_on_error)
    954         return ret
    955 
--> 956     return rc_params_from_file(fname, fail_on_error)
    957 
    958 

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py in rc_params_from_file(fname, fail_on_error, use_default_template)
   1067         parameters specified in the file. (Useful for updating dicts.)
   1068     """
-> 1069     config_from_file = _rc_params_in_file(fname, fail_on_error)
   1070 
   1071     if not use_default_template:

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py in _rc_params_in_file(fname, fail_on_error)
    993     cnt = 0
    994     rc_temp = {}
--> 995     with _open_file_or_url(fname) as fd:
    996         for line in fd:
    997             cnt += 1

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/contextlib.py in __enter__(self)
     57     def __enter__(self):
     58         try:
---> 59             return next(self.gen)
     60         except StopIteration:
     61             raise RuntimeError("generator didn't yield") from None

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/matplotlib/__init__.py in _open_file_or_url(fname)
    978         f.close()
    979     else:
--> 980         with io.open(fname, encoding=locale.getdefaultlocale()[1]) as f:
    981             yield f
    982 

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py in getdefaultlocale(envvars)
    554     else:
    555         localename = 'C'
--> 556     return _parse_localename(localename)
    557 
    558 

/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/locale.py in _parse_localename(localename)
    482     elif code == 'C':
    483         return None, None
--> 484     raise ValueError('unknown locale: %s' % localename)
    485 
    486 def _build_localename(localetuple):

ValueError: unknown locale: UTF-8

In [6]: 

Is it possible that I may have overlooked a language setting in matplotlibrc? I looked but did not find one.

[KDruzhkin]

I had a similar problem, and found the solution here:
http://blog.remibergsma.com/2012/07/10/setting-locales-correctly-on-mac-osx-terminal-application/

In MacOS Terminal Preferences | Settings | Advanced: uncheck the button "set environment variables on startup".

[HubertHolin]

Thanks for the reminder (I just got bitten again on a new machine)! For completeness, under MacOS X 10.11 ("El Capitan"), the setting is now in Terminal Preferences | Profiles | Advanced.

[mdboom]

Yeah -- Apple is not following what POSIX says about what LANG should be here. Either Python needs to update to support that non-standard value, or Apple needs to follow the standard.

[ghost]

I have the same problem. But it still exists even I do what you said(uncheck the button "set environment variables on startup")
When I import locale; locale.getdefaultlocale() I get (None, None), but when I type "locale" in the terminal I get : LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

Could you please help me to fix this?
Thanks a lot!

[HubertHolin]

You have to restart the terminal after having unchecked that button: this setting is not taken into account until the next session.

As well, once this is done and you want to also print some unicode in the terminal, you should use the following in your script before the first print statement (found in Stack Overflow - my thanks!):

sys.stdout = open(1, 'w', encoding='utf-8', closefd=False)

[HubertHolin]

As an addendum, the above method has (at least) one drawback: it is no longer possible to input the whole range of unicode character at the Terminal's prompt.

It thus appears preferable to do the following:

• in your ".bash_profile" file (see http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html for the MacOS X take on ".bash_profile" vs. ".bashrc") set LC_CTYPE to something matplotlib will like (for instance with LC_CTYPE=en_us)
• check the "Preferences | Settings | Advanced: 'set environment variables on startup'" button
• restart "Terminal"
• in your scripts, use sys.stdout = open(1, 'w', encoding='utf-8', closefd=False) before your first print statement

At least you will be able to input unicode characters in the terminal (even though in this exemple, LC_CTYPE is 'en_us'…)l, print them from your script, and matplotlib will not bomb out (further tests still needed to check all is well…). You may also set LANGUAGE if you wish, but that has no bearing on the current problem.