import error when non-ascii characters are present in cwd or user name (windows) #3516

mmagnuski · 2014-09-14T19:02:26Z

Ahoy!
I have recently updated matplotlib through conda to version 14.0 (np19py27_0).
I am using windows 8, python 2.7.7 (Anaconda distribution 2.0.2 32-bit)
Unfortunatelly my windows user name contains non-ascii caracter (ś) and importing matplotlib gives errors. 😢 (previous matplotlib versions I used did not have any problem with this)

The error comes in two flavors:
⚫ If my current working directory contains non-ascii character then:

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-1-82be63b7783c> in <module>()
----> 1 import matplotlib

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in <module>()
   1046 
   1047 # this is the instance used by the matplotlib classes
-> 1048 rcParams = rc_params()
   1049 
   1050 if rcParams['examples.directory']:

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in rc_params(fail_on_error)
    895     default matplotlib rc file.
    896     """
--> 897     fname = matplotlib_fname()
    898     if not os.path.exists(fname):
    899         # this should never happen, default in mpl-data should always be found

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in matplotlib_fname()
    746       system-defined copy.
    747     """
--> 748     fname = os.path.join(os.getcwd(), 'matplotlibrc')
    749     if os.path.exists(fname):
    750         return fname

D:\Python\Anaconda\lib\ntpath.pyc in join(path, *paths)
     82         if result_path and result_path[-1] not in '\\/':
     83             result_path = result_path + '\\'
---> 84         result_path = result_path + p_path
     85     return result_drive + result_path
     86 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x9c in position 11: ordinal not in range(128)

⚫ If I change my current directory so that it does not contain any ascii characters the error transforms itself a little:

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-6-1d1446f6fa64> in <module>()
----> 1 from matplotlib import pyplot as plt

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in <module>()
   1046 
   1047 # this is the instance used by the matplotlib classes
-> 1048 rcParams = rc_params()
   1049 
   1050 if rcParams['examples.directory']:

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in rc_params(fail_on_error)
    895     default matplotlib rc file.
    896     """
--> 897     fname = matplotlib_fname()
    898     if not os.path.exists(fname):
    899         # this should never happen, default in mpl-data should always be found

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in matplotlib_fname()
    757                 return fname
    758 
--> 759     configdir = _get_configdir()
    760     if configdir is not None:
    761         fname = os.path.join(configdir, 'matplotlibrc')

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in _get_configdir()
    628     5. A writable directory could not be found or created; return None.
    629     """
--> 630     return _get_config_or_cache_dir(_get_xdg_config_dir())
    631 
    632 get_configdir = verbose.wrap('CONFIGDIR=%s', _get_configdir, always=False)

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in _get_xdg_config_dir()
    553     path = os.environ.get('XDG_CONFIG_HOME')
    554     if path is None:
--> 555         path = get_home()
    556         if path is not None:
    557             path = os.path.join(path, '.config')

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in wrapper(*args, **kwargs)
    321         assert six.callable(func)
    322         def wrapper(*args, **kwargs):
--> 323             ret = func(*args, **kwargs)
    324 
    325             if (always or not wrapper._spoke):

D:\Python\Anaconda\lib\site-packages\matplotlib\__init__.py in _get_home()
    507     """
    508     try:
--> 509         path = os.path.expanduser("~")
    510     except ImportError:
    511         # This happens on Google App Engine (pwd module is not present).

D:\Python\Anaconda\lib\ntpath.pyc in expanduser(path)
    271         userhome = join(dirname(userhome), path[1:i])
    272 
--> 273     return userhome + path[i:]
    274 
    275 

UnicodeDecodeError: 'ascii' codec can't decode byte 0x9c in position 13: ordinal not in range(128)

The text was updated successfully, but these errors were encountered:

tacaswell · 2014-09-14T20:25:47Z

Partial duplicate of #3487.

mdboom · 2014-09-16T17:48:06Z

I think this is a Python bug that is reportedly is unfixable on Python 2. See this:

http://bugs.python.org/issue13207

It looks difficult to workaround, since even if we do the home directory lookup ourselves, you can't get a unicode environment variable on Windows on Python 2. There is a proposed ctypes solution, but ick.

One possible workaround is to define the MPLCONFIGDIR environment variable to an ascii-clean path, and the configuration file for matplotlib will be stored there rather than your home directory.

cimarronm · 2014-09-16T18:52:47Z

Yeah, tracebacks point to the problem in the python 2 library itself.

Did previous version not check for anything in the user home path? @mmagnuski, which previous version of matplotlib worked for you in the past and were you using python2?

mmagnuski · 2014-09-17T22:28:35Z

I changed to matplotlib 1.3.1 and importing works fine. I don't remember which version I used originally but most probably something around 1.3
But my non-unicode home path is now a problem for pip, I regret using a polish sign in my username, eh...

tacaswell · 2014-09-18T00:39:21Z

If the issue is the underlying python bug, why does 1.3.1 work?

Where did you install mpl on your system?

efiring · 2014-09-18T00:43:30Z

@tacaswell, maybe we see the problem now because we are using unicode_literals, so we are injecting unicode where previously we were injecting native strings? In my own code porting, I concluded that using unicode_literals causes more problems than it solves. In the case of this issue, though, I am just guessing.

tacaswell · 2014-09-18T00:45:27Z

I tracked this back via blame and it looks like what happened is that in ca6cd19 the except was changed to only catch ImportErrors.

Older version of mpl are just silently snarfing the error.

closes matplotlib#3516 by ignoring the real problem. The real problem seems to be in the core libary ( http://bugs.python.org/issue13207) to which the response was 'upgrade to python 3'.

mmagnuski · 2014-09-18T10:31:28Z

If this is relevant - I do have a .matplotlib directory present in my C:\Users\Mikuś\ path with matplotlib 1.3.1. The path has non-ascii signs but matplotlib was somehow able to create a dir there.

tacaswell · 2014-09-18T12:17:25Z

Can you see if this the patch in #3532 works?

The next block of code after this one tries to get the home directory from the envs for evar in ('HOME', 'USERPROFILE', 'TMP'): which might handle the unicode paths correctly.

wernerfb · 2014-09-29T09:05:03Z

A user of my app just run into the same problem with 1.4, downgraded my app to mpl 1.3 which does not show this problem.

tacaswell · 2014-09-29T12:55:53Z

@wernerfb Can you check if the patch in #3532 fixes it for your user?

wernerfb · 2014-09-29T15:21:59Z

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
#3516 (comment).

Hi Thomas,

I can test it here, but only on my test machine, so need to py2exe app -
which takes a bit.

Unfortunately this doesn't help, I now get the exception:
twcbsrc.app_base - ERROR - app_base.pyo - 292 - Traceback (most recent
call last):
File "twcbsrc\controllers\app_cb.pyo", line 403, in onTBStats
File "importlib__init__.pyo", line 37, in import_module
File "zipextimporter.pyo", line 82, in load_module
File "twcbsrc\controllers\app_stats.pyo", line 29, in
File "zipextimporter.pyo", line 82, in load_module
File "twcbsrc\views\statistics.pyo", line 19, in
File "zipextimporter.pyo", line 82, in load_module
File "matplotlib__init__.pyo", line 1051, in
File "matplotlib__init__.pyo", line 900, in rc_params
File "matplotlib__init__.pyo", line 762, in matplotlib_fname
File "matplotlib__init__.pyo", line 633, in get_configdir
File "matplotlib__init_.pyo", line 558, in get_xdg_config_dir
File "matplotlib__init_.pyo", line 326, in wrapper
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 32:
invalid start byte

What is strange is that this worked with 1.3 and if I do
'os.path.expanduser('~')' in the py2exe'd application using a test
Python prompt I don't get an exception either and the path shown in the
prompt is "C:\Dokumente und Einstellungen\j\xf6rg f\xfcr tests"

Could it have to do with "from future import unicode_literals" ?

Werner

wernerfb · 2014-09-29T15:24:45Z

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
#3516 (comment).

Hhm, just did a little test on the Python prompt within the py2exe'd
app and yes it is the future import.

from future import unicode_literals
os.path.expanduser('~')

wernerfb · 2014-09-29T15:46:41Z

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
#3516 (comment).

looking at this thread:
https://stackoverflow.com/questions/23888120/an-alternative-to-os-path-expanduser

I tried on the test Python prompt without getting an exception and the
returned value looks correct.

from future import unicode_literals
os.path.expanduser(b'~').decode(sys.getfilesystemencoding())

wernerfb · 2014-09-29T16:03:16Z

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
#3516 (comment).

Making the following change works for me on the dev machine running
Windows 8.1, Python 2.7.8, wxPython 2.9.5 and running the py2exe'd app
on Windows XP.

```
   path = os.path.expanduser("~")
```

   path = os.path.expanduser(b"~").decode(sys.getfilesystemencoding())

tacaswell · 2014-09-29T17:20:49Z

Could you put that in a PR?

On Mon, Sep 29, 2014 at 12:03 PM, Werner F Bruhin notifications@github.com
wrote:

On 9/29/2014 14:55, Thomas A Caswell wrote:

@wernerfb https://github.com/wernerfb Can you check if the patch in
#3532 #3532 fixes it
for your user?

—
Reply to this email directly or view it on GitHub
<
#3516 (comment)
.

Making the following change works for me on the dev machine running
Windows 8.1, Python 2.7.8, wxPython 2.9.5 and running the py2exe'd app
on Windows XP.

path = os.path.expanduser("~")

path = os.path.expanduser(b"~").decode(sys.getfilesystemencoding())

—
Reply to this email directly or view it on GitHub
#3516 (comment)
.

Thomas Caswell
tcaswell@gmail.com

tacaswell · 2014-09-30T16:47:15Z

closed by #3594

tacaswell modified the milestones: v1.4.x, v1.4.1 Sep 14, 2014

tacaswell mentioned this issue Sep 17, 2014

Issue with non-ascii paths in font look up #3517

Closed

tacaswell added topic: text/unicode status: upstream fix required and removed status: upstream fix required labels Sep 17, 2014

tacaswell mentioned this issue Sep 18, 2014

BUG : skip looking up expanduser if unicode error #3532

Closed

tacaswell added the has_patch label Sep 18, 2014

tacaswell closed this as completed Sep 30, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

import error when non-ascii characters are present in cwd or user name (windows) #3516

import error when non-ascii characters are present in cwd or user name (windows) #3516

mmagnuski commented Sep 14, 2014

tacaswell commented Sep 14, 2014

mdboom commented Sep 16, 2014

cimarronm commented Sep 16, 2014

mmagnuski commented Sep 17, 2014

tacaswell commented Sep 18, 2014

efiring commented Sep 18, 2014

tacaswell commented Sep 18, 2014

mmagnuski commented Sep 18, 2014

tacaswell commented Sep 18, 2014

wernerfb commented Sep 29, 2014

tacaswell commented Sep 29, 2014

wernerfb commented Sep 29, 2014

wernerfb commented Sep 29, 2014

wernerfb commented Sep 29, 2014

wernerfb commented Sep 29, 2014

tacaswell commented Sep 29, 2014

tacaswell commented Sep 30, 2014

import error when non-ascii characters are present in cwd or user name (windows) #3516

import error when non-ascii characters are present in cwd or user name (windows) #3516

Comments

mmagnuski commented Sep 14, 2014

tacaswell commented Sep 14, 2014

mdboom commented Sep 16, 2014

cimarronm commented Sep 16, 2014

mmagnuski commented Sep 17, 2014

tacaswell commented Sep 18, 2014

efiring commented Sep 18, 2014

tacaswell commented Sep 18, 2014

mmagnuski commented Sep 18, 2014

tacaswell commented Sep 18, 2014

wernerfb commented Sep 29, 2014

tacaswell commented Sep 29, 2014

wernerfb commented Sep 29, 2014

wernerfb commented Sep 29, 2014

wernerfb commented Sep 29, 2014

wernerfb commented Sep 29, 2014

tacaswell commented Sep 29, 2014

tacaswell commented Sep 30, 2014