matplotlib failing to import when running from Unicode path

[hraesvelgr]

downloaded https://github.com/pyfa-org/Pyfa/releases/tag/v1.28.1

but graph is still greyed out. tried to change the "gui" file in "library zip", but 7zip refused this with an error.

so no graphs for me?

---

@blitzmann says:

WORKAROUND: This is a known issue for the graphing library that we ship with (matplotlib v1.4.3). It's been fixed in a higher version, however we haven't been able to update this library just yet. In the meantime, please ensure that pyfa runs from a non-Unicode directory (ASCII only)

[Ebag333]

You can either use an earlier version, or wait until the next build. There should be one soon (tm).

What OS are you using?

[hraesvelgr]

win 7, 64 bit.

[blitzmann]

tried to change the "gui" file in "library zip"

I'm not sure what this means... gui is a directory that contains our GUI definitions. :P

Can you take a screenshot of Help > About as well as the graph menu being disabled? Additionally, can you open the library.zip/gui/graphFrame.py and post the contents (feel free to extract library.zip to your desktop or something)? I'm grasping at straws here, there really shouldn't be a situation in which graphs are disabled for the 1.28.1 windows release - everything is bundled correctly and should for all intents and purposes work for everyone.

Also check %userprofile%/.pyfa/ for log files (Pyfa*.log) and post them here

[hraesvelgr]

@blitzmann i tried this solution: #1001 (comment)

logfile: https://gist.github.com/hraesvelgr/1566c1e4a74d1dd51afc202512b4695c

content of the graphframe-data:

# =============================================================================
# Copyright (C) 2010 Diego Duclos
#
# This file is part of pyfa.
#
# pyfa is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# pyfa is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with pyfa.  If not, see <http://www.gnu.org/licenses/>.
# =============================================================================

import os
from logbook import Logger

# noinspection PyPackageRequirements
import wx

from service.fit import Fit
import gui.display
import gui.mainFrame
import gui.globalEvents as GE
from gui.graph import Graph
from gui.bitmapLoader import BitmapLoader
import traceback

pyfalog = Logger(__name__)

try:
    import matplotlib as mpl

    mpl_version = int(mpl.__version__[0]) or -1
    if mpl_version >= 2:
        mpl.use('wxagg')
        mplImported = True
    else:
        mplImported = False
    from matplotlib.patches import Patch

    from matplotlib.backends.backend_wxagg import FigureCanvasWxAgg as Canvas
    from matplotlib.figure import Figure

    graphFrame_enabled = True
    mplImported = True
except ImportError as e:
    pyfalog.warning("Matplotlib failed to import.  Likely missing or incompatible version.")
    mpl_version = -1
    Patch = mpl = Canvas = Figure = None
    graphFrame_enabled = False
    mplImported = False
except Exception:
    # We can get exceptions deep within matplotlib. Catch those.  See GH #1046
    tb = traceback.format_exc()
    pyfalog.critical("Exception when importing Matplotlib. Continuing without importing.")
    pyfalog.critical(tb)
    mpl_version = -1
    Patch = mpl = Canvas = Figure = None
    graphFrame_enabled = False
    mplImported = False


class GraphFrame(wx.Frame):
    def __init__(self, parent, style=wx.DEFAULT_FRAME_STYLE | wx.NO_FULL_REPAINT_ON_RESIZE | wx.FRAME_FLOAT_ON_PARENT):
        global graphFrame_enabled
        global mplImported
        global mpl_version

        self.legendFix = False

        if not graphFrame_enabled:
            pyfalog.warning("Matplotlib is not enabled. Skipping initialization.")
            return

        try:
            cache_dir = mpl._get_cachedir()
        except:
            cache_dir = os.path.expanduser(os.path.join("~", ".matplotlib"))

        cache_file = os.path.join(cache_dir, 'fontList.cache')

        if os.access(cache_dir, os.W_OK | os.X_OK) and os.path.isfile(cache_file):
            # remove matplotlib font cache, see #234
            os.remove(cache_file)
        if not mplImported:
            mpl.use('wxagg')

        graphFrame_enabled = True
        if int(mpl.__version__[0]) < 1:
            print("pyfa: Found matplotlib version ", mpl.__version__, " - activating OVER9000 workarounds")
            print("pyfa: Recommended minimum matplotlib version is 1.0.0")
            self.legendFix = True

        mplImported = True

        wx.Frame.__init__(self, parent, title=u"pyfa: Graph Generator", style=style, size=(520, 390))

        i = wx.IconFromBitmap(BitmapLoader.getBitmap("graphs_small", "gui"))
        self.SetIcon(i)
        self.mainFrame = gui.mainFrame.MainFrame.getInstance()
        self.CreateStatusBar()

        self.mainSizer = wx.BoxSizer(wx.VERTICAL)
        self.SetSizer(self.mainSizer)

        sFit = Fit.getInstance()
        fit = sFit.getFit(self.mainFrame.getActiveFit())
        self.fits = [fit] if fit is not None else []
        self.fitList = FitList(self)
        self.fitList.SetMinSize((270, -1))

        self.fitList.fitList.update(self.fits)

        self.graphSelection = wx.Choice(self, wx.ID_ANY, style=0)
        self.mainSizer.Add(self.graphSelection, 0, wx.EXPAND)

        self.figure = Figure(figsize=(4, 3))

        rgbtuple = wx.SystemSettings.GetColour(wx.SYS_COLOUR_BTNFACE).Get()
        clr = [c / 255. for c in rgbtuple]
        self.figure.set_facecolor(clr)
        self.figure.set_edgecolor(clr)

        self.canvas = Canvas(self, -1, self.figure)
        self.canvas.SetBackgroundColour(wx.Colour(*rgbtuple))

        self.subplot = self.figure.add_subplot(111)
        self.subplot.grid(True)

        self.mainSizer.Add(self.canvas, 1, wx.EXPAND)
        self.mainSizer.Add(wx.StaticLine(self, wx.ID_ANY, wx.DefaultPosition, wx.DefaultSize, wx.LI_HORIZONTAL), 0,
                           wx.EXPAND)

        self.gridPanel = wx.Panel(self)
        self.mainSizer.Add(self.gridPanel, 0, wx.EXPAND)

        dummyBox = wx.BoxSizer(wx.VERTICAL)
        self.gridPanel.SetSizer(dummyBox)

        self.gridSizer = wx.FlexGridSizer(0, 4)
        self.gridSizer.AddGrowableCol(1)
        dummyBox.Add(self.gridSizer, 0, wx.EXPAND)

        for view in Graph.views:
            view = view()
            self.graphSelection.Append(view.name, view)

        self.graphSelection.SetSelection(0)
        self.fields = {}
        self.select(0)
        self.sl1 = wx.StaticLine(self, wx.ID_ANY, wx.DefaultPosition, wx.DefaultSize, wx.LI_HORIZONTAL)
        self.mainSizer.Add(self.sl1, 0, wx.EXPAND)
        self.mainSizer.Add(self.fitList, 0, wx.EXPAND)

        self.fitList.fitList.Bind(wx.EVT_LEFT_DCLICK, self.removeItem)
        self.mainFrame.Bind(GE.FIT_CHANGED, self.draw)
        self.Bind(wx.EVT_CLOSE, self.close)

        self.Fit()
        self.SetMinSize(self.GetSize())

    def handleDrag(self, type, fitID):
        if type == "fit":
            self.AppendFitToList(fitID)

    def close(self, event):
        self.fitList.fitList.Unbind(wx.EVT_LEFT_DCLICK, handler=self.removeItem)
        self.mainFrame.Unbind(GE.FIT_CHANGED, handler=self.draw)
        event.Skip()

    def getView(self):
        return self.graphSelection.GetClientData(self.graphSelection.GetSelection())

    def getValues(self):
        values = {}
        for fieldName, field in self.fields.iteritems():
            values[fieldName] = field.GetValue()

        return values

    def select(self, index):
        view = self.getView()
        icons = view.getIcons()
        labels = view.getLabels()
        sizer = self.gridSizer
        self.gridPanel.DestroyChildren()
        self.fields.clear()

        # Setup textboxes
        for field, defaultVal in view.getFields().iteritems():

            textBox = wx.TextCtrl(self.gridPanel, wx.ID_ANY, style=0)
            self.fields[field] = textBox
            textBox.Bind(wx.EVT_TEXT, self.onFieldChanged)
            sizer.Add(textBox, 1, wx.EXPAND | wx.ALIGN_CENTER_VERTICAL | wx.ALL, 3)
            if defaultVal is not None:
                if not isinstance(defaultVal, basestring):
                    defaultVal = ("%f" % defaultVal).rstrip("0")
                    if defaultVal[-1:] == ".":
                        defaultVal += "0"

                textBox.ChangeValue(defaultVal)

            imgLabelSizer = wx.BoxSizer(wx.HORIZONTAL)
            if icons:
                icon = icons.get(field)
                if icon is not None:
                    static = wx.StaticBitmap(self.gridPanel)
                    static.SetBitmap(icon)
                    imgLabelSizer.Add(static, 0, wx.ALL | wx.ALIGN_CENTER_VERTICAL, 1)

            if labels:
                label = labels.get(field)
                label = label if label is not None else field
            else:
                label = field

            imgLabelSizer.Add(wx.StaticText(self.gridPanel, wx.ID_ANY, label), 0,
                              wx.LEFT | wx.RIGHT | wx.ALIGN_CENTER_VERTICAL, 3)
            sizer.Add(imgLabelSizer, 0, wx.ALIGN_CENTER_VERTICAL)
        self.draw()

    def draw(self, event=None):
        global mpl_version

        values = self.getValues()
        view = self.getView()
        self.subplot.clear()
        self.subplot.grid(True)
        legend = []

        for fit in self.fits:
            try:
                success, status = view.getPoints(fit, values)
                if not success:
                    # TODO: Add a pwetty statys bar to report errors with
                    self.SetStatusText(status)
                    return

                x, y = success, status

                self.subplot.plot(x, y)
                legend.append(fit.name)
            except:
                pyfalog.warning("Invalid values in '{0}'", fit.name)
                self.SetStatusText("Invalid values in '%s'" % fit.name)
                self.canvas.draw()
                return

        if mpl_version < 2:
            if self.legendFix and len(legend) > 0:
                leg = self.subplot.legend(tuple(legend), "upper right", shadow=False)
                for t in leg.get_texts():
                    t.set_fontsize('small')

                for l in leg.get_lines():
                    l.set_linewidth(1)

            elif not self.legendFix and len(legend) > 0:
                leg = self.subplot.legend(tuple(legend), "upper right", shadow=False, frameon=False)
                for t in leg.get_texts():
                    t.set_fontsize('small')

                for l in leg.get_lines():
                    l.set_linewidth(1)
        elif mpl_version >= 2:
            legend2 = []
            legend_colors = {
                0: "blue",
                1: "orange",
                2: "green",
                3: "red",
                4: "purple",
                5: "brown",
                6: "pink",
                7: "grey",
            }

            for i, i_name in enumerate(legend):
                try:
                    selected_color = legend_colors[i]
                except:
                    selected_color = None
                legend2.append(Patch(color=selected_color, label=i_name), )

            if len(legend2) > 0:
                leg = self.subplot.legend(handles=legend2)
                for t in leg.get_texts():
                    t.set_fontsize('small')

                for l in leg.get_lines():
                    l.set_linewidth(1)

        self.canvas.draw()
        self.SetStatusText("")
        if event is not None:
            event.Skip()

    def onFieldChanged(self, event):
        self.draw()

    def AppendFitToList(self, fitID):
        sFit = Fit.getInstance()
        fit = sFit.getFit(fitID)
        if fit not in self.fits:
            self.fits.append(fit)

        self.fitList.fitList.update(self.fits)
        self.draw()

    def removeItem(self, event):
        row, _ = self.fitList.fitList.HitTest(event.Position)
        if row != -1:
            del self.fits[row]
            self.fitList.fitList.update(self.fits)
            self.draw()


class FitList(wx.Panel):
    def __init__(self, parent):
        wx.Panel.__init__(self, parent)
        self.mainSizer = wx.BoxSizer(wx.VERTICAL)
        self.SetSizer(self.mainSizer)

        self.fitList = FitDisplay(self)
        self.mainSizer.Add(self.fitList, 1, wx.EXPAND)
        fitToolTip = wx.ToolTip("Drag a fit into this list to graph it")
        self.fitList.SetToolTip(fitToolTip)


class FitDisplay(gui.display.Display):
    DEFAULT_COLS = ["Base Icon",
                    "Base Name"]

    def __init__(self, parent):
        gui.display.Display.__init__(self, parent)

[hraesvelgr]

logfile ist too big to post (character-limit)

[Ebag333]

Use gist (link up at the top

[hraesvelgr]

done.

[Ebag333]

You have to share the link with us. We won't know what it is. :)

[hraesvelgr]

https://gist.github.com/hraesvelgr/1566c1e4a74d1dd51afc202512b4695c

Lunk already has been shared with all the other data you needed :)
#1075 (comment)

[blitzmann]

[2017-03-28 06:35:45.830000] CRITICAL: gui.graphFrame: Traceback (most recent call last):
  File "C:\Users\André\Documents\eve trade tools\pyfa\library.zip\gui\graphFrame.py", line 37, in <module>
    import matplotlib as mpl
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 1100, in <module>
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 947, in rc_params
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 789, in matplotlib_fname
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 325, in wrapper
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 693, in _get_data_path_cached
  File "C:\python-2.7.10\lib\site-packages\matplotlib\__init__.py", line 661, in _get_data_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 13: ordinal not in range(128)

Here's the error. It's due to having unicode characters in the path (André). I don't think we're providing matplotlib with any of our paths, so this may very well be an issue with matplotlib internally... I'll verify tonight and try to work on a fix. :)

[blitzmann]

@Ebag333 excellent work btw on the logging stuff. Wouldn't have this error without it. Kudos :D

[Ebag333]

Here's the error. It's due to having unicode characters in the path (André). I don't think we're providing matplotlib with any of our paths, so this may very well be an issue with matplotlib internally... I'll verify tonight and try to work on a fix. :)

I'm actually working on the locale tests currently. They're all sorts of messed up because of the revert. I suspect that it has to do with the unicode chars, but some stuff in git thinks they exist and some doesn't. Super frustrating.

Since you haven't submitted a PR on the unicode fix I'll probably tackle that next (unless you have an active branch working on it?)

[blitzmann]

@hraesvelgr also, as a work around, don't use pyfa from a directory with unicode characters. If you install pyfa with the .exe, it should install to Program Files, which should work around the issue and give you support for graphs.

For future reference for researching this issue, quick google results:

http://stackoverflow.com/questions/30095006/python-unicode-decode-error-when-importing-matplotlib
matplotlib/matplotlib#3618
matplotlib/matplotlib#3487

[blitzmann]

@Ebag333

Locale tests for pyfa won't really help for this issue if it's an internal matplotlib, which I suspect it may be. This is happening during the import of matplotlib, which tells me that it's something that are doing. Again, will have more info once I look into it.

[Ebag333]

Um, no, they wouldn't.

path = os.sep.join([os.path.dirname(__file__), 'mpl-data'])

They don't set __file__ to be unicode, so that's why it breaks.

Quick search doesn't turn up an issue:
https://github.com/matplotlib/matplotlib/issues

[hraesvelgr]

So... The issue is because of my name (i.e. the name of the directory) and a problem with ascii-table?

Ok, and if i simply change the name of my directory with "simple" letters?

[blitzmann]

python2, which pyfa is written on, has shit support for unicode characters (rather, the applications have shit support if unicode is not kept in mind while developing). matplotlib has these same limitations - if their development team doesn't keep in mind unicode paths, then it's easy to run into problem, as seems to be the case here

Ok, and if i simply change the name of my directory with "simple" letters?

I strongly advise against changing the actual directory name. There are a lot of things on your system that utilize this and will break if you just arbitrarily change it. :)

Here are your options until this can be verified and fixed (or determined unfixable for some reason):

• Use the pyfa installer and install to the Program Files (x86) directory (which is the default path). I'm not sure if different locales have a different name for this directory
• Run pyfa from somewhere else on the C:\ drive if you have permissions to do so. So, for example, C:\pyfa\.
• Run pyfa off another drive (hard drive, flash drive, what have you), so that the path is simply drive:\pyfa\

I'll keep progress posted in this thread as to what I find out. Thanks :)

[hraesvelgr]

Much appreciated!

[blitzmann]

Can confirm this is a matplotlib issue. The issue has been fixed (and I've tested personally that it works) in later versions of matplotlib, v1.5.0+. Unfortunately pyfa ships with v1.4.3.

matplotlib/matplotlib@3ba4f5a
matplotlib/matplotlib@7c12fa0

Updating matplotlib is on the list of things to do when we roll out new binaries, so this should be solved at that time. Or I can look into releasing new binaries just for Windows (much easier than OS X).

Until any of that happens, though, if you're interested in using graphing support, please ensure that pyfa doesn't run from a directory which contains Unicode characters.

[blitzmann]

I'm going to assume this issue is not longer a thing with the new 2.x versions for the following reasons:

• Python 3 support, which deals with unicode paths better
• Updated to matplotlib 2.x, the original issue was that the version were were running didn't have the fix in it, and later versions did.

Going to close this issue as solved, if something else creeps up, please open a new issue :)

[ine16]

On windows 7 32 bit and matplotlib 2.2.2 the problem still persists

[blitzmann]

@ine16 can you provide some more details? Are you sure it's due to locale issue, or possibly another reason why graph option might be grayed out?