libfreetype not found if installed at uncommon path

[toobaz]

For various reasons (inability to install software as admin), my libfreetype install resides at an unusual path. This was not a problem while building matplotlib, since the command

PKG_CONFIG_PATH="/home/pietro/my/unusual/path/lib/pkgconfig" python3 setup.py install

worked perfectly. But now instead any

from matplotlib import pyplot

results in

In [2]: from matplotlib import pyplot
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-a371ea114746> in <module>()
----> 1 from matplotlib import pyplot

/home/pietroba/software/python/.localpython/lib/python3.4/site-packages/matplotlib-1.5.dev1-py3.4-linux-x86_64.egg/matplotlib/pyplot.py in <module>()
     25 
     26 import matplotlib
---> 27 import matplotlib.colorbar
     28 from matplotlib import style
     29 from matplotlib import _pylab_helpers, interactive

/home/pietroba/software/python/.localpython/lib/python3.4/site-packages/matplotlib-1.5.dev1-py3.4-linux-x86_64.egg/matplotlib/colorbar.py in <module>()
     32 import matplotlib.artist as martist
     33 import matplotlib.cbook as cbook
---> 34 import matplotlib.collections as collections
     35 import matplotlib.colors as colors
     36 import matplotlib.contour as contour

/home/pietroba/software/python/.localpython/lib/python3.4/site-packages/matplotlib-1.5.dev1-py3.4-linux-x86_64.egg/matplotlib/collections.py in <module>()
     25 import matplotlib.artist as artist
     26 from matplotlib.artist import allow_rasterization
---> 27 import matplotlib.backend_bases as backend_bases
     28 import matplotlib.path as mpath
     29 from matplotlib import _path

/home/pietroba/software/python/.localpython/lib/python3.4/site-packages/matplotlib-1.5.dev1-py3.4-linux-x86_64.egg/matplotlib/backend_bases.py in <module>()
     54 
     55 import matplotlib.tight_bbox as tight_bbox
---> 56 import matplotlib.textpath as textpath
     57 from matplotlib.path import Path
     58 from matplotlib.cbook import mplDeprecation

/home/pietroba/software/python/.localpython/lib/python3.4/site-packages/matplotlib-1.5.dev1-py3.4-linux-x86_64.egg/matplotlib/textpath.py in <module>()
     13 from matplotlib.path import Path
     14 from matplotlib import rcParams
---> 15 import matplotlib.font_manager as font_manager
     16 from matplotlib.ft2font import FT2Font, KERNING_DEFAULT, LOAD_NO_HINTING
     17 from matplotlib.ft2font import LOAD_TARGET_LIGHT

/home/pietroba/software/python/.localpython/lib/python3.4/site-packages/matplotlib-1.5.dev1-py3.4-linux-x86_64.egg/matplotlib/font_manager.py in <module>()
     56 import matplotlib
     57 from matplotlib import afm
---> 58 from matplotlib import ft2font
     59 from matplotlib import rcParams, get_cachedir
     60 from matplotlib.cbook import is_string_like

ImportError: libfreetype.so.6: cannot open shared object file: No such file or directory

matplotlib's setup.py did correctly find my pkgconfig catalog "freetype2.pc", and such catalog correctly specifies "libdir=/home/pietro/my/unusual/path/lib", but this libdir doesn't seem to be used anywhere. If I set

export LD_LIBRARY_PATH=/home/pietro/my/unusual/path/lib

before invoking python, everything works fine.

[tacaswell]

Is there something we can do during the build process to fix this or should this just be documented?

[toobaz]

Good question... I'm not an expert at all, if I had to solve this problem in a pure Python library I would

• make FreeType (in setupext.py) use pkg-config to get (and store somewhere) also "libdir"
• load the .so as shown here

... but instead the code using the shared library is C++, so I have no idea...

[WeatherGod]

The other problem with that approach is that it breaks the "dynamic" part
of "dynamic library loading".

IIRC, Python does not respect LD_LIBRARY_PATH at all for finding .so files
(or at least, the behavior is different if you are trying to find a library
versus load a library). Although, LD_LIBRARY_PATH is used by the OS to load
up any other objects that are dynamically linked to the object being loaded
by Python: https://bugs.python.org/issue2936

On Mon, Mar 30, 2015 at 4:07 AM, Pietro Battiston notifications@github.com
wrote:

Good question... I'm not an expert at all, if I had to solve this problem
in a pure Python library I would

• make FreeType (in setupext.py) use pkg-config to get (and store
  somewhere) also "libdir"
• load the .so as shown here
  http://stackoverflow.com/a/21665823/2858145

... but instead the code using the shared library is C++, so I have no
idea...

—
Reply to this email directly or view it on GitHub
#4296 (comment)
.

[mdboom]

This seems like standard UNIX behavior -- loading dynamic modules from an unusual location requires the setting of LD_LIBRARY_PATH.

@WeatherGod: Python extensions are loaded using the standard import mechanism (the same as for regular Python modules). Any C shared libraries that those extensions used are loaded using the standard OS conventions, and LD_LIBRARY_PATH is respected.

[toobaz]

Il giorno lun, 30/03/2015 alle 07.00 -0700, Benjamin Root ha scritto:

The other problem with that approach is that it breaks the "dynamic"
part
of "dynamic library loading".

I would say that if any, the opposite is true! Allowing matplotlib to
be loaded from a custom path even though at execution time cpython was
unaware of anything related to it sounds to me the most "dynamic"
loading possible.

By the way, this (or the current mechanism) could be used as fallback.

But that said, I can understand if it's too much work for a feature
with limited use. I would still document it because using pkg-config at
setup.py time but not honouring the paths it provides is unexpected.

Pietro

[mdboom]

Installing libraries in non-standard location on a UNIX system comes with a certain set of standard requirements, and one of those things is setting LD_LIBRARY_PATH. I don't think there's much matplotlib can do here, and I don't know if filling the installation docs with non-standard use cases (which are nonetheless solved by the standard UNIX conventions) is helpful in the general case.

[WeatherGod]

except Unix-y systems are working on moving away from LD_LIBRARY_PATH.

http://xahlee.info/UnixResource_dir/_/ldpath.html

I now avoid LD_LIBRARY_PATH like the plague. I have been bitten too many
times.

On Wed, Apr 1, 2015 at 7:50 AM, Michael Droettboom <notifications@github.com

wrote:

Installing libraries in non-standard location on a UNIX system comes with
a certain set of standard requirements, and one of those things is setting
LD_LIBRARY_PATH. I don't think there's much matplotlib can do here, and I
don't know if filling the installation docs with non-standard use cases
(which are nonetheless solved by the standard UNIX conventions) is helpful
in the general case.

—
Reply to this email directly or view it on GitHub
#4296 (comment)
.

[toobaz]

@mdboom: matplotlib's setup.py uses pkg-config, which looks to me like a pretty standard - and sufficient for avoiding further //manual// interventions - requirement for finding libraries in UNIX systems...

(and by the way, python's behavior with libraries is clearly even better, so we tend to be spoiled... but that's another story)

[mdboom]

I'm not sure what you're asking for at this point. You want a way to not have to specify LD_LIBRARY_PATH, even though that is the standard way to find libraries that are in non-standard locations?

To clarify, pkg-config is for build-time configuration, and what you're talking about is run-time --- when the dynamic loading actually happens. That is generally handled by LD_LIBRARY_PATH.

I'm open to suggestions, but I have spent a lot of time looking into this problem in the past, and there aren't really any good solutions. One could argue that this is the reason things like anaconda and ureka exist, to kind of workaround these shortcomings with dynamic linking by doing everything in a sandbox. But if you can point to such a better solution, I'm happy to have a look. The best solution I can think of would be to add an option to statically link against freetype, libpng etc. which would solve this problem.

[toobaz]

As I said, I do not have the skills to solve such a problem. But your last message suggests I was even conceptually not clear.
So: I think matplotlib would ideally

1. try to load freetype as it is currently doing,
2. if this fails, load it (dynamically, of course!) from the path provided by pkg-config at install time.

As far as I understand, this is both the most useful and the "most UNIX" approach, and it would be feasible in Python: if it is not feasible in C++ (or requires too much work)... sorry for the noise.

(an option to statically link libraries would certainly solve my specific case... but then I do wonder whether it would be worth the effort)

[mdboom]

The issue is that we don't do (2). The OS' dynamic linker does.

[tacaswell]

Closing this issue as there is nothing that we can do, this is an issue for the system linker. On linux you can use the ldd command to check what shared libraries are being linked to, ex:

✔ ~/other_source/matplotlib/build/lib.linux-x86_64-3.4/matplotlib [repl_hook|⚑ 1] 
16:46 $ ldd ft2font.cpython-34m.so 
        linux-vdso.so.1 (0x00007ffff5f96000)
        libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007fa4523d9000)
        libpython3.4m.so.1.0 => /usr/lib/libpython3.4m.so.1.0 (0x00007fa451f45000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fa451c36000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007fa451930000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fa45171a000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fa4514fd000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007fa451159000)
        libz.so.1 => /usr/lib/libz.so.1 (0x00007fa450f43000)
        libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007fa450d33000)
        libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007fa450afc000)
        libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0x00007fa45089f000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fa45069b000)
        libutil.so.1 => /usr/lib/libutil.so.1 (0x00007fa450497000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007fa4528e0000)
        libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007fa450189000)
        libgraphite2.so.3 => /usr/lib/libgraphite2.so.3 (0x00007fa44ff6a000)
        libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007fa44fcfb000)