integrate/port/document find_libpython to find libpython for pyInitLibPath automatically

[timotheecour]

it's surprisingly hard to find libpython in a robust way; this does the trick but relies on an external python package find_libpython:

# ubuntu:
$ pip3 install find_libpython
$ python3 -c 'import find_libpython; print(find_libpython.find_libpython())'
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0

# osx:
$ python3 -c 'import find_libpython; print(find_libpython.find_libpython())'
/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Python

# this also works on osx directly via shell, but somehow not on ubuntu (probably because I installed it inside a miniconda environment):
find_libpython
/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Python

$ file /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Python
/usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Python: Mach-O 64-bit dynamically linked shared library x86_64

links

• the main code is this file: https://github.com/ktbarrett/find_libpython/blob/master/src/find_libpython/__init__.py#L284
• findPythonDLL + friends in nimpy could be updated based on find_libpython

other

• this snippet extracted from it works on OSX but not ubuntu:

import ctypes.util
import os
import sys

class Dl_info(ctypes.Structure):
    _fields_ = [
        ("dli_fname", ctypes.c_char_p),
        ("dli_fbase", ctypes.c_void_p),
        ("dli_sname", ctypes.c_char_p),
        ("dli_saddr", ctypes.c_void_p),
    ]

def _linked_libpython_unix():
    libdl = ctypes.CDLL(ctypes.util.find_library("dl"))
    libdl.dladdr.argtypes = [ctypes.c_void_p, ctypes.POINTER(Dl_info)]
    libdl.dladdr.restype = ctypes.c_int

    dlinfo = Dl_info()
    retcode = libdl.dladdr(
        ctypes.cast(ctypes.pythonapi.Py_GetVersion, ctypes.c_void_p),
        ctypes.pointer(dlinfo))
    if retcode == 0:  # means error
        return None
    path = os.path.realpath(dlinfo.dli_fname.decode())
    if path == os.path.realpath(sys.executable):
        return None
    return path

print(_linked_libpython_unix())

find_library is described here: https://docs.python.org/2.5/lib/ctypes-finding-shared-libraries.html (https://github.com/python/cpython/blob/master/Lib/ctypes/util.py#L72) and would be useful to port/wrap to nim (more generally)

• before finding find_libpython I had also tried parsing otool -L (ldd on linux) directly on OSX but this doesn't seem robust, but at least avoids calling an external python program:

import std/[osproc,strformat,os,strutils]
proc getPythonDll*(path: string): string =
  when defined(osx):
    let result = execCmdEx(fmt"otool -L {path.quoteShell}")
    if result.exitCode != 0: return ""
    let s = result.output.splitLines
    # dbg s
    if s.len < 2: return ""
    let s2 = s[1].split(maxsplit=2)
    if s2.len < 2: return ""
    return s2[1]

when isMainModule:
  let exe = "python3".findExe
  let s = getPythonDll(exe)
  echo (exe, s)

proposal

• step 1: link to find_libpython in the README
• step 2: port find_libpython to nim; likely based on std/dyncalls etc

[yglukhov]

step 1: link to find_libpython in the README
step 2: port find_libpython to nim; likely based on std/dyncalls etc

Sounds good to me

[cvanelteren]

Ran into this recently, is there any update on this? Manually specifying the python did fix it, but is rather uggly to include in scripts. This happened under MacOS with one of my students.