Add an option to specify which Python to use?

[narimiran]

I've tried calling Python from Nim (as shown in the Readme), just to realize it calls system's python, not the version I use for developing (installed via conda) which has all the libraries I usually use (e.g. numpy, matplotlib, etc.)

Could there be a way to specify which Python to use?

• if nothing is specified, use the system Python,
• if user provides a path, use that version of Python.

Maybe something like this?

import nimpy

pyPath("/path/to/my/python")
let np = pyImport("numpy")

[narimiran]

Update:

It seems that nimpy works with python's/conda's virtualenvs.

But if the base/root env is activated, then nimpy doesn't use that one, and uses system's Python.
Only if you activate some other env, nimpy will use that one.

EDIT:

No, it is more crazy than that (at least on my end):
Nimpy always uses system's python, but if you have a virtual env with the same version of Python — it can import libraries which are installed in that env!

[yglukhov]

Nimpy needs to find python library if loaded not from within python, and here's how it does that: https://github.com/yglukhov/nimpy/blob/master/nimpy.nim#L729-L744. I could add an explicit initialization function to be called by the user, but maybe it would be possible to adjust the "automagic" search to suite your needs?

[narimiran]

Here it is once again the situation I have faced, this time explained in more detail, just to make sure we are on the same page.

I have system's python and conda's python:

$ /usr/bin/python
Python 3.7.0 (default, Jul 15 2018, 10:44:58) 
[GCC 8.1.1 20180531] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

$ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

As you can see, when I run python, conda version is run. And it differs from the system's python (3.6.6 vs 3.7.0).

Conda Python is the Python I work with, and it has all the libraries I need and use, e.g. numpy, matplotlib, etc.

---

If I try to run Python from Nim using Nimpy, Nimpy calls system's Python (with no extra libraries installed) so I cannot use Nimpy with numpy, matplotlib, etc.

import nimpy

let pysys = pyImport("sys")
echo pysys.version

let np = pyImport("numpy")
let n = np.array([1, 3, 2, 5, 6, 4])

echo "mean ", n.mean()

3.7.0 (default, Jul 15 2018, 10:44:58) 
[GCC 8.1.1 20180531]
(...)
nimpy.nim(1373)          pyImport
nimpy.nim(1273)          raisePythonError
Error: unhandled exception: <class 'ModuleNotFoundError'>: No module named 'numpy' [Exception]

---

But now comes the funny/tricky/strange part.

If I create a virtual env with Python 3.7 (the same version as system's Python) and there I install the libraries I want to use, what Nimpy does is: it uses system's Python with the libraries from virtual env!

$ conda create -yn py37env python=3.7 numpy
$ conda activate py37env
$ python                                                                  (py37env)
Python 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Running the same file from above gives now:

3.7.0 (default, Jul 15 2018, 10:44:58)
[GCC 8.1.1 20180531]
mean 3.5

Notice the Python version (it is system's Python), and there was no problems with importing and using numpy.

[narimiran]

I could add an explicit initialization function to be called by the user, but maybe it would be possible to adjust the "automagic" search to suite your needs?

I would rather keep the 'automagic' as the default behaviour (no explicit initialization), as it makes nimpy very elegant to show (off :)) and use.

But I'm not sure how to deal with the behaviour explained in the previous comment. If there is a possibility to add an override of the default behaviour by specifying custom Python path, I think that might solve the issue.

[yglukhov]

I've never used conda and don't know how it works, so bear with me. Also let me emphasize it once again: running python from nim involves searching for libpython.so, and once found, sys.path is whatever the python library sets it to be, nimpy does no attempt to configure that. Now we need to know:

1. whether conda activate changes the resulting libpython.so that nimpy discovers. Log the result here: https://github.com/yglukhov/nimpy/blob/master/nimpy.nim#L740
2. how is sys.path affected by conda. Run python -c 'import sys; print(sys.path)' in all your listed cases above

[narimiran]

how is sys.path affected by conda. Run python -c 'import sys; print(sys.path)' in all your listed cases above

$ /usr/bin/python -c 'import sys; print(sys.path)'
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3.7/site-packages']

$ python -c 'import sys; print(sys.path)'
['', '/home/miran/miniconda3/lib/python36.zip', '/home/miran/miniconda3/lib/python3.6', '/home/miran/miniconda3/lib/python3.6/lib-dynload', '/home/miran/miniconda3/lib/python3.6/site-packages']

$ conda activate py37env
$ python -c 'import sys; print(sys.path)'                           (py37env) 
['', '/home/miran/miniconda3/envs/py37env/lib/python37.zip', '/home/miran/miniconda3/envs/py37env/lib/python3.7', '/home/miran/miniconda3/envs/py37env/lib/python3.7/lib-dynload', '/home/miran/miniconda3/envs/py37env/lib/python3.7/site-packages']

[yglukhov]

Huh, completely different, ok. What about the libpython.sos?
How does find /usr/lib -name 'libpython*.so*' -exec ls -ls '{}' + && find /usr/local/lib -name 'libpython*.so*' -exec ls -ls '{}' + behave in same cases?

[narimiran]

In all three cases it is the same:

$ find /usr/lib -name 'libpython*.so*' -exec ls -ls '{}' + ; find /usr/local/lib -name 'libpython*.so*' -exec ls -ls '{}' +
   0 lrwxrwxrwx 1 root root      19 lip  27 15:05 /usr/lib/libpython2.7.so -> libpython2.7.so.1.0
2504 -r-xr-xr-x 1 root root 2563528 lip  27 15:05 /usr/lib/libpython2.7.so.1.0
   0 lrwxrwxrwx 1 root root      20 srp  15 12:44 /usr/lib/libpython3.7m.so -> libpython3.7m.so.1.0
3408 -rwxr-xr-x 1 root root 3486008 srp  15 12:44 /usr/lib/libpython3.7m.so.1.0
   8 -rwxr-xr-x 1 root root    5592 srp  15 12:44 /usr/lib/libpython3.so
  28 -rwxr-xr-x 1 root root   26392 kol  20 18:14 /usr/lib/libreoffice/program/libpythonloaderlo.so
   0 lrwxrwxrwx 1 root root      21 lip  27 15:05 /usr/lib/python2.7/config/libpython2.7.so -> ../../libpython2.7.so

[yglukhov]

Ok, so it seems conda doesn't change the dylibs. And one mystery remains as to how py37env is picked up by nimpy. I'm out of ideas for now. As a quick and dirty workaround, you can just add the needed paths to sys.path:

let sp = pyImport("sys").path
for p in ["/home/miran/miniconda3/lib/python3.6", "/home/miran/miniconda3/lib/python3.6/lib-dynload", "/home/miran/miniconda3/lib/python3.6/site-packages"]:
  sp.append(p)

[narimiran]

I'm out of ideas for now.

Thanks for your help nonetheless!

[yglukhov]

Added pyInitLibPath in 2dd06c8

[TheDoctorAI]

Added pyInitLibPath in 2dd06c8

How to use it to set the path? thanks

[timotheecour]

workaround: use find_libpython to reliably find the libpython shared library, see #171