Numpy Support

[whilo]

Hey,

can you lay out some steps to get my feet wet in direction of Numpy Support? Maybe some small "junior" jobs would help in general, since this both involves Python-C-binding and Jython/JVM-C-binding experience.

Cheers,
Christian

[Stewori]

Hello Christian,
I am currently preparing a major update of the repository (might take another one or two weeks). The update will fix some issues with exception support and will introduce support for heap types. Ideally, it will support Tkinter, so there will be a popular real-world extension runnable with JyNI.

So before actually starting any work, I suggest to await this update. However, until then you could get an overview how JyNI works and try to figure out, where Numpy-loading fails. As far as I remember, it fails on loading other extensions it depends on. If this is the case, you could go through these dependencies and see, which are loadable by JyNI and which fail (and why).

[whilo]

Ok, I will try to do that.

[Stewori]

I uploaded the promised update today. Try the Tkinter demo and tell me if it works :-) .

[whilo]

Nice! I checked out your Tkinter demo and it works fine.

When I try to import numpy, this happens:

>>> import sys
import sys
>>> sys.path.append('/usr/lib/python2.7/dist-packages')
sys.path.append('/usr/lib/python2.7/dist-packages')
>>> import numpy as np
import numpy as np
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 137, in <module>
    import add_newdocs
  File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 9, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 4, in <module>
    from type_check import *
  File "/usr/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 5, in <module>
    import multiarray
ImportError: No module named multiarray

The relevant files seem to be:
./dist-packages/numpy/core/include/numpy/__multiarray_api.h
./dist-packages/numpy/core/include/numpy/multiarray_api.txt
./dist-packages/numpy/core/multiarray.so

I also tried to load:

import numpy.core.tests.test_multiarray
but the error is the same:

import numpy.core.tests.test_multiarray
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 137, in <module>
    import add_newdocs
  File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 9, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 4, in <module>
    from type_check import *
  File "/usr/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 8, in <module>
    import numpy.core.numeric as _nx
  File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 5, in <module>
    import multiarray
ImportError: No module named multiarray

Can I get better debugging information, e.g. where loading of the multiarray-lib fails? I can instrument a numpy build accordingly if this is really necessary, but it is a lot of C code and I am not yet sure where to dig in.

[Stewori]

Nice! I checked out your Tkinter demo and it works fine.

Happy to hear that :-) Could you state what Linux distribution and version you used (and whether 32 or 64 bit)?

ImportError: No module named multiarray

Obviously it can't find multiarray.so. Check the python-path to see if it is on the list. Also make sure that JyNI is actually loaded (f.i. by importing datetime and checking the docstring - it must read "Fast implementation of the datetime type."; see datetime demo). JyNI is currently not able to handle multiarray.so, but the error msg should be different (unknown symbol or segfault or something).
I suggest to do these experiments directly with the command "import multiarray" rather than "import numpy".

Another path is to set up a custom numpy and exclude the failing line "import multiarrray" from the general import procedure. This way you can see, where else it fails. Maybe it fails due to missing multiarray, then you can exclude those parts too one after each other, until an unrelated error comes or the import is over. This would obtain an overview what else needs to be done.

Since you only need to modify numpy's Python code for this, you would get around building the C-part. However, in the long-run you won't get around building a custom numpy for debugging. Maybe the easiest way is to stick to numpy's own build-scripts. Can you figure out (maybe with help of google), how to build numpy with debug symbols? Being able to use the C-debugger would be a nice initial step.

[whilo]

Sorry, got distracted by my main project. I have Ubuntu 13.10 amd64. I will have a look into it soon when I continue to work on a Python stack.

[whilo]

Hey, I am still interested in getting Python 2.7 with native extensions on the JVM (and in fact integrate it with Clojure). I just retried loading numpy, but it still has a problem with loading multiarray.so. import imp does not seem to work in Jython 2.7. I wanted to try to load multiarray.so directly with imp.

>>> import numpy
<single-top>: numpy= 
import: trying source /mnt/slow/src/JyNI/numpy
import: trying precompiled with no source /mnt/slow/src/JyNI/numpy$py.class
import: trying source /mnt/slow/src/JyNI/Lib/numpy
import: trying precompiled with no source /mnt/slow/src/JyNI/Lib/numpy$py.class
import: trying numpy in packagemanager for path None
import: trying numpy as java class in SysPathJavaLoader
import: trying numpy as java class in context class loader, for backwards compatibility
import: trying source /mnt/slow/src/JyNI/__classpath__/numpy
import: trying precompiled with no source /mnt/slow/src/JyNI/__classpath__/numpy$py.class
import: trying numpy/__init__$py.class in parent class loader
import: trying numpy/__init__.py in parent class loader
import: trying numpy$py.class in parent class loader
import: trying numpy.py in parent class loader
import: trying source /mnt/slow/src/JyNI/__pyclasspath__/numpy
import: trying precompiled with no source /mnt/slow/src/JyNI/__pyclasspath__/numpy$py.class
import: trying source /mnt/slow/src/JyNI/Lib/site-packages/numpy
import: trying precompiled with no source /mnt/slow/src/JyNI/Lib/site-packages/numpy$py.class
import: trying precompiled /usr/lib/python2.7/dist-packages/numpy/__init__$py.class
import: import numpy # precompiled from /usr/lib/python2.7/dist-packages/numpy/__init__$py.class
import: trying source /usr/lib/python2.7/dist-packages/numpy/add_newdocs
import: trying precompiled /usr/lib/python2.7/dist-packages/numpy/add_newdocs$py.class
import: import numpy.add_newdocs # precompiled from /usr/lib/python2.7/dist-packages/numpy/add_newdocs$py.class
import: trying precompiled /usr/lib/python2.7/dist-packages/numpy/lib/__init__$py.class
import: import numpy.lib # precompiled from /usr/lib/python2.7/dist-packages/numpy/lib/__init__$py.class
import: trying source /mnt/slow/src/JyNI/type_check
import: trying precompiled with no source /mnt/slow/src/JyNI/type_check$py.class
import: trying source /mnt/slow/src/JyNI/Lib/type_check
import: trying precompiled with no source /mnt/slow/src/JyNI/Lib/type_check$py.class
import: trying type_check in packagemanager for path None
import: trying type_check as java class in SysPathJavaLoader
import: trying type_check as java class in context class loader, for backwards compatibility
import: trying source /mnt/slow/src/JyNI/__classpath__/type_check
import: trying precompiled with no source /mnt/slow/src/JyNI/__classpath__/type_check$py.class
import: trying type_check/__init__$py.class in parent class loader
import: trying type_check/__init__.py in parent class loader
import: trying type_check$py.class in parent class loader
import: trying type_check.py in parent class loader
import: trying source /mnt/slow/src/JyNI/__pyclasspath__/type_check
import: trying precompiled with no source /mnt/slow/src/JyNI/__pyclasspath__/type_check$py.class
import: trying source /mnt/slow/src/JyNI/Lib/site-packages/type_check
import: trying precompiled with no source /mnt/slow/src/JyNI/Lib/site-packages/type_check$py.class
import: trying source /usr/lib/python2.7/dist-packages/type_check
import: trying precompiled with no source /usr/lib/python2.7/dist-packages/type_check$py.class
import: trying source /usr/lib/python2.7/dist-packages/numpy/core/type_check
import: trying precompiled with no source /usr/lib/python2.7/dist-packages/numpy/core/type_check$py.class
import: trying source /usr/lib/python2.7/dist-packages/numpy/type_check
import: trying precompiled with no source /usr/lib/python2.7/dist-packages/numpy/type_check$py.class
import: trying source /usr/lib/python2.7/dist-packages/numpy/lib/type_check
 mintypecode: typechars=P typeset=P default=P typecodes= _[61_17]= isinstance,f t= str,f asarray,f intersection= _[63_20]= l= i= _typecodes_by_elsize,f 
 asfarray: a=P dtype=P _nx,f issubclass,f asarray,f 
 real: val=P asanyarray,f 
 imag: val=P asanyarray,f 
 iscomplex: x=P ax= asanyarray,f issubclass,f _nx,f res= zeros,f bool,f 
 isreal: x=P imag,f 
 iscomplexobj: x=P issubclass,f asarray,f _nx,f 
 isrealobj: x=P issubclass,f asarray,f _nx,f 
 _getmaxmin: t=P getlimits= f= 
 nan_to_num: x=P t= AttributeError,f obj2sctype,f type,f issubclass,f _nx,f nan_to_num,f y= array,f scalar= True,f False,f are_inf= isposinf,f are_neg_inf= isneginf,f are_nan= isnan,f maxf= minf= _getmaxmin,f 
 real_if_close: a=P tol=P asanyarray,f issubclass,f _nx,f getlimits= f= 
 asscalar: a=P 
 typename: char=P _namefromtype,f 
 common_type: arrays=P is_complex= False,f precision= a= t= iscomplexobj,f True,f issubclass,f _nx,f p= array_precision,f None,f TypeError,f max,f array_type,f 
<file-top>: division= absolute_import= print_function= __all__= _nx= asarray= asanyarray= array= isnan= obj2sctype= zeros= isneginf= isposinf= _typecodes_by_elsize= mintypecode= asfarray= real= imag= iscomplex= isreal= iscomplexobj= isrealobj= _getmaxmin= nan_to_num= real_if_close= asscalar= _namefromtype= typename= array_type= array_precision= common_type= 
 mintypecode: typechars=P typeset=P default=P typecodes= _[61_17]= isinstance t= str asarray intersection= _[63_20]= l= i= _typecodes_by_elsize 
 asfarray: a=P dtype=P _nx issubclass asarray 
 real: val=P asanyarray 
 imag: val=P asanyarray 
 iscomplex: x=P ax= asanyarray issubclass _nx res= zeros bool 
 isreal: x=P imag 
 iscomplexobj: x=P issubclass asarray _nx 
 isrealobj: x=P issubclass asarray _nx 
 _getmaxmin: t=P getlimits= f= 
 nan_to_num: x=P t= AttributeError obj2sctype type issubclass _nx nan_to_num y= array scalar= True False are_inf= isposinf are_neg_inf= isneginf are_nan= isnan maxf= minf= _getmaxmin 
 real_if_close: a=P tol=P asanyarray issubclass _nx getlimits= f= 
 asscalar: a=P 
 typename: char=P _namefromtype 
 common_type: arrays=P is_complex= False precision= a= t= iscomplexobj True issubclass _nx p= array_precision None TypeError max array_type 
import: 'type_check' as /usr/lib/python2.7/dist-packages/numpy/lib/type_check.py
 _ufunc_reconstruct: module=P name=P mod= __import__,f getattr,f 
 _ufunc_reduce: func=P whichmodule= name= _ufunc_reconstruct,f 
<file-top>: division= absolute_import= print_function= __doc__= __version__= multiarray= umath= _internal= nt= numeric= *= fromnumeric= char= rec= chararray= scalarmath= function_base= machar= getlimits= shape_base= max= min= round= abs= __all__= Tester= test= bench= _ufunc_reconstruct= _ufunc_reduce= sys= copyreg= ufunc 
 _ufunc_reconstruct: module=P name=P mod= __import__ getattr 
 _ufunc_reduce: func=P whichmodule= name= _ufunc_reconstruct 
import: 'numpy.core' as /usr/lib/python2.7/dist-packages/numpy/core/__init__.py
import: trying source /usr/lib/python2.7/dist-packages/numpy/core/info
<file-top>: division= absolute_import= print_function= depends= global_symbols= 
import: 'numpy.core.info' as /usr/lib/python2.7/dist-packages/numpy/core/info.py
import: trying source /usr/lib/python2.7/dist-packages/numpy/core/multiarray
import: trying precompiled with no source /usr/lib/python2.7/dist-packages/numpy/core/multiarray$py.class
import: trying numpy as java class in SysPathJavaLoader
import: trying numpy as java class in context class loader, for backwards compatibility
import: trying source /usr/lib/python2.7/dist-packages/numpy/core/multiarray
import: trying precompiled with no source /usr/lib/python2.7/dist-packages/numpy/core/multiarray$py.class
import: trying numpy as java class in SysPathJavaLoader
import: trying numpy as java class in context class loader, for backwards compatibility
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 153, in <module>
    from . import add_newdocs
  File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 6, in <module>
    from . import multiarray
ImportError: cannot import name multiarray

What is your current status of the work? I have read your paper and seen your presentation, which was nice. I still feel the pain of language fragmentation on runtime level and would try to help you, but some specific pointers would help.

[Stewori]

Looking at /usr/share/pyshared/numpy/init.py, I obtained the following list of imports for numpy:

import core
from core import *
import compat
import lib
from lib import *
import linalg
import fft
import polynomial
import random
import ctypeslib
import ma
import matrixlib as _mat
from matrixlib import *

I don't know which of them need C-API, but I asume it's several of them if not all. Note that
"ctypeslib" is numpy's wrapper around ctypes, so also ctypes is a dependency.

Looking at /usr/share/pyshared/numpy/core/init.py reveals even more (difficult) imports.

import multiarray
import umath
import internal # for freeze programs
import numerictypes as nt
multiarray.set_typeDict(nt.sctypeDict)
import numeric
from numeric import *
import fromnumeric
from fromnumeric import *
import defchararray as char
import records as rec
from records import *
from memmap import *
from defchararray import chararray
import scalarmath
import function_base
from function_base import *
import machar
from machar import *
import getlimits
from getlimits import *
import shape_base
from shape_base import *
del nt
from fromnumeric import amax as max, amin as min,
round as round
from numeric import absolute as abs

Some of them are internal dependencies, some of them are external. In order to support numpy, we must
ensure support for all of these imports one by one. You could help by starting to work through these
imports, extend this list by further subsequent imports and finally try to classify all imports into

1. is Python module and needs no C-extension
   1a) and works with Jython
   1b) and does not work with Jython

2. is C-extension or depends on one (i.e. imports one)
   2a) and works with JyNI
   2b) and does not work with JyNI

In any cases 1a and 2a please just write a note here with the name of the extension.
In case 1b) I'd suggest to file a bug-report at bugs.jython.org
In case 2b) please open a separate issue here.

You don't need to investigate ctypes, as it is currently my top-priority anyway. I think having
ctypes support would be already very valuable for Jython and we will need it for numpy anyway.
Main show stopper for it is support of PyWeakRef, which I am planning to support along with
garbage collection hopefully until end of this summer.

[Stewori]

With 54c8e16 I am able to call import numpy properly on my system. I'd appreciate if you could reproduce this. It only works with numpy 1.12 (current github version). Still fails with 1.11.1 (latest release) and 1.8 (native numpy on my system) and probably with all other versions too.

[Stewori]

To get an impression (also some things that not yet work):

stefan@stefan-x200 ~/eclipseWorkspace/JyNI $ java -cp jython.jar:build/JyNI.jar org.python.util.jython
Jython 2.7.1b3 (, Jun 27 2016, 16:27:24) 
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_91
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/data/workspace/linux/numpy1.12-dev/numpy')
>>> import numpy as np
>>> a = np.array([4, -1, 2])
>>> b = np.array([7, -3, 1])
>>> a
array([4, -1, 2])
>>> 3*b
array([21, -9, 3])
>>> np.outer(a, b)
array([[28, -12, 4],
       [-7, 3, -1],
       [14, -6, 2]])
>>> np.dot(a, b)
33
>>> np.identity(4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'FloatFormat' object has no attribute 'special_fmt'
>>> 3.0*a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'FloatFormat' object has no attribute 'special_fmt'
>>> a*b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for *: 'numpy.ndarray' and 'numpy.ndarray'
>>> 

[whilo]

This progress is the coolest thing that happened to me in the last weeks! Thank you so much for your work! I have seen your talk on JVMLS, funny that you are also working on neuro science :). I have started to use Python with PyNN and the Nest simulator for my Bachelor-Thesis and ever since wanted to integrate the Python libs with my Clojure stack. I hope I can check the numpy support soon.

[wlievens]

Looks cool! Is printing floating point numbers somehow difficult, or is that just an example of a function for which the wrapper hasn't been ported yet?

[Stewori]

@wlievens It's just an example that not everything is working yet. The initializer code of 'FloatFormat' object aborts with some silently caught exception before it can init the attribute 'special_fmt'. I need to find the cause for this and solve it. Things like this can be easy or difficult, sometimes I get it solved in one evening, sometimes it takes several weeks, you can never know beforehand. The floating point-issue is surely the next I will look into.

@whilo Thanks a lot for these words! I have heard about the Nest simulator, once helped a friend to compile it, set it up etc. I remember rumors they wanted to switch to CFFI or so. But sure, we can take a look, whether it is workable with JyNI. However NumPy should be working reliably first. It would be great if you could reproduce the numpy-import; you are running Linux, aren't you?

[whilo]

@Stewori I can reproduce your example code above exactly with current numpy from github (d3e3d91) :D. I am on Ubuntu 14.04 64bit.

[whilo]

Also I don't work with nest anymore, as I am doing more machine learning and math now. So I would be more interested in scikit-learn, theano, tensorflow etc. and I think they would bring a lot of leverage to the JVM (as do many Python libs, but those are certainly popular). Atm. there is deeplearning4j, which is nice, but most stuff in machine learning is happening in Python. It would also be nice to let scientific people program Python on the JVM, but expose application interfaces (e.g. Java interfaces), to manage long term state.

[Stewori]

stefan@stefan-x200 ~/eclipseWorkspace/JyNI $ java -cp jython.jar:build/JyNI.jar org.python.util.jython
Jython 2.7.1b3 (, Jul 20 2016, 05:30:21) 
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_91
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/data/workspace/linux/numpy1.12')
>>> import numpy as np
>>> a = np.array([2, -5, 7])
>>> a
array([2, -5, 7])
>>> b = np.array([-1.2, 3.5, 2.8])
>>> b
array([ -1.20000000e+000,   3.50000000e+000,   2.80000000e+000])
>>> a+b
array([  8.00000000e-001,  -1.50000000e+000,   9.80000000e+000])
>>> a*b
array([ -2.40000000e+000,  -1.75000000e+001,   1.96000000e+001])
>>> a.dot(b)
-0.30000000000000071
>>> a[::-1]
array([7, -5, 2])
>>> V = np.outer(a, b)
>>> V
array([[ -2.40000000e+000,   7.00000000e+000,   5.60000000e+000],
       [  6.00000000e+000,  -1.75000000e+001,  -1.40000000e+001],
       [ -8.40000000e+000,   2.45000000e+001,   1.96000000e+001]])
>>> V.dot(a)
array([ -6.00000000e-001,   1.50000000e+000,  -2.10000000e+000])
>>> np.linalg.eig(V)
(array([ -3.00000000e-001+0.j,  -5.77315973e-015+0.j,   0.00000000e+000+0.j]), array([[ -2.26455407e-001+0.j,  -1.99350120e-001+0.j,  -4.69277056e-002+0.j],
       [  5.66138517e-001+0.j,   5.69571772e-001+0.j,   6.14146784e-001+0.j],
       [ -7.92593924e-001+0.j,  -7.97400481e-001+0.j,  -7.87795353e-001+0.j]]))
>>> 

I don't know if it's art, but I think I like it :)

[Stewori]

*the presented code works as of 1dd0c6e

**you might notice that float-formatting is somewhat ill-configured; currently I can't tell how this comes, but however numbers are still accurate

[whilo]

Really nice. I have added the python sys.path of my system and could load numpy, but get some slice errors:

>>> sys.path
['', '/usr/src/JyNI/build/JyNI.jar', '/usr/src/JyNI/Lib', '/usr/src/JyNI/jython.jar/Lib', '__classpath__', '__pyclasspath__/', '', '/home/void/.local/lib/python2.7/site-packages/ropemode-0.2-py2.7.egg', '/home/void/.local/lib/python2.7/site-packages/rope-0.9.4-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/dot2tex-2.8.7-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/pyparsing-1.5.6-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/reimport-1.3dev_r77-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/gensim-0.8.6-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/clojure_py-0.2.4-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/pysistence-0.4.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/pystache-0.5.3-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/astor-0.3-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/rply-0.7.2-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/hy-0.10.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/benchmark-0.1.5-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/peewee-2.3.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/SpikeBasedSampling-1.0.0-py2.7-linux-x86_64.egg', '/usr/local/lib/python2.7/dist-packages/epc-0.0.5-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/sexpdata-0.0.3-py2.7.egg', '/home/void/notebook/src/traitlets', '/home/void/notebook/src/jupyter-core', '/home/void/notebook/src/nbformat', '/home/void/notebook/src/jupyter-client', '/home/void/notebook/src/ipython', '/home/void/notebook/src/ipykernel', '/usr/local/lib/python2.7/dist-packages/Lasagne-0.1.dev0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/Keras-0.1.1-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/progressbar2-2.7.3-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/picklable_itertools-0.1.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/GPy-0.8.8-py2.7-linux-x86_64.egg', '/opt/nest-2.4.2/lib/python2.7/site-packages', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/void/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode']
>>> import numpy as np
>>> foo = np.array([1,2,3])
>>> foo
array([1, 2, 3])
>>> bar = np.array([3,4,5])
>>> np.outer(foo,bar)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/numeric.py", line 1085, in outer
    return multiply(a.ravel()[:, newaxis], b.ravel()[newaxis,:], out)
IndexError: invalid slice
>>> foo
array([1, 2, 3])
>>> bar
array([3, 4, 5])
>>> np.dot(foo,bar)
26
>>> np.linalg
<module 'numpy.linalg' from '/usr/local/lib/python2.7/dist-packages/numpy/linalg/__init__.py'>
>>> np.linalg.svd
<function svd at 0x7>
>>> np.identity(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/numeric.py", line 2214, in identity
    return eye(n, dtype=dtype)
  File "/usr/local/lib/python2.7/dist-packages/numpy/lib/twodim_base.py", line 240, in eye
    m[:M-k].flat[i::M+1] = 1
IndexError: invalid slice
>>> import scipy
>>> import scipy.sparse as s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/__init__.py", line 215, in <module>
    from .lil import *
  File "/usr/local/lib/python2.7/dist-packages/scipy/sparse/lil.py", line 16, in <module>
    from . import _csparsetools
ImportError: /usr/local/lib/python2.7/dist-packages/scipy/sparse/_csparsetools.so: undefined symbol: PyByteArray_Type

But I really like where this is going. :)

[Stewori]

Hey Christian, thanks for testing! Unfortunately I cannot reproduce this issue on my system:

stefan@stefan-x200 ~/eclipseWorkspace/JyNI $ java -cp jython.jar:build/JyNI.jar org.python.util.jython
Jython 2.7.1b3 (, Jul 20 2016, 05:30:21) 
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_91
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/data/workspace/linux/numpy1.12-2016-07-21/numpy')
>>> import numpy as np
>>> foo = np.array([1,2,3])
>>> foo
array([1, 2, 3])
>>> bar = np.array([3,4,5])
>>> np.outer(foo,bar)
array([[3, 4, 5],
       [6, 8, 10],
       [9, 12, 15]])
>>> np.dot(foo,bar)
26
>>> np.linalg
<module 'numpy.linalg' from '/data/workspace/linux/numpy1.12-2016-07-21/numpy/numpy/linalg/__init__$py.class'>
>>> np.linalg.svd
<function svd at 0x8>
>>> np.identity(3)
array([[  1.00000000e+000,   0.00000000e+000,   0.00000000e+000],
       [  0.00000000e+000,   1.00000000e+000,   0.00000000e+000],
       [  0.00000000e+000,   0.00000000e+000,   1.00000000e+000]])
>>> 

I am using a fresh numpy 1.12 clone from this morning. Precisely speaking cloned after commit numpy/numpy@93240e0
However I also tested this with a clone I obtained two weeks ago or so, getting the same results. Anyway - I'd recommend to go on with exactly the same numpy versions to figure this out.
So I'd like you to follow these steps:

• make sure to use the same numpy as I do. Let's stick to the commit 93240e0d8ea23644a5b7874037e658c54966ff54 mentioned above.
• let's focus on one issue at a time, let's start with np.outer.
• please open a separate issue for this, so we can keep this one in a roadmap-fashion
• try to figure out as much as you can about the issue and share it there

I just started digging a bit:
np.outer is implemented in numpy/core/numeric.py and essentially is implemented like this:

a = asarray(a)
b = asarray(b)
return multiply(a.ravel()[:, newaxis], b.ravel()[newaxis,:], out)

Given that foo and bar are arrays we can skip asarray. Also ravel has no effect in this case:

>>> foo
array([1, 2, 3])
>>> foo.ravel()
array([1, 2, 3])
>>> 

It appears that newaxis is None, so in this case np.outer is equivalent to np.multiply(foo[:, None], bar[None, :]):

>>> np.multiply
<ufunc 'multiply'>
>>> np.multiply(foo[:, None], bar[None, :])
array([[3, 4, 5],
       [6, 8, 10],
       [9, 12, 15]])
>>> 

I suppose, already the slicing goes wrong, so could you please try plain foo[:, None]?
Also play around with various slicing. What is the simplest case that fails? Or does slicing fail in any case?

[whilo]

Sorry for the confusion. Numpy 1.12 works for me, too. I just tried to use the same python path as CPython out of curiousity, which is Numpy 1.10.4. Again I should have made that clear, sorry, I was just too happy to try some things I'd like to do, after I saw that numpy seems to work better now :). Why is a new numpy version needed btw.?

[Stewori]

Good to hear that with Numpy 1.12 it still works!
Regarding Numpy 1.10.4 I am actually surprised that it even imports without error, given that 1.11.1 and 1.8 do not. However Python's C-API is so complex that there will always be some edges.

I actually cannot tell you why older Numpy versions don't work now; they obviously just don't for some reason.
In JyNI-land finding out why something fails is usually 95% of the work to fix it. So in very general here the question "why does xy not work" is almost pointless. Instead ask "is it worth to allocate time on making xy work" or convince me that it is.

In this sense, should I spend time on supporting older Numpy or should I instead spend this time on making support for 1.12 better, e.g. fix this float-printing format thing? Another one: Even with 1.12 this currently does not work on OSX (not yet speaking of Windows). So should I focus on older Numpy for Linux, or should I focus on getting 1.12 work on OSX?

For what reason ever, Numpy versions are different enough to require extra effort to make each of them work, so I'd rather choose one version and attempt to make it work as good as possible instead of spreading my capacity over several versions. So, which one to choose? Obviously the newest I'd say, so we'll have the maximal version-life-time to benefit from it. I suppose that Numpy 1.12 will have been released for quite some time when JyNI transists out of beta state one day. Even then I'd allocate my time with priority to keep up with 1.13 etc than fixing support for legacy versions.

Anyway. If someone comes up with a good reason to support some specific older version I will reconsider, but so far this didn't happen. Also, feel free to investigate by yourself if you whish to get an older version to run. Open a version-specific issue and I will guide through this process as good as I can and as my capacity allows.
However I'd recommend also for you and everybody else to rather focus on 1.12 now. Help finding issues, order them by relevance (i.e. identify the aspects you need in order to achieve something specific, getting your personal project running or your favorit Numpy-dependent extension), create threads here etc.

[whilo]

Yes, I was just curious what happens if I take my Python path and plug it in there. I agree, focus on 1.12 to get a first stack working. I was just curious why there are differences between versions. I can imagine that the interface is hairy. Do you have a description/screencast of your working environment somewhere? I cannot promise to do anything significant atm. as I am occupied with other tasks. But if I can do something with a few hours effort from time to time, I would help. It is difficult to get an environment for JyNI (JNI debugging etc.) setup though.

[Stewori]

can imagine that the interface is hairy

Yes, and also NumPy significantly changed over time. IronPython devs also suffer from this with their Ironclad. Once upon a time Ironclad was able to load NumPy and pass most NumPy unittests. There are discussions elsewhere about NumPy's change policy and some people are actually very concerned about it. E.g. NumPy has even made backwards-incompatible changes also from Python perspective without bumping major version.
But - of course - every older Numpy not runnable with JyNI finally points to a bug in JyNI or to a still unsupported piece of API. Sometimes it might be also due to an issue in NumPy that was fixed. Or they moved from a semi-official API part to a more official one. To tell this in detail one would have to debug each issue for itself. I admit/agree that in theory this stuff should be fixed in JyNI; in practice this whole thing is just about priorities.

Regarding setup and debugging instructions, please open a separate issue for that. Ideally we will turn the information collected there into documentation once it's sufficiently useful, something like a JyNI-dev guide.

But if I can do something with a few hours effort from time to time, I would help

Just identify the next issue that bothers you most, open an issue-thread here, try to reduce it to a minimal example and trace the origin as far as you can.

Btw the Float-formating issue is due to lacking support for __cmp__ in PyCPeer. Because of this arrayprint.py misconcepts the threshold for using exponent-notation. However if I add support for __cmp__ this makes NumPy run into a code-branch that uses iterator-access on a numpy.ndarray, which is not yet supported. So I will have to add iterator support in order to fix this, but that will take some time, a week or two I guess (I had originally planned iterator support for JyNI alpha.5, but now it might already go into 4).

[Stewori]

stefan@stefan-x200 ~/eclipseWorkspace/JyNI $ java -cp jython.jar:build/JyNI.jar org.python.util.jython
Jython 2.7.1b3 (, Jul 20 2016, 05:30:21) 
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java1.8.0_91
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/data/workspace/linux/numpy1.12-2016-07-21/numpy')
>>> import numpy as np
>>> a = np.array([2.1, 51.2, 7.1, 1.2])
>>> a
array([  2.1,  51.2,   7.1,   1.2])
>>> type(a)
<type 'numpy.ndarray'>
>>> b = np.array([-12.4, -32.5, -13.3, 7])
>>> a+b
array([-10.3,  18.7,  -6.2,   8.2])
>>> a*b
array([  -26.04, -1664.  ,   -94.43,     8.4 ])
>>> b[1:3]
array([-32.5, -13.3])
>>> a[::-1]
array([  1.2,   7.1,  51.2,   2.1])
>>> 3*a
array([   6.3,  153.6,   21.3,    3.6])
>>> b+3
array([ -9.4, -29.5, -10.3,  10. ])
>>> np.array.__module__
'numpy.core.multiarray'
>>> np.dot(a, b)
-1776.0699999999999
>>> a.dot(b)
-1776.0699999999999
>>> V = np.outer(a, a)
>>> V
array([[  4.41000000e+00,   1.07520000e+02,   1.49100000e+01,
          2.52000000e+00],
       [  1.07520000e+02,   2.62144000e+03,   3.63520000e+02,
          6.14400000e+01],
       [  1.49100000e+01,   3.63520000e+02,   5.04100000e+01,
          8.52000000e+00],
       [  2.52000000e+00,   6.14400000e+01,   8.52000000e+00,
          1.44000000e+00]])
>>> V[2:, :2]
array([[  14.91,  363.52],
       [   2.52,   61.44]])
>>> V.dot(b)
array([ -3729.747, -90934.784, -12610.097,  -2131.284])
>>> np.linalg.eig(V)
(array([  0.00000000e+00,   2.67770000e+03,  -1.56207180e-15,
         2.00035169e-15]), array([[ -9.99176193e-01,   4.05824568e-02,   5.53501614e-01,
         -5.98317464e-01],
       [  4.01869697e-02,   9.89438948e-01,   5.04683914e-02,
          1.10594419e-01],
       [  5.57280244e-03,   1.37207354e-01,  -6.21052451e-01,
         -5.19106166e-01],
       [  9.41882102e-04,   2.31899753e-02,   5.52614475e-01,
         -6.00261506e-01]]))
>>> 

Floats print well now. As of b4ffbd5 output looks pretty much like in CPython.
Note how it auto-configures float-format when slicing V omits most extreme values.
Next step is to get this working on OSX.

[whilo]

Very nice work!

[whilo]

Thanks also for pointing out how fragile the numpy ctype binding is. I pictured the cpython interface to be rather stable and solid, but having browsed the large CPython codebase once, I was worried about Python as a language/runtime. (compared to the fairly small and clean Clojure reference implementation on the JVM).

[Stewori]

As of f1edab2 the code snippets from above also work on OSX (more testing required, feel free to do so!).

[whilo]

Just a quick status update with the new release: On Ubuntu 17.04 with the new release of JyNI and Jython 2.7.1rc2 I can import my system's numpy 1.11.2 without problems. scipy also seems to work basically! 💃

Matplotlib fails still:

>>> import matplotlib.pylab as p
JyNI-Warning: numpy.dtype occurred as type of a non-static object!
!!! Segfault-WARNING: JyNI_GC_ObtainJyGCHead 1749 (140281117581696)

As does sklearn (which is expected due to the missing PByteArray support):

>>> import sklearn
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/sklearn/__init__.py", line 57, in <module>
    from .base import clone
  File "/usr/lib/python2.7/dist-packages/sklearn/base.py", line 10, in <module>
    from scipy import sparse
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/__init__.py", line 223, in <module>
    from .lil import *
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/lil.py", line 16, in <module>
    from . import _csparsetools
ImportError: /usr/lib/python2.7/dist-packages/scipy/sparse/_csparsetools.x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type

You can close this issue from my side, I can open new ones. I am really happy that you keep pushing this forward, I will take a closer look at the matplotlib issue.

[whilo]

Matplotlib actually crashes with (which is also expected due to missing PyFile support):

>>> import matplotlib.pyplot as plt
java: symbol lookup error: /usr/lib/python2.7/dist-packages/matplotlib/ft2font.x86_64-linux-gnu.so: undefined symbol: PyFile_AsFile

[whilo]

So for me the matplotlib issue would be most important, do you think it is reasonable to tackle the PyFile issue? Where should I start?

I also just tried to import torch (which I could use for my master thesis), and it fails with importing multiprocessing. Tensorflow fails with a name clash of import google, since that seems to import a Java Package and not the protocol buffer one of python.

[Stewori]

Whilo, good to hear that it is relatively well workable! SciPy also fails for me complaining about PyByteArray. So does CFFI. However, PyByteArray is much involved with BufferProtocol, so I would implement that all together.

The support for PyFile is kind of an "easy issue", involving some work though.

You can maybe tackle the google issue by moving the location of tensorflow to the top of sys.path.
However, this is more a Jython problem than a JyNI one. Consider to bring it up at bugs.jython.org.

Regarding the Segfault-WARNING: Solving that requires some refactoring of JyNI's memory management. I have that on the agenda, but cannot tell when I will come to that. For now, focus is on Windows support.

That said, please open separate issues for these, so we can better keep track. I wouldn't close this issue yet, because things like the following still don't work:

class subarr(np.ndarray):
	pass

inst = np.array(7.5).view(subarr)

print inst, type(inst)
inst.foo = 'bar'
print inst.foo

(e.g. astropy crashes because of this)

Thank you very much for testing!

[whilo]

I have commented in

FILE *
PyFile_AsFile(PyObject *f)
{
    printf("Accessing file.\n");
    if (f == NULL || !PyFile_Check(f))
        return NULL;
    else
        return ((PyFileObject *)f)->f_fp;
}

Which is called and immediately SIG_ABORTS. I get a dump file from the JVM:

Register to memory mapping:

RAX=0x00007fb7cbdd8280: PyFile_Type+0 in /home/christian/Development/JyNI/build/libJyNI.so at 0x00007fb7cbaa9000
RBX=0x00007fb7f8070958 is an unknown value
RCX=0x00007fb7c055d390 is an unknown value
RDX=0x0000000000000001 is an unknown value
RSP=0x00007fb8278190e8 is pointing into the stack for thread: 0x00007fb82000a000
RBP=0x00007fb7f8032e30 is an unknown value
RSI=0x00007fb7c055d390 is an unknown value
RDI=0x00007fb7f8032e30 is an unknown value
R8 =0x0000000000000000 is an unknown value
R9 =0x0000000000000004 is an unknown value
R10=0x0000000000000319 is an unknown value
R11=0x00007fb7cbb08b65: PyErr_SetString+0 in /home/christian/Development/JyNI/build/libJyNI.so at 0x00007fb7cbaa9000
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000000 is an unknown value
R14=0x00007fb827819130 is pointing into the stack for thread: 0x00007fb82000a000
R15=0x0000000000000001 is an unknown value


Stack: [0x00007fb82771f000,0x00007fb827820000],  sp=0x00007fb8278190e8,  free space=1000k
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 3935  JyNI.JyNI.callPyCPeer(JLorg/python/core/PyObject;Lorg/python/core/PyObject;J)Lorg/python/core/PyObject; (0 bytes) @ 0x00007fb81183612c [0x00007fb8118360c0+0x6c]
j  JyNI.PyCPeerType.__call__([Lorg/python/core/PyObject;[Ljava/lang/String;)Lorg/python/core/PyObject;+36
J 2911 C1 org.python.core.PyObject.__call__(Lorg/python/core/PyObject;)Lorg/python/core/PyObject; (16 bytes) @ 0x00007fb811ae0fec [0x00007fb811ae0e80+0x16c]
J 5454 C2 org.python.core.PyObject.__call__(Lorg/python/core/ThreadState;Lorg/python/core/PyObject;)Lorg/python/core/PyObject; (6 bytes) @ 0x00007fb8125b22e0 [0x00007fb8125b22a0+0x40]
j  matplotlib.font_manager$py.createFontList$16(Lorg/python/core/PyFrame;Lorg/python/core/ThreadState;)Lorg/python/core/PyObject;+673
j  matplotlib.font_manager$py.call_function(ILorg/python/core/PyFrame;Lorg/python/core/ThreadState;)Lorg/python/core/PyObject;+368

Just returning NULL is not sufficient as a quick hack and I guess that I need to wrap a Jython class for PyFile instead of accessing the pointer directly. You said it would be easy to fix, but in the comment you say:

//Providing the following method via JyNI would be very hard or impossible.
//However, we could create a FILE-Pointer to the same file and also adjust
//io mode and seek-position to the values of the given PyFile.
//But this may lead to conflicts in systems not supporting multiple file
//access (i.e. windows) and would not stand equality-checks via pointer-wise "==".

I had some frustrations with filesystem APIs lately for my kv-store implementation, so I would also be careful with double access. What are your suggestions to proceed? Also do you have some chat channel? This might be easier to get started.

[Stewori]

Chrisitan, please open a new issue for that,. If possible move the last post to that new issue.
With "easy issue" I meant that it is doable with guidance. Adding new stuff sometimes requires adjustments at other places and the JyNI Code base is pretty messy (I apologize). I can guide you through in a separate issue thread.
That said, I didin't remember that comment when I wrote that. Indeed PyFile_AsFile would be hard to implement, but still maybe doable. It might require changes in Jython though. A quick search brought up this link, which might yield interesting ideas too: http://www.kfu.com/~nsayer/Java/jni-filedesc.html

In that sense "easy" only refers to all other PyFile methods more or less. Maybe you could do a text-search of "PyFile_AsFile" in SciPy codebase...? In the lucky case that SciPy wouldn't depend on this exact function, we could postpone it.

Regarding chat-channel I am usually online in Jython's irc channel (stewori). We can meet there and move to a private Chat thread.

[whilo]

I have moved the PyFile issue over here: #11

[whilo]

IRC requires to run a client all the time to get offline messages (or using fancy messageserv services, browsing plaintext backlogs of 3rd parties etc). It also is really primitive compared to modern richtext chat media. I say this as a Unix and former IRC lover. In my experience this keeps many young and non-Unix people out, so I would suggest to use either gitter, via activating https://gitter.im/stewori/jyni or some other web-based solution (slack or an open-source one like mattermost). I have joined the IRC channel now, hopefully we can talk soon :)

[Stewori]

For now let's stick to the official infrastructure. Regarding offline messages, the channel is logged at http://www.extreme.st/jython.extreme.st/irclogs/index.php, regarding rich text might be more a matter of the client. For offline/async communication keep in mind there is still email...

[hanslovsky]

This is exciting and looks very promising! I was able to download the binaries and run them on my machine (Arch Linux). I was able to import numpy and tested basic functionality, e.g. this works:

import numpy
img = numpy.zeros( (3, 4) ) + 3
random = np.random.rand()

I was not able to generate a random int, though:

>>> np.random.randint(3)
Traceback (most recent call last):
  File "mtrand.pyx", line 970, in mtrand.RandomState.randint (numpy/random/mtrand/mtrand.c:15903)
TypeError: argument of type 'dict' is not iterable

Is this the appropriate location or should I create a separate issue?
If you have an intuition about where this would happen, could you point me to it? I would have a look at it myself, then.

[Stewori]

Philipp, thanks for testing JyNI!
Regarding the issue you observe, take a look at https://github.com/Stewori/JyNI#2-current-state. The "Planned" section lists the item "Support for remaining iterator types, e.g. TupleIter, ListIter".
I guess there is also something like PyDictIter under this scope. Support for this iterator-stuff would likely fix the issue you observe. That said, yes please open a separate issue for TypeError: argument of type 'dict' is not iterable. It's no numpy specific issue, but could arise in various situations. In case you want to help on this front (very welcome!), we can discuss further steps there.

[rjv2dlv]

Hi, I am trying to run the bumpy library in python from java. I see that jython 2.7.0 does not support this. I am not sure if jyni still supports this. Can you please let me know what is the best way to do this.

[Stewori]

bumpy is about deployment and does not seem to involve CExtensions. So this sounds a bit off-topic here. If there are issues with bumpy on Jython (please use 2.7.1 rather than 2.7.0 these days!), please bring it up at Jython issue tracker, or maybe -even better- at bumpy issue tracker.

If it was a typo and you're talking about numpy, make sure to use JyNI-alpha.4, Jython2.7.1 and NumPy >= 1.12. Download the JyNI binaries for your platform, if available. Ensure that JyNI.jar, the so-files and the NumPy folder are on Jython's sys.path. Also add CPython 2.7 dynload folder to sys.path, because NumPy requires ctypes. If you encounter issues, please open seperate threads for them, so we can better keep track.

Oh, and better try it with Jython-standalone for now.

[ghost]

@Stewori how's implementation of PyByteArray going? I am trying to get pandas to work

>>> import pandas as pd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vagrant/pandas-0.22.0/pandas/__init__.py", line 32, in <module>
    raise ImportError("C extension: {0} not built. If you want to import "
ImportError: C extension: /home/vagrant/pandas-0.22.0/pandas/_libs/tslib.so: undefined symbol: PyByteArray_Type not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.
>>> 

I am using jython-standalone-2.7.1.jar and JyNI-2.7-alpha.5-bin-linux-x86_64

[Stewori]

@ysz I currently cannot work much on JyNI, just doing the most necessary maintenance. However, I would happily review and accept PRs towards PyByteArray, BufferProtocol, SciPy and Pandas support.
Indeed, PyByteArray would be kind of the next step in this direction, but that should be implemented in context of overall support for the BufferProtocol. If you want to work on that, I can give hints and guide to get you started. Please open a separate issue for BufferProtocol related discussion.

[MarkoPaul0]

Hi everyone, I am hoping to get some help from someone in this thread.

My setup is pretty simple, I have a written a java application with a maven dependency on Jython-2.7.2. The java app calls a python script, and all things work as expected. The problem comes in when I want the script to use Numpy.

Now, before even trying to get the java/python app to work with numpy, I wanted to simply run a jython interpreter and see if I could import numpy from there, and I am not even able to pip install numpy. I tried various things here, but nothing seems to work. I would appreciate any help to get me started. At least getting to a point where I could do something like this successfully:

~/jython2.7.2 » java -cp "/Users/markus/Downloads/JyNI.jar:/Users/markus/Downloads/JyNI-2.7-alpha.5-bin-macosx-10.12-intel:jython.jar" org.python.util.jython                                                                  127 ↵ markus@mkbox
Jython 2.7.2 (v2.7.2:925a3cc3b49d, Mar 21 2020, 10:03:58)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java11.0.6
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named numpy
>>> 

In the example above, my understanding is that before I do the import, I should add the path to the numpy module, but as I said before, I cannot even install numpy as running the following fails:

~/jython2.7.2/bin/pip install 'numpy==1.13.3' 

Big thanks in advance to the charitable soul who's gonna be willing to help me on this :)

[Stewori]

Jython still prevents pip-installation of C-extensions, even if JyNI is running. Given that support is still fragile, it was so far not a priority to improve this. On Linux, easiest way is to build NumPy yourself in-place and add the paths to Jython's pythonpath by hand. On Windows, building NumPy is a pain. There it is easiest to install it for CPython and add the paths. I recommend to use a virtual env when installing NumPy 13.3 for CPython because you won't mess up your system NumPy then. Also, make sure to do this using Python 2.7.
See #21 (comment) and surrounding posts for some more hints.