umath fails to load

[lokuz]

umath fails to load with the message: AttributeError: _ARRAY_API not found

Cause for this is a bug in JyNI-C/src/Python/import.c in the function PyObject * PyImport_ImportModule(const char *name). The current implementation calls directly the import method from Jythons' builtin class. This leads to the fact, that function calls like
PyImport_ImportModule("A.B.C")
will return the top module A instead of A.B.C as it should be.

In umath this is location in the numpy/core/src/umath/umathmodule.c file within the numpy package
. In the function initumath in the line

PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray");

the variable numpy will ironically be the numpy package and not the multiarray subpackage. Which leads to the observed error, if later the attribute "_ARRAY_API" is checked with this line:

c_api2 = PyObject_GetAttrString(numpy, "_ARRAY_API");
The solution to this bug is to change the function PyImport_ImportModule(const char name) to call PyImport_Import(PyObject module_name) as in the cpython implementation and uncomment the original PyImport_Import(PyObject* module_name) in JyNI-C/src/Python/import.c and let it call the Jython methods after the silly_list has been built up.

In order to do this a comment how these lines

r = PyObject_CallFunction(import, "OOOOi", module_name, globals,
                              globals, silly_list, 0, NULL);

should be called in Jython would be very helpful.

[Stewori]

After adding .../numpy/core folder to sys.path I can do
import umath
without failure. Like with issues #7 and #8, can you provide a minimal code sample that triggers this issue for you?

[Stewori]

Same behavior with self-built current NumPy.
Also: I scanned through umathmodule.c and it appears there is no line
PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray");
and actually no call PyImport_ImportModule in the entire file.
Is there a typo in the issue description?

[Stewori]

With the latest commit that enables raising of import-exceptions in JyNI imports the issue is now also visible on my system. Here it shows up as
ImportError: numpy.core.multiarray failed to import
I'll now try to follow your hints for the cause...

[Stewori]

You did quite a good job in identifying the source of this issue. I think I fixed loading of dotted module names in 66a74e0.
This actually solves the _ARRAY_API problem.

However umath still doesn't load; now it segfaults at a later point. I'll continue on this front tomorrow or so.

[lokuz]

Great. Just another step forward. I will try your fix at the weekend and report the behaviour with my setup. Same with the other issue.

I would then continue to look at the next bug on the way to load umath. I will keep you up to date to avoid duplicate work.

[Stewori]

Sounds good. I already started investigation a bit, but it appears to be a difficult bug. It segfaults at about four different possible locations and has also a small chance of passing through, like your reported about multiarray.
Only thing I found out so far is that the crash always relates to the type
numpy.ufunc
Sometimes during creation, sometimes during cleanup. My suggestion would be to reduce the function initumath step by step from the end until a working partial implementation was discovered. Then we can expand from that point and fix problems one by one. Hopefully this then sorts the possible points of failure a bit.

[Stewori]

Also note that since import umath triggers an import numpy.core.multiarray, Jython will also attempt to import numpy and numpy.core, i.e. it will process numpy/__init__.py and numpy/core/__init__.py, which is both currently not feasible. To focus on umath in this thread I mostly out-commented these init-procedures and you should do so as well. Of course they need to be made workable too, but that should be investigated in distinct issue threads.

[Stewori]

Okay, already fixed it as of 238ba94 I suppose. Took some time, but I traced it around until I concluded it's because ufunc uses PyMem_Malloc (indirectly via PyArray_Malloc) for object allocation. My original assumption was that only PyObject_Malloc would be used for PyObject allcoation while PyMem_Malloc was only intended for low-level purpose memory. However once recognized it could be fixed by modifying PyMem_Malloc and friends accordingly.
Now umath module loads without error on my system, given that __init__.py module initializers of numpy and numpy.core are stripped down to a workable portion for now.
(If you can confirm this you can close this issue then...)

[lokuz]

In my case the program stops, when trying to load _ctypes. But JyNIctypesDemo.sh is also not running. It looks right now, that my setup with the ctypes library is not working. I have to look into that before I a can give an answer to the status of this issue.

[Stewori]

In case it's not just because of an invalid setup, please file a ctypes-scoped issue here. Getting ctypes support stable has even higher priority than NumPy.

But JyNIctypesDemo.sh is also not running.

What about JyNI-Demo/src/test_JyNI_ctypes.py? (as called by JyNI_unittest.sh)

[lokuz]

Was just a setup error which I had fixed for the last fork as well, but forgot to mention. On Suse Linux the dyn-load path is different.

This line

lib_dynload = '/usr/lib64/python2.7/lib-dynload'

in JyNI-Lib/ctypes/init.py fixes it. Just add "64".

The error message was: ImportError: No module named _ctypes

[Stewori]

Yes, managing Python-path/Jython-path/Java-classpath whatever properly for JyNI is a topic for beta phase I suppose. In general this topic is more complex than just adding the right path, because we will also have to resolve conflicts between native modules and Jython-supplied versions (e.g. datetime and also ctypes so far). Actually this is something we should maybe also discuss on Jython-dev at its time.

I suppose your message implies, umath now loads on your system...?

[lokuz]

An import of umath is not succesfull. The program just stops at one point. Reminds me of the deadlock I described in another issue, but I have seen you added the fix. :) Have to look into the details to give a meaningful answer.

BTW: import numpy results in this error: NameError: name 'NUMPY_SETUP' is not defined

[Stewori]

Okay. Looking forward to hear about the location it hangs on! Try to add debug output to ENTER_JYNI and LEAVE_JYNI in JyNI.h if not yet done.

Regarding NUMPY_SETUP', let's first identify the script location that triggers this (in a distinct issue would be ideal).

[lokuz]

Still looking into this. Debug output is influencing the behavior of the program control. Therefore limiting down the location is possible, but does not lead to the cause of the error.

[Stewori]

Did you apply jstack to identify the deadlock when it is hanging? (Please consider to post the plain jstack-output here too, so I can also have a look.)
Sensitivity to log-output points to a parallelity-related issue (usually this is main thread vs something gc-related). In such a case I usually out-comment code step-wise to identify the latest stable point. Also, feel free to post to what you limited down the location.
I am currently working to fix another deadlock bug (gc again) that popped up in a JyOpenGL test. Probably it's better if you wait for this commit, just in case it's the same bug. Hope to get it done until tomorrow evening.

[Stewori]

(fixed the other (maybe unrelated) deadlock issue as of the last two commits)

[Stewori]

BTW: import numpy results in this error: NameError: name 'NUMPY_SETUP' is not defined

This side-issue is fixed as of ed06aa7.

Now import numpy fails at a later point with this symbol-error:
symbol lookup error: /data/workspace/linux/numpy/numpy/core/umath.so: undefined symbol: PyThreadState_GetDict

[Stewori]

Fixed the symbol error
symbol lookup error: /data/workspace/linux/numpy/numpy/core/umath.so: undefined symbol: PyThreadState_GetDict as of 9fa540f.

Now there are issues regarding static vs. dynamic memory detection. scalartypes.c declares boolean-singletons:

NPY_NO_EXPORT PyBoolScalarObject _PyArrayScalar_BoolValues[] = {
    {PyObject_HEAD_INIT(&PyBoolArrType_Type) 0},
    {PyObject_HEAD_INIT(&PyBoolArrType_Type) 1},
};

JyNI segfaults on these, because its current detection approach misconcepts them as dynamically allocated PyObjects. It appears that numpy always calls PyObject_Init on dynamically created objects. I will try to improve the detection mechanism based on this fact.

[Stewori]

With 54c8e16 import numpy passes properly on my system. It would be good if you could check this issue after a fresh pull. Also see #2 (comment)
(Only works with latest numpy 1.12 dev-version from repository)

[Stewori]

Given that JyNI significantly changed since the last activity here and I cannot reproduce this issue any more for two months or so now, I will close it.
If you encounter problems again, feel free to re-open.