PyArray_API is *not* a unique symbol. (Trac #1928) #2521

Closed
numpy-gitbot opened this Issue Oct 19, 2012 · 3 comments

Comments

Projects
None yet
3 participants
@numpy-gitbot

Original ticket http://projects.scipy.org/numpy/ticket/1928 on 2011-08-10 by trac user phil@..., assigned to @pearu.

numpy makes the API to its array type available through the symbol PyArray_API. But it also allows users to make it available through some other symbol by defining PY_ARRAY_UNIQUE_SYMBOL to be whatever other symbol is desired. (Why anyone would want to use a different symbol name never became clear to me, but some do.)

__multiarray_api.h has some C preprocessor magic that defines the symbol one way if PY_ARRAY_UNIQUE_SYMBOL is defined, and another way if PY_ARRAY_UNIQUE_SYMBOL is not defined. It all works well, except...

...In the numpy header fortranobject.h, PY_ARRAY_UNIQUE_SYMBOL is defined as PyArray_API. This makes it look as if a different symbol is being used, but the "different" symbol being used is the same as the "real" symbol for the API. This results in anyone including fortranobject.h (scipy does) having a definition of PyArray_API that does not play well at all with the "real" global definition of PyArray_API.

I was sometimes getting a crash when invoking "import scipy.signal" I tracked it down to:

  • signal imports special (directly or indirectly)
  • import of special results in call of C function initspecfun()
  • initspecfun() calls _import_array
  • _import_array executes this line of code:
    PyArray_API = (void **)PyCObject_AsVoidPtr(c_api);

It was this statement that was causing trouble. My debugger was telling me that the value being returned from PyCObject_AsVoidPtr was the same as the address of PyArray_API. In other words, after the above statement, &PyArray_API == PyArray_API is true. I'm pretty sure that's not good. It resulted in the first attempt to access the contents of PyArray_API (a call to PyArray_GetNDArrayCVersion() a few lines later) crashing.

The crash only happens if I do the "import scipy.signal" after I import my own C-python module that uses numpy arrays. I suppose it could be a bug in my code, but I can't see how. My source file just has a "#include <numarray/libnumarray.h>", without define any optional symbols (no PY_ARRAY_UNIQUE_SYMBOL, no NO_IMPORT or NO_IMPORT_ARRAY). It does not anywhere (directly) reference PyArray_API. The only thing the init routine does with numpy is call _import_array. Of course, my module has routines that do stuff with numpy arrays, but I don't call any of those between importing my module and importing scipy.signal to get the crash. If I am doing something wrong, then consider this to be a bug report on the docs, which have not made it clear what I am doing wrong. (I've looked. A bit.)

I tried to fix the problem by removing the #define of PY_ARRAY_UNIQUE_SYMBOL in fortranobject.h. That did indeed prevent the crash when I import my module followed by scipy.signal. However, if I import scipy.signal without first importing my module, the attempt to import scipy.signal raises in ImportError with complaint of undefined symbol PyArray_API.

I then tried to fix it by changing the define of PY_ARRAY_UNIQUE_SYMBOL in fortranobject.h to something (anything) other than PyArray_API. That worked. I am now able to import scipy.signal whether or not I first import my module. Is that the "right" fix? You decide.

Python 2.5.4
numpy 1.5.1rc2
scipy 0.8.0

@numpy-gitbot

This comment has been minimized.

Show comment
Hide comment
@numpy-gitbot

numpy-gitbot Oct 19, 2012

trac user phil@... wrote on 2011-08-11

I looked into the problem a little more. It was indeed something my module was doing that was causing the problem, but it had nothing to do with my module using numpy arrays. It was because the python wrapper around my C-python module was sharing symbols across modules. The problem is easily reproduced thusly:

>>> import sys
>>> import DLFCN
>>> sys.setdlopenflags(sys.getdlopenflags() | DLFCN.RTLD_GLOBAL)
>>> import scipy.signal
Segmentation fault (core dumped)

What I said about changing the definition of PY_ARRAY_UNIQUE_SYMBOL in fortranobject.h still fixes it, and I still think it might be a good idea to check in the change. But that's just my unexpert opinion...

trac user phil@... wrote on 2011-08-11

I looked into the problem a little more. It was indeed something my module was doing that was causing the problem, but it had nothing to do with my module using numpy arrays. It was because the python wrapper around my C-python module was sharing symbols across modules. The problem is easily reproduced thusly:

>>> import sys
>>> import DLFCN
>>> sys.setdlopenflags(sys.getdlopenflags() | DLFCN.RTLD_GLOBAL)
>>> import scipy.signal
Segmentation fault (core dumped)

What I said about changing the definition of PY_ARRAY_UNIQUE_SYMBOL in fortranobject.h still fixes it, and I still think it might be a good idea to check in the change. But that's just my unexpert opinion...

@bfroehle

This comment has been minimized.

Show comment
Hide comment
@bfroehle

bfroehle Feb 24, 2013

Contributor

For a project I'm working on I'm currently needing to build a statically compiled Python binary with several extension modules built in (NumPy, SciPy, etc).

I'm experiencing this crash as well when importing a SciPy module:

bfroehle@hopper10:~> gdb --quiet --args pycomplete-frozen -c "import scipy.sparse.linalg"
Reading symbols from /global/project/projectdirs/m1080/hopper/bin/pycomplete-frozen...done.
(gdb) run
Starting program: /global/project/projectdirs/m1080/hopper/bin/pycomplete-frozen -c import\ scipy.sparse.linalg
[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
0x00000000054f7708 in PyArray_API ()
(gdb) print PyArray_API
$1 = (void **) 0x54f7700
(gdb) print PyArray_API[0]@10
$2 = {0x54f7700, 0x5738420, 0x54f7fe0, 0x54f8180, 0x54f8320, 0x54f84c0, 0x54f8660, 0x5736d40, 0x54f8800, 0x54f89a0}
(gdb) watch *0x54f7700
Hardware watchpoint 1: *0x54f7700
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /global/project/projectdirs/m1080/hopper/bin/pycomplete-frozen -c import\ scipy.sparse.linalg
[Thread debugging using libthread_db enabled]
Hardware watchpoint 1: *0x54f7700

Old value = 12983584
New value = 89093888
_import_array () at /global/project/projectdirs/m1080/hopper/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1564
1564      Py_DECREF(c_api);
(gdb) print (unsigned int (*)(void)) 12983584
$3 = (unsigned int (*)()) 0xc61d20 <PyArray_GetNDArrayCVersion>
(gdb) print (void**) 89093888
$4 = (void *) 0x54f7700
(gdb) up
#1  init_iterative () at build/src.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/sparse/linalg/isolve/iterative/_iterativemodule.c:5739
5739      import_array();
(gdb) down
#0  _import_array () at /global/project/projectdirs/m1080/hopper/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1564
1564      Py_DECREF(c_api);
(gdb) list
1559          Py_DECREF(c_api);
1560          return -1;
1561      }
1562      PyArray_API = (void **)PyCObject_AsVoidPtr(c_api);
1563    #endif
1564      Py_DECREF(c_api);
1565      if (PyArray_API == NULL) {
1566          PyErr_SetString(PyExc_RuntimeError, "_ARRAY_API is NULL pointer");
1567          return -1;
1568      }

The cause appears to be the same. If PyArray_API == NULL on entering the routine, things seem to be okay. Otherwise somehow we end up with PyArray_API[0] == PyArray_API which is clearly wrong.

Contributor

bfroehle commented Feb 24, 2013

For a project I'm working on I'm currently needing to build a statically compiled Python binary with several extension modules built in (NumPy, SciPy, etc).

I'm experiencing this crash as well when importing a SciPy module:

bfroehle@hopper10:~> gdb --quiet --args pycomplete-frozen -c "import scipy.sparse.linalg"
Reading symbols from /global/project/projectdirs/m1080/hopper/bin/pycomplete-frozen...done.
(gdb) run
Starting program: /global/project/projectdirs/m1080/hopper/bin/pycomplete-frozen -c import\ scipy.sparse.linalg
[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
0x00000000054f7708 in PyArray_API ()
(gdb) print PyArray_API
$1 = (void **) 0x54f7700
(gdb) print PyArray_API[0]@10
$2 = {0x54f7700, 0x5738420, 0x54f7fe0, 0x54f8180, 0x54f8320, 0x54f84c0, 0x54f8660, 0x5736d40, 0x54f8800, 0x54f89a0}
(gdb) watch *0x54f7700
Hardware watchpoint 1: *0x54f7700
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /global/project/projectdirs/m1080/hopper/bin/pycomplete-frozen -c import\ scipy.sparse.linalg
[Thread debugging using libthread_db enabled]
Hardware watchpoint 1: *0x54f7700

Old value = 12983584
New value = 89093888
_import_array () at /global/project/projectdirs/m1080/hopper/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1564
1564      Py_DECREF(c_api);
(gdb) print (unsigned int (*)(void)) 12983584
$3 = (unsigned int (*)()) 0xc61d20 <PyArray_GetNDArrayCVersion>
(gdb) print (void**) 89093888
$4 = (void *) 0x54f7700
(gdb) up
#1  init_iterative () at build/src.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/sparse/linalg/isolve/iterative/_iterativemodule.c:5739
5739      import_array();
(gdb) down
#0  _import_array () at /global/project/projectdirs/m1080/hopper/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1564
1564      Py_DECREF(c_api);
(gdb) list
1559          Py_DECREF(c_api);
1560          return -1;
1561      }
1562      PyArray_API = (void **)PyCObject_AsVoidPtr(c_api);
1563    #endif
1564      Py_DECREF(c_api);
1565      if (PyArray_API == NULL) {
1566          PyErr_SetString(PyExc_RuntimeError, "_ARRAY_API is NULL pointer");
1567          return -1;
1568      }

The cause appears to be the same. If PyArray_API == NULL on entering the routine, things seem to be okay. Otherwise somehow we end up with PyArray_API[0] == PyArray_API which is clearly wrong.

@bfroehle

This comment has been minimized.

Show comment
Hide comment
@bfroehle

bfroehle Feb 24, 2013

Contributor

Related to #1746 as well.

Contributor

bfroehle commented Feb 24, 2013

Related to #1746 as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment