ENH: _lib: add common machinery for low-level callback functions #6509

Merged
merged 32 commits into from Jan 4, 2017

Conversation

Projects
None yet
6 participants
@pv
Member

pv commented Aug 22, 2016

Replace the various approaches to combined Python and low-level compiled callback functions (e.g. in integrate.quad, ndimage, ...) with something better.

We need this part of a FFI layer in Scipy, because the FFI solutions in Python are not standard, and we want to interoperate with all of them. Moreover, FFI is not standard even across Scipy, and we want a reusable solution.
Supported are:

  • pure-python callbacks
  • cython "api" exported C functions
  • ctypes function pointers
  • pycapsule'd function pointers
  • cffi function pointers.

Caller/thunk code can be implemented in C or Cython.

I ended up with a design where the callback functions need to be explicitly wrapped by the user in LowLevelCallable object. This is because parsing e.g. ctypes callback signatures can take a few µs, so users may want to "precompile" them. Another reason is that if sometime in the future, PyPy becomes more important and Scipy switches to cffi etc, changing the design probably is easier.

The specific use cases in integrate.quad and ndimage still handle also the current "legacy" formats, to preserve backward compat.

Fixes gh-4831, gh-5002

  • test reentrancy/threadsafety rigorously
  • write cython .pxd wrapper for ccallback.h
  • decide if the (module, "name") cython syntax makes sense
  • decide if the (func, user_data) syntax makes sense
  • convert ndimage callbacks
  • maybe move everything from _ccallback.c to Cython? --- no need to do that in the end, leave some test code in C as a coding example

@pv pv referenced this pull request Aug 22, 2016

Merged

ENH: New ODE solvers #6326

2 of 2 tasks complete

@pv pv added the needs-work label Aug 22, 2016

@pv pv removed the needs-work label Sep 3, 2016

@pv pv changed the title from WIP: ENH: _lib: add common machinery for low-level callback functions to ENH: _lib: add common machinery for low-level callback functions Sep 3, 2016

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Sep 3, 2016

Member

I think this is more or less ready, comments welcome.
More controversial point may be where to put LowLevelCallable --- I put it in the top level namespace.

Member

pv commented Sep 3, 2016

I think this is more or less ready, comments welcome.
More controversial point may be where to put LowLevelCallable --- I put it in the top level namespace.

@nmayorov

This comment has been minimized.

Show comment
Hide comment
@nmayorov

nmayorov Sep 6, 2016

Contributor

It looks like a really great work.

I wonder if it would be possible to make an extended tutorial for LowLevelCallable where usage of each option from ctypes, cffi, Cython, or contained in PythonPyCapsuleobjects. will be shown on a single problem? It will be helpful for people who not well familiar with Python-C interaction (like myself).

Contributor

nmayorov commented Sep 6, 2016

It looks like a really great work.

I wonder if it would be possible to make an extended tutorial for LowLevelCallable where usage of each option from ctypes, cffi, Cython, or contained in PythonPyCapsuleobjects. will be shown on a single problem? It will be helpful for people who not well familiar with Python-C interaction (like myself).

@rgommers

This comment has been minimized.

Show comment
Hide comment
@rgommers

rgommers Sep 19, 2016

Member

More controversial point may be where to put LowLevelCallable --- I put it in the top level namespace.

We never put anything there, but in this case I think I agree - there's no other obvious place.

Member

rgommers commented Sep 19, 2016

More controversial point may be where to put LowLevelCallable --- I put it in the top level namespace.

We never put anything there, but in this case I think I agree - there's no other obvious place.

@rgommers

This comment has been minimized.

Show comment
Hide comment
@rgommers

rgommers Sep 19, 2016

Member

I wonder if it would be possible to make an extended tutorial

@nmayorov the last commit adds documentation which looks quite nice to me. See https://926-1460385-gh.circle-artifacts.com/0//home/ubuntu/scipy/doc/build/html-scipyorg/ccallback.html

Member

rgommers commented Sep 19, 2016

I wonder if it would be possible to make an extended tutorial

@nmayorov the last commit adds documentation which looks quite nice to me. See https://926-1460385-gh.circle-artifacts.com/0//home/ubuntu/scipy/doc/build/html-scipyorg/ccallback.html

@rgommers

This comment has been minimized.

Show comment
Hide comment
@rgommers

rgommers Sep 19, 2016

Member

Looks impressive @pv. Reviewing this PR is nontrivial - it's massive and pretty low-level. I'm tempted to just start using it and see if anything unexpected happens. Starting at the top of the diff somehow doesn't make too much sense to me. And I'll ping the people on gh-4831 who were interested.

Member

rgommers commented Sep 19, 2016

Looks impressive @pv. Reviewing this PR is nontrivial - it's massive and pretty low-level. I'm tempted to just start using it and see if anything unexpected happens. Starting at the top of the diff somehow doesn't make too much sense to me. And I'll ping the people on gh-4831 who were interested.

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Sep 19, 2016

Member

For reviewing, I would suggest starting by looking at:
.

  1. The new C code in __quadpack.h, compared to the old.
  2. The changes in C code in nd_image.c
  3. The contents of ccallback.h
  4. The suggested usage in the examples e.g. tutorial
  5. Think about whether the LowLevelCallable should be put inside quad()
    so that you could just pass in cffi/ctypes callables. But I think the
    current way is a good approach, because it solves the problem of how to
    pass in the void* pointer to additional data without having to add new
    keyword arguments to functions.
    .
    The rest of the code is mostly for testing the feature.
    .
    As I see it, the alternative to this would be converting our wrapper
    codes to cffi --- it appears to me to be the best of the FFI options for
    this purpose. However, I don't see how it could work together with Cython.
Member

pv commented Sep 19, 2016

For reviewing, I would suggest starting by looking at:
.

  1. The new C code in __quadpack.h, compared to the old.
  2. The changes in C code in nd_image.c
  3. The contents of ccallback.h
  4. The suggested usage in the examples e.g. tutorial
  5. Think about whether the LowLevelCallable should be put inside quad()
    so that you could just pass in cffi/ctypes callables. But I think the
    current way is a good approach, because it solves the problem of how to
    pass in the void* pointer to additional data without having to add new
    keyword arguments to functions.
    .
    The rest of the code is mostly for testing the feature.
    .
    As I see it, the alternative to this would be converting our wrapper
    codes to cffi --- it appears to me to be the best of the FFI options for
    this purpose. However, I don't see how it could work together with Cython.
doc/source/tutorial/ndimage.rst
--------------------------------------------
+ user_data = ctypes.c_double(shift)
+ ptr = ctypes.cast(ctypes.pointer(user_data), ctypes.c_void_p)
+ callback = LowLevelCallable.from_cython(example, "transform", ptr)

This comment has been minimized.

@person142

person142 Sep 19, 2016

Member

Doesn't this example also require adding a .pxd file to export the C api?

@person142

person142 Sep 19, 2016

Member

Doesn't this example also require adding a .pxd file to export the C api?

@person142

This comment has been minimized.

Show comment
Hide comment
@person142

person142 Sep 19, 2016

Member

Would it make sense to collect the examples into a single new tutorial page? Once this is in I imagine use of it is going to proliferate (seems like it's already in the works for ODE solvers), and it might save replication to have an authoritative guide to point to.

Member

person142 commented Sep 19, 2016

Would it make sense to collect the examples into a single new tutorial page? Once this is in I imagine use of it is going to proliferate (seems like it's already in the works for ODE solvers), and it might save replication to have an authoritative guide to point to.

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Sep 19, 2016

Member
Member

pv commented Sep 19, 2016

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Sep 19, 2016

Member
Member

pv commented Sep 19, 2016

@person142

This comment has been minimized.

Show comment
Hide comment
@person142

person142 Sep 19, 2016

Member

No, I think "cdef api" does the same magic.

So it does. That's a good trick to know about.

Member

person142 commented Sep 19, 2016

No, I think "cdef api" does the same magic.

So it does. That's a good trick to know about.

@dhirschfeld

This comment has been minimized.

Show comment
Hide comment
@dhirschfeld

dhirschfeld Sep 20, 2016

I tried building the branch but with no luck :( First I had to downgrade setuptools because of incorrect path quoting (IIUC) after which it got a lot further but finally failed with the below errors:

building 'scipy.ndimage._nd_image' extension
compiling C sources
creating build\temp.win-amd64-3.5\Release\scipy\ndimage
creating build\temp.win-amd64-3.5\Release\scipy\ndimage\src
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Iscipy\ndimage\src -IC:\bin\Anaconda3\lib\site-packages\numpy\core\include -Iscipy\_lib\src -IC:\bin\Anaconda3\lib\site-packages\numpy\core\include -IC:\bin\Anaconda3\include -IC:\bin\Anaconda3\include -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include\intel64" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mpi\intel64\bin\..\..\intel64\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\tbb\bin\..\include" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include\intel64" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mpi\intel64\bin\..\..\intel64\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\tbb\bin\..\include" /Tcscipy\ndimage\src\nd_image.c /Fobuild\temp.win-amd64-3.5\Release\scipy\ndimage\src\nd_image.obj
nd_image.c
c:\bin\anaconda3\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
C:\bin\Anaconda3\lib\site-packages\numpy\core\include\numpy/npy_3kcompat.h(179): warning C4244: '=': conversion from 'Py_ssize_t' to 'int', possible loss of data
scipy\_lib\src\ccallback.h(133): error C2061: syntax error: identifier 'jmp_buf'
scipy\_lib\src\ccallback.h(140): error C2059: syntax error: '}'
scipy\_lib\src\ccallback.h(260): warning C4133: 'function': incompatible types - from 'PyTypeObject *' to 'PyObject *'
scipy\_lib\src\ccallback.h(275): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(276): error C2037: left of 'c_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(277): error C2037: left of 'user_data' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(278): error C2037: left of 'signature_index' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(284): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(285): error C2037: left of 'c_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(286): error C2037: left of 'user_data' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(287): error C2037: left of 'signature_index' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(307): error C2037: left of 'signature_index' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(308): error C2037: left of 'signature_index' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(325): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(326): error C2037: left of 'c_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(327): error C2037: left of 'user_data' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(335): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(339): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(354): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(355): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(355): error C2198: 'ccallback__set_thread_local': too few arguments for call
scipy\_lib\src\ccallback.h(357): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(358): error C2037: left of 'c_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(359): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(284): error C2037: left of 'info_p' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(296): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(296): error C2198: 'PyObject_Call': too few arguments for call
scipy\ndimage\src\nd_image.c(320): error C2079: 'callback' uses undefined struct 'ccallback'
scipy\ndimage\src\nd_image.c(327): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(328): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(360): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(365): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(368): error C2224: left of '.info_p' must have struct/union type
scipy\ndimage\src\nd_image.c(373): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(374): error C2224: left of '.user_data' must have struct/union type
scipy\ndimage\src\nd_image.c(382): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(382): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(383): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(396): error C2037: left of 'info_p' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(407): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(407): error C2198: 'PyObject_Call': too few arguments for call
scipy\ndimage\src\nd_image.c(428): error C2079: 'callback' uses undefined struct 'ccallback'
scipy\ndimage\src\nd_image.c(435): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(436): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(468): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(473): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(476): error C2224: left of '.info_p' must have struct/union type
scipy\ndimage\src\nd_image.c(481): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(482): error C2224: left of '.user_data' must have struct/union type
scipy\ndimage\src\nd_image.c(489): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(489): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(490): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(572): error C2037: left of 'info_p' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(588): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(588): error C2198: 'PyObject_Call': too few arguments for call
scipy\ndimage\src\nd_image.c(614): error C2079: 'callback' uses undefined struct 'ccallback'
scipy\ndimage\src\nd_image.c(621): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(622): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(658): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(663): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(666): error C2224: left of '.info_p' must have struct/union type
scipy\ndimage\src\nd_image.c(671): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(672): error C2224: left of '.user_data' must have struct/union type
scipy\ndimage\src\nd_image.c(682): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(682): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(683): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
error: Command "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Iscipy\ndimage\src -IC:\bin\Anaconda3\lib\site-packages\numpy\core\include -Iscipy\_lib\src -IC:\bin\Anaconda3\lib\site-packages\numpy\core\include -IC:\bin\Anaconda3\include -IC:\bin\Anaconda3\include -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include\intel64" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mpi\intel64\bin\..\..\intel64\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\tbb\bin\..\include" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include\intel64" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mpi\intel64\bin\..\..\intel64\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\tbb\bin\..\include" /Tcscipy\ndimage\src\nd_image.c /Fobuild\temp.win-amd64-3.5\Release\scipy\ndimage\src\nd_image.obj" failed with exit status 2

NB: I quite possibly wasn't doing something right but thought I'd post the error in case it points to an actual problem with the windows build
NBB: This was with the latest Intel parallel studio 2017

dhirschfeld commented Sep 20, 2016

I tried building the branch but with no luck :( First I had to downgrade setuptools because of incorrect path quoting (IIUC) after which it got a lot further but finally failed with the below errors:

building 'scipy.ndimage._nd_image' extension
compiling C sources
creating build\temp.win-amd64-3.5\Release\scipy\ndimage
creating build\temp.win-amd64-3.5\Release\scipy\ndimage\src
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Iscipy\ndimage\src -IC:\bin\Anaconda3\lib\site-packages\numpy\core\include -Iscipy\_lib\src -IC:\bin\Anaconda3\lib\site-packages\numpy\core\include -IC:\bin\Anaconda3\include -IC:\bin\Anaconda3\include -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include\intel64" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mpi\intel64\bin\..\..\intel64\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\tbb\bin\..\include" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include\intel64" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mpi\intel64\bin\..\..\intel64\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\tbb\bin\..\include" /Tcscipy\ndimage\src\nd_image.c /Fobuild\temp.win-amd64-3.5\Release\scipy\ndimage\src\nd_image.obj
nd_image.c
c:\bin\anaconda3\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
C:\bin\Anaconda3\lib\site-packages\numpy\core\include\numpy/npy_3kcompat.h(179): warning C4244: '=': conversion from 'Py_ssize_t' to 'int', possible loss of data
scipy\_lib\src\ccallback.h(133): error C2061: syntax error: identifier 'jmp_buf'
scipy\_lib\src\ccallback.h(140): error C2059: syntax error: '}'
scipy\_lib\src\ccallback.h(260): warning C4133: 'function': incompatible types - from 'PyTypeObject *' to 'PyObject *'
scipy\_lib\src\ccallback.h(275): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(276): error C2037: left of 'c_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(277): error C2037: left of 'user_data' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(278): error C2037: left of 'signature_index' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(284): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(285): error C2037: left of 'c_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(286): error C2037: left of 'user_data' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(287): error C2037: left of 'signature_index' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(307): error C2037: left of 'signature_index' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(308): error C2037: left of 'signature_index' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(325): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(326): error C2037: left of 'c_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(327): error C2037: left of 'user_data' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(335): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(339): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(354): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(355): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(355): error C2198: 'ccallback__set_thread_local': too few arguments for call
scipy\_lib\src\ccallback.h(357): error C2037: left of 'prev_callback' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(358): error C2037: left of 'c_function' specifies undefined struct/union 'ccallback'
scipy\_lib\src\ccallback.h(359): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(284): error C2037: left of 'info_p' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(296): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(296): error C2198: 'PyObject_Call': too few arguments for call
scipy\ndimage\src\nd_image.c(320): error C2079: 'callback' uses undefined struct 'ccallback'
scipy\ndimage\src\nd_image.c(327): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(328): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(360): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(365): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(368): error C2224: left of '.info_p' must have struct/union type
scipy\ndimage\src\nd_image.c(373): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(374): error C2224: left of '.user_data' must have struct/union type
scipy\ndimage\src\nd_image.c(382): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(382): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(383): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(396): error C2037: left of 'info_p' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(407): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(407): error C2198: 'PyObject_Call': too few arguments for call
scipy\ndimage\src\nd_image.c(428): error C2079: 'callback' uses undefined struct 'ccallback'
scipy\ndimage\src\nd_image.c(435): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(436): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(468): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(473): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(476): error C2224: left of '.info_p' must have struct/union type
scipy\ndimage\src\nd_image.c(481): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(482): error C2224: left of '.user_data' must have struct/union type
scipy\ndimage\src\nd_image.c(489): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(489): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(490): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(572): error C2037: left of 'info_p' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(588): error C2037: left of 'py_function' specifies undefined struct/union 'ccallback'
scipy\ndimage\src\nd_image.c(588): error C2198: 'PyObject_Call': too few arguments for call
scipy\ndimage\src\nd_image.c(614): error C2079: 'callback' uses undefined struct 'ccallback'
scipy\ndimage\src\nd_image.c(621): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(622): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(658): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
scipy\ndimage\src\nd_image.c(663): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(666): error C2224: left of '.info_p' must have struct/union type
scipy\ndimage\src\nd_image.c(671): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(672): error C2224: left of '.user_data' must have struct/union type
scipy\ndimage\src\nd_image.c(682): error C2224: left of '.py_function' must have struct/union type
scipy\ndimage\src\nd_image.c(682): error C2224: left of '.c_function' must have struct/union type
scipy\ndimage\src\nd_image.c(683): warning C4133: 'function': incompatible types - from 'int *' to 'ccallback_t *'
error: Command "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Iscipy\ndimage\src -IC:\bin\Anaconda3\lib\site-packages\numpy\core\include -Iscipy\_lib\src -IC:\bin\Anaconda3\lib\site-packages\numpy\core\include -IC:\bin\Anaconda3\include -IC:\bin\Anaconda3\include -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include\intel64" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mpi\intel64\bin\..\..\intel64\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\tbb\bin\..\include" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" -I"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" -I"C:\Program Files (x86)\Windows Kits\10\include\10.0.10240.0\ucrt" -I"C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\shared" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\um" -I"C:\Program Files (x86)\Windows Kits\8.1\include\\winrt" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\include\intel64" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mpi\intel64\bin\..\..\intel64\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\mkl\include" -I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\tbb\bin\..\include" /Tcscipy\ndimage\src\nd_image.c /Fobuild\temp.win-amd64-3.5\Release\scipy\ndimage\src\nd_image.obj" failed with exit status 2

NB: I quite possibly wasn't doing something right but thought I'd post the error in case it points to an actual problem with the windows build
NBB: This was with the latest Intel parallel studio 2017

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Oct 13, 2016

Member

@dhirschfeld: may be just missing #include <setjmp.h>, added in 56f0925

Member

pv commented Oct 13, 2016

@dhirschfeld: may be just missing #include <setjmp.h>, added in 56f0925

@dhirschfeld

This comment has been minimized.

Show comment
Hide comment
@dhirschfeld

dhirschfeld Oct 17, 2016

Thanks @pv - that fixed the compilation so I can now test this branch on py35/win64.

I'm very interested in this branch as I have an expensive integral which I'm hoping to speedup using the numba cfunc functionality. I haven't been able to make full use of it so far because quad hasn't allowed passing in extra parameters correctly when using the ctypes interface (IIUC).

In [4]: from scipy.integrate import quad

In [5]: def f1(x, mu):
   ...:     return np.exp(-x**2)/(1+np.exp(-(x-mu)))

In [6]: quad(f1, -np.inf, np.inf, args=(1.1,))
Out[6]: (0.47866266846942096, 5.656862062202448e-09)

In [7]: quad(f1, -np.inf, np.inf, args=(1.9,))
Out[7]: (0.26575640518956256, 2.7307507646266408e-09)

I implemented this using numbas cfunc as follows:

In [58]: from numba import types
    ...: from numba import carray, cfunc, float64
    ...: from scipy import LowLevelCallable

In [59]: @cfunc(float64(float64, types.voidptr))
    ...: def f2(x, ptr_mu):
    ...:     mu = carray(ptr_mu, (1,), dtype=float64)[0]
    ...:     return np.exp(-x**2)/(1+np.exp(-(x-mu)))

In [60]: mu = ctypes.c_double(1.1)
    ...: mu_ptr = ctypes.cast(ctypes.pointer(mu), ctypes.c_void_p)
    ...: f = LowLevelCallable(f2.ctypes, mu_ptr)

In [61]: quad(f, -np.inf, np.inf)
Out[61]: (0.47866266846942096, 5.656862062202448e-09)

In [62]: mu = ctypes.c_double(1.9)
    ...: mu_ptr = ctypes.cast(ctypes.pointer(mu), ctypes.c_void_p)
    ...: f = LowLevelCallable(f2.ctypes, mu_ptr)

In [63]: quad(f, -np.inf, np.inf)
Out[63]: (0.26575640518956256, 2.7307507646266408e-09)

...with a resulting 16x speedup!

In [73]: %timeit quad(f, -np.inf, np.inf)
10000 loops, best of 3: 53.4 µs per loop

In [74]: %timeit quad(f1, -np.inf, np.inf, args=(1.9,))
1000 loops, best of 3: 859 µs per loop

In [75]: quad(f1, -np.inf, np.inf, args=(1.9,))[0] == quad(f, -np.inf, np.inf)[0]
Out[75]: True

🎉

Thanks @pv - that fixed the compilation so I can now test this branch on py35/win64.

I'm very interested in this branch as I have an expensive integral which I'm hoping to speedup using the numba cfunc functionality. I haven't been able to make full use of it so far because quad hasn't allowed passing in extra parameters correctly when using the ctypes interface (IIUC).

In [4]: from scipy.integrate import quad

In [5]: def f1(x, mu):
   ...:     return np.exp(-x**2)/(1+np.exp(-(x-mu)))

In [6]: quad(f1, -np.inf, np.inf, args=(1.1,))
Out[6]: (0.47866266846942096, 5.656862062202448e-09)

In [7]: quad(f1, -np.inf, np.inf, args=(1.9,))
Out[7]: (0.26575640518956256, 2.7307507646266408e-09)

I implemented this using numbas cfunc as follows:

In [58]: from numba import types
    ...: from numba import carray, cfunc, float64
    ...: from scipy import LowLevelCallable

In [59]: @cfunc(float64(float64, types.voidptr))
    ...: def f2(x, ptr_mu):
    ...:     mu = carray(ptr_mu, (1,), dtype=float64)[0]
    ...:     return np.exp(-x**2)/(1+np.exp(-(x-mu)))

In [60]: mu = ctypes.c_double(1.1)
    ...: mu_ptr = ctypes.cast(ctypes.pointer(mu), ctypes.c_void_p)
    ...: f = LowLevelCallable(f2.ctypes, mu_ptr)

In [61]: quad(f, -np.inf, np.inf)
Out[61]: (0.47866266846942096, 5.656862062202448e-09)

In [62]: mu = ctypes.c_double(1.9)
    ...: mu_ptr = ctypes.cast(ctypes.pointer(mu), ctypes.c_void_p)
    ...: f = LowLevelCallable(f2.ctypes, mu_ptr)

In [63]: quad(f, -np.inf, np.inf)
Out[63]: (0.26575640518956256, 2.7307507646266408e-09)

...with a resulting 16x speedup!

In [73]: %timeit quad(f, -np.inf, np.inf)
10000 loops, best of 3: 53.4 µs per loop

In [74]: %timeit quad(f1, -np.inf, np.inf, args=(1.9,))
1000 loops, best of 3: 859 µs per loop

In [75]: quad(f1, -np.inf, np.inf, args=(1.9,))[0] == quad(f, -np.inf, np.inf)[0]
Out[75]: True

🎉

@dhirschfeld

This comment has been minimized.

Show comment
Hide comment
@dhirschfeld

dhirschfeld Oct 17, 2016

...since it's slightly non-trivial it might make for a good example - but that's possibly beter located in the numba docs?

cc: @pitrou

...since it's slightly non-trivial it might make for a good example - but that's possibly beter located in the numba docs?

cc: @pitrou

@dhirschfeld

This comment has been minimized.

Show comment
Hide comment
@dhirschfeld

dhirschfeld Oct 18, 2016

When there's more than one extra argument to the callback it gets even more convoluted:

In [46]: def f1(x, mu, alpha):
    ...:     return np.exp(-alpha*x**2)/(1+np.exp(-(x-mu)))

In [47]: mu = 1.9
    ...: alpha = 0.77
    ...: args = np.asarray((mu, alpha), dtype=[('mu', np.float64),('alpha', np.float64)])

In [48]: quad(f1, -np.inf, np.inf, args=(mu, alpha))
Out[48]: (0.3138771115843293, 3.516492370046489e-09)
In [49]: dtype = args.dtype

In [50]: @cfunc(float64(float64, types.voidptr))
    ...: def f2(x, args):
    ...:     args = carray(args, (1,), dtype=dtype)
    ...:     mu = args.mu[0]
    ...:     alpha = args.alpha[0]
    ...:     return np.exp(-alpha*x**2)/(1+np.exp(-(x-mu)))

In [51]: f = LowLevelCallable(f2.ctypes, ctypes.c_voidp(args.ctypes.data))
    ...: quad(f, -np.inf, np.inf)
Out[51]: (0.31387711158432935, 3.5164923763348615e-09)
In [52]: %timeit quad(f1, -np.inf, np.inf, args=(mu, alpha))
1000 loops, best of 3: 661 µs per loop

In [53]: %timeit quad(f, -np.inf, np.inf)
10000 loops, best of 3: 41.8 µs per loop

AFAICS, since there's no concept of a void pointer to a struct of data in python, it's fairly awkward to use with e.g. numba.

I'm wondering if, instead of the signature LowLevelCallable(function, void*) it was instead LowLevelCallable(function, *args) - if your c-func did actually take a void* then you could pass that as the single arg, otherwise you could pass whatever types the callback was expecting. Would that even be possible?

When there's more than one extra argument to the callback it gets even more convoluted:

In [46]: def f1(x, mu, alpha):
    ...:     return np.exp(-alpha*x**2)/(1+np.exp(-(x-mu)))

In [47]: mu = 1.9
    ...: alpha = 0.77
    ...: args = np.asarray((mu, alpha), dtype=[('mu', np.float64),('alpha', np.float64)])

In [48]: quad(f1, -np.inf, np.inf, args=(mu, alpha))
Out[48]: (0.3138771115843293, 3.516492370046489e-09)
In [49]: dtype = args.dtype

In [50]: @cfunc(float64(float64, types.voidptr))
    ...: def f2(x, args):
    ...:     args = carray(args, (1,), dtype=dtype)
    ...:     mu = args.mu[0]
    ...:     alpha = args.alpha[0]
    ...:     return np.exp(-alpha*x**2)/(1+np.exp(-(x-mu)))

In [51]: f = LowLevelCallable(f2.ctypes, ctypes.c_voidp(args.ctypes.data))
    ...: quad(f, -np.inf, np.inf)
Out[51]: (0.31387711158432935, 3.5164923763348615e-09)
In [52]: %timeit quad(f1, -np.inf, np.inf, args=(mu, alpha))
1000 loops, best of 3: 661 µs per loop

In [53]: %timeit quad(f, -np.inf, np.inf)
10000 loops, best of 3: 41.8 µs per loop

AFAICS, since there's no concept of a void pointer to a struct of data in python, it's fairly awkward to use with e.g. numba.

I'm wondering if, instead of the signature LowLevelCallable(function, void*) it was instead LowLevelCallable(function, *args) - if your c-func did actually take a void* then you could pass that as the single arg, otherwise you could pass whatever types the callback was expecting. Would that even be possible?

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Oct 18, 2016

Member
Member

pv commented Oct 18, 2016

doc/source/tutorial/integrate.rst
1.) Write an integrand function in C with the function signature
-``double f(int n, double args[n])``, where ``args`` is an array containing the
+``double f(int n, double *args)``, where ``args`` is an array containing the

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

That's imprecise (and also forgets mentioning the void * user data). What "arguments" are those? The integration variables or any ancillary arguments?

@pitrou

pitrou Oct 18, 2016

That's imprecise (and also forgets mentioning the void * user data). What "arguments" are those? The integration variables or any ancillary arguments?

scipy/_lib/_ccallback.py
+
+try:
+ import cffi
+ ffi = cffi.FFI()

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

Beware that this can be costly:

>>> %time import cffi; cffi.FFI().CData
CPU times: user 24 ms, sys: 0 ns, total: 24 ms
Wall time: 21.7 ms
@pitrou

pitrou Oct 18, 2016

Beware that this can be costly:

>>> %time import cffi; cffi.FFI().CData
CPU times: user 24 ms, sys: 0 ns, total: 24 ms
Wall time: 21.7 ms
scipy/_lib/src/_test_ccallback.c
+ * This also is an internal "best-practices" code example on how to write
+ * low-level callback code.
+ *
+ * The *thunk_simple* function shows how to write the wrapper callback

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

Who would need to write such a thunk function?

@pitrou

pitrou Oct 18, 2016

Who would need to write such a thunk function?

scipy/_lib/src/_test_ccallback.c
+
+/* Thunks for the different cases considered */
+
+static double test_thunk_simple(double a, int *error_flag, void *data)

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

Why void *? Is this signature normalized somewhere?

@pitrou

pitrou Oct 18, 2016

Why void *? Is this signature normalized somewhere?

scipy/_lib/src/_test_ccallback.c
+ PyGILState_Release(state);
+ }
+ else {
+ if (callback->signature_index == 0) {

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

Hmm, this looks obscure to me.

@pitrou

pitrou Oct 18, 2016

Hmm, this looks obscure to me.

+ }
+
+ if (error) {
+ *error_flag = 1;

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

I'm a bit surprised that *error_flag isn't set to 0 explicitly otherwise. Is it best practice?

@pitrou

pitrou Oct 18, 2016

I'm a bit surprised that *error_flag isn't set to 0 explicitly otherwise. Is it best practice?

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Oct 18, 2016

Member
Member

pv commented Oct 18, 2016

+ return (ccallback_t *)ccallback__get_thread_local();
+}
+
+#else

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

MSVC also has a thread-local extension that you can use: https://github.com/numba/numba/blob/master/numba/_random.c#L108

@pitrou

pitrou Oct 18, 2016

MSVC also has a thread-local extension that you can use: https://github.com/numba/numba/blob/master/numba/_random.c#L108

scipy/_lib/src/ccallback.h
+ lowlevelcallable_type = (PyTypeObject *)PyObject_GetAttrString(module, "LowLevelCallable");
+ if (lowlevelcallable_type == NULL) {
+ goto error;
+ }

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

You may want to decref the module object after this.

@pitrou

pitrou Oct 18, 2016

You may want to decref the module object after this.

+ }
+
+ if ((flags & CCALLBACK_PARSE) && !PyObject_TypeCheck(callback_obj, lowlevelcallable_type)) {
+ /* Parse callback */

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

Out of curiosity, why is this conditioned on CCALLBACK_PARSE?

@pitrou

pitrou Oct 18, 2016

Out of curiosity, why is this conditioned on CCALLBACK_PARSE?

+ goto error;
+ }
+
+ callback_obj = callback_obj2;

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

If I'm not mistaken, callback_obj2 is a new reference, while callback_obj was borrowed.

@pitrou

pitrou Oct 18, 2016

If I'm not mistaken, callback_obj2 is a new reference, while callback_obj was borrowed.

This comment has been minimized.

@rgommers

rgommers Jan 3, 2017

Member

@pv this comment still seems open, not clear to me that it was addressed

@rgommers

rgommers Jan 3, 2017

Member

@pv this comment still seems open, not clear to me that it was addressed

+
+ if (PyCallable_Check(callback_obj)) {
+ /* Python callable */
+ callback->py_function = callback_obj;

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

This seems to imply callback_obj should be an owned reference (not borrowed).

@pitrou

pitrou Oct 18, 2016

This seems to imply callback_obj should be an owned reference (not borrowed).

This comment has been minimized.

@rgommers

rgommers Jan 3, 2017

Member

@pv this comment still seems open, not clear to me that it was addressed

@rgommers

rgommers Jan 3, 2017

Member

@pv this comment still seems open, not clear to me that it was addressed

scipy/_lib/src/ccallback.h
+ callback->signature_index = -1;
+ }
+ else if (PyObject_TypeCheck(callback_obj, lowlevelcallable_type) &&
+ PyTuple_Check(callback_obj) &&

This comment has been minimized.

@pitrou

pitrou Oct 18, 2016

I'm probably misreading this but... how can callback_obj be a tuple and a LowLevelCallable instance at the same time? (unless LowLevelCallable is a tuple subclass)

Edit: ok, LowLevelCallable is a tuple subclass :-) Then the check is redundant (though it did wake my curiosity).

@pitrou

pitrou Oct 18, 2016

I'm probably misreading this but... how can callback_obj be a tuple and a LowLevelCallable instance at the same time? (unless LowLevelCallable is a tuple subclass)

Edit: ok, LowLevelCallable is a tuple subclass :-) Then the check is redundant (though it did wake my curiosity).

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Oct 18, 2016

Member
Member

pv commented Oct 18, 2016

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Oct 18, 2016

Member
Member

pv commented Oct 18, 2016

pv added some commits Aug 24, 2016

ENH: _lib: move ccallback parsing code to pure Python
The performance difference is not very significant (a few us).  Add a
public LowLevelCallable class that can be used to "precompute" the
low-level function pointer and its signature.
ENH: _lib/ccallback: specify signatures with an auxiliary value rathe…
…r than index

Since several signatures can be equivalent, it's more convenient to
refer to them by user-specified codes, rather than by their index in the
list of signatures.

@rgommers rgommers added this to the 0.19.0 milestone Jan 3, 2017

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Jan 4, 2017

Member
Member

pv commented Jan 4, 2017

@rgommers

This comment has been minimized.

Show comment
Hide comment
@rgommers

rgommers Jan 4, 2017

Member

Okay, thanks for confirming.

Member

rgommers commented Jan 4, 2017

Okay, thanks for confirming.

@rgommers

This comment has been minimized.

Show comment
Hide comment
@rgommers

rgommers Jan 4, 2017

Member

Let's get this in then - has been thoroughly reviewed, seems to be in good shape, and would be good to have it in master for a few weeks before branching 0.19.x.

Member

rgommers commented Jan 4, 2017

Let's get this in then - has been thoroughly reviewed, seems to be in good shape, and would be good to have it in master for a few weeks before branching 0.19.x.

@rgommers

This comment has been minimized.

Show comment
Hide comment
@rgommers

rgommers Jan 4, 2017

Member

Or do you still want to do anything about this comment regarding numba ints:#4933 ?

Member

rgommers commented Jan 4, 2017

Or do you still want to do anything about this comment regarding numba ints:#4933 ?

@pv

This comment has been minimized.

Show comment
Hide comment
@pv

pv Jan 4, 2017

Member
Member

pv commented Jan 4, 2017

@rgommers rgommers merged commit cc9f6b2 into scipy:master Jan 4, 2017

3 checks passed

ci/circleci Your tests passed on CircleCI!
Details
codecov/project 80.64% (target 1.00%)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@rgommers

This comment has been minimized.

Show comment
Hide comment
@rgommers

rgommers Jan 4, 2017

Member

Okay in it goes. Thanks Pauli! And thanks everyone for the reviews!

Member

rgommers commented Jan 4, 2017

Okay in it goes. Thanks Pauli! And thanks everyone for the reviews!

@gfyoung gfyoung referenced this pull request in pandas-dev/pandas Jan 4, 2017

Closed

ENH: Accept callable for skiprows in read_csv #15059

@jreback jreback referenced this pull request in pandas-dev/pandas2 Mar 9, 2017

Open

ufunc specifications #68

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment