Skip to content

Implement back end of faster multivariate integration #3262

Closed
wants to merge 6 commits into from

10 participants

@BrianNewsom

This PR implements my recent work on the backend of scipy.integrate, adding:
A lower level (C) method of handling multivariate functions so that they may be passed to the quadpack routines
C Wrappers for all double fortran quadpack routines supporting this function type
A simple additional wrapper that allows the wrappers to be incorporated into the existing SciPy integration, for testing purposes.

1.) Currently Scipy’s integration is inhibited in that it’s multivariate integration requires nested calls between python and fortran that are slow. The intent of my work (which is not fully completed) is to allow the movement of the innermost-loop integration entirely to compiled languages, which should be much faster.

The first step of my solution is the cwrapper.h file, which converts a multivariate function of the form f(int nargs, double args[nargs]) to a function of a single variable - the form expected by quadpack. This is handled nearly exactly as it is currently done in Python, storing the original function and any additional arguments to global variables, then using them from inside a generic “call” function passed into quadpack.

An example function of this type would be:

    double f(int nargs, double args[nargs]){
        args[0] * args[1] + args[2] * args[2]
     }

Which would be equivalent to f(x) = xy+zz but now allows us to pass the number of extra arguments (2) and their values in the args array so that they may be integrated in fortran.

2.) This file then additionally includes C wrappers for each of quadpack’s double routines that handle this evaluation. I added wrappers for every routine, for future use if necessary, though the current schema seems to only use about 6 of them. These can be removed if superfluous. Between these two elements the foundation is laid for handling the n-dimensional integration at this lower level.

3.) Additionally, so that these wrappers may be tested and used, an additional wrapper has been added. This wrapper allows the currently handled python and ctypes functions to be evaluated in the f(nargs,args) form with another simple initialization and call. These functions are funcwrapper_init and funcwrapper, and these have been applied in the __quadpack.h file inside each routine so that the cwrapper is being applied as will be necessary with future additions. These functions should be removed, once SciPy’s integration module is adapted to use the new-style wrappers, as described earlier.

What does this mean for the end user?
As of now, nothing. No functionality has changed, quad works exactly the same way as before, passing all existing unit tests going through my wrappers. It can still handle ctypes or python functions of any number of variables, and this handling is still quite slow. Because of this, no additional documentation has been included.

Why does this PR matter, then?
This code adds a framework for doing exactly what I described earlier on, moving the handling of multiple variables down into a compiled language, which should provide a significant speed advantage to the power user, once a framework is written that allows ctypes or cython (or both) functions of this new format to be passed directly into the wrappers, without changing anything for the traditional user.

I am a first time contributor to SciPy but believe this will be an important contribution and am open to any advice, criticism, or comments on the code or process.

@rgommers
SciPy member
rgommers commented Feb 1, 2014

This seems to have a merge issue. Could you rebase it onto latest master?

@rgommers
SciPy member
rgommers commented Feb 1, 2014
@rgommers
SciPy member
rgommers commented Feb 1, 2014

If there's no user-visible changes yet, I'd prefer to merge this after branching 0.14.x (planned for ~ last week of Feb)

@pv
SciPy member
pv commented Feb 1, 2014

I think it will be necessary to have a more complete plan for the rest of the feature, before we can consider merging this. The important questions:

  • how does the user pass in low-level functions to quad()?
  • how are these represented on the Python side? ctypes?

Notes on the present code:

  • use of global variables like this is not reentrant or threadsafe
  • that non-reentrancy doesn't break the test suite is due to the fact that funcwrapper_init is always called with the same function quad_function. The following test program crashes now:
from scipy import integrate
import ctypes
lib = ctypes.CDLL('libm.so')
lib.sin.restype = ctypes.c_double
lib.sin.argtypes = (ctypes.c_double,)
def b(y):
    return y + integrate.quad(lib.sin, 0, 1)[0]
print(integrate.quad(b, 0, 1))
  • C coding style needs refinement. Please run the cwrapper.h through GNU indent with indent -kr -bad -sc -nce cwrapper.h to make it more readable. Also, prefer foo_bar_quux over foobarquux in variable naming.
  • The functions dq* are not directly callable by users from C code, so it's not clear what is gained by adding another wrapper layer to this set of functions.
@woodscn
woodscn commented Feb 1, 2014

I'm working with Brian on this, and I think I can answer some of the questions. We suspected the reentrancy problem, but neither of us is really clear on how to resolve it. We'll be looking at Oliphant's original implementation to see if we can figure it out, but any direction we can get on the issue would be welcome.

As for the plan going forward. We think the preferred end-goal is to allow users to code their functions in cython, since that's the direction SciPy seems to be going, however getting there from the status quo isn't trivial, and will likely take a few steps. Our immediate next-step would be a very small change in end-user functionality: getting a multivariate ctypes function of the form f(nargs,args[nargs]) to pass directly to quadpack, in much the same way as is possible now. This builds on already-implemented concepts and ideas, and is a small enough project to be manageable for us. Additionally, it provides a good entry point for any future improvements, since any new code will have to provide a pointer to a function f(nargs args[nargs]), anyway, if it's going to use this backend.

This is deep water for us already, so any advice and/or help would be welcome.

@pv
SciPy member
pv commented Feb 3, 2014

The trick in making it re-entrant is to save a copy of the global state before calling the function, and restore it afterward (this is what the code currently in quad does). This is most convenient to do by putting all of the state in a struct (or, better, making the global a pointer to a struct). To make it threadsafe, one should use thread-local storage instead, but I'm not sure what's the portable way to do that.

The simplest way to proceed for the rest is probably (i) write a class ScipyLowLevelFunction in Python, that does the checking and preprocessing currently done in get_func_type in the quad() wrappers, plus extend it for functions accepting extra arguments. (ii) make a clone of the mechanism currently in quad() and modify it so that it makes use of the checked results computed by ScipyLowLevelFunction, in addition to normal Python functions. For dealing with extra arguments, a number of trampoline functions similar to your call are necessary, one for each combination of possible additional argument types (plus one for usual Python functions).

@coveralls

Coverage Status

Changes Unknown when pulling d1fc3c3 on BrianNewsom:cwrapping into ** on scipy:master**.

@BrianNewsom

Sorry about the messed up history, everyone involved. Very strange things are happening and I am working on getting them fixed.

Brian

@coveralls

Coverage Status

Changes Unknown when pulling 1bb717d on BrianNewsom:cwrapping into ** on scipy:master**.

@coveralls

Coverage Status

Changes Unknown when pulling 1bb717d on BrianNewsom:cwrapping into ** on scipy:master**.

@coveralls

Coverage Status

Changes Unknown when pulling 2803f3e on BrianNewsom:cwrapping into ** on scipy:master**.

@coveralls

Coverage Status

Changes Unknown when pulling 2803f3e on BrianNewsom:cwrapping into ** on scipy:master**.

@coveralls

Coverage Status

Changes Unknown when pulling 2803f3e on BrianNewsom:cwrapping into ** on scipy:master**.

@BrianNewsom

Hello all,
I believe that I have taken in all of your comments and remedied the situations. I rebased and fixed the ugly commits.

I have also fixed the re-entrancy issue as described by @pv . The cwrapper.h file has been removed, as further examination shows that it creates overhead and the solution doesn't necessitate those wrappers.

So now I just have the extra two wrappers both allowing re-entrancy so that functions of form f(n, args[n]) can be accepted. I agree that steps should be taken towards getting that function pointer in cython, but as @thezealite described, immediate next steps should be allowing ctypes of this form as is done now for f(*x) functions.

Brian

@pv pv added the PR label Feb 19, 2014
@BrianNewsom

Hello,

This PR now is functional, as it detects the ctypes function signature and handles it appropriately, producing a 2x increase in speed in the integrations I've tested so far.

Use:
Function written in c:

double new(int n, double args[n]){
    return args[0]*args[0] + args[1]*args[1] + args[0] * args[1] * args[2] + args[3]*args[3]*args[3]      +args[4];
}

Compile to library
Set restype and argtypes as in ctypes 1d version in Python:

lib = ctypes.CDLL('lib.so')
lib.new.restype = ctypes.c_double
lib.new.argtypes = (ctypes.c_int,ctypes.c_double*4)

As in 1d ctypes, run nquad on the function:

integrate.nquad(testlib.new,[[-1,1],[-1,1],[-1,1],[-1,1],[1,1]])

For this specific example the integration occurs at about a 1.9x faster speed.

I will try and get some test cases into the integrate unit testing when I get a chance. In the meantime, feel free to try it out and/or take a look at the code, and please bring up any concerns or deficiencies found.

Thanks,
Brian

@coveralls

Coverage Status

Coverage remained the same when pulling 0be6bc0 on BrianNewsom:cwrapping into dfd2e2b on scipy:master.

@coveralls

Coverage Status

Coverage remained the same when pulling 0be6bc0 on BrianNewsom:cwrapping into dfd2e2b on scipy:master.

@woodscn
woodscn commented Feb 28, 2014

Brian, I notice that the Travis CI build failed. You should try to get that to finish.

@coveralls

Coverage Status

Coverage remained the same when pulling e3704e5 on BrianNewsom:cwrapping into c81c978 on scipy:master.

@coveralls

Coverage Status

Changes Unknown when pulling 6eaefc7 on BrianNewsom:cwrapping into ** on scipy:master**.

@coveralls

Coverage Status

Changes Unknown when pulling fb42e11 on BrianNewsom:cwrapping into ** on scipy:master**.

@coveralls

Coverage Status

Changes Unknown when pulling dd8d3d9 on BrianNewsom:cwrapping into ** on scipy:master**.

@coveralls

Coverage Status

Changes Unknown when pulling 7b1e4e6 on BrianNewsom:cwrapping into ** on scipy:master**.

@woodscn
woodscn commented Apr 9, 2014

Building a unit test for this feature seems difficult, since it would require a C library that could be loaded by ctypes. How would such a library be distributed? The need for a cython version of this is becoming more and more clear.

Any further comments? Has anyone had a chance to look over this more closely?

@coveralls

Coverage Status

Changes Unknown when pulling 5e06ea6 on BrianNewsom:cwrapping into ** on scipy:master**.

@coveralls

Coverage Status

Changes Unknown when pulling d8fa178 on BrianNewsom:cwrapping into ** on scipy:master**.

@coveralls

Coverage Status

Changes Unknown when pulling 611615d on BrianNewsom:cwrapping into ** on scipy:master**.

@rkern
SciPy member
rkern commented Apr 29, 2014

Please follow the stdlib's ctypes test more closely. Make _test_multivariate an actual Python extension module. Import this module in the test suite. Use its __file__ attribute to pass to CDLL(); you don't need to do all of that filename munging to guess the name. You must make sure to EXPORT the functions that you are exposing for this to work on Windows; again look at the stdlib's ctypes test module for the appropriate macros.

@woodscn
@woodscn
woodscn commented Apr 30, 2014

All right, first things first. I've been feeling somewhat ungrateful, especially to @rkern, who has been extremely helpful the last few days with this. Once I figured out what was going on (and why), your comments seem spot on, so thanks again for being patient with me.

Second, I noticed that the way modules are implemented in the C API changed with Python 3. I think I've figured it out for Python 2, which you can see in a PR to Brian's fork. I'm not sure if/how that might work with Python 3, though, or how that problem should be handled.

Nathan Woods

@coveralls

Coverage Status

Coverage remained the same when pulling fcea3af on BrianNewsom:cwrapping into d9a8c21 on scipy:master.

@coveralls

Coverage Status

Coverage remained the same when pulling c68abf7 on BrianNewsom:cwrapping into d9a8c21 on scipy:master.

@coveralls

Coverage Status

Coverage remained the same when pulling 0847c38 on BrianNewsom:cwrapping into d9a8c21 on scipy:master.

@rgommers
SciPy member

I've done some testing, and it works for me on 32-bit Linux (after the build fix I sent BrianNewsom#3 for). Except for this spurious failure, which happened only once in ~100 test runs:

======================================================================
FAIL: test_improvement (test_quadpack.TestMultivariateCtypesQuad)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rgommers/.local/lib/python2.7/site-packages/numpy/testing/decorators.py", line 146, in skipper_func
    return f(*args, **kwargs)
  File "/home/rgommers/Code/scipy/scipy/integrate/tests/test_quadpack.py", line 121, in test_improvement
    assert_(fast < .666 * slow, (fast, slow))
  File "/home/rgommers/.local/lib/python2.7/site-packages/numpy/testing/utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: (0.004848003387451172, 0.005922079086303711)

It looks the same as gh-2209. You see that fast is still faster than slow, just not fast enough. This kind of test should probably simply use inputs that cause longer runtimes; 5 ms isn't a lot compared to timing variations due to other processes. Or time.time is just not that reliable. I suggest to change the test to run 10x slower, then merge and see if there are still failures reported.

@rgommers rgommers added this to the 0.15.0 milestone May 28, 2014
@rgommers
SciPy member

Can you rebase this? Then it'll be mergeable again, and all the merge commits will be gone. It would be good to also remove c5b5db9 and 5d648a0

@woodscn
woodscn commented May 28, 2014

Brian pointed out to me that there is an additional test that sees a much higher speedup using more complex functions. Should we just remove the simple math.sin() test entirely, and rely on the more complex one?

@rgommers
SciPy member

Sure, that makes sense. Please do verify that the check has some room for timing jitter. Right now the comment says 20+ times speedup and the check is assert_(fast < 0.05*slow).

@WarrenWeckesser

My $0.02: A unit test of relative performance does not seem like a good idea. How do you determine what the passing threshold should be? If you are just running a comparison now and adjusting the relative improvement with a fudge factor, the test will be brittle. It might fail in the future, because of changes in the platform, or the hardware used, or the version of python, or whatever. Is it really a "failure" if the calculation is only 5 times or even 1.5 times as fast instead of 10 or 20 times?

These comparisons should be benchmarks and not unit tests.

But I respect @rgommers and @pv's judgement. If they think the tests are solid, I won't complain beyond this comment.

@rgommers
SciPy member

@WarrenWeckesser this PR just copies the approach from gh-190. On that PR I made pretty much the same comment about the timing test, but no one came up with a better idea. Travis had a rationale for keeping this test.

@rkern
SciPy member
rkern commented May 28, 2014

If you want to test that it's using the function pointer instead of doing normal Python function call, implement a C function that records when it gets called.

@argriffing

vbench was written for checking performance regressions

@rkern
SciPy member
rkern commented May 28, 2014

The purpose of the test is not to track performance regressions, but to make sure that the fast path of calling the function pointer is being chosen rather than the slow path of calling the Python-side of the ctypes function.

@rgommers
SciPy member

@rkern thanks, that's of course a much better idea.

I propose to not require that test change to be done in this PR though, given that this copies an existing test strategy.

@coveralls

Coverage Status

Changes Unknown when pulling 8a0bd5d on BrianNewsom:cwrapping into ** on scipy:master**.

@woodscn
woodscn commented May 28, 2014

Sorry, we're struggling with git at the moment, trying to remove the reverted commit from the history.

@rkern
SciPy member
rkern commented May 28, 2014

@rgommers Never mind, it doesn't make sense. No matter which branch quad() goes down, the C function always get called. Hmm.

@woodscn
woodscn commented May 28, 2014

Is this what you had in mind as far as removing the commits? https://github.com/woodscn/scipy/commits/cwrapping
Working on the rebase.

@woodscn
woodscn commented May 28, 2014

The rebase is giving me awful headaches. The best I've come up with so far is to (by hand) selectively apply the appropriate commits on top of an updated master using git cherry-pick. git rebase has dozens of confusing conflicts, and we're just generally not sure how to deal with it all.

Simply pulling the changes from scipy master works fine, so there's no real conflict there, just something rebase doesn't like.

cherry-pick is how we got rid of the extra commits, too.

Is there some easy way of doing this that I'm just not understanding?

N

@rgommers
SciPy member

@rkern right, you would then have to do something slightly ugly. Maybe make the C function take a callback which can be a Python function that uses sys._getframe to check that we went straight to C.

The time.time approach starts looking good:)

@rgommers
SciPy member

@woodscn something went wrong here, I'll have a look at the rebase.

@rgommers
SciPy member

@pv are your concerns regarding this PR resolved?

@rgommers
SciPy member

Rebase at https://github.com/rgommers/scipy/tree/pr/3262-rebase. The problem is that you merged scipy master into your branch several times, you should never do that. Either just keep working on your branch, or rebase it on master at some point.

I still think that the rebased branch doesn't look good, there's the same commits a bunch of times, with reverts in between. I can squash it a bit more or just re-commit the final patch in a few sensible parts. Anyone have a preference?

@woodscn
woodscn commented May 29, 2014

Yeah, although using rebase makes perfect sense to me now that I've thought about it, the strong preference for merge in most of the online git guides makes it confusing. Anyway, now I know.

No preference here about how to handle the final patch, though.

@BrianNewsom

I agree, the weird commits and repeated commits and things look really ugly. I know I somehow managed to cause that, but I'm not sure how, I think when I was rebasing earlier on or something. I apologize for my ineptness when it comes to the nontrivial functionalities of git.

I don't have any preference, whichever is cleaner and/or easier to do.

@rgommers
SciPy member

@BrianNewsom no apology needed. I think we all have made a mess with git at some point - it's not that hard to do:)

I have a preference for cleaning this up by re-committing things in logical chucks. I'll do that once we are ready to merge this, unless someone disagrees with that approach.

@woodscn
woodscn commented May 30, 2014

Just so we're clear, did you want us to clean this all up, or were you doing it? We're more than happy to take care of it, if you like.

@rgommers
SciPy member

@woodscn if you want to do it, that would be helpful. I was thinking maybe 3 commits:

  • new functionality
  • tests
  • THANKS and 0.15.0 release notes (the latter is not in here yet, a couple of sentences are justified though)

You should use git commit --author=... to ensure the right author name is kept. Looks like that should be Brian for functionality and Nathan for tests.

@woodscn
woodscn commented May 31, 2014

I've put things together and pushed it to https://github.com/woodscn/scipy/tree/fix_cwrapping. I've asked Brian to look over how I split up author attribution before it's final, but I'd like to make sure that I got the right idea, if anyone wants to look it over quickly.

@rgommers
SciPy member
rgommers commented Jun 1, 2014

That branch looks good to me.

@BrianNewsom

Looks great to me!

@rgommers
SciPy member
rgommers commented Jun 2, 2014

@BrianNewsom could you push that branch to this PR?

BrianNewsom and others added some commits May 31, 2014
@BrianNewsom BrianNewsom ENH: Implement faster multivariate integration in scipy.integrate.
The numerical integration routines in SciPy use a well-tested, fast, Fortran
library. The computational cost of integration with these routines can be
greatly reduced if the function to be integrated is compiled and passed to
the library directly, avoiding callbacks to the Python interpreter.
2af2939
@BrianNewsom BrianNewsom TST: Preliminary implementation of unit tests for faster integration. 6325033
@woodscn woodscn TST: Cross-platform build and install of unit tests. ecd1a52
@woodscn woodscn DOC: Updated THANKS.txt and scipy 0.15.0 release notes. 4b01be5
@woodscn woodscn TST: fix mistake in release notes 51bbf4b
@BrianNewsom

@rgommers I believe it's done now!

@coveralls

Coverage Status

Coverage increased (+0.0%) when pulling 51bbf4b on BrianNewsom:cwrapping into 9f89371 on scipy:master.

@coveralls

Coverage Status

Coverage increased (+0.0%) when pulling 51bbf4b on BrianNewsom:cwrapping into 9f89371 on scipy:master.

@ev-br
SciPy member
ev-br commented Jun 2, 2014

A comment from the peanut gallery for this PR: it would be good to have some tutorial / user-manual type docs for this feature. For example, I as a user have an expensive function to integrate, what do I do? If I already have a function in cython (or Fortran), I can always wrap it up as a python callable and feed it to quad, is there a better way now? What about multivariate functions, what is the status of dblquad, tplquad, nquad?

@coveralls

Coverage Status

Coverage increased (+0.0%) when pulling 51bbf4b on BrianNewsom:cwrapping into 9f89371 on scipy:master.

@woodscn
woodscn commented Jun 3, 2014

As a quick answer to Evgeni, the docs for quad and nquad include give some quick instructions. From nquad:

  • If speed is desired, this function may be a ctypes function of
    • the form:: +
    • f(int n, double args[n]) +
    • where n is the number of extra parameters and args is an array
    • of doubles of the additional parameters. This function may then
    • be compiled to a dynamic/shared library then imported through
    • ctypes, setting the function's argtypes to (c_int, c_double),
    • and the function's restype to (c_double). Its pointer may then be
    • passed into nquad normally.

Unfortunately, ctypes is the only interface that works right now, so if you have a Cython or Fortran function, you would have to find some way to wrap it with ctypes first. With Fortran that's not so much of a problem. We've been looking at Cython, but haven't had any luck with that yet.

This was explicitly designed for use with multivariate functions, especially with nquad. Since the changes are the level of the interface between quad and quadpack, they should work just fine with dblquad and tplquad too.

The performance increases here come from two main sources. First, faster evaluation of compiled vs interpreted functions. Second, reduced overhead because the quadpack library doesn't have to callback to Python to evaluate the integrand function. If those are bottlenecks in a particular application, then it may be worthwhile for people to use the extra ctypes path. In my case, I need to evaluate volume integrals of messy analytic functions, and the speed improvements from compiling the integrand and cutting overhead in the innermost loop are significant.

As for including a tutorial of sorts, we could try to put something together, but my first thought is that there are a lot of ways for someone to get a ctypes pointer, so it would either be incredibly general, or else it would encourage specific design choices.

What are everyone's thoughts? If there's nothing else, is this ready to merge?

@BrianNewsom

@ev-br What @woodscn is saying is correct. Although I would like to expand on the cython question.

I believe this route provides two (primary) advantages over just compiling to Cython and passing it into quad - as this will work already.

1.) Removal of Python callbacks. I'm pretty sure that although Cython compiles to C, the way it's handled requires calls to Python for evaluation at each inner iteration of the integration loop. The compilation will provide speed increases, but the time necessary for callbacks between langauges remains because of the way the quadpack code is written as of now. This may be possible to change, but that is another topic.

2.) Ease of use. For users that have not worked with Cython much (e.g. me), the ctypes method presented here consists of a fairly concise set of steps that can be repeated without delving too much into what is going on below. Maybe that is just my familiarity with Ctypes, but it allows a user to avoid the Cython building infrastructure which can be confusing.

Everything I wrote applying to Cython also is true for Fortran using, say, F2PY. I have run timing tests and such on this and this method DOES provide a ~2x increase on simply compiled fortran, where this is provided by the removal of python/c callbacks.

@ev-br
SciPy member
ev-br commented Jun 3, 2014

@BrianNewsom @woodscn You seem to be implying that I criticize your design decisions or question the choice of ctypes or advocate for switching over to cython --- I am not.

All I am saying is that I think it would make sense to add an example of using this feature to eg this tutorial: http://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html
This addition need not be large. A slightly expanded example from the docstrings of quad and nquad would already be very helpful for us mere mortals with nasty functions to integrate and without extensive experience with ctypes pointers.

Note that I am not saying this needs to be done, and I do not mean to block or postpone merging this PR. I just think this feature should be advertised more, and that now seems to be a good time to write things up.

@BrianNewsom

@ev-br I apologize if that came off as defensive, I was just trying to explain why this could be an improvement over cython - why it is a viable new option.

I absolutely agree that we could add something into the tutorial, it is definitely a slightly confusing feature meant for adventurous users and it would be great to throw a more explicit and elaborate explanation into the documentation. Thanks for the suggestion!

@BrianNewsom

Added a more explicit explanation in the tutorials section. Take a look, let me know if this is helpful.

@coveralls

Coverage Status

Coverage increased (+0.0%) when pulling c8f742d on BrianNewsom:cwrapping into 9f89371 on scipy:master.

@ev-br
SciPy member
ev-br commented on c8f742d Jun 4, 2014

It's definitely helpful, thanks! I intend to try it out --- when I've time (tm), and it is unlikely to happen until next week at the earliest. At any rate, it shouldn't postpone merging this PR. The doc tweaks can be done separately.

@woodscn
woodscn commented Jun 8, 2014

Is this just waiting for merge now?

@rgommers
SciPy member
rgommers commented Jun 8, 2014

@woodscn I think so. I'll try to make some time this week (but I'm traveling, may take a few days).

@woodscn
woodscn commented Jun 17, 2014

Pinging again.

@BrianNewsom

If you'd like me to rollback the most recent commit (it was suggested to have it done separately) just let me know. Otherwise I'll leave it.

@rgommers
SciPy member

Leave it in I'd say - doc updates are an integral part of PRs like this, and it helps me to play with the code.

@woodscn
woodscn commented Jul 2, 2014

Ping!

@rgommers
SciPy member
rgommers commented Jul 5, 2014

Tutorial looks good to me. I verified the statements on speedup: example given is ~1.5x while np.sin(args[0] - np.exp(-args[1])) is more than 10x faster.

@rgommers rgommers added a commit that referenced this pull request Jul 5, 2014
@rgommers rgommers Merge branch 'pr/3262' into master.
Adds support for direct callbacks to C functions via ctypes
for multivariate integration.

Reviewed at #3262
04cc74f
@rgommers
SciPy member
rgommers commented Jul 5, 2014

Squashed the two commits that touched the release notes, fixed tutorial formatting and merged in 04cc74f.

Thanks a lot all!

@rgommers rgommers closed this Jul 5, 2014
@rgommers
SciPy member
rgommers commented Jul 5, 2014

Build fails on Python 3.4, which has been enabled on TravisCI after running the last tests on this PR.

I'll try to fix it tomorrow (football will get in the way tonight). Should be straightforward:

In file included from scipy/integrate/_quadpackmodule.c:4:0:
scipy/integrate/quadpack.h: In function ‘c_array_from_tuple’:
scipy/integrate/quadpack.h:131:5: error: ISO C90 forbids mixed declarations and
code [-Werror=declaration-after-statement]
@woodscn
woodscn commented Jul 6, 2014

I tried to see if I could fix it, but building on my machine didn't reproduce the error, and I didn't want to mess without being able to check the results.

@rgommers
SciPy member
rgommers commented Jul 6, 2014

Looks like this is due to http://bugs.python.org/issue21121; -Werror=declaration-after-statement was by accident added to flags that distutils passes on for 3.4. Is already reversed. I can't reproduce it either, but I'll still send a fix.

@rgommers
SciPy member
rgommers commented Jul 6, 2014

fixed in gh-3774

@pitrou pitrou commented on the diff Apr 14, 2016
scipy/integrate/quadpack.h
+ store->global1 = global_n_args;
+ store->arg = global_args;
+
+ /*Set new parameters */
+ if ((global_function = get_ctypes_function_pointer(f)) == NULL) {
+ PyErr_SetString(quadpack_error,
+ "Ctypes function not correctly initialized");
+ return NPY_FAIL;
+ }
+ if ((global_args = c_array_from_tuple(args)) == NULL) {
+ PyErr_SetString(quadpack_error,
+ "Extra Arguments must be in a tuple");
+ return NPY_FAIL;
+ }
+ int n_args_ref = PyTuple_Size(args);
+ global_n_args = &n_args_ref;
@pitrou
pitrou added a note Apr 14, 2016

This stores a global pointer to a local variable that will soon go out of scope. Clearly the value dereferenced later will be non-sensical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.