-
Notifications
You must be signed in to change notification settings - Fork 201
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
boost::python::numpy::initialize() incompatibilities with numpy>=1.21 #376
Comments
This is connected to changes in numpy type casting implemented in these two numpy PRs: numpy/numpy#17401 and numpy/numpy#18676. For hacking around with the Numpy sources the following Dockerfile might be convenient: Dockerfile
After numpy commit ba77419b10, this procedure gives the segfault described here; then after e108c76 the Not sure where to take it from here, but I hope this is helpful... |
Made some progress on this. First things first, this Dockerfile will provide debug symbols for numpy, boost_python, python, etc: Dockerfile
Here is the segfault:
This is an excerpt from the function body:
In my understanding
and this is what we get instead when
So it really looks like a numpy bug in the new type casting implementation. I'm not sure exactly what should be the correct behavior, it appears there's a legacy system that has been preserved so maybe that is still usable for types that don't have a data member |
This fixes it! Will open a numpy PR.
If an urgent fix is needed it's possible to clone numpy sources, apply this diff, rebuild numpy and symlink the result inside e.g. |
cc @rgommers (for awareness) |
Maybe floating it here is more helpful: The fix above seems like a bad idea to me, it is a small fix to cover up some of the symptoms of a much larger problem. I am still struggling to have a good local reproducer, +but the problem comes from here: Lines 161 to 162 in f5d14ef
The second line seems OK (if shady). The first line gets some NB: The |
To be fair, it seems likely that whatever that But I would still like to understand what triggers this problem and clarify that whatever triggers means that this is an invalid |
Just to see if I can get this moving here:
The fix is to just delete the I may add a compat hack to NumPy, but even if, that cannot stick around forever and it is clearly a |
I am proposing a "fix" in NumPy in numpy/numpy#20616. If merged, it should prevent the crash, but behave identically to just deleting the whole " Quite simply: this must be fixed in |
@stefanseefeld do you have thoughts here? 🙂 |
I'm looking into it. Meanwhile, @TallJimbo , do you remember the origin (and rationale) behind this code ? |
Thanks. The code makes some sense: NumPy scalars can wrap e.g. C Trying to use |
@seberg nailed the original motivation. It is entirely possible that my use of |
This reflects the resolution of #627 as discussed in several other issues and PRs: - boostorg/python#376 - numpy/numpy#20507 - numpy/numpy#20616 Leaving this "bibliography" here because the fix in numpy PR 20616 is considered temporary; thus someday we may have to revisit this to fix the underlying bug in boost::python. Co-authored-by: Billy Poon <bkpoon@lbl.gov>
This reflects the resolution of #627 as discussed in several other issues and PRs: - boostorg/python#376 - numpy/numpy#20507 - numpy/numpy#20616 Leaving this "bibliography" here because the fix in numpy PR 20616 is considered temporary; thus someday we may have to revisit this to fix the underlying bug in boost::python. Co-authored-by: Billy Poon <bkpoon@lbl.gov>
Sorry to bring this up again, but this error is appearing again in the numpy 2.0 release candidates because the fix was removed in this commit numpy/numpy@a914726#diff-71aadf0228c9cd61ea260748af1dcd4a726eea8476b8cbd667367466c1e39077 What can be a way forward? Thanks! |
Would it be super hard to just fix it in boost::python? I don't really like the idea of dragging around a hack in NumPy for another 3 years and 2.0 needs everyone to bump boost versions anyway?! EDIT: To be clear: this should be easy to fix so the main question may be how hard it is to make a bug-fix release that includes it soon enough to not block projects on. |
I cleared some items off my plate so I can try to solve this over the next few weeks. On conda-forge, it should be pretty easy to patch the currently pinned version of the Boost conda package since compatibility with Numpy 2.0 was previously added (conda-forge/boost-feedstock#198). @jakirkham would know more. I'm guessing the approach would be to change the |
@bkpoon, the no-op change is to just use |
Hi all, Sorry for the delay, I was traveling for a conference. I made a pull request to reduce the check to directly comparing the pointers (thanks @seberg). It does seem to work for me. I built a patched version of the Boost conda packages and tested it on my use cases (we were getting segmentation faults for C++ arrays we were exposing to Python via Boost). It does work for us. Prior to the fix, our Numpy 2 migration on conda-forge fails. I adjusted that PR to pull my patched Boost libraries from my channel and the segmentation faults go away. The patched Boost packages are available here. @stefanseefeld can you review the PR? I completely removed the old code path, but we can use the Thanks! |
Thanks for all the work ! (And despite the actual patch to look quite simple, I have been removed from low-level Python & Numpy code in recent years that I find it increasingly difficult to review such changes. Let's see when the next window of opportunity is to merge the |
I'll get the pull requests for @stefanseefeld Feel free to ping me if there are issues. Thanks! |
Summary
The following anomaly applies to boost::python operating with NumPy version >= 1.21.0; prior to this version the degradation doesn't occur:
Symptom
If a
BOOST_PYTHON_MODULE
callsboost::python::numpy::initialize()
, boost::python's parameter type detection feature is partially-defeated. This affects any method or function type checking and is particularly noticeable in overloads. If an incorrect parameter type (such as an object) ispassed to a method/function that expects an intrinsic int, float, or string, a segfault occurs instead of
the helpful boost error throw. In the event of an overload, the overload-dispatcher may send a legitimate parameter to the incorrect overload instance.
These cases are exercised in Python example 1 and Python example 2 below, respectively.
Relevant Software Versions Information:
For Failure:
(several were tried, this is representative)
For Success(expected behavior):
just change numpy:
C++ sample:
minipy.cpp
Build with standard boost libraries/paths. in this case for me, conda venv = compile2:
Observations:
Python example 1:
(This is fast to reproduce, so a command line python may be simpler)
Expected output:
Incorrect output:
Segmentation fault: 11
Python example 2 -- boost overloading failure:
Expected output:
Incorrect output:
Observation: It appears from this example that the numpy registration in numpy >= 1.21
is defeating boost::python's type matching for overloads. If the overloads are declared
in the opposite order, the code works regardless of the numpy version. Given the last-in-first-checked order
of boost::python dispatcher, add(double) is checked first in the code example above. Normally this is fine, but if numpy>=1.21 registration has occurred the
m.A object is incorrectly considered convertible to double, then leading to the segmentation fault.
The text was updated successfully, but these errors were encountered: