Make py::vectorize() ignore non-vectorizable arguments & support member functions #762

jagerman · 2017-03-23T19:09:02Z

Edit: PR amended from this initial description, see below.

This commit adds py::vectorize() support for class methods, so that you can write:

    class_<MyClass>(m, "MyClass")
        .def("method", py::vectorize(&MyClass::method));

Currently the only way to do this is via a lambda method wrapper with the exclusion argument technique described in the documentation. That way still works—this PR also adds a test for it—and is still needed if you want to wrap a lambda method wrapper.

Fixes #761

jagerman · 2017-03-23T19:10:28Z

include/pybind11/numpy.h

-detail::vectorize_helper<Func, Return, Args...>
-vectorize(const Func &f, Return (*) (Args ...)) {
-    return detail::vectorize_helper<Func, Return, Args...>(f);
-}


Moved this into detail as it didn't seem to be an intentionally exposed part of the API.

wjakob · 2017-03-23T19:18:40Z

I'm wondering if this is the nicest way of solving the issue with #761. I think the cleanest solution would be for vectorize() to simply to act as a pass-through for non array arguments. This allows for a very general and much shorter implementation. If you want, I can invite you to another project that does something similar with pybind11 in the context of SIMD vectorization. (currently still in private mode, so I can't post a public link)

jagerman · 2017-03-23T19:43:05Z

Yeah, I'm not entirely happy with how much needed to be added to make this work. Making it skip anything non-array-convertible would indeed be nicer.

jagerman · 2017-03-23T19:45:10Z

(Though it would still need some of this code--the current vectorize code won't accept a member function pointer).

jagerman · 2017-03-23T20:02:36Z

I'm not really seeing how this could work in practice.

Say vectorize(f) is called with double f(T1 a, T2 b); it will then generate a lambda-like callable object with operator()(array_t<T1>, array_t<T2>), so the templated implementation has the needed array types. So given arbitrary T1 and T2 types, what we would consider a non-array arguments?

wjakob · 2017-03-23T20:05:58Z

Something like !std::is_arithmetic or more general: !std::is_pod?

jagerman · 2017-03-27T20:45:13Z

I've updated the PR; copying and pasting the main commit message here:

This extends py::vectorize to automatically pass through non-vectorizable arguments. This removes the need for the documented "explicitly exclude an argument" workaround.

Vectorization now applies to arithmetic, std::complex, and POD types, passed as plain value or by const lvalue reference (previously only pass-by-value types were supported). Non-const lvalue references and any other types are passed through as-is.

Functions with rvalue reference arguments (whether vectorizable or not) are explicitly prohibited: an rvalue reference is inherently not something that can be passed multiple times and is thus unsuitable to being in a vectorized function.

The vectorize returned value is also now more sensitive to inputs: previously it would return by value when all inputs are of size 1; this is now amended to having all inputs of size 1 and 0 dimensions. Thus if you pass in, for example, [[1]], you get back a 1x1, 2D array, while previously you got back just the resulting single value.

Vectorization of member function specializations is now also supported via py::vectorize(&Class::method); this required passthrough support for the initial object pointer on the wrapping function pointer.

This commit also removes the documentation about manually dealing with buffers; that feature is much better covered by the direct access implementation (which is the subsequent section).

wjakob

This looks great, just a minor comment about the docs.

wjakob · 2017-05-24T11:15:45Z

docs/advanced/pycpp/numpy.rst

-        }
-    );
-
-In cases where the computation is too complicated to be reduced to


It might still be good to keep the piece of code below as an complete self-contained example how how to deal with arrays when a function being vectorized is not a universal function.

Yeah. Looking at it now, I don't know why I removed it; I'm guessing it was a mistake of just removing too much along with the above.

No, scratch that. I took it out deliberately, but on second thought I see the merits in keeping it (it doesn't only apply to py::array arguments); I've added it back in.

wjakob · 2017-05-24T11:19:22Z

include/pybind11/common.h

@@ -504,6 +512,14 @@ struct is_input_iterator<T, void_t<decltype(*std::declval<T &>()), decltype(++st
 /// Ignore that a variable is unused in compiler warnings
 inline void ignore_unused(const int *) { }

+/// Apply a function over each element of a parameter pack
+#ifdef __cpp_fold_expressions
+#define PYBIND11_EXPAND_SIDE_EFFECTS(PATTERN) (((PATTERN), void()), ...)


wjakob · 2017-05-24T11:22:22Z

include/pybind11/numpy.h

+    //     - BIndex is a incremental sequence (beginning at 0) of the same size as VIndex, so that
+    //       we can store vectorized buffer_infos in an array (argument VIndex has its buffer at
+    //       index BIndex in the array).
+    template <size_t... Index, size_t... VIndex, size_t... BIndex> object run(


cool, that's a nice trick (with select_indices).

This allows calling of functions (typically void) over a parameter pack, replacing usage such as: bool unused[] = { (voidfunc(param_pack_arg), false)..., false }; (void) unused; with a much cleaner: PYBIND11_EXPAND_SIDE_EFFECTS(voidfunc(param_pack_arg));

Adds `remove_reference_t` and converts various `typename std::remove_reference<...>::type` to using it.

Currently if you construct an `array_t<T, array::f_style>` with a shape but not strides you get a C-style array; the only way to get F-style strides was to calculate the strides manually. This commit fixes that by adding logic to use f_style strides when the flag is set. This also simplifies the existing c_style stride logic.

This extends py::vectorize to automatically pass through non-vectorizable arguments. This removes the need for the documented "explicitly exclude an argument" workaround. Vectorization now applies to arithmetic, std::complex, and POD types, passed as plain value or by const lvalue reference (previously only pass-by-value types were supported). Non-const lvalue references and any other types are passed through as-is. Functions with rvalue reference arguments (whether vectorizable or not) are explicitly prohibited: an rvalue reference is inherently not something that can be passed multiple times and is thus unsuitable to being in a vectorized function. The vectorize returned value is also now more sensitive to inputs: previously it would return by value when all inputs are of size 1; this is now amended to having all inputs of size 1 *and* 0 dimensions. Thus if you pass in, for example, [[1]], you get back a 1x1, 2D array, while previously you got back just the resulting single value. Vectorization of member function specializations is now also supported via `py::vectorize(&Class::method)`; this required passthrough support for the initial object pointer on the wrapping function pointer.

wjakob · 2017-05-24T21:51:03Z

LGTM!

jagerman commented Mar 23, 2017

View reviewed changes

jagerman force-pushed the vectorize-methods branch from 6c78bd5 to 5b194b6 Compare March 27, 2017 20:42

jagerman changed the title ~~Make py::vectorize() work on class methods~~ Make py::vectorize() ignore non-vectorizable arguments & support member functions Mar 27, 2017

jagerman force-pushed the vectorize-methods branch from 5b194b6 to e2acf16 Compare April 1, 2017 20:32

jagerman mentioned this pull request Apr 1, 2017

Add a scope guard call policy #740

Merged

jagerman force-pushed the vectorize-methods branch 3 times, most recently from cfd0d23 to 67d9614 Compare April 10, 2017 22:32

jagerman mentioned this pull request May 20, 2017

Convenience constructor templates for buffer_info #860

Merged

jagerman force-pushed the vectorize-methods branch from 67d9614 to 1c75420 Compare May 20, 2017 16:13

wjakob reviewed May 24, 2017

View reviewed changes

jagerman added 4 commits May 24, 2017 13:14

Add and use detail::remove_reference_t

0c972e4

Adds `remove_reference_t` and converts various `typename std::remove_reference<...>::type` to using it.

jagerman force-pushed the vectorize-methods branch from 1c75420 to 92a75e2 Compare May 24, 2017 17:28

jagerman merged commit f3ce00e into pybind:master May 25, 2017

dean0x7d modified the milestone: v2.2 Aug 13, 2017

rwgk mentioned this pull request Feb 9, 2023

FWD pybind11 google/pybind11k#762

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make py::vectorize() ignore non-vectorizable arguments & support member functions #762

Make py::vectorize() ignore non-vectorizable arguments & support member functions #762

jagerman commented Mar 23, 2017 •

edited

Loading

jagerman Mar 23, 2017

wjakob commented Mar 23, 2017

jagerman commented Mar 23, 2017

jagerman commented Mar 23, 2017

jagerman commented Mar 23, 2017

wjakob commented Mar 23, 2017

jagerman commented Mar 27, 2017

wjakob left a comment

wjakob May 24, 2017

jagerman May 24, 2017

jagerman May 24, 2017

wjakob May 24, 2017

wjakob May 24, 2017

wjakob commented May 24, 2017

Make py::vectorize() ignore non-vectorizable arguments & support member functions #762

Make py::vectorize() ignore non-vectorizable arguments & support member functions #762

Conversation

jagerman commented Mar 23, 2017 • edited Loading

jagerman Mar 23, 2017

Choose a reason for hiding this comment

wjakob commented Mar 23, 2017

jagerman commented Mar 23, 2017

jagerman commented Mar 23, 2017

jagerman commented Mar 23, 2017

wjakob commented Mar 23, 2017

jagerman commented Mar 27, 2017

wjakob left a comment

Choose a reason for hiding this comment

wjakob May 24, 2017

Choose a reason for hiding this comment

jagerman May 24, 2017

Choose a reason for hiding this comment

jagerman May 24, 2017

Choose a reason for hiding this comment

wjakob May 24, 2017

Choose a reason for hiding this comment

wjakob May 24, 2017

Choose a reason for hiding this comment

wjakob commented May 24, 2017

jagerman commented Mar 23, 2017 •

edited

Loading