ARROW-9164: [C++] Add embedded documentation to compute functions #8457

pitrou · 2020-10-13T17:13:56Z

Also re-use the embedded documentations in Python to generate nicer signatures and docstrings.

github-actions · 2020-10-13T17:20:16Z

https://issues.apache.org/jira/browse/ARROW-9164

kszucs · 2020-10-14T20:55:28Z

python/pyarrow/compute.py

-        def wrapper(left, right, *, memory_pool=None):
-            return func.call([left, right], None, memory_pool)
+        template = _wrapper_template
+    exec(template.format(func_name=name, args_sig=args_sig), globals(), ns)


Wouldn't it be better to set the __signature__ property instead?

bkietz

Altogether this looks like a great improvement in the usability of the compute module, thanks for doing this!

cpp/src/arrow/compute/function.h

python/pyarrow/compute.py

bkietz · 2020-10-14T20:54:38Z

python/pyarrow/compute.py

+    option_class = _get_options_class(func)
+    arg_names = _get_arg_names(func)
+    args_sig = ', '.join(arg_names)
+


Suggested change

if option_class is None:

def wrapper(*args, *, memory_pool=None):

return func.call(args, None, memory_pool)

wrapped_template = 'lambda {}, *, memory_pool=None: ()'

else:

def wrapper(*args, *, options=None, memory_pool=None, **kwargs):

options = _handle_options(name, option_class, options, kwargs)

return func.call(args, options, memory_pool)

wrapped_template = 'lambda {}, *, options=None, memory_pool=None, **kwargs: ()'

wrapper.__name__ = name

wrapper.__qualname__ = name

wrapper.__wrapped__ = eval(wrapped_template.format(args_sig))

Hmm, I didn't know that __wrapped__ was taken up by inspect.signature. Interesting. I'd rather examine this later though.

ARROW-10316

jorisvandenbossche

This looks great! Also the python docstrings ;)

I mostly did a review of the docstrings' content now (and not yet the generation machinery)

cpp/src/arrow/compute/kernels/aggregate_var_std.cc

cpp/src/arrow/compute/kernels/scalar_arithmetic.cc

cpp/src/arrow/compute/kernels/scalar_boolean.cc

cpp/src/arrow/compute/kernels/scalar_arithmetic.cc

cpp/src/arrow/compute/kernels/scalar_validity.cc

jorisvandenbossche · 2020-10-15T13:47:55Z

cpp/src/arrow/compute/kernels/vector_nested.cc

+FunctionDoc list_parent_indices_doc(
+    "Compute parent indices of nested list values",
+    ("`lists` must have a list-like type.\n"
+     "For each value in each list of `lists`, the top-level list index\n"


Not for this PR, but I think that for conceptually more complex functions (like this one, for me), it might be useful to include a small example. But that might get verbose to write in this form ..

For example, for this case: [[1, 2], [3, 4, 5], [6]] returns [0, 0, 1, 1, 1, 2]

Yes, it's not obvious how to make this too unwiedly to write.

Going further down the docstring rabbithole: we could introduce a doctest-like class (of which a FunctionDoc contains a vector), which could just be a set of lambdas which generate a set of inputs, options, and maybe the expected output. These can then be rendered lazily in python

cpp/src/arrow/compute/kernels/vector_sort.cc

pitrou · 2020-10-15T16:49:54Z

@nealrichardson @kou This may help for R and Ruby.

kou · 2020-10-16T02:24:40Z

Wow! Great!
I want not only argument name but also acceptable argument types to generate methods automatically in Ruby. (It'll be a follow-up task.)
For example, if the first argument of function A (x of A(x, y, z)) accepts a table and a record batch, we can add an A method to table and record batch classes:

x = table
x.A(y, z)

x = record_batch
x.A(y, z)

pitrou · 2020-10-19T11:57:15Z

Will rebase and merge.

Also re-use the embedded documentations in Python to generate nicer signatures and docstrings.

Expose CountOptions. Improve introspectability of option classes.

pitrou · 2020-10-19T12:55:04Z

Travis-CI build: https://travis-ci.org/github/pitrou/arrow/builds/737045923

CI failures are unrelated.

Also re-use the embedded documentations in Python to generate nicer signatures and docstrings. Closes #8457 from pitrou/ARROW-9164-compute-func-docs Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

Also re-use the embedded documentations in Python to generate nicer signatures and docstrings. Closes apache#8457 from pitrou/ARROW-9164-compute-func-docs Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Antoine Pitrou <antoine@python.org>

pitrou requested review from wesm and bkietz and removed request for wesm October 13, 2020 17:14

pitrou force-pushed the ARROW-9164-compute-func-docs branch 2 times, most recently from 2c75b60 to 8010c4e Compare October 13, 2020 18:10

kszucs reviewed Oct 14, 2020

View reviewed changes

bkietz requested changes Oct 14, 2020

View reviewed changes

jorisvandenbossche reviewed Oct 15, 2020

View reviewed changes

pitrou force-pushed the ARROW-9164-compute-func-docs branch from 8010c4e to 2b0e393 Compare October 15, 2020 16:27

pitrou added 2 commits October 19, 2020 13:57

ARROW-9164: [C++] Add embedded documentation to compute functions

01f8338

Also re-use the embedded documentations in Python to generate nicer signatures and docstrings.

Address review comments.

61fc098

Expose CountOptions. Improve introspectability of option classes.

pitrou force-pushed the ARROW-9164-compute-func-docs branch from 2b0e393 to 61fc098 Compare October 19, 2020 12:00

pitrou closed this in 69b444b Oct 19, 2020

pitrou deleted the ARROW-9164-compute-func-docs branch October 19, 2020 12:56

This was referenced Oct 19, 2020

[C++] Provide APIs for adding "docstrings" to arrow::compute::Function classes that can be accessed by bindings #25270

Closed

[Python] Consider using __wrapped__ for compute function introspection #26307

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-9164: [C++] Add embedded documentation to compute functions #8457

ARROW-9164: [C++] Add embedded documentation to compute functions #8457

pitrou commented Oct 13, 2020

github-actions bot commented Oct 13, 2020

kszucs Oct 14, 2020

bkietz left a comment

bkietz Oct 14, 2020

pitrou Oct 15, 2020

pitrou Oct 15, 2020

jorisvandenbossche left a comment

jorisvandenbossche Oct 15, 2020

pitrou Oct 15, 2020

bkietz Oct 15, 2020

pitrou commented Oct 15, 2020

kou commented Oct 16, 2020

pitrou commented Oct 19, 2020

pitrou commented Oct 19, 2020

+        if option_class is None:
+            def wrapper(*args, *, memory_pool=None):
+                return func.call(args, None, memory_pool)
+            wrapped_template = 'lambda {}, *, memory_pool=None: ()'
+        else:
+            def wrapper(*args, *, options=None, memory_pool=None, **kwargs):
+                options = _handle_options(name, option_class, options, kwargs)
+                return func.call(args, options, memory_pool)
+            wrapped_template = 'lambda {}, *, options=None, memory_pool=None, **kwargs: ()'
+        wrapper.__name__ = name
+        wrapper.__qualname__ = name
+        wrapper.__wrapped__ = eval(wrapped_template.format(args_sig))

ARROW-9164: [C++] Add embedded documentation to compute functions #8457

ARROW-9164: [C++] Add embedded documentation to compute functions #8457

Conversation

pitrou commented Oct 13, 2020

github-actions bot commented Oct 13, 2020

kszucs Oct 14, 2020

Choose a reason for hiding this comment

bkietz left a comment

Choose a reason for hiding this comment

bkietz Oct 14, 2020

Choose a reason for hiding this comment

pitrou Oct 15, 2020

Choose a reason for hiding this comment

pitrou Oct 15, 2020

Choose a reason for hiding this comment

jorisvandenbossche left a comment

Choose a reason for hiding this comment

jorisvandenbossche Oct 15, 2020

Choose a reason for hiding this comment

pitrou Oct 15, 2020

Choose a reason for hiding this comment

bkietz Oct 15, 2020

Choose a reason for hiding this comment

pitrou commented Oct 15, 2020

kou commented Oct 16, 2020

pitrou commented Oct 19, 2020

pitrou commented Oct 19, 2020