ENH: support for clip and quantile ops on DoubleColumns #1090

jreback · 2017-08-05T20:21:24Z

No description provided.

jreback · 2017-08-05T20:22:25Z

ibis/pandas/tests/test_operations.py

+        (methodcaller('quantile', 0.5), methodcaller('quantile', 0.5)),
+    ]
+)
+def test_arrayfunctions(t, df, ibis_func, pandas_func):


@cpcloud where do you testsfor exceptions that bubble up from the impl (e.g. an out-of-range for quantile) for instance.?

I would just have a separate test that only contains failures (using with pytest.raises). Easier to debug that way.

cpcloud · 2017-08-05T20:23:38Z

ibis/expr/operations.py

@@ -349,6 +349,17 @@ def output_type(self):
            return rules.shape_like(arg, 'double')


+class Clip(ValueOp):
+    input_type = [number(name='lower', allow_boolean=False, optional=True),
+                  number(name='upper', allow_boolean=False, optional=True)]


What is the behavior when neither is passed?

just returns the original. not great. I could raise if not at least 1 is passed.

Yeah, probably should raise during execution since we don't have a rule to express "at least one argument" that I'm aware of.

cpcloud · 2017-08-05T20:25:18Z

ibis/expr/operations.py

+    input_type = [number(name='lower', allow_boolean=False, optional=True),
+                  number(name='upper', allow_boolean=False, optional=True)]
+
+    def output_type(self):


The output type here should be the same as the input type. You can use rules.type_of_arg(0) to get the output type of the first input argument. You should also be able to just assign that directly to output_type like this:

class FooNode(...): output_type = rules.type_of_arg(0)

cpcloud · 2017-08-05T20:27:05Z

ibis/pandas/execution.py

@@ -170,6 +170,20 @@ def execute_series_natural_log(op, data, scope=None):
    return np.log(data)


+@execute_node.register(
+    ops.Clip, pd.Series, (float, int, type(None)), (float, int, type(None))


For int use integer_types which includes numpy integers and six.integer_types. If that's not imported you can import it from ibis/pandas/core.py.

Also, does Series.clip allow Series as input? Do we want to allow other table columns as inputs?

it does, will add

cpcloud · 2017-08-05T20:27:37Z

ibis/pandas/execution.py

+@execute_node.register(
+    ops.Quantile, pd.Series, float
+)
+def execute_series_quantile(op, data, q, scopy=None):


Misspelling here: s/scopy/scope/

Should we validate that 0 <= q <= 1 here?

cpcloud · 2017-08-05T20:28:58Z

ibis/pandas/tests/test_operations.py

+        (methodcaller('quantile', 0.5), methodcaller('quantile', 0.5)),
+    ]
+)
+def test_arrayfunctions(t, df, ibis_func, pandas_func):


Maybe have two tests here? I know there's a few examples where we dump a bunch into one, but these are distinct enough that it probably makes sense to separate them.

cpcloud · 2017-08-05T20:29:42Z

ibis/expr/operations.py

@@ -696,6 +707,12 @@ class Mean(Reduction):
    output_type = rules.scalar_output(_mean_output_type)


+class Quantile(Reduction):
+
+    input_type = [number(name='q', allow_boolean=False)]


Maybe give this a slightly longer name :)

cpcloud · 2017-08-05T21:11:07Z

ibis/expr/api.py

+    -------
+    clipped : type depending on input
+      Decimal values: yield decimal
+      Other numeric values: yield integer (int32)


I don't think you need to enumerate any types here. Something like "The type of the returned column is the same as type of the input" or something similar.

jreback · 2017-08-05T21:42:12Z

pushed, but still have to handle this

In [3]: pd.Series([1, 2, 3]).quantile(0.5)
Out[3]: 2.0

In [4]: pd.Series([1, 2, 3]).quantile([0.5, 0.75])
Out[4]: 
0.50    2.0
0.75    2.5
dtype: float64

input type is easy, and output type is a the type of inputscalar column, But we don't have an Index (e.g. to annotate the quantiles). How do you handle this? (return a Table)?

cpcloud · 2017-08-05T22:17:19Z

ibis/pandas/tests/test_operations.py

+    ]
+)
+def test_arraylike_functions_transforms(t, df, ibis_func, pandas_func):
+    if isinstance(pandas_func, Exception):


Why do you need this? It looks like every pandas_func is a lambda.

changed in folllowing push

cpcloud · 2017-08-05T22:19:03Z

ibis/expr/operations.py

+
+    input_type = [value,
+                  number(name='quantile', allow_boolean=False),
+                  string(name='interpolation', optional=True)]


I think we want to use rules.string_options(list_of_possible_interpolation_values) so that we can limit these before executing.

cpcloud · 2017-08-05T22:21:52Z

ibis/expr/operations.py

+    input_type = [value,
+                  number(name='quantile', allow_boolean=False),
+                  string(name='interpolation', optional=True)]
+    output_type = rules.scalar_output(_array_reduced_type)


This is correct for multiple quantiles, but not for a single quantile. In this case the output_type is actually the same as the second argument: T -> T or array<T> -> array<T>. rules.type_of_arg(1) should cover this case.

I don't remember how to spell the correct type for the input though, which is "any scalar or array of scalars".

cpcloud · 2017-08-05T22:33:00Z

ibis/expr/operations.py

+class Quantile(Reduction):
+
+    input_type = [value,
+                  number(name='quantile', allow_boolean=False),


We don't have a nice way of describing "numeric scalar or array of numeric scalars". I'll put up a PR to do this now.

This is actually pretty tricky because of the distinction between a generic list of values (whose elements could each be of a different type) and an array which is a list of values all with the same type.

I'll make an issue about this to refactor this part of the code.

In the meantime, you can work around this by creating an additional node type, maybe MultipleQuantile and then in the def quantile function try to construct one and if that fails validation construct the other.

jreback · 2017-08-05T22:41:28Z

ibis/pandas/execution.py

+
+
+@execute_node.register(
+    ops.Quantile, pd.Series, float,


I repeated this w/o the optional arg as I couldn' get rules.string_options to be optional....?

Did you try dispatching on six.string_types + (type(None),)?

You should be able to cover this in one function by dispatching on six.string_types + (type(None),) for the interpolation parameter.

jreback · 2017-08-05T22:42:02Z

ibis/expr/operations.py

+                  rules.string_options(
+                      ['linear', 'lower', 'higher',
+                       'midpoint', 'nearest'],
+                      name='interpolation', optional=True)]


e.g. here I think the optional is being ignored

Hm, could be a bug. Looking into it.

cpcloud · 2017-08-06T14:59:18Z

@jreback let's handle the array of quantiles case in a follow up. So for now we only allow a scalar double as the quantile argument. There's some other things to fix before we can properly handle the array case.

jreback · 2017-08-06T15:08:58Z

sgtm

will fix up the doc string and add some tests

jreback · 2017-08-06T20:57:42Z

pushed

jreback · 2017-08-06T20:57:58Z

docs/source/release.rst

@@ -7,6 +7,17 @@ Release Notes
    interesting. Point (minor, e.g. 0.5.1) releases will generally not be found
    here and contain only bug fixes.

+0.12.0 (???)


leave out? change to 0.11.3?

Yep, this should be 0.11.3

cpcloud · 2017-08-07T14:24:27Z

docs/source/release.rst

+0.12.0 (???)
+------------
+
+This release brings initial Pandas backend support along with a number of


Maybe just leave the section blank for now.

cpcloud · 2017-08-07T14:25:18Z

ibis/expr/api.py

+
+    Parameters
+    ----------
+    quantile : float, default 0.5 (50% quantile)


Doesn't look like there's a default, so should remove this text.

cpcloud · 2017-08-07T14:26:04Z

ibis/expr/operations.py

+class Quantile(Reduction):
+
+    input_type = [value,
+                  number(name='quantile', allow_boolean=False),


This should be rules.double()

technically this should work on all numeric (and datetimelikes) actually

Oh interesting re datetime.

cpcloud · 2017-08-07T14:29:07Z

@jreback few more small things. other than that this looks good to go.

jreback · 2017-08-07T18:09:27Z

@cpcloud if you'd have a look. I added default='linear' for Quantile. but still looking for original signature (and now that I have deleted it, the testsfail :>)

jreback · 2017-08-07T20:48:07Z

pushed with updates

cpcloud · 2017-08-07T20:50:39Z

I'll merge on green

jreback · 2017-08-07T22:48:10Z

green now

cpcloud · 2017-08-07T22:50:32Z

thanks @jreback !

xref ibis-project#1090

xref #1090 Author: Jeff Reback <jeff@reback.net> Closes #1094 from jreback/multiple and squashes the following commits: 0e235cc [Jeff Reback] allow quantile to accept int input specify output type as double if integer input 2337321 [Jeff Reback] support for passing multiple quantiles in .quantile()

jreback added the feature Features or general enhancements label Aug 5, 2017

jreback commented Aug 5, 2017

View reviewed changes

cpcloud self-requested a review August 5, 2017 21:04

cpcloud added this to the 0.11.3 milestone Aug 5, 2017

cpcloud requested changes Aug 5, 2017

View reviewed changes

cpcloud added expressions Issues or PRs related to the expression API pandas The pandas backend labels Aug 5, 2017

cpcloud reviewed Aug 5, 2017

View reviewed changes

jreback commented Aug 5, 2017

View reviewed changes

jreback force-pushed the funcs branch from d9c9196 to 84e8801 Compare August 6, 2017 20:57

jreback commented Aug 6, 2017

View reviewed changes

cpcloud reviewed Aug 7, 2017

View reviewed changes

jreback force-pushed the funcs branch from 84e8801 to 9544739 Compare August 7, 2017 18:07

ENH: support for clip and quantile ops on DoubleColumns

aba7544

jreback force-pushed the funcs branch from 2111a42 to aba7544 Compare August 7, 2017 20:47

cpcloud approved these changes Aug 7, 2017

View reviewed changes

cpcloud closed this in 6388baf Aug 7, 2017

jreback added a commit to jreback/ibis that referenced this pull request Aug 8, 2017

support for passing multiple quantiles in .quantile()

71bb190

xref ibis-project#1090

jreback mentioned this pull request Aug 8, 2017

support for passing multiple quantiles in .quantile() #1094

Closed

jreback added a commit to jreback/ibis that referenced this pull request Aug 8, 2017

support for passing multiple quantiles in .quantile()

2337321

xref ibis-project#1090

jreback mentioned this pull request Sep 15, 2017

support for .quantile() #1152

Closed

ENH: support for clip and quantile ops on DoubleColumns #1090

ENH: support for clip and quantile ops on DoubleColumns #1090

Conversation

jreback commented Aug 5, 2017

jreback Aug 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud Aug 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Aug 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud Aug 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud Aug 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Aug 6, 2017

jreback commented Aug 6, 2017

jreback commented Aug 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Aug 7, 2017

jreback commented Aug 7, 2017

jreback commented Aug 7, 2017

cpcloud commented Aug 7, 2017

jreback commented Aug 7, 2017

cpcloud commented Aug 7, 2017

jreback Aug 5, 2017 •

edited

Loading

cpcloud Aug 5, 2017 •

edited

Loading

cpcloud Aug 6, 2017 •

edited

Loading

cpcloud Aug 5, 2017 •

edited

Loading