Rules refactor #1366

kszucs · 2018-02-24T00:10:48Z

No description provided.

cpcloud

This is looking pretty good so far. We need to avoid any unnecessary breakages, so that downstream users that have extended the library outside of upstream don't experience unnecessary breakage.

cpcloud · 2018-02-26T18:46:29Z

ibis/bigquery/client.py

 def bq_param_integer(param, value):
    return bq.ScalarQueryParameter(param._name, 'INT64', value)


-@bigquery_param.register(ir.DoubleScalar, float)
+@bigquery_param.register(ir.FloatingScalar, float)


Make sure there are tests for these

Eg passing in the different int and float subclasses

I've removed those subclasses.

Why did you remove them?

Because they are only required to be able to bind methods on them (otherwise evenScalarExpr(DataType) and ColumnExpr(DataType) would be enough).

The actual work is done by the datatype passed to ValueExpr as an argument. Also see DataType.scalar and DataType.column class properties.

cpcloud · 2018-02-26T18:48:21Z

ibis/expr/analysis.py

+
+
+def is_reduction(expr):
+    # Aggregations yield typed scalar expressions, since the result of an


Let's put this in the docstring

cpcloud · 2018-02-26T18:54:46Z

ibis/expr/datatypes.py

@@ -348,12 +360,16 @@ def __str__(self):
    def _equal_part(self, other, cache=None):
        return self.precision == other.precision and self.scale == other.scale

+    def largest(self):
+        return Decimal(self.precision, 38)


A scale of 38 implies a precision of 38, because precision = scale + number of digits to the left of the decimal point so unless self.precision == 38 this type is impossible.

There's also the issue of what you mean by largest here. The largest decimal value that can be constructed can only be constructed with a decimal type of decimal(38, 0) and that's the integer with 9 repeated 38 times. Also, we shouldn't assume 38 digits of precision.

Ported from rules, used at the following places:

Sum

VarianceBase

CumulativeMean

CumulativeSum

Decimal largest is clearly wrong, but what should I use instead?

How about we define largest for a given scale? In that case the definition of largest becomes Decimal(38, self.scale)

Let's assume 38 digits of precision for now. If we have users that desire more precision than that we can revisit in the future.

I've got confused https://github.com/ibis-project/ibis/blob/master/ibis/expr/tests/test_decimal.py#L39 :)

cpcloud · 2018-02-26T18:56:12Z

ibis/expr/datatypes.py

@@ -559,6 +584,9 @@ def _equal_part(self, other, cache=None):
 any = Any()
 null = Null()
 boolean = Boolean()
+integer = Integer()
+floating = Floating()


What does it mean for a user to specify this as the type of a column (exposing them this way implies that they are part of the user-facing API). If it does have a meaning, are there tests for it?

They were required previously, I'll drop them.

cpcloud · 2018-02-26T18:56:38Z

ibis/expr/datatypes.py

@@ -559,6 +584,9 @@ def _equal_part(self, other, cache=None):
 any = Any()
 null = Null()
 boolean = Boolean()
+integer = Integer()
+floating = Floating()
+decimal = Decimal(12, 2)


Why 12, 2 as the precision and scale, respectively?

cpcloud · 2018-02-26T21:02:10Z

ibis/expr/operations.py

+class OperationMeta(type):
+
+    @classmethod
+    def __prepare__(metacls, name, bases, **kwds):


This doesn't exist on Python 2.7, so we'll need to find another way of writing this class. We may have a module that we import it from depending on the version of python so that we can continue to use __prepare__ in Python >=3

cpcloud · 2018-02-26T21:03:14Z

ibis/expr/operations.py

+        return False
+
+    def flat_args(self):
+        for arg in self.args:


I feel like we have a flatten function laying around somewhere.

There isn't one, should it go to ibis.util?

Let's leave this here for now.

One wrinkle here is that this only flattens one nesting level. So a general flatten function would probably break some things. Let's revisit flatten after this PR is merged.

cpcloud · 2018-02-26T21:03:28Z

ibis/expr/operations.py

+        if (self, other) in cache:
+            return cache[(self, other)]
+
+        if id(self) == id(other):


This can be changed to self is other.

cpcloud · 2018-02-26T21:07:24Z

ibis/expr/operations.py

-        HasSchema.__init__(self, schema, name=name)
+    name = rlz.instanceof(six.string_types)
+    schema = rlz.schema
+    source = rlz.noop


These noops should be instances of ibis.client.Client, right?

Yes, there are a lot of noops currently which we should eventually configure.

cpcloud · 2018-02-26T21:07:53Z

ibis/expr/operations.py

-        TableNode.__init__(self, [schema, name])
-        HasSchema.__init__(self, schema, name=name)
+    schema = rlz.schema
+    name = rlz.optional(rlz.instanceof(six.string_types), default=genname)


Can you document that callables are allowed for defaults if you haven't already?

kszucs · 2018-02-28T23:18:42Z

ibis/expr/analytics.py

-        self.buckets = buckets
-        self.closed = _validate_closed(closed)
+    __slots__ = ('arg', 'buckets', 'closed', 'close_extreme', 'include_under',
+                 'include_over')


@cpcloud I've added slots for each operations to support argument order on py2 too. While this is a bit more boilerplate, I wouldn't consider bad.
I can use plain __init__ methods as well as the counter trick we've discussed, in case of the latter we need to enforce calling the rules.

Which approach do You prefer?
a. __slot__ or argnames or ...?
b. counter trick and rlz.any() or rlz.value(dt.any)
c. plain but idiomatic __init__ functions*?

which is not not that bad, but op.args would require another order definition

Here is another one:

class ArraySlice(ValueOp): arg = Argument(rlz.value(dt.Array(dt.any))) start = Argument(rlz.integer) stop = Argument(rlz.integer, default=None) output_type = rlz.typeof('arg')

Based on gitter conversation we ended up with the following API:

class StringFind(ValueOp): arg = Arg(rlz.string) start = Arg(rlz.integer, default=None) end = Arg(rlz.integer, default=None) return_ = Return(like=arg, dtype=dt.int64)

This will alse make possible to support operation definition with python typing.

kszucs · 2018-02-28T23:19:27Z

ibis/expr/api.py

-                             MapValue, MapScalar, MapColumn,
-                             StructValue, StructScalar, StructColumn,
-                             CategoryValue, unnamed, as_value_expr, literal,
-                             param, null, sequence)


These were not exposed in __all__, so I've removed them.

kszucs · 2018-02-28T23:20:38Z

ibis/expr/api.py

 from ibis.compat import PY2, to_time, to_date
+from ibis.expr.types import Expr, null, param, literal, sequence, as_value_expr


We might factor out null, param, literal, sequence, as_value_expr to here, but we can do that incrementally :)

kszucs · 2018-02-28T23:22:02Z

ibis/expr/datatypes.py

    def castable(self, target, **kwargs):
        return castable(self, target, **kwargs)

    def cast(self, target, **kwargs):
        return cast(self, target, **kwargs)

    def scalar_type(self):
-        import ibis.expr.types as ir
-        return getattr(ir, '{}Scalar'.format(self.name))
+        return partial(self.scalar, dtype=self)


@cpcloud These are actually "factories" not types. Are You OK with these?

Yes, that's fine with me.

kszucs · 2018-02-28T23:42:51Z

ibis/expr/operations.py

-        case_expr = as_value_expr(case_expr)
-        result_expr = as_value_expr(result_expr)
+        case_expr = ir.as_value_expr(case_expr)
+        result_expr = ir.as_value_expr(result_expr)


I think we should get rid of as_value_expr too, all the value coercions should be handled by rules.

I definitely agree with that.

kszucs · 2018-02-28T23:49:03Z

ibis/expr/types.py

@@ -915,502 +567,239 @@ def group_by(self, by=None, **additional_grouping_expressions):
 # with: an instance of each is well-typed and includes all valid methods
 # defined for each type.

+# TODO: __slots__?


Define or not to define? Use a metaclass?

cpcloud · 2018-03-11T00:26:05Z

ibis/expr/analytics.py

-        self.include_under = bool(include_under)
-
-        if not len(buckets):
+    arg = Arg(rlz.noop)


If I understand correctly, Arg(rlz.noop) is equivalent to (the old) rules.value?

No, noop is toolz.identity, just a placeholder where no rule was defined previously.

I see. LGTM.

cpcloud · 2018-03-11T00:28:32Z

ibis/expr/api.py

+import ibis.expr.analysis as _L
+import ibis.expr.datatypes as dt
+import ibis.expr.analytics as _analytics
+import ibis.expr.operations as ops


Is there any reason to change the _ops and _com imports? The changes in this file are rather noisy because it looks like most of them just renaming _ops to ops and _com to com. The reason these are prefixed with an underscore is to hide them from a star import.

Api.py has __all__ defined - which protects against star import, and I thought ops is cleaner.

cpcloud · 2018-03-11T00:31:10Z

ibis/expr/datatypes.py

@@ -239,11 +230,16 @@ def __repr__(self):


 class SignedInteger(Integer):
-    pass
+
+    def largest(self):


Any reason not to make these @propertys?

No particular. When should I prefer a property over a method?

cpcloud · 2018-03-11T00:46:09Z

ibis/expr/datatypes.py

@@ -1143,6 +1188,12 @@ def can_cast_floats(source, target, upcast=False, **kwargs):
    return True


+@castable.register(Decimal, Decimal)
+def cas_cast_decimals(source, target, **kwargs):


Should this be can_cast_decimals?

cpcloud · 2018-03-11T00:50:00Z

ibis/expr/signature.py

+@six.add_metaclass(AnnotableMeta)
+class Annotable(object):
+
+    __slots__ = tuple()


This can just be __slots__ = ().

cpcloud · 2018-03-11T00:51:52Z

ibis/expr/signature.py

+            self.default == other.default
+        )
+
+    def optional(self):


Should this be a @property?

cpcloud · 2018-03-11T00:52:50Z

ibis/expr/signature.py

+                if self.default is None:
+                    return None
+                elif util.is_function(self.default):
+                    value = self.default()


I wasn't aware that we support default argument values that are functions.

That is for table genname:

class UnboundTable(PhysicalTable): schema = Arg(sch.Schema) name = Arg(six.string_types, default=genname)

I can remove it though.

No, let's leave it. It's fine for now.

cpcloud · 2018-03-11T00:55:54Z

ibis/expr/signature.py

+            elif name in kwargs:
+                value = argument.validate(kwargs[name], name=name)
+            else:
+                value = argument.validate(name=name)


Do you think it might be possible to bind arguments using inspect.signature here so we don't have to recreate Python's calling convention algorithm?

Good question. Isn't inspect.signature py3 only?

Yep, but there's the funcsigs package for that :)

I'm going to add funcsigs as a dependency for #1277, so don't worry about adding it here

Converted to issue: #1396

cpcloud · 2018-03-11T00:57:33Z

ibis/expr/tests/test_datatypes.py

@@ -182,22 +176,6 @@ def test_primitive(spec, expected):
    assert dt.dtype(spec) == expected


-def test_precedence_with_no_arguments():


Did you move these tests to another file?

cpcloud · 2018-03-11T01:00:58Z

ibis/tests/all/test_client.py

+    schema = query.schema()
+
+    # clickhouse columns has been defined as non-nullable
+    # whereas other backends don't support non-nullable columns yet


I believe Impala supports this, and possiblty other backends.

cpcloud · 2018-03-11T01:01:37Z

ibis/sql/transforms.py

-        ir.Node.__init__(self, [foreign_table, predicates])
+class NotExistsSubquery(ops.Node):
+    foreign_table = Arg(rlz.noop)
+    predicates = Arg(rlz.noop)


Nice to see this finally getting rolled into the rules system.

We need to revisit all Args with rlz.noop :)

cpcloud · 2018-03-11T01:19:27Z

ibis/expr/types.py

-            r = find_base_table(arg)
-            if isinstance(r, TableExpr):
-                return r
+unnamed = UnnamedMarker()


I remember deleting this. Am I mistaken? If not, why bring it back?

Probably, it's present in the master

cpcloud · 2018-03-11T01:21:22Z

ibis/expr/tests/test_table.py

@@ -1288,7 +1289,7 @@ def g(x):
 def test_pickle_table_expr():
    schema = [('time', 'timestamp'), ('key', 'string'), ('value', 'double')]
    t0 = ibis.table(schema, name='t0')
-    raw = pickle.dumps(t0)
+    raw = pickle.dumps(t0, protocol=2)


Why are you passing a specific protocol version here?

On py27 serializeing object with __slots__ requires protocol 2.

cpcloud · 2018-03-11T01:25:16Z

ibis/expr/tests/test_signature.py

+        arg = Argument(lambda x: x)
+
+    class StringOp(Op):
+        arg = Argument(str)  # inherit


Should this be Arg instead of Argument?

In signature.py it's defined as Argument and aliased as Arg during import. Do You mean renaming in signature.py?

I see. No, leaving it here as Argument is fine.

cpcloud · 2018-03-11T01:37:23Z

ibis/expr/operations.py

+
+    def _repr(self, memo=None):
+        if memo is None:
+            from ibis.expr.format import FormatMemo


Probably ok to just import in the function, this is a little strange.

There are circular imports.

cpcloud · 2018-03-11T01:41:14Z

ibis/expr/operations.py

+        cache[(self, other)] = True
+        return True
+
+    def is_ancestor(self, other):


Is this being used anywhere? It seems like it's really not that different from just a straight-up call to equals.

cpcloud · 2018-03-11T01:43:09Z

ibis/expr/operations.py

-    else:
-        return rules.shape_like(arg, 'int64')
+    """Absolute value"""
+    output_type = rlz.typeof('arg')


Can you not pass arg directly here? It is in scope after all.

arg is actually an Argument instance without any knowledge of the underlying data, also doesn't know its name until the metaclass finishes signature construction.
A solution would be to set the argument's name in the metaclass or use descriptors and _sunder slots.

Actually You could give me a couple of ideas how should we abstract return_ definition.

cpcloud · 2018-03-11T01:45:28Z

ibis/expr/operations.py



 class Capitalize(StringUnaryOp):
-    pass
+    """Return a capitalized version of input string"""


Thanks for adding the documentation here.

cpcloud · 2018-03-11T01:50:34Z

ibis/expr/operations.py



 class Union(TableNode, HasSchema):
+    left = Arg(rlz.noop)


Should probably be TableExpr here, but leave it as is if making that change doesn't trivially work.

cpcloud · 2018-04-05T16:03:57Z

Bombs away!

kszucs changed the title ~~Prainbow~~ [WIP] Rules refactor Feb 24, 2018

kszucs force-pushed the prainbow branch 2 times, most recently from 2eb37b7 to 8a19a88 Compare February 26, 2018 16:31

cpcloud requested changes Feb 26, 2018

View reviewed changes

kszucs commented Feb 28, 2018

View reviewed changes

cpcloud reviewed Mar 11, 2018

View reviewed changes

kszucs force-pushed the prainbow branch from 15791a7 to 38c3cd3 Compare March 11, 2018 00:42

cpcloud reviewed Mar 11, 2018

View reviewed changes

ibis/expr/signature.py Outdated

self.default == other.default

)

def optional(self):

Copy link

Member

cpcloud Mar 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a @property?

cpcloud reviewed Mar 11, 2018

View reviewed changes

kszucs added 26 commits April 5, 2018 15:56

shape_like

9fb8915

shape_like for impala udfs

0b1bcd3

factor out signature

45c8b5a

fix tests

99d8180

fix optional

7ce42d0

fix bigquery tests

68d6243

resolve conflicts; added tests using singleton dt.null

1dad6bf

fix a couple of review issues

5a059c5

signature optional property

6bbc822

typo

d129e76

remove unnecessary constructors

0297cd9

rebase

b7048b4

remove operation is_ancestor

be5968b

remove specific pickle protocol

f9e28c3

pickle with protocol 2 for objects with __slots__ defined

c7805e7

remove todo note

332fa50

resolve conflicts; simplified table rule

16dbb4e

TypeSignature.from_dtypes for pandas UDFs

9892db5

flake8

d47849d

test nullable propagation

3356427

reintroduce value expr imports in public api

ba36429

release notes

3b6b9c4

syntax error in release notes

d292008

fixing custom operation notebook

eed2ec7

missing newlines

80b85c4

missing import

63f0786

kszucs force-pushed the prainbow branch from 66ed0b8 to 63f0786 Compare April 5, 2018 13:56

cpcloud closed this in 9394436 Apr 5, 2018

kszucs deleted the prainbow branch May 30, 2022 12:40



		def is_reduction(expr):
		# Aggregations yield typed scalar expressions, since the result of an

		from ibis.compat import PY2, to_time, to_date
		from ibis.expr.types import Expr, null, param, literal, sequence, as_value_expr

		@@ -182,22 +176,6 @@ def test_primitive(spec, expected):
		assert dt.dtype(spec) == expected


		def test_precedence_with_no_arguments():

Rules refactor #1366

Rules refactor #1366

Conversation

kszucs commented Feb 24, 2018

cpcloud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Feb 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Mar 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Mar 11, 2018 • edited Loading

Choose a reason for hiding this comment

kszucs Feb 28, 2018 •

edited

Loading

kszucs Mar 11, 2018 •

edited

Loading

kszucs Mar 11, 2018 •

edited

Loading