Added rules for validating that tables have certain columns and for c… #1298

DiegoAlbertoTorres · 2018-01-24T20:51:13Z

…olumn type checking.

DiegoAlbertoTorres · 2018-01-29T16:18:38Z

ibis/expr/rules.py

+    Parameters
+    ----------
+    satisfying : iterable
+        An iterable of column rules.


Ah crap, have to finish this docstring.

DiegoAlbertoTorres · 2018-01-31T18:51:47Z

ibis/expr/rules.py

+        self.doc = doc
+        self.validator = validator
+
+    def _validate(self, args, i):


@cpcloud I just realized that with the current implementation a table can pass validation simply by having enough columns to pass the rules. We can of course control the amount of columns with lambda rules, but this might be counterintuitive. Maybe we should have an argument like allow_extra, which changes the behavior for allowing columns that do not match a rule.

I implemented this.

jreback · 2018-02-02T23:31:14Z

ibis/expr/rules.py

+        An iterable of lambda expressions, which will be used to validate
+        arguments. Each lambda expression must take a table as its single
+        argument, validate, and then return ``True`` for tables that
+        pass validation and ``False`` otherwise.


is it possible to have satisfying or schema (or both as you have them now)?

iow what if i don’t care about a specific schema
i could validate in satisfying (or do some general rule inside that)

DiegoAlbertoTorres · 2018-02-06T18:28:16Z

I cleaned up the example a bit. Once tests pass I think this will be ready.

DiegoAlbertoTorres · 2018-02-06T22:52:24Z

ibis/expr/rules.py

+    validator : ???
+        ???
+    doc : ???
+        ???


@cpcloud I couldn't figure what to do with these (the validator and doc args). I don't use them but stuff breaks if I don't set them to None. Maybe I could set them to None manually (without turning them into args), but first I want to make sure that it makes sense. Do you know how these are used?

doc lets you give a docstring to the argument:

In [1]: class Foo(ibis.expr.operations.ValueOp): ...: input_type = [ibis.expr.rules.double(name='foo', doc='bar bar double')] ...: In [2]: x = Foo(1) In [3]: x.a? Object `x.a` not found. In [4]: x.foo? Type: property String form: <property object at 0x7fedee9b2278> Docstring: bar bar double

Looks like validators let you pass in your own validation function, see line 239 in this file.

I see. Should we leave it? Can't think of any use case for it, if users want a custom validator they should just write one, right?

bump @cpcloud

jreback · 2018-02-07T16:16:27Z

ibis/expr/rules.py

+        arguments. Each lambda expression must take a table as its single
+        argument, validate, and then return ``True`` for tables that
+        pass validation and ``False`` otherwise.
+    schema : iterable


mark as optional

jreback · 2018-02-07T16:16:35Z

ibis/expr/rules.py

+        The name of the table argument.
+    optional : bool
+        Whether this table argument is optional or not.
+    satisfying : iterable


mark as optional

jreback · 2018-02-07T16:17:34Z

ibis/expr/tests/test_rules.py

+        MyOp(123)
+
+
+def test_table_invalid_satisfying():


can you add a test that doesn't pass either satisfying or schema

There were already some tests, but I just added more.

jreback · 2018-02-11T15:43:33Z

ibis/expr/rules.py

+            # Check column schema
+            rules_matched = 0
+            for column_rule in self.schema:
+                if not isinstance(column_rule, Argument):


can you add some comments on the various sections on what is going on / what you are checking

jreback · 2018-02-11T15:44:21Z

ibis/expr/rules.py

+    ...                lambda t: t.schema().types.count(dt.Int64()) >= 2],
+    ...        )]
+    ...    output_type = rules.type_of_arg(0)
+    """


add a Notes section about satisfying, schema, allow_extra can be mixed / matched to produce a validator and not all are needed

jreback · 2018-02-11T15:45:25Z

ibis/expr/tests/test_rules.py

+            Op(table)
+
+
+@pytest.mark.parametrize(


I would separate the tests that raise from the ones that don't and don't use check_op_input at all. Much more readable.

I tried that and it was a mess, because the same schemas would be repeated twice, once for tests passing validation and those not passing.

I think it is waaay clearer to say "given this schema, here are things that pass and things that don't pass".

Why would they be repeated? There's a set of schemas for passing, and one for failing. This should definitely be separated into two tests: test_table_with_schema and test_table_with_invalid_schema. The True/False is hard to follow. If the name of the function indicates what's supposed to happen, then it's very clear what the test is testing.

Sorry, didn't explain this right. Indeed, the schemas for the tables being validated would not be repeated. Bu the schema we are comparing against will be repeated.

This would end up looking as:

@pytest.mark.parametrize( "... many passing test cases here ...") def test_table_with_schema(): "<validation code with a schema>" @pytest.mark.parametrize( "... failing test cases here ...") def test_table_with_schema_failing(): with pytest.raises(IbisTypeError): "<same validation code with the same schema>"

Eh, did it like that. Its not so bad.

DiegoAlbertoTorres · 2018-02-16T21:54:52Z

@jreback bump

jreback · 2018-02-16T22:00:51Z

lgtm. needs a rebase. if @cpcloud is ok.

cpcloud · 2018-02-19T01:07:45Z

@DiegoAlbertoTorres @jreback

I don't understand why the complexity of this is needed. As a possible strawman, what's wrong with doing the validation of the table in the operation itself, like we currently do with joins?

Column rules are convenient because most operations are column operations and there are a ton of them, but there are very few operations that take tables as input and if they do, they are (and should be) unrestricted. Sure, we may want to write more operations that takes tables as input, but there aren't so many that having a generic table rule this complex is warranted.

How about this?

We define the following classmethod on the Table object:

def with_column_subset(...):
    """must have at least these columns"""

and that's it. I think we can get pretty far without things like allow_extra and satisfying. We should start simple and go from there.

I don't think there are any operations (yet) where something like satisfying is needed. If something like that comes up, we'll add the functionality then.

Sound good?

jreback · 2018-02-19T01:16:50Z

i agree this is more complex than needed atm
but the core operation we need to write is

a table that for example

has a column named foo of type time
has a column of type int32
has at one or more double columns

cpcloud · 2018-02-19T01:26:40Z

I think the use case you have in mind for the third case--one or more double columns--can be solved by giving it a different API than one that would require such a rule.

The with_column_subset method would cover cases 1 and 2 (unnamed and named columns with a particular type).

cpcloud · 2018-02-19T01:34:48Z

The API with this pared down version would be:

rules.table.with_column_subset([
    rules.column(name='foo', value_type=dt.timestamp),
    rules.column(value_type=int32)
])

If what you meant by an int32 column was "exactly one int32 column", that would be another method exactly_one_of(rules.column(value_type=dt.int32)), maybe we define an and_ method that allows you to compose them.

jreback · 2018-02-19T01:35:03Z

the key is i need a conjunction of all of these in a particular table ; separately they r straightforward

DiegoAlbertoTorres · 2018-02-28T16:28:15Z

Having with_column_subset and exactly_one_of and composing them sounds pretty much like having schema and allow_extra. Maybe we just want to keep these two and do away with satisfying?

…olumn type checking.

…ns. Fixed tests.

…olumnValidator instance.

jreback

lgtm. some minor comments

jreback · 2018-03-07T21:38:52Z

ci/requirements-dev-3.6.yml

@@ -26,7 +26,7 @@ dependencies:
  - requests
  - six
  - sqlalchemy>=1.0.0,<1.1.15
-  - thrift
+  - thrift>=0.10.0,<0.11.0


is this orthogonal change?

Actually @DiegoAlbertoTorres something is strange, since @jreback's code is passing with this commit: 60c5806. You shouldn't need to update this since he didn't have to.

Yeah, tests were refusing to pass (and still are).

Got rid of the pin.

jreback · 2018-03-07T21:40:56Z

ibis/expr/tests/test_rules.py

+     ([('group', dt.int64),
+       ('value', dt.double),
+       ('value2', dt.double)], False)])
+def test_table_with_schema(schema, raises):


do you have an example of using with_coumn_subset with a rules that is NOT a column? (should raise)

Just added one.

DiegoAlbertoTorres · 2018-03-08T18:47:34Z

CI passes!!

cpcloud · 2018-02-09T02:27:19Z

ibis/expr/rules.py

+
+    def _validate(self, args, i):
+        arg = args[i]
+        if isinstance(arg, ir.TableExpr):


Let raise if it's not a table as the first thing, then we can remove a level of indentation.

I do this now, I think you commented on an outdated diff?

cpcloud · 2018-02-09T02:27:42Z

ibis/expr/rules.py

+            if not self.allow_extra:
+                # Count rules
+                rules = [x for x in self.schema if isinstance(x, Argument)]
+                if ((len(rules) != 0) and


You can just say if rules here

Outdated diff, this is gone!

cpcloud · 2018-02-09T02:30:50Z

ibis/expr/rules.py

+                    column = arg[column_rule.name]
+                except IbisTypeError:
+                    if column_rule.optional:
+                        break


I'm not sure I follow the logic of using break here. This will exit the loop. Don't we want to check additional column rules that may not be optional?

This is gone too

cpcloud · 2018-02-09T02:32:14Z

ibis/expr/rules.py

+                rules = [x for x in self.schema if isinstance(x, Argument)]
+                if ((len(rules) != 0) and
+                        (len(arg.columns) > rules_matched)):
+                    raise IbisTypeError('Extra columns not allowed.')


How about "Additional columns beyond {} not allowed".format(list of names)?

cpcloud · 2018-03-07T21:55:04Z

ci/requirements-dev-3.6.yml

@@ -26,7 +26,7 @@ dependencies:
  - requests
  - six
  - sqlalchemy>=1.0.0,<1.1.15
-  - thrift
+  - thrift>=0.10.0,<0.11.0


Actually @DiegoAlbertoTorres something is strange, since @jreback's code is passing with this commit: 60c5806. You shouldn't need to update this since he didn't have to.

cpcloud · 2018-03-07T21:55:18Z

ibis/expr/rules.py

+    def __str__(self):
+        fields = {'name', 'doc', 'optional'}
+        return str({k: v for k, v in self.__dict__.items()
+                    if (k in fields) and (v is not None)})


Don't need parens here.

cpcloud · 2018-03-07T21:56:00Z

ibis/expr/rules.py

@@ -801,3 +807,120 @@ def _validate(self, args, i):

 def comparable(left, right):
    return ir.castable(left, right) or ir.castable(right, left)
+
+
+@six.add_metaclass(abc.ABCMeta)


Do this as:

class TableColumnValidator(six.with_metaclass(object, abc.ABCMeta)): ...

Fixed this like we said in person.

cpcloud · 2018-03-07T21:58:43Z

ibis/expr/tests/test_rules.py

+            Op(table)
+
+
+@pytest.mark.parametrize(


Why would they be repeated? There's a set of schemas for passing, and one for failing. This should definitely be separated into two tests: test_table_with_schema and test_table_with_invalid_schema. The True/False is hard to follow. If the name of the function indicates what's supposed to happen, then it's very clear what the test is testing.

cpcloud · 2018-03-09T22:40:20Z

ibis/expr/rules.py

+                    if column_rule.optional:
+                        break
+                    else:
+                        raise IbisTypeError(


This whole thing can be written much more clearly:

if not column_rule.optional and column_rule.name not in arg: raise IbisTypeError(...)

Still have to skip if optional. Will do:

if column_rule.name not in arg: if column_rule.optional: continue else: raise IbisTypeError( 'No column with name {}.'.format(column_rule.name))

cpcloud · 2018-03-09T22:41:32Z

ibis/expr/rules.py

+            # Check that columns match the schema first
+            for column_rule in self.rules:
+                # Members of a schema are arguments with a name
+                if not isinstance(column_rule, Argument):


There's a thing that makes stuff into Argument instances somewhere so you shouldn't have to do this. It's very hard to get to the meat of the code here because of all the checking.

Do you mean _to_argument? It seems like most argument subclasses call _to_argument directly, so we wouldn't even get it for free if I made SubsetValidator a child of Argument. Also, we always want to take an Argument here (can't think of anything we may want to upcast), so I think it is a good idea to check it explicitly.

Although I agree it is hard to get to the meat of the code with all the validation. I will split the rule checking from the table validation.

Ok, I made a new function to validate rules. Also it is now called in the constructor, not during validation.

(I think this is the only outstanding-ish comment).

cpcloud · 2018-03-09T22:42:08Z

ibis/expr/rules.py

+    ...    output_type = rules.type_of_arg(0)
+    """
+    def __init__(self, name=None, optional=False, schema=None, doc=None,
+                 validator=None, allow_extra=False, **arg_kwds):


Do we still need allow_extra? I thought we had decided to get rid of that.

:( forgot to remove it here srry

Removed it. 👍

cpcloud · 2018-03-13T21:19:02Z

@DiegoAlbertoTorres Merged! Thanks!

DiegoAlbertoTorres force-pushed the rules branch 4 times, most recently from 0a398a5 to a609965 Compare January 29, 2018 16:13

DiegoAlbertoTorres changed the title ~~WIP: Added rules for validating that tables have certain columns and for c…~~ Added rules for validating that tables have certain columns and for c… Jan 29, 2018

DiegoAlbertoTorres commented Jan 29, 2018

View reviewed changes

DiegoAlbertoTorres commented Jan 31, 2018

View reviewed changes

jreback reviewed Feb 2, 2018

View reviewed changes

DiegoAlbertoTorres commented Feb 6, 2018

View reviewed changes

jreback reviewed Feb 7, 2018

View reviewed changes

DiegoAlbertoTorres force-pushed the rules branch from e2e9ff7 to 42f292c Compare February 8, 2018 20:18

cpcloud added this to the 0.13 milestone Feb 9, 2018

jreback added the feature Features or general enhancements label Feb 11, 2018

jreback suggested changes Feb 11, 2018

View reviewed changes

jreback approved these changes Feb 16, 2018

View reviewed changes

DiegoAlbertoTorres added 6 commits March 1, 2018 17:30

Added rules for validating that tables have certain columns and for c…

1422db0

…olumn type checking.

Extended docstring, added invalid input test.

5ae0bfc

Table rule now takes a schema and a list of custom validation functio…

f8268e3

…ns. Fixed tests.

Fixed bug: table argument overwriting other arguments within same rule.

6916c38

Fix Python 2 compatibility.

8b3d6a4

Style fixup.

2c5943e

DiegoAlbertoTorres added 6 commits March 1, 2018 17:30

Added import to example.

058a9a9

Cleaned up example description.

78e0ab1

Extended tests.

5f30e56

Added notes to Table rule and comments.

4d838a6

Simplified table rules. Removed satisfying, schema now takes a TableC…

28cfe76

…olumnValidator instance.

Fixed Python2 compatibility. Added test for wrong schema class.

5aa137f

DiegoAlbertoTorres force-pushed the rules branch from 9a21f1a to 5aa137f Compare March 1, 2018 22:30

Pinned thrift to 0.10.x on ci/requirements-dev-3.6.yml

8257d63

DiegoAlbertoTorres closed this Mar 7, 2018

DiegoAlbertoTorres reopened this Mar 7, 2018

DiegoAlbertoTorres closed this Mar 7, 2018

DiegoAlbertoTorres reopened this Mar 7, 2018

jreback reviewed Mar 7, 2018

View reviewed changes

Added test for bad with_column_subset input.

8a14440

cpcloud requested changes Mar 9, 2018

View reviewed changes

cpcloud reviewed Mar 9, 2018

View reviewed changes

DiegoAlbertoTorres added 2 commits March 12, 2018 17:50

CR changes.

644e54f

Linted.

1813903

cpcloud closed this Mar 13, 2018

cpcloud reopened this Mar 13, 2018

cpcloud approved these changes Mar 13, 2018

View reviewed changes

cpcloud closed this in a885e70 Mar 13, 2018

Added rules for validating that tables have certain columns and for c… #1298

Added rules for validating that tables have certain columns and for c… #1298

Conversation

DiegoAlbertoTorres commented Jan 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DiegoAlbertoTorres commented Feb 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud Feb 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DiegoAlbertoTorres Feb 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DiegoAlbertoTorres commented Feb 16, 2018

jreback commented Feb 16, 2018

cpcloud commented Feb 19, 2018 • edited Loading

jreback commented Feb 19, 2018

cpcloud commented Feb 19, 2018

cpcloud commented Feb 19, 2018

jreback commented Feb 19, 2018

DiegoAlbertoTorres commented Feb 28, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DiegoAlbertoTorres commented Mar 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DiegoAlbertoTorres Mar 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Mar 13, 2018

cpcloud Feb 9, 2018 •

edited

Loading

DiegoAlbertoTorres Feb 8, 2018 •

edited

Loading

cpcloud commented Feb 19, 2018 •

edited

Loading

DiegoAlbertoTorres Mar 12, 2018 •

edited

Loading