BUG: Fix subqueries with parameters #1303

cpcloud · 2018-01-25T21:31:47Z

closes #1300
closes #1331

This was a somewhat invasive change, but nets a lot fewer places where the params argument has to be used outside of the compile and execute functions and methods on clients. Now, params are part of the context and context is never None so it's easier to reason about where parameters are coming from.

cpcloud · 2018-01-30T18:36:34Z

@wesm @kszucs can you review when you get a chance?

kszucs

A short summary on top of ibis/sql/compiler.py would be great, a general overview of the classes and the relationships between them.

It was pretty hard to understand the abstractions here when I wrote the clickhouse backend.

kszucs · 2018-01-30T21:11:23Z

ibis/bigquery/client.py

@@ -46,14 +55,30 @@ def __exit__(self, exc_type, exc_value, traceback):

 class BigQuery(Query):

+    def __init__(self, *args, **kwargs):


How about explicit args?

def __init__(self, client, ddl, params=None): super(BigQuery, self).__init__(client, ddl) self.query_parameters = params or []

Good idea, will change. This originally was just so that I could do the default initialization and be robust to signature changes in the parent.

kszucs · 2018-01-30T21:13:16Z

ibis/bigquery/client.py

+    def __init__(self, *args, **kwargs):
+        query_parameters = kwargs.pop('query_parameters', [])
+        super(BigQuery, self).__init__(*args, **kwargs)
+        self.query_parameters = query_parameters


I might just hate typing but ibis.bigquery.BigQuery.query_parameters is a bit verbose to me.

Users don't interact with ibis at this level, and this is an instance variable anyway so the most you'd ever type is self.query_parameters

kszucs · 2018-01-30T21:23:05Z

ibis/bigquery/compiler.py

@@ -260,6 +245,13 @@ class BigQueryExprTranslator(impala_compiler.ImpalaExprTranslator):
    _registry = _operation_registry
    _rewrites = impala_compiler.ImpalaExprTranslator._rewrites.copy()

+    context_class = BigQueryContext


This is inconsistent with the BigQuerySelect.translator which is not postfixed with _class

There's context_class which is a class variable, and context which is an instance of context_class, so they are consistent, unless I'm misunderstanding you here.

I mean BigQuerySelect.translator is holding a class too: BigQueryExprTranslator.

I'll make a follow up issue to fix the inconsistency. In the meantime, I'm going to merge this next.

Great! :) You might check #1315 next

kszucs · 2018-01-30T21:26:10Z

ibis/client.py

@@ -243,7 +234,8 @@ def execute(self, expr, params=None, limit='default', async=False):

    def _execute_query(self, ddl, async=False):
        klass = self.async_query if async else self.sync_query
-        return klass(self, ddl).execute()
+        inst = klass(self, ddl)
+        return inst.execute()


The previous one was nicer :)

Yeah, but this is +epsilon easier to debug.

kszucs · 2018-01-30T21:38:35Z

ibis/sql/compiler.py

-            context = self.context
+    @property
+    def translator(self):
+        return self.context.dialect.translator


It's quite hard to distinguish between Context, Dialect and Translator. Too many indirections for me.
What is the relationship between them?

I'll add some documentation for this in ibis/sql/compiler.py.

cpcloud · 2018-02-05T01:57:21Z

ibis/bigquery/tests/test_client.py

+    param = ibis.param('int64')
+    expr = alltypes[alltypes.string_col.cast('int64') == param]
+
+    string_value = 0


I need to change these names to reflect the actual values.

kszucs · 2018-02-07T12:46:36Z

ibis/tests/backends.py

@@ -57,6 +57,14 @@ def batting(self):
    def awards_players(self):
        return self.connection.database().awards_players

+    @classmethod
+    def make_context(cls, params=None):
+        module_name = cls.module


There is a property for getting the module's name: self.name
We should use either explicit module variables or a method which converts the classname.

We should use either explicit module variables or a method which converts the classname.

I'm not sure I follow this. Isn't self.name enough?

cpcloud · 2018-02-07T21:16:12Z

merging on green

cpcloud self-assigned this Jan 25, 2018

cpcloud requested a review from wesm January 25, 2018 21:31

cpcloud added the bug Incorrect behavior inside of ibis label Jan 25, 2018

cpcloud added this to the 0.13 milestone Jan 25, 2018

cpcloud force-pushed the fix-scalar-params branch 3 times, most recently from b607536 to 94bb3e2 Compare January 27, 2018 20:34

cpcloud changed the title ~~BUG: Fix subqueries with parameters~~ WIP/BUG: Fix subqueries with parameters Jan 27, 2018

cpcloud force-pushed the fix-scalar-params branch from 10ca2ef to 61c699d Compare January 28, 2018 00:38

cpcloud added the bigquery label Jan 28, 2018

cpcloud force-pushed the fix-scalar-params branch 2 times, most recently from 3909737 to 6c4aee3 Compare January 28, 2018 16:51

kszucs reviewed Jan 30, 2018

View reviewed changes

cpcloud force-pushed the fix-scalar-params branch 4 times, most recently from ff16f0b to 0d5247e Compare February 4, 2018 22:08

cpcloud commented Feb 5, 2018

View reviewed changes

cpcloud force-pushed the fix-scalar-params branch 4 times, most recently from f8c7261 to 3330226 Compare February 6, 2018 16:41

kszucs reviewed Feb 7, 2018

View reviewed changes

cpcloud force-pushed the fix-scalar-params branch from d02cca5 to 13f2f8d Compare February 7, 2018 15:02

cpcloud changed the title ~~WIP/BUG: Fix subqueries with parameters~~ BUG: Fix subqueries with parameters Feb 7, 2018

BUG: Fix scalar parameters

eb285de

cpcloud force-pushed the fix-scalar-params branch from f5f4042 to eb285de Compare February 7, 2018 20:40

cpcloud mentioned this pull request Feb 7, 2018

Fix inconsistency between context_class and translator #1340

Closed

cpcloud closed this in 780084d Feb 7, 2018

cpcloud deleted the fix-scalar-params branch February 7, 2018 23:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Fix subqueries with parameters #1303

BUG: Fix subqueries with parameters #1303

cpcloud commented Jan 25, 2018 •

edited

cpcloud commented Jan 30, 2018

kszucs left a comment

kszucs Jan 30, 2018

cpcloud Feb 7, 2018

kszucs Jan 30, 2018 •

edited

cpcloud Feb 7, 2018

kszucs Jan 30, 2018

cpcloud Feb 7, 2018

kszucs Feb 7, 2018 •

edited

cpcloud Feb 7, 2018

kszucs Feb 7, 2018

kszucs Jan 30, 2018

cpcloud Feb 7, 2018

kszucs Jan 30, 2018 •

edited

cpcloud Feb 7, 2018

kszucs Feb 7, 2018

cpcloud Feb 5, 2018

kszucs Feb 7, 2018

cpcloud Feb 7, 2018

cpcloud commented Feb 7, 2018

		@@ -46,14 +55,30 @@ def __exit__(self, exc_type, exc_value, traceback):

		class BigQuery(Query):

		def __init__(self, args, *kwargs):

BUG: Fix subqueries with parameters #1303

BUG: Fix subqueries with parameters #1303

Conversation

cpcloud commented Jan 25, 2018 • edited

cpcloud commented Jan 30, 2018

kszucs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Jan 30, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Feb 7, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Jan 30, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Feb 7, 2018

cpcloud commented Jan 25, 2018 •

edited

kszucs Jan 30, 2018 •

edited

kszucs Feb 7, 2018 •

edited

kszucs Jan 30, 2018 •

edited