ENH: Upgrade google-cloud-bigquery dependency to 1.0.0 #1424

cpcloud · 2018-04-14T22:18:49Z

Upgrade to google-cloud-bigquery 1.0.0

tswast

Thanks! A comments, but overall looks good.

tswast · 2018-04-15T03:50:55Z

ibis/bigquery/client.py

@@ -106,11 +100,13 @@ def __init__(self, query):
        self.query = query

    def fetchall(self):
-        return list(self.query.fetch_data())
+        result = self.query.result()


Note: the QueryJob is itself an iterator, and will call result() under the covers as soon as you try to iterate over it. No need to update, it's fine to call result() to be more explicit that the code will block to wait for the query to complete.

Cool, good to know. Thanks!

tswast · 2018-04-15T03:53:59Z

ibis/bigquery/client.py

@@ -189,28 +181,18 @@ def get_datasets(self):
        return list(self.client.list_datasets())

    def get_dataset(self, dataset_id):
-        return self.client.dataset(dataset_id)
+        dataset_ref = self.client.dataset(dataset_id)
+        return self.client.get_dataset(dataset_ref)


Excellent.

Sidenote: the previous version (based on 0.27 client) didn't actually make a GET call. To do that you would have had to also call .reload() on the Dataset object.

Update: Looking at my subsequent comments, maybe we don't actually want to GET a dataset resource? I guess the function name makes it sound like we should. Maybe what we really need is a def dataset_ref(self, dataset_id) function?

I've gone ahead and removed the BigQueryAPIProxy in favor of calls into the client. The additional level of indirection was unnecessary.

This made it easier to use refs as much as possible, so a dataset_ref method shouldn't be necessary anymore.

tswast · 2018-04-15T03:56:03Z

ibis/bigquery/client.py

-        table = self.client.dataset(dataset_id).table(table_id)
-        if reload:
-            table.reload()
+        table_ref = self.get_dataset(dataset_id).table(table_id)


You don't actually have to call get_dataset() here. All you need is a reference, so self.client.dataset(dataset_id).table(table_id) should work. Then you're only doing 1 API call to GET the Table instead of 2 to GET the Dataset and the Table.

Excellent, I'll make this change.

tswast · 2018-04-15T03:57:03Z

ibis/bigquery/client.py

-        query.run()
+        job_config = bq.job.QueryJobConfig()
+        job_config.query_parameters = query_parameters or []
+        job_config.use_legacy_sql = False


Since 0.28, the client has set standard SQL as the default, but okay to keep this line to be explicit about it.

Great. I will leave this here for now to be explicit.

tswast · 2018-04-15T03:58:57Z

ibis/bigquery/client.py

+        job_config = bq.job.QueryJobConfig()
+        job_config.query_parameters = query_parameters or []
+        job_config.use_legacy_sql = False
+        query = self._proxy.client.query(stmt, job_config=job_config)

        # run_sync_query is not really synchronous: there's a timeout


You could call query.result() here to wait. It effectively does this, but with a little more retry logic. If you don't explicitly pass in a timeout to result() it will wait indefinitely as you are doing here.

This sounds like a good idea.

tswast · 2018-04-15T04:00:13Z

ibis/bigquery/client.py

@@ -381,11 +363,13 @@ def exists_table(self, name, database=None):

    def list_tables(self, like=None, database=None):
        dataset = self._proxy.get_dataset(database or self.dataset_id)


You don't actually need to make a GET call to fetch a Dataset resource. All you need to list is a reference, so you could do dataset = client.dataset(database or self.dataset_id)

Great, I'll make this change.

tswast · 2018-04-15T04:04:29Z

ibis/bigquery/client.py

-        assert fields
-        names = [el.name for el in fields]
-        ibis_types = [_discover_type(el) for el in fields]
-        ibis_type = dt.Struct(names, ibis_types)


Note: to_dataframe() in the google-cloud-bigquery library just uses object as the datatype for record and array columns. (I think because Pandas doesn't have these datatypes) Will that cause a problem for Ibis? Is there actually a Pandas Struct and Array type now that we should be using in that library?

There's no struct/array type in pandas. This won't affect ibis, since we already assume this to be true.

👍

Would it make sense for google-cloud-bigquery to add a to_arrow() method like to_dataframe() so that these types have better support here?

tswast · 2018-04-15T04:10:53Z

ibis/bigquery/client.py

-    except BadRequest:
-        pass
+    if table.partitioning_type is not None:
+        pairs.append((NATIVE_PARTITION_COL, dt.timestamp))


Note: this won't be correct for tables that use the new column-based time partitioning.

Unfortunately google-cloud-bigquery still doesn't have the partitioning field property yet googleapis/google-cloud-python#4658 Whenever we add that there, this logic should change to only add the NATIVE_PARTITION_COL when the partitioning field is None.

Is there a surefire way (even if either is a workaround) to 1) tell if a table is partitioned, and 2) what column(s) are being used for partitioning?

Update: I see that there's some more info in the issue, I'll read through that :)

cpcloud · 2018-04-15T18:53:03Z

ibis/bigquery/client.py

 class BigQueryClient(SQLClient):

    sync_query = BigQueryQuery
    database_class = BigQueryDatabase
-    proxy_class = BigQueryAPIProxy
+    table_class = BigQueryTable


The functionality to make this work isn't yet implemented. I've got a PR in the works to do this.

cpcloud · 2018-04-15T19:14:10Z

@tswast This is ready for another review if you're up for it. In particular these two functions (https://github.com/ibis-project/ibis/pull/1424/files#diff-8e85aac896fd8a71d7151fe44de24b5fR257, https://github.com/ibis-project/ibis/pull/1424/files#diff-8e85aac896fd8a71d7151fe44de24b5fR91) that are mucking around in _properties to figure out partition fields.

I'll add some more tests to make sure this works with column based partitions.

kszucs · 2018-04-15T20:30:41Z

ibis/expr/schema.py

 def schema_from_pairs(lst):
    return Schema.from_tuples(lst)


-@schema.register((tuple, list), (tuple, list))
+@schema.register(collections.Iterable, collections.Iterable)


tswast

BigQuery stuff looks great.

Yes, bq_table._properties.get('timePartitioning', None) is the correct workaround.

tswast · 2018-04-16T15:12:05Z

ibis/bigquery/client.py

-        assert fields
-        names = [el.name for el in fields]
-        ibis_types = [_discover_type(el) for el in fields]
-        ibis_type = dt.Struct(names, ibis_types)


👍

Would it make sense for google-cloud-bigquery to add a to_arrow() method like to_dataframe() so that these types have better support here?

tswast · 2018-04-16T15:17:20Z

ibis/bigquery/tests/test_client.py

+def test_parted_column(client, kind, option, expected_fn):
+    table_name = '{}_column_parted'.format(kind)
+    option_key = 'bigquery.partition_col'
+    with ibis.config.option_context(option_key, option):


Why does Ibis have an option to rename the partition column? Or does this do something else?

This option is a convenience to allow users to refer to _PARTITIONTIME, since refering to it without renaming generates an error in BigQuery. For column-based partitioning this isn't a problem, but for tables that are implicitly partitioned we want to do this.

cpcloud · 2018-04-18T12:05:49Z

@tswast regarding to_arrow I think that's a great idea. I'll open an issue over on the bigquery python tracker.

cpcloud · 2018-04-18T13:41:07Z

Merging on green.

cpcloud · 2018-04-18T15:20:13Z

@tswast Created googleapis/google-cloud-python#5204

cpcloud added this to the 0.14 milestone Apr 14, 2018

cpcloud self-assigned this Apr 14, 2018

cpcloud added feature Features or general enhancements docs Documentation related issues or PRs bigquery labels Apr 14, 2018

tswast reviewed Apr 15, 2018

View reviewed changes

cpcloud force-pushed the bigquery-1.0.0-upgrade branch 2 times, most recently from e40386d to 2a59ddf Compare April 15, 2018 18:51

cpcloud commented Apr 15, 2018

View reviewed changes

cpcloud force-pushed the bigquery-1.0.0-upgrade branch from 2a59ddf to d0bdbc7 Compare April 15, 2018 18:54

kszucs reviewed Apr 15, 2018

View reviewed changes

cpcloud force-pushed the bigquery-1.0.0-upgrade branch from d0bdbc7 to 097abdb Compare April 16, 2018 14:34

tswast approved these changes Apr 16, 2018

View reviewed changes

tswast mentioned this pull request Apr 16, 2018

google library versions need to be specified for proxies? #1319

Closed

ENH: Upgrade google-cloud-bigquery dependency to 1.0.0

fa7f0f8

cpcloud force-pushed the bigquery-1.0.0-upgrade branch from d91ae8d to fa7f0f8 Compare April 18, 2018 13:40

cpcloud closed this in 63c9274 Apr 18, 2018

cpcloud deleted the bigquery-1.0.0-upgrade branch April 18, 2018 15:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Upgrade google-cloud-bigquery dependency to 1.0.0 #1424

ENH: Upgrade google-cloud-bigquery dependency to 1.0.0 #1424

cpcloud commented Apr 14, 2018

tswast left a comment

tswast Apr 15, 2018

cpcloud Apr 15, 2018

tswast Apr 15, 2018

cpcloud Apr 15, 2018

cpcloud Apr 15, 2018 •

edited

Loading

tswast Apr 15, 2018

cpcloud Apr 15, 2018

tswast Apr 15, 2018

cpcloud Apr 15, 2018

tswast Apr 15, 2018

cpcloud Apr 15, 2018

tswast Apr 15, 2018

cpcloud Apr 15, 2018

tswast Apr 15, 2018

cpcloud Apr 15, 2018

tswast Apr 16, 2018

tswast Apr 15, 2018

cpcloud Apr 15, 2018

cpcloud Apr 15, 2018

cpcloud Apr 15, 2018

cpcloud commented Apr 15, 2018

kszucs Apr 15, 2018

tswast left a comment

tswast Apr 16, 2018

tswast Apr 16, 2018

cpcloud Apr 16, 2018

cpcloud commented Apr 18, 2018

cpcloud commented Apr 18, 2018

cpcloud commented Apr 18, 2018

		@@ -381,11 +363,13 @@ def exists_table(self, name, database=None):

		def list_tables(self, like=None, database=None):
		dataset = self._proxy.get_dataset(database or self.dataset_id)

ENH: Upgrade google-cloud-bigquery dependency to 1.0.0 #1424

ENH: Upgrade google-cloud-bigquery dependency to 1.0.0 #1424

Conversation

cpcloud commented Apr 14, 2018

tswast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud Apr 15, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Apr 15, 2018

Choose a reason for hiding this comment

tswast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Apr 18, 2018

cpcloud commented Apr 18, 2018

cpcloud commented Apr 18, 2018

cpcloud Apr 15, 2018 •

edited

Loading