ENH: Add to_timestamp support in BigQuery and Pandas #1410

cpcloud · 2018-04-10T23:29:50Z

Update our testing data for parquet

cpcloud · 2018-04-13T16:18:16Z

@kszucs can you review this when you get a chance?

kszucs

Mainly general comments, thoughts. If You agree with them we could convert them to issues.

kszucs · 2018-04-15T19:52:26Z

ibis/bigquery/types.py

@@ -0,0 +1,70 @@
+import six


Just a couple of general thoughts:

We currently have a pretty solid datatype abstraction in expr/datatypes.py, but our conversions between backend and ibis datatypes could be more consistent, see clickhouse, pandas, alchemy
Shouldn't we standardize that?

I've found the mixed usage of types and datatypes identifiers really confusing, so I suggest to rename this file to datatypes.py to be a clear mapping between expr/datatypes.py and backend/datatypes.py (dt -> dt instead of dt -> ir), we should also factor out the datatype conversion codes in all backends.

We currently have a pretty solid datatype abstraction in expr/datatypes.py, but our conversions between backend and ibis datatypes could be more consistent,

Definitely agree with that. infer works well for going from backend type to ibis type, but we need another API (maybe the to_ibis function you mention below) to make it convenient to go from ibis type to backend type.

I've found the mixed usage of types and datatypes identifiers really confusing, so I suggest to rename this file to datatypes.py

Fully agree, I'll rename the file.

kszucs · 2018-04-15T19:55:22Z

ibis/bigquery/types.py

+    __slots__ = ()
+
+
+ibis_type_to_bigquery_type = Dispatcher('ibis_type_to_bigquery_type')


I know that explicit is always better than implicit, but this is a bit verbose to me. I've used from_ibis and to_ibis in other backends inside datatypes scope, which might be too implicit. Is there a golden section?

Let's refactor this in another PR, I'll make an issue.

kszucs · 2018-04-15T19:57:54Z

ibis/bigquery/udf/core.py

-def trans_integer(t):
-    # BigQuery doesn't accept integer types because javascript doesn't
+@ibis_type_to_bigquery_type.register(dt.Integer, UDFContext)
+def trans_integer(t, context):


Could You write a short reasoning?

Yes, sorry, I thought I had a comment there. I'll add.

kszucs · 2018-04-15T20:01:19Z

ibis/bigquery/types.py

+import ibis.expr.datatypes as dt
+
+
+class TypeTranslationContext(object):


At first look this class seems unused, until the UDFContext definition. A little docstring would be handy :)

kszucs · 2018-04-15T20:04:02Z

ibis/bigquery/types.py

+    return ibis_type_to_bigquery_type(t, TypeTranslationContext())
+
+
+@ibis_type_to_bigquery_type.register(six.string_types)


I would not support string types, that's rather the responsibility of the caller, see my comment in api.py.

Cool, will change.

kszucs · 2018-04-15T20:04:27Z

ibis/bigquery/udf/api.py

@@ -156,12 +158,14 @@ def compiles_udf_node(t, expr):
 return {name}({args});
 """;'''.format(
            name=f.__name__,
-            return_type=ibis_type_to_bigquery_type(output_type),
+            return_type=ibis_type_to_bigquery_type(
+                output_type, type_translation_context),


How about

ibis_type_to_bigquery_type(dt.dtype(output_type), type_translation_context)

?

kszucs · 2018-04-15T20:05:05Z

ibis/bigquery/udf/api.py

            source=source,
            signature=', '.join(
                '{name} {type}'.format(
                    name=name,
-                    type=ibis_type_to_bigquery_type(type)
+                    type=ibis_type_to_bigquery_type(
+                        type, type_translation_context)


Same as above:

ibis_type_to_bigquery_type(dt.dtype(type), type_translation_context)

kszucs · 2018-04-15T20:14:11Z

ibis/impala/compiler.py

@@ -734,15 +734,19 @@ def _truncate(translator, expr):
    return "trunc({}, '{}')".format(arg_formatted, unit)


+TIMESTAMP_UNIT_CONVERSIONS = {


After factoring out _convert_unit could we reuse it here?

I'll look into that.

kszucs · 2018-04-15T20:18:16Z

ibis/bigquery/compiler.py

+@compiles(ops.Floor)
+def compiles_floor(t, e):
+    bigquery_type = ibis_type_to_bigquery_type(e.type())
+    arg, = e.op().args


Simply arg = e.op().arg?
I guess we should favor using named arguments instead of unpacking args.

Agreed, will change.

cpcloud added this to the 0.14 milestone Apr 10, 2018

cpcloud self-assigned this Apr 10, 2018

cpcloud force-pushed the timestamp-from-unix-bigquery branch 5 times, most recently from 795f2f1 to 1a35039 Compare April 13, 2018 14:43

cpcloud force-pushed the timestamp-from-unix-bigquery branch from 1a35039 to 53dfb14 Compare April 15, 2018 19:32

kszucs approved these changes Apr 15, 2018

View reviewed changes

cpcloud added 20 commits April 15, 2018 18:44

ENH: Add to_timestamp support in BigQuery and Pandas

888b32a

Rename types.py to datatypes.py

2741254

Explain the need to translate int64 to float64 in udfs

b6cb3a0

Docstring for type translation

59ad6ac

Remove string dispatches

9b7ec4b

use named args

b82330c

Factor out convert_unit

855197f

Call dtype

7aa4a45

Test udf context

2f5f953

Docstring for convert_unit

b8df182

Remove copyright header

7ea1ac9

fix impala compiler tests

ef2c514

Add doctest example

cba682a

Do not convert if the units are the same

a024734

Fix a bug where slicing was incorrect

fd70bed

code formatting

3080adf

Add an error doctest

7ac4485

Formatting

8e685c7

More readable error message

23ef10c

Remove dead code

a7a4392

cpcloud force-pushed the timestamp-from-unix-bigquery branch from 9d33c09 to a7a4392 Compare April 15, 2018 22:58

cpcloud closed this in f917f7c Apr 16, 2018

cpcloud deleted the timestamp-from-unix-bigquery branch April 16, 2018 14:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Add to_timestamp support in BigQuery and Pandas #1410

ENH: Add to_timestamp support in BigQuery and Pandas #1410

cpcloud commented Apr 10, 2018 •

edited

Loading

cpcloud commented Apr 13, 2018

kszucs left a comment

kszucs Apr 15, 2018

cpcloud Apr 15, 2018

kszucs Apr 15, 2018

cpcloud Apr 15, 2018

cpcloud Apr 15, 2018

kszucs Apr 15, 2018

cpcloud Apr 15, 2018

kszucs Apr 15, 2018

cpcloud Apr 15, 2018

kszucs Apr 15, 2018

cpcloud Apr 15, 2018

kszucs Apr 15, 2018

cpcloud Apr 15, 2018

kszucs Apr 15, 2018

cpcloud Apr 15, 2018

kszucs Apr 15, 2018

cpcloud Apr 15, 2018

kszucs Apr 15, 2018

cpcloud Apr 15, 2018

		__slots__ = ()


		ibis_type_to_bigquery_type = Dispatcher('ibis_type_to_bigquery_type')

		import ibis.expr.datatypes as dt


		class TypeTranslationContext(object):

		return ibis_type_to_bigquery_type(t, TypeTranslationContext())


		@ibis_type_to_bigquery_type.register(six.string_types)

		@@ -734,15 +734,19 @@ def _truncate(translator, expr):
		return "trunc({}, '{}')".format(arg_formatted, unit)


		TIMESTAMP_UNIT_CONVERSIONS = {

ENH: Add to_timestamp support in BigQuery and Pandas #1410

ENH: Add to_timestamp support in BigQuery and Pandas #1410

Conversation

cpcloud commented Apr 10, 2018 • edited Loading

cpcloud commented Apr 13, 2018

kszucs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Apr 10, 2018 •

edited

Loading