Dtype, schema inference and implicit casting #1269

kszucs · 2017-12-15T10:09:31Z

Relates to #1221

cpcloud · 2017-12-15T22:15:42Z

ibis/expr/datatypes.py

+
+@castable.register(Interval, Interval)
+def can_cast_intervals(source, target):
+    return castable(source.value_type, target.value_type)


Hm, this is tricky because there are a few cases where casting between intervals becomes ambiguous, like casting anything longer than a week (that isn't a week itself) to weeks, e.g., casting 1 month to weeks. Some months may have more weeks than others.

Casting seconds to months is also fraught with annoyances.

I think we will need to check the units to make sure these are valid casts.

cpcloud · 2017-12-22T21:23:32Z

@kszucs Looks like you need to rebase.

… operations

cpcloud · 2017-12-26T15:22:10Z

ibis/expr/operations.py

-        self._assert_can_compare()
-        return rules.shape_like_args(self.args, 'boolean')
+        if not rules.comparable(self.left, self.right):
+            raise TypeError('Arguments are not comparable')


Can you show self.left.type() and self.right.type() in the error message?

cpcloud · 2017-12-26T15:25:17Z

ibis/expr/tests/test_schema.py

+    ([pd.Timedelta('1 days'),
+      pd.Timedelta('-1 days 2 min 3us'),
+      pd.Timedelta('-2 days +23:57:59.999997')], dt.Interval('ns')),
+    # (pd.Categorical(['a', 'b', 'c', 'a']), 'category')


Let's mark this as pytest.mark.xfail with a specific exception.

Actually It passes locally. How should I create a categorical column working in each build?

cpcloud · 2017-12-26T15:30:30Z

ibis/util.py

 from ibis.config import options


+@contextmanager
+def ignoring(*exceptions):


This exists in the standard library as contextlib.suppress since Python 3.4.

How about something inibis/compat.py like:

try: from contextlib import suppress except ImportError: import contextlib @contextlib.contextmanager def suppress(...): ...

Great :) I wasn't aware.

cpcloud · 2017-12-26T15:36:09Z

ibis/impala/tests/test_udf.py

            ('string', [], self.all_cols[:7] + self.all_cols[8:]),
            ('timestamp', [], self.all_cols[:-1]),
-            ('decimal', [], self.all_cols[:4] + self.all_cols[7:])
+            ('decimal', self.all_cols[:7], self.all_cols[7:])


We really should clean up this test, it's quite hard to understand what's being tested here. I'll make an issue.

Probably we should port these to the new test suite

cpcloud · 2017-12-26T15:50:04Z

ibis/expr/datatypes.py

+    def issubtype(self, parent):
+        return issubtype(self, parent)
+
+    def castable(self, target):


Can you add a docstring here indicating that castable here means implicitly castable?

Sure, I might be more verbose.

cpcloud · 2017-12-26T15:54:41Z

ibis/expr/datatypes.py

+    return True
+
+
+@castable.register(UnsignedInteger, UnsignedInteger)


You can chain decorators here so you don't have to repeat the code for signed and unsigned integers, like this:

@castable.register(UnsignedInteger, UnsignedInteger) @castable.register(SignedInteger, SignedInteger)

In fact, any of these implementations that take the same number of arguments and return True can all be chained into one function to avoid repetition.

cpcloud · 2017-12-26T15:59:42Z

ibis/expr/datatypes.py

+    return TypeParser(value).parse()
+
+
+infer = Dispatcher('infer')


Let's add a docstring to this (using the doc= argument to Dispatcher)

To clarify: from reading the code it looks like dtype is for mapping types from other systems (e.g., pandas and numpy) into ibis's type system, and infer is for getting the ibis schema or ibis type of data values such as literals, lists and DataFrames. Is that correct?

Can you make sure to clarify the distinction between these two functions in their respective docstrings. Some examples in the docstrings will likely help here.

Would You please convert this to an issue?

cpcloud · 2017-12-26T16:00:12Z

ibis/expr/datatypes.py

+})
+
+
+dtype = Dispatcher('dtype')


Docstring here

cpcloud · 2017-12-26T16:05:22Z

ibis/expr/tests/test_datatypes.py

-    assert customers.schema() == expected
+@pytest.mark.parametrize(('pandas_dtype', 'ibis_dtype'), [
+    (DatetimeTZDtype(tz='US/Eastern', unit='ns'), dt.Timestamp('US/Eastern')),
+    # (CategoricalDtype(['a', 'b']), dt.Category(2))  compat problem


let's make this pytest.mark.xfail with a specific exception.

Same thing with the categorical column, how can I create a categorical dtype?

cpcloud · 2018-01-08T17:25:26Z

LGTM, merging.

cpcloud · 2018-01-08T17:34:31Z

@kszucs Thank you!

kszucs changed the title ~~[WIP] Dtype, schema inference~~ [WIP] Dtype, schema inference and implicit casting Dec 15, 2017

cpcloud reviewed Dec 15, 2017

View reviewed changes

cpcloud added the feature Features or general enhancements label Dec 18, 2017

cpcloud added this to the 0.13 milestone Dec 18, 2017

kszucs added 25 commits December 24, 2017 14:45

infer_dtype draft

926943b

make multipledispatch mandatory dependency

d332d8a

refactored datatype implicit casting

27471f9

multipledispatch to conda build reqs

11ecaf7

fixed failed decimal to floating casts

7b6faf4

resolve builtins float name conflict

1aaa292

allow double->float because float literal is inferred as double

a4f7874

factor out _can_implicit_cast and _implicit_cast calls for comparison…

d97432d

… operations

factor out ir types' _can_compare methods

7ea04f4

infer numpy generic types

a0ab226

integer to category cast

191a476

tests for implicit dtype cast

a54f662

infer pandas schema

c2d82a8

simplify higher precedence dtype

774c317

forbid double to float implicit cast

6c49f56

higher precedence only upcasts

43a581e

fix doctest errors

fb04c84

pass literal value to implicit casts; fix failing udf test;

0b646f6

numpy binary dtype

13a09f7

comment out array casting

27cae31

little rename; trigger builds

4bd5c5b

subtype checking

e7aecc5

infer pandas timestamp

ee0a27c

fix pandas types namespace

337430d

another attempt to fix pandas types namespace

acd27ec

kszucs added 2 commits December 24, 2017 14:45

fix DatetimeTZDtype

affff8b

support for pandas categorical

ef2439e

kszucs force-pushed the infer_schema branch from e7534c0 to ef2439e Compare December 24, 2017 14:02

kszucs added 3 commits December 24, 2017 15:59

flake8

8b3102e

resolve compat issue

a461e99

comment out test case

bcc6d25

cpcloud reviewed Dec 26, 2017

View reviewed changes

ibis/expr/datatypes.py

})

dtype = Dispatcher('dtype')

Copy link

Member

cpcloud Dec 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docstring here

cpcloud reviewed Dec 26, 2017

View reviewed changes

kszucs added 6 commits December 27, 2017 18:47

resolve a cuple of review issues

b6369c9

missing contextmanager decorator for compat.suppress

4ccb42e

to_pandas skeleton

85beb27

categorical

3558d95

schema.to_pandas test

2a1578a

revert to_pandas

93bd201

kszucs changed the title ~~[WIP] Dtype, schema inference and implicit casting~~ Dtype, schema inference and implicit casting Dec 28, 2017

kszucs added a commit to kszucs/ibis that referenced this pull request Dec 29, 2017

dtype, schema inference and implicit casting ibis-project#1269

bdee9c4

kszucs added a commit to kszucs/ibis that referenced this pull request Dec 29, 2017

Dtype, schema inference and implicit casting ibis-project#1269

9f7c201

cpcloud approved these changes Jan 8, 2018

View reviewed changes

cpcloud closed this in 71fc552 Jan 8, 2018

kszucs deleted the infer_schema branch May 30, 2022 12:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dtype, schema inference and implicit casting #1269

Dtype, schema inference and implicit casting #1269

kszucs commented Dec 15, 2017

cpcloud Dec 15, 2017 •

edited

Loading

cpcloud commented Dec 22, 2017

cpcloud Dec 26, 2017

kszucs Dec 27, 2017

cpcloud Dec 26, 2017

kszucs Dec 27, 2017

cpcloud Dec 26, 2017

kszucs Dec 27, 2017

cpcloud Dec 26, 2017

kszucs Dec 27, 2017

cpcloud Dec 26, 2017

kszucs Dec 27, 2017

cpcloud Dec 26, 2017

cpcloud Dec 26, 2017

kszucs Dec 27, 2017

cpcloud Dec 26, 2017

cpcloud Dec 26, 2017

kszucs Dec 27, 2017

kszucs Dec 27, 2017

cpcloud Dec 26, 2017

cpcloud Dec 26, 2017

kszucs Dec 27, 2017

cpcloud commented Jan 8, 2018

cpcloud commented Jan 8, 2018

		return True


		@castable.register(UnsignedInteger, UnsignedInteger)

Dtype, schema inference and implicit casting #1269

Dtype, schema inference and implicit casting #1269

Conversation

kszucs commented Dec 15, 2017

cpcloud Dec 15, 2017 • edited Loading

Choose a reason for hiding this comment

cpcloud commented Dec 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Jan 8, 2018

cpcloud commented Jan 8, 2018

cpcloud Dec 15, 2017 •

edited

Loading