Interval support #1243

kszucs · 2017-11-26T18:36:31Z

cpcloud · 2017-11-26T22:18:12Z

ibis/expr/tests/test_datatypes.py

@@ -300,6 +300,10 @@ def test_timestamp_with_timezone_parser_invalid_timezone():
    assert str(ts) == "timestamp('US/Ea')"


+def test_interval():
+    dt.validate_type("interval('Y')")


This should assert that this returns the same thing as Interval('Y').

cpcloud · 2017-11-26T22:19:24Z

ibis/expr/datatypes.py

+        self.unit = unit
+
+    def valid_literal(self, value):
+        return isinstance(value, six.string_types + (datetime.timedelta,))


Should this list the specific characters that are valid?

cpcloud · 2017-11-27T15:12:53Z

ibis/expr/datatypes.py

+
+    __slots__ = 'unit',
+
+    _valid_units = set([


Let's make this frozenset.

cpcloud · 2017-11-28T17:07:54Z

ibis/expr/tests/test_temporal.py

@@ -201,21 +201,10 @@ def test_interval(literal):


 def test_interval_repr():
-    repr(api.interval(weeks=3)).splitlines()[0] == 'Literal[interval]'
+    assert repr(api.interval(weeks=3)) == 'Literal[interval]\n  3'


We should put the unit in the repr as well.

cpcloud · 2017-11-28T17:18:48Z

ibis/expr/tests/test_datatypes.py

+])
+def test_interval(unit):
+    definition = "interval('{}')".format(unit)
+    dt.Interval(unit) == dt.validate_type(definition)


needs an assert here

cpcloud · 2017-11-28T17:19:42Z

ibis/expr/rules.py

@@ -709,6 +710,10 @@ def time(**arg_kwds):
    return ValueTyped(ir.TimeValue, 'not time', **arg_kwds)


+def interval(**arg_kwds):


Do we also need a way to specify exactly unit rules as well? Something like:

interval_with_unit('s')

cpcloud · 2017-11-28T17:20:21Z

ibis/expr/rules.py

+def interval(**arg_kwds):
+    return ValueTyped(ir.IntervalValue, 'not an interval', **arg_kwds)
+
+
 def timedelta(**arg_kwds):


I wonder if this can be removed (or just aliased to interval for backwards compat)

cpcloud · 2017-11-28T17:21:00Z

ibis/expr/rules.py

@@ -452,22 +452,23 @@ def output_type(self):
    return output_type


-def numeric_highest_promote(i):
+# TODO: UNUSED
+# def numeric_highest_promote(i):


If things are unused, you can delete them. We can always resurrect them later with git if needed.

cpcloud · 2017-11-28T17:23:24Z

ibis/expr/operations.py

+class IntervalAdd(ValueOp):  # __radd__
+
+    input_type = [rules.temporal, rules.interval]
+    output_type = rules.shape_like_arg(0, 'timestamp')


I think the output type here is a bit more complicated since date + interval doesn't always return a timestamp. You can either perform the checking in the output_type method or you can make three different add classes that directly express the output type. The latter is a bit easier to reason about IMO.

cpcloud · 2017-11-28T17:26:05Z

ibis/expr/types.py

+        print('can_implicit_cast')
+        op = arg.op()
+        if isinstance(op, Literal):
+            if isinstance(op.value, six.integer_types):


Arbitrary integers shouldn't be implicitly castable to IntervalValues since integers do not have units. Only other interval values should be implicitly castable to interval values.

cpcloud · 2017-11-29T15:47:29Z

ibis/expr/api.py

+    if len(defined_units) > 1:
+        raise ValueError('Only one arg can be specified')
+    elif len(defined_units) < 1:
+        raise ValueError('At least one of the arguments must be specified')


Probably just len(defined_units) != 1, with a message saying Exactly one argument is required.

Is there any reason why we need to enforce this? Postgres, for example, allows multiple values to the interval constructor.

Postgres is more flexible than others

Fair enough, just wondering.

cpcloud · 2017-11-29T15:48:09Z

ibis/expr/api.py

+    else:
+        unit, value = defined_units[0]
+        type = dt.Interval(unit)
+        return ir.IntervalScalar(ir.literal(value, type=type).op())


Should this be ir.literal(value, type=type).op().to_expr()?

cpcloud · 2017-11-29T15:49:35Z

ibis/expr/datatypes.py

+        self.unit = unit
+
+    def valid_literal(self, value):
+        return isinstance(value, six.integer_types + (datetime.timedelta,))


I think only timedelta instances are valid literals, otherwise that would mean we're giving a default unit to integers.

cpcloud · 2017-11-29T15:51:03Z

ibis/expr/datatypes.py

@@ -778,6 +808,7 @@ def valid_literal(self, value):
 date = Date()
 time = Time()
 timestamp = Timestamp()
+interval = Interval()


Do we want aliases for each of the interval values like year, month, day, etc.?

Fine if not, just a thought.

I was just following the patterns in datatypes.py
None of the parametric types has specified aliases, so I guess not.

kszucs · 2017-12-04T15:54:12Z

@cpcloud Would You please review before I finalize this PR?

cpcloud · 2017-11-30T15:47:51Z

ibis/sql/tests/test_compiler.py

-WHERE `timestamp_col` < months_add('2010-01-01 00:00:00', 3) AND
-      `timestamp_col` < days_add(now(), 10)"""
-        assert result == expected
+#     def test_where_analyze_scalar_op(self):


We should be able to keep this test passing if we alias the day, month, etc functions to be calls to the top level interval function.

cpcloud · 2017-12-04T20:19:14Z

ibis/clickhouse/operations.py

@@ -367,19 +386,14 @@ def _timestamp_from_unix(translator, expr):
    return _call(translator, 'toDateTime', arg)


-def _timestamp_delta(translator, expr):
+def _timestamp_add(translator, expr):


Where is this being used? It looks like you're mapping ops.TimestampAdd to binary_infix_op('+') and not this. Is that intentional?

cpcloud · 2017-12-04T20:22:05Z

ibis/expr/api.py

+_interval_floordiv = _binop_expr('__floordiv__', _ops.IntervalFloorDivide)
+
+_interval_value_methods = dict(
+    to_unit=_to_unit,


Can this be done with cast?

Do You mean interval(hours=3).cast("interval('s')")?

Yep. If this is for backward compat, then we can leave it.

cpcloud · 2017-12-04T20:24:09Z

ibis/expr/rules.py

+    """Infers the output type of the binary interval operation
+
+    This is a slightly modified version of BinaryPromoter, it converts
+    back and forth between the interval and its innner value.


Small typo here, innner has one too many n's

cpcloud · 2017-12-04T20:34:16Z

ibis/clickhouse/operations.py

@@ -261,6 +261,23 @@ def _value_list(translator, expr):
    return '({0})'.format(', '.join(values_))


+def _interval_format(translator, expr):
+    units = {


Can you use Interval._valid_units here so we don't have to repeat this? Maybe we can use Interval._valid_units everywhere we would hard code unit values/labels.

cpcloud · 2017-12-04T20:35:59Z

ibis/expr/datatypes.py

+        return False
+
+    def valid_literal(self, value):
+        return isinstance(value, six.integer_types + (datetime.timedelta,))


Are the semantics here such that:

interval(days=2) + 1 == interval(days=3)

?

That's required to pass https://github.com/ibis-project/ibis/blob/master/ibis/expr/types.py#L1587

For a type e.g. interval<int32>('s') a python integer should be a valid literal.

The operation fails:

In [1]: import ibis In [2]: ibis.interval(days=2) + 1 == ibis.interval(days=3) --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-2-3abf187ff76a> in <module>() ----> 1 ibis.interval(days=2) + 1 == ibis.interval(days=3) TypeError: unsupported operand type(s) for +: 'IntervalScalar' and 'int'

Ah right. Carry on then, this looks good.

cpcloud · 2017-12-04T20:37:03Z

ibis/expr/datatypes.py

+        'ns'   # Nanosecond
+    ])
+
+    def __init__(self, value_type=None, unit='s', nullable=True):


We probably want the unit first here, so that it's easy to construct this type without using a keyword:

# current Interval(unit='s') # a little easier to read IMO Interval('s')

cpcloud · 2017-12-05T21:22:53Z

@kszucs Looking great. Everything ready to go on your side?

kszucs · 2017-12-05T22:07:20Z

@cpcloud Yes! Thank You!

kszucs added 3 commits November 26, 2017 19:13

interval datatype skeleton

50a25ef

parse interval unit

d29778f

flake8: trailing whitespaces

dd5b0b0

cpcloud self-requested a review November 26, 2017 22:16

cpcloud added the feature Features or general enhancements label Nov 26, 2017

cpcloud added this to the 0.13 milestone Nov 26, 2017

cpcloud reviewed Nov 26, 2017

View reviewed changes

kszucs added 2 commits November 27, 2017 09:41

validate interval units; second as default unit

ca6b64b

test for unsupported interval units and parsing failure

3f11173

cpcloud reviewed Nov 27, 2017

View reviewed changes

ibis/expr/datatypes.py Outdated

__slots__ = 'unit',

_valid_units = set([

Copy link

Member

cpcloud Nov 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make this frozenset.

kszucs added 4 commits November 27, 2017 18:23

skeleton for interval arithmetics

ef95969

fixed test cases

c72b915

IntervalAdd arithmetic

dcb96d4

fixed interval repr test case; marked unused methods

9013e9c

cpcloud reviewed Nov 28, 2017

View reviewed changes

cpcloud reviewed Nov 29, 2017

View reviewed changes

kszucs added 4 commits November 30, 2017 01:23

date, time timestamp arithmetics with proper rules

bf918c2

missed adding compat.py

4238c7c

cleanup unused method

24087bf

temp disable backend test cases using timedeltas

dc8a9d0

kszucs added 19 commits December 2, 2017 22:37

fixed timestamp test case

084183a

adapt impala compiler to support interval type

ba4b2da

flake8

51b9b11

clickhouse interval implementation

3b8e484

fix temporal add operations

62fdf5d

right temporal add operations

953b477

re-enabled temporal tests

fe26028

added columns to table fixture

f679655

impala timestamp tests

92d19f0

attempt to fix impala tests

1a16d8a

uppercase impala intervals

b5d498a

uppercase plural S

83d798d

cast timestamp arg of impala's date_add and date_sub

36013bf

flake8

1ccc8c9

sql compiler test upper unit

9b549f0

cast impala timestamp

0ac2d3d

flake8

d1d03a6

integer.to_interval conversion

a0299c6

interval can_implicit_cast and flake8[again]

fb87e94

kszucs added 2 commits December 5, 2017 13:07

interval comparisons; complex tests; timestamp date operator

6f91095

fix failed tests

22846ff

cpcloud reviewed Dec 5, 2017

View reviewed changes

resolved review issues

34e1b43

cpcloud approved these changes Dec 5, 2017

View reviewed changes

kszucs changed the title ~~[WIP] Interval support~~ Interval support Dec 5, 2017

cpcloud closed this in 48306f8 Dec 6, 2017

kszucs deleted the interval branch December 29, 2017 14:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Interval support #1243

Interval support #1243

kszucs commented Nov 26, 2017 •

edited

Loading

cpcloud Nov 26, 2017

cpcloud Nov 26, 2017

cpcloud Nov 27, 2017

cpcloud Nov 28, 2017

cpcloud Nov 28, 2017

cpcloud Nov 28, 2017

cpcloud Nov 28, 2017

cpcloud Nov 28, 2017

cpcloud Nov 28, 2017

cpcloud Nov 28, 2017

cpcloud Nov 29, 2017

kszucs Nov 29, 2017 •

edited

Loading

cpcloud Nov 29, 2017

cpcloud Nov 29, 2017

cpcloud Nov 29, 2017

cpcloud Nov 29, 2017

cpcloud Nov 29, 2017

kszucs Nov 29, 2017

kszucs commented Dec 4, 2017

cpcloud Nov 30, 2017

cpcloud Dec 4, 2017

cpcloud Dec 4, 2017

kszucs Dec 5, 2017

cpcloud Dec 5, 2017

cpcloud Dec 4, 2017

cpcloud Dec 4, 2017

cpcloud Dec 4, 2017

kszucs Dec 5, 2017 •

edited

Loading

cpcloud Dec 5, 2017

cpcloud Dec 4, 2017

cpcloud commented Dec 5, 2017

kszucs commented Dec 5, 2017

		@@ -709,6 +710,10 @@ def time(**arg_kwds):
		return ValueTyped(ir.TimeValue, 'not time', **arg_kwds)


		def interval(**arg_kwds):

Interval support #1243

Interval support #1243

Conversation

kszucs commented Nov 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Nov 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs commented Dec 4, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Dec 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Dec 5, 2017

kszucs commented Dec 5, 2017

kszucs commented Nov 26, 2017 •

edited

Loading

kszucs Nov 29, 2017 •

edited

Loading

kszucs Dec 5, 2017 •

edited

Loading