creation of datetime objects in riak ts record sets should be optional [JIRA: CLIENTS-832]

[seanjensengrey]

When returning record sets from Riak TS queries, the client automatically converts Riak TS timestamps into datetime objects and vice versa.

For many applications where Python is an intermediate layer, converting into a datetime is not required.

Additionally there are some timestamps that do not roundtrip properly.

The first of which is 1001 (code taken from https://github.com/basho/riak-python-client/blob/2.4.2/riak/transports/pbc/codec.py#L86 )

In [69]: d._unix_time_millis(d._datetime_from_unix_time_millis(1001))
Out[69]: 1000

It would be nice if the conversion to datetime objects as optional.

[lukebakken]

What version of Python are you using? I found that some versions have a bug that prevent correct conversion:

https://github.com/basho/riak-python-client#riak-ts-timeseries

[seanjensengrey]

All on OSX (10.10.5), I can repro the 1001 case using non-system Python versions

Python 2.7.11
Python 3.5.1
PyPy 5.0.1 (2.7.10 compatible)

import datetime

epoch = datetime.datetime.utcfromtimestamp(0)

def _unix_time_millis(dt):
    td = dt - epoch
    return int(td.total_seconds() * 1000.0)


def _datetime_from_unix_time_millis(ut):
    return datetime.datetime.utcfromtimestamp(ut / 1000.0)

print("1001 ==",_unix_time_millis(_datetime_from_unix_time_millis(1001)))

My hunch is that floats|datetime are unable to represent time values that are that small. Some time values will not be representable without error.

[lukebakken]

That's just super, thanks Python. I'll figure out a place to put in that option, unless you have suggestions.

[seanjensengrey]

Yes, love hate. Python has never done a great job of insulating the underlying platform away. I don't know of a good place to put the option. That is the conundrum.

for an interface how about

client = RiakClient(transport_opts = { 'ts_datetime_conversion' : False })

how that makes it down to codec.py I am not sure.

[lukebakken]

Great, that's exactly what I was thinking. Do you think it should "convert by default"?

I'm also going to try some different math with regard to ms timestamps.

[seanjensengrey]

I would prefer if the client did not convert by default.

1. Some folks might want to use Arrow
2. In my case, conversion is unnecessary and a little wasteful as the middle tier service can use the raw timestamps just fine.

Thoughts from other Python Riak TS users?

[lukebakken]

I'm pretty sure I can fix the conversion by using Decimal objects, since float is the devil's type:

import datetime
import decimal

epoch = datetime.datetime.utcfromtimestamp(0)

def unix_time_millis(dt):
    td = dt - epoch
    # print('unix_time_millis td: {}'.format(td))
    ts = td.seconds
    tm = td.microseconds // 1000
    # print('td.seconds: {} td.milliseconds: {}, types: {} {}'.format(ts, tm, type(ts), type(tm)))
    rv = decimal.Decimal((ts * 1000) + tm)
    # print('unix_time_millis rv: {} type: {}'.format(rv, type(rv)))
    return rv


def datetime_from_unix_time_millis(ut):
    # print('datetime_from_unix_time_millis ut: {} type ut: {}'.format(ut, type(ut)))
    d = decimal.Decimal(ut) / 1000
    # print('ut / 1000: {} type: {}'.format(d, type(d)))
    rv = datetime.datetime.utcfromtimestamp(d)
    # print('datetime_from_unix_time_millis rv: {} type rv: {}'.format(rv, type(rv)))
    return rv

dt_1001 = datetime_from_unix_time_millis(1001)
print('dt_1001:', dt_1001)
utm_1001 = unix_time_millis(dt_1001)
print('utm_1001:', utm_1001)

@rcgenova - what do you think?

[seanjensengrey]

That code passes both python2 and python3 on my machine. Not that it matters much but it fails on pypy

(x.env) basho:riak-ts-client-time-bug basho$ pypy test1.py 
Traceback (most recent call last):
  File "test1.py", line 25, in <module>
    dt_1001 = datetime_from_unix_time_millis(1001)
  File "test1.py", line 21, in datetime_from_unix_time_millis
    rv = datetime.datetime.utcfromtimestamp(d)
  File "/usr/local/Cellar/pypy/5.0.1/libexec/lib_pypy/datetime.py", line 1472, in utcfromtimestamp
    return cls._from_timestamp(_time.gmtime, t, None)
  File "/usr/local/Cellar/pypy/5.0.1/libexec/lib_pypy/datetime.py", line 1479, in _from_timestamp
    us = _round(frac * 1e6)
TypeError: unsupported operand type(s) for *: 'Decimal' and 'float'

[rcgenova]

+1 to returning epoch millis by default (i.e., not converting)

[lukebakken]

@seanjensengrey - can you try this on pypy?

import datetime
import decimal
import platform

is_cpython = platform.python_implementation() == 'CPython'
epoch = datetime.datetime.utcfromtimestamp(0)

def unix_time_millis(dt):
    td = dt - epoch
    # print('unix_time_millis td: {}'.format(td))
    ts = td.seconds
    tm = td.microseconds // 1000
    # print('td.seconds: {} td.milliseconds: {}, types: {} {}'.format(ts, tm, type(ts), type(tm)))
    rv = decimal.Decimal((ts * 1000) + tm)
    # print('unix_time_millis rv: {} type: {}'.format(rv, type(rv)))
    return rv


def datetime_from_unix_time_millis(ut):
    # print('datetime_from_unix_time_millis ut: {} type ut: {}'.format(ut, type(ut)))
    d = decimal.Decimal(ut) / 1000
    #print('ut / 1000: {} type: {}'.format(d, type(d)))
    if is_cpython:
        rv = datetime.datetime.utcfromtimestamp(d)
    else:
        rv = datetime.datetime.utcfromtimestamp(float(d))
    #print('datetime_from_unix_time_millis rv: {} type rv: {}'.format(rv, type(rv)))
    return rv

dt_1001 = datetime_from_unix_time_millis(1001)
print('dt_1001:', dt_1001)
utm_1001 = unix_time_millis(dt_1001)
print('utm_1001:', utm_1001)

[seanjensengrey]

$ pypy test2.py 
('dt_1001:', datetime.datetime(1970, 1, 1, 0, 0, 1, 1000))
('utm_1001:', Decimal('1001'))