Issues with nanoseconds

[slazarov]

I am writing the data in nanoseconds but upon retrieving it has lost the precision:

import pystore
import pandas as pd

pystore.set_path("/tmp")
store = pystore.store('test')
collection = store.collection('tick')

ts = '2019-06-15 00:00:12.868214001+00:00'
ts = pd.to_datetime(ts, format="%Y-%m-%d %H:%M:%S.%f")
df = pd.DataFrame({'ts': [ts],
                   'data': [100]})
df.set_index('ts', inplace=True)

name = 'ns_test'
collection.write(name, df, overwrite=True)

result = collection.item(name).to_pandas()
print(f'Before write: {df.index}')
print(f'After read: {result.index}')
print(f'Difference: {df.index.values.astype(int) - result.index.values.astype(int)}')
----
Before write: DatetimeIndex(['2019-06-15 00:00:12.868214001+00:00'], dtype='datetime64[ns, UTC]', name='ts', freq=None)
After read: DatetimeIndex(['2019-06-15 00:00:12.868214+00:00'], dtype='datetime64[ns, UTC]', name='ts', freq=None)
Difference: [1]

[ChristophRose]

You have to explicitly tell FastParquet to store the timestamp as an int96 when working with nanosecond resolution timestamps.
Luckily PyStore and Dask pass on the argument.

collection.write(name, df, overwrite=True, times="int96")

[slazarov]

Thank you for the clarification.

Perhaps this could be added into the example file.