Localized dates cause crash with load_table_columnar method

[stets]

While attempting to load a table into OmniSci via pymapd's columnar load with con.load_table_columnar(mapdtable, df, preserve_index=False, col_names_from_schema=True)

I receive back:

Traceback (most recent call last):
  File "/main.py", line 135, in <module>
    main()
  File "/main.py", line 123, in main
    con.load_table_columnar(mapdtable, df, preserve_index=False, col_names_from_schema=True)
  File "/usr/local/lib/python3.6/site-packages/pymapd/connection.py", line 593, in load_table_columnar
    col_names=col_names
  File "/usr/local/lib/python3.6/site-packages/pymapd/_pandas_loaders.py", line 136, in build_input_columnar
    data = thrift_cast(data, mapd_type, 0)
  File "/usr/local/lib/python3.6/site-packages/pymapd/_pandas_loaders.py", line 84, in thrift_cast
    return datetime_to_seconds(data)
  File "/usr/local/lib/python3.6/site-packages/pymapd/_utils.py", line 26, in datetime_to_seconds
    elif arr.dtype == 'object' or 'datetime64[ns,' in arr.dtype:
TypeError: argument of type 'DatetimeTZDtype' is not iterable

This is from latest master release pymapd==0.10.1.dev11+ga3fd5b9

Replication

Create a table with a single column as type TIMESTAMP(0):

create table example(test TIMESTAMP(0))

Run the following python code:

import os 
import pymapd
import pandas as pd
import numpy as np
from datetime import datetime, timedelta

mapdhost =  'localhost'
mapdport = '9090'
mapduser = 'mapd'
mapdpass =  'HyperInteractive')
mapddbname = 'mapd'
mapdprotocol = 'http'

mapdcon = pymapd.connect(user=mapduser, password=mapdpass, host=mapdhost, dbname=mapddbname, port=mapdport, protocol=mapdprotocol)

# create a dataframe of random dates
date_today = datetime.now()
days = pd.date_range(date_today, date_today + timedelta(7), freq='D')
np.random.seed(seed=1111)
data = np.random.randint(1, high=100, size=len(days))
df = pd.DataFrame({'test': days})
# convert the col test to UTC localized
df['test'] = df['test'].dt.tz_localize('UTC')
# load table
mapdcon.load_table_columnar('example', df, preserve_index=False, col_names_from_schema=True)

[randyzwitch]

Can you provide a reproducible example? Even a printed example of what the 'DatetimeTZDtype column looks like would be helpful.

[stets]

Thanks @randyzwitch -- I'll get this replicated and upload some code to show the issue.

[stets]

@randyzwitch Just added a bit more detail, I'm not sure if this is expected behavior or not since the columns in OmniSci are TIMESTAMP(0) types and the Dataframe objects have to be of type datetime64[ns, UTC]. I believe this worked in the previous versions.

[randyzwitch]

We should think through what the desired behavior is here. In your example, you are converting to UTC to get a DatetimeTZDtype, but what should we do if the data aren't UTC? I think the answer is to do the conversion to UTC, because that's what the backend expects, but we'd need to double check this since silent casting of data is frequently undesired.