Incorrect type when odo-ing to a list from the SQL backend wrapped in Data

[cpcloud]

create table t (a integer);
insert into t values (1), (2), (3);

In [10]: from blaze import Data

In [11]: from odo import odo, resource

In [12]: d = Data('postgresql://localhost::t')

In [13]: d.a
Out[13]:
   a
0  1
1  2
2  3

In [14]: odo(d.a, list)
Out[14]: [1.0, 2.0, 3.0]

In [15]: d.a.dshape
Out[15]: dshape("var * ?int32")

[cpcloud]

I suspect we're going through pandas or numpy at some point in the odo graph

[kwmsmith]

@cpcloud you're right -- odo is selecting a path from sqla.Select -> pd.DataFrame -> np.ndarray -> list.

I'm able to fix this by dropping the cost of odo.backends.sql.select_to_iterator(). Its original cost was 300, although it is doing less work than odo.backends.sql.select_or_selectable_to_frame(), whose cost is 200. I dropped it to 100, but this feels completely arbitrary. Thoughts?

This issue exposes another bug in odo.convert.dataframe_to_numpy()--or perhaps elsewhere. The dshape for t.a is "var * ?int32". When this is passed through odo.numpy_dtype.dshape_to_numpy(), it comes back as float64. It looks like when resolving a datashape.Option type, dshape_to_numpy() casts it to a float64. Is this the right behavior? I presume it's upcasting to a more general type because of the Option.

When converting the DB's column to a dataframe, odo.backends.sql.select_or_selectable_to_frame() already does the right thing -- why can't we just do a df.to_records(index=False) and ignore the dshape from that point forward? That way we would let Pandas / NumPy figure out the largest type necessary.