Read_csv should raise error on bad dtypes

[hayd]

OPs code from http://stackoverflow.com/questions/16988526/pandas-reading-csv-as-string-type

Add: auto conversion from a passed str or np.string_ to np.object might be ok

df
           A         B
1A  0.209059  0.275554
1B  0.742666  0.721165

df.to_csv(savefile)

df_read = pd.read_csv(savefile, dtype=str, index_col=0)
   A  B
B  (  <

Actually I get something different on dev:

In [101]: pd.read_csv('a', index_col=0, dtype=str)
Out[101]:
 A B

[cpcloud]

@jreback suggesting should return a frame with floats converted to str (which pandas will then cvt to object)? or convert to object (which still leaves them as floats)...

[jreback]

@jreback

op wanted a string dtype.....

the answer is to do:

pd.read_csv('a',index_col=0,dtype=np.object')

I am just suggesting

a) if we see a dtype==str (string, str, or np.str), convert it to np.object
b) validate that dtype is not some weird thing (which actually happens I think already), it
just tries to coerce and if it fails its left as object

so this would just avoid the np.str trap

however I think there is also a case of

dtype='datetime64[ns]' (or variants of) which should be ignored/raise an error (as you really need to pass parse_dates)

[hayd]

It has to be a valid type (at least) ... I like the idea of special casing str.

With regards to last point I guess conceivably you may know the things you're passing are datetime64[ns] already... ?

[jreback]

yes...I could see something like:

read_csv(....dtype = { 'A' : np.str, 'B' : 'datetime64' })

being equivalent to

read_csv(dtype = { 'A' : np.object }, parse_dates=['B'])

for consistency then you could just construct a dtype dict (rather than having to separate out the date fields)
(and then you could accept dtype-like string that basically all go to datetime64[ns], e.g. datetime64 is not a valid dtype

(also showing the conversion of the 'string-like' dtypes

[cpcloud]

i think special casing str is probably the way 2 go. what happens if someone passes read_csv(dtype={'A': str, 'B': 'datetime64'}, parse_dates=['B'])

@jreback what's the reason again for not using numpy's string type? i've sort of dogmatically accepted object just because it doesn't really affect anything that i do, but now i'm a bit curious. does it have to with the fact that there's basically different dtype for every string length?

[jreback]

hmmm...the first case should probably be an error

np.str is special case that has fixed width for each item, while object is better because it allows variable length string (and any object actually). You get more perf by using np.str but less flexibility.

[jreback]

fyi...looking at this closer, passing 'str' as the dtype actually works, but since the width of the string is 0 it does weird stuff....so that's the case to handle

e.g. np.dtype('S10') 'works' (and returns object), but np.dtype('S') (which is what 'str') give you is an error
this is all in parser.pyx/_convert_with_dtype, pretty easy fix for the string issue

[majkrzak]

Passing 'str' or str as dtype= causes columnt to be converted to dtype='int64' if data in column looks like a number.