BUG: replace not converting dtypes

[jreback]

I believe replace should do a convert_objects(copy=False) after replacement to provide dtype soft-conversion

In [1]: df = DataFrame([['foo','bar','bah'],['bar','foo','bah']])

In [2]: df
Out[2]: 
     0    1    2
0  foo  bar  bah
1  bar  foo  bah

In [3]: m = { 'foo' : 1, 'bar' : 2, 'bah' : 3 }

In [5]: df.replace(m)
Out[5]: 
   0  1  2
0  1  3  2
1  2  1  3

In [6]: df.replace(m).dtypes
Out[6]: 
0    object
1    object
2    object
dtype: object

In [8]: df.replace(m).convert_objects().dtypes
Out[8]: 
0    int64
1    int64
2    int64
dtype: object

[jreback]

@cpcloud for you!

[cpcloud]

there's an infer_types param that does this already...should it default to True?

[jreback]

yes...didn't realize it was False default (and I would put copy=False)

[cpcloud]

i thought copy didn't do anything in internals.py, changed recently?

[cpcloud]

yep docs said true but i think forgot to change when i finally submitted the original pr

[cpcloud]

ah i c copy is doing something ...

[jreback]

actually this is tricky....because if say nothing changes then you don't need to copy, otherwise I guess you do.....maye just leave copy==True (if needs conversion, IOW there are ANY object blocks);
acutally this could be an option to convert, e.g. convert='needed' ?

[jreback]

Because I believe you copy a block in replace (if needed), so no point in copying twice...

[jreback]

then again, say you don't actually replace anything....should STILL copy I guess

[cpcloud]

i copy if inplace=True (the default)...

[jreback]

ok...then its easy, I would drop infer_types; and always convert_objects, passing copy=not inplace

[cpcloud]

sounds good

[cpcloud]

btw is there an iterable_but_not_string(obj) function somewhere?

[cpcloud]

easy enough for me to put one in common.py if not...

[cpcloud]

would do in a separate PR

[jreback]

com.is_list_like

[cpcloud]

thanks

[cpcloud]

problem is that now convert clobbers strings in object arrays with nans when u originally replace nans with numbers..

[cpcloud]

i.e., mixed_frame.fillna(value=10.0) and mixed_frame.replace(nan, 10.0) should be equivalent

[cpcloud]

nvm fixed it

[cpcloud]

@jreback i see that is_list_like has a check for hasattr(arg, 'len'). Should that be hasattr(arg, '__len__')?

[cpcloud]

also _is_sequence looks redundant with is_list_like should I remove that as well?

[jreback]

in theory yes, but I'll be it works anyhow (and python may do some translation on that)

[jreback]

check where things are used....I think _is_sequence is slightly different

[cpcloud]

hm, hasattr([], 'len') != hasattr([], '__len__') so prolly should chnage that one

[cpcloud]

hm or just removed entirely

[jreback]

thanks for the fix