BUG: shift() on Series of dtype bool makes dtype into object, so HDFStore won't append #4618

Closed
bigtonylewis opened this Issue Aug 21, 2013 · 2 comments

Comments

Projects
None yet
2 participants
import pandas as pd

# '0.11.0'
print pd.__version__

df = pd.DataFrame(index=range(0,5), columns=['mybool'])
df.mybool = False
store = pd.HDFStore('shiftedbool.h5')

# this works
store.append('noshiftedbool', df)

# now add a shifted bool, backfilled
df['shiftedbool'] = df.mybool.shift(1).fillna(method='bfill')
print df

# causes exception:
store.append('shiftedbool', df)

This can be worked around; for now I'm using 0.0 and 1.0. But this seems like an issue that could be improved.

My first issue submitted; I hope it's up to scratch

Contributor

jreback commented Aug 21, 2013

#4610 fixes this ; the backfilled arry doesn't downcast (to bool dtype) correctly
the exception is correct by HDFstore

another workaround is to do this:

In [1]: s = Series([True],list(range(0,5)))

In [2]: s
Out[2]: 
0    True
1    True
2    True
3    True
4    True
dtype: bool

In [3]: s.shift(1)
Out[3]: 
0     NaN
1    True
2    True
3    True
4    True
dtype: object

In [4]: s.shift(1).fillna(False)
Out[4]: 
0    False
1     True
2     True
3     True
4     True
dtype: object

In [5]: s.shift(1).fillna(False).convert_objects()
Out[5]: 
0    False
1     True
2     True
3     True
4     True
dtype: bool

astype(bool) will also work here rather than convert_objects

Contributor

jreback commented Aug 21, 2013

closed by #4610

jreback closed this Aug 21, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment