Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Unstacking MultiIndex with datetime tz-aware data raises ValueError: cannot create a DatetimeTZBlock without a tz #18338

Closed
cancan101 opened this issue Nov 17, 2017 · 5 comments

Comments

Projects
None yet
5 participants
@cancan101
Copy link
Contributor

commented Nov 17, 2017

Code Sample, a copy-pastable example if possible

df = pd.DataFrame({
    'timestamp': [pd.Timestamp('2017-08-27 01:00:00.709949+0000', tz='UTC')],
    'a': ['a'],
    'b': ['b'],
    'c': ['c'],
})
df.set_index(['a', 'b']).unstack()

Problem description

Exception thrown. This used to work pre 0.21 and works with non Timestamp columns

ValueError                                Traceback (most recent call last)
<ipython-input-13-469280bd9dc1> in <module>()
      5     'c': ['c'],
      6 })
----> 7 df.set_index(['a', 'b']).unstack()

/usr/local/lib/python3.5/dist-packages/pandas/core/frame.py in unstack(self, level, fill_value)
   4544         """
   4545         from pandas.core.reshape.reshape import unstack
-> 4546         return unstack(self, level, fill_value)
   4547 
   4548     _shared_docs['melt'] = ("""

/usr/local/lib/python3.5/dist-packages/pandas/core/reshape/reshape.py in unstack(obj, level, fill_value)
    467     if isinstance(obj, DataFrame):
    468         if isinstance(obj.index, MultiIndex):
--> 469             return _unstack_frame(obj, level, fill_value=fill_value)
    470         else:
    471             return obj.T.stack(dropna=False)

/usr/local/lib/python3.5/dist-packages/pandas/core/reshape/reshape.py in _unstack_frame(obj, level, fill_value)
    480         unstacker = partial(_Unstacker, index=obj.index,
    481                             level=level, fill_value=fill_value)
--> 482         blocks = obj._data.unstack(unstacker)
    483         klass = type(obj)
    484         return klass(blocks)

/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py in unstack(self, unstacker_func)
   4337                 partial(unstacker_func,
   4338                         value_columns=self.items[blk.mgr_locs.indexer]),
-> 4339                 new_columns)
   4340 
   4341             new_blocks.extend(blocks)

/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py in _unstack(self, unstacker_func, new_columns)
   1808 
   1809         blocks = [self.make_block_same_class(vals, [place])
-> 1810                   for vals, place in zip(new_values, new_placement)]
   1811         return blocks, mask
   1812 

/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py in <listcomp>(.0)
   1808 
   1809         blocks = [self.make_block_same_class(vals, [place])
-> 1810                   for vals, place in zip(new_values, new_placement)]
   1811         return blocks, mask
   1812 

/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py in make_block_same_class(self, values, placement, fastpath, **kwargs)
    222             placement = self.mgr_locs
    223         return make_block(values, placement=placement, klass=self.__class__,
--> 224                           fastpath=fastpath, **kwargs)
    225 
    226     @mgr_locs.setter

/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py in make_block(values, placement, klass, ndim, dtype, fastpath)
   2948     elif klass is DatetimeTZBlock and not is_datetimetz(values):
   2949         return klass(values, ndim=ndim, fastpath=fastpath,
-> 2950                      placement=placement, dtype=dtype)
   2951 
   2952     return klass(values, ndim=ndim, fastpath=fastpath, placement=placement)

/usr/local/lib/python3.5/dist-packages/pandas/core/internals.py in __init__(self, values, placement, ndim, **kwargs)
   2610 
   2611         if values.tz is None:
-> 2612             raise ValueError("cannot create a DatetimeTZBlock without a tz")
   2613 
   2614         super(DatetimeTZBlock, self).__init__(values, placement=placement,

ValueError: cannot create a DatetimeTZBlock without a tz

Expected Output

   c timestamp
b  b         b
a             
a  c       2017-08-27 01:00:00.709949+00:00

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-1039-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: C.UTF-8
LANG: C.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: 3.2.3
pip: 9.0.1
setuptools: 36.6.0
Cython: 0.27.2
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.9999999
sqlalchemy: 1.1.15
pymysql: None
psycopg2: 2.7.3.2 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@jreback

This comment has been minimized.

Copy link
Contributor

commented Nov 17, 2017

There are a number of related issue w.r.t. reshaping and dti-tz-aware. https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3AReshaping+label%3ATimezones

but i'll mark it. if you want investigate what needs to be done to fix would be great.

@jreback jreback added this to the Next Major Release milestone Nov 17, 2017

@jreback jreback changed the title ValueError: cannot create a DatetimeTZBlock without a tz BUG: ValueError: cannot create a DatetimeTZBlock without a tz Nov 17, 2017

@MikeSpreitzer

This comment has been minimized.

Copy link

commented Jan 27, 2018

Is there any work-around for this?

@mroeschke

This comment has been minimized.

Copy link
Member

commented Jul 29, 2018

xref #19420

@mroeschke mroeschke changed the title BUG: ValueError: cannot create a DatetimeTZBlock without a tz BUG: Unstacking MultiIndex with datetime tz-aware data raises ValueError: cannot create a DatetimeTZBlock without a tz Jul 29, 2018

@ronid

This comment has been minimized.

Copy link

commented Dec 26, 2018

any update?

@mroeschke

This comment has been minimized.

Copy link
Member

commented Jan 4, 2019

This looks to work on master now; could use a test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.