BUG: unstack doesn't preserve categorical dtype

[pijucha]

Code Sample, a copy-pastable example if possible

idx = pd.MultiIndex.from_product([['A'], [0, 1]])
df = pd.DataFrame({'cat': pd.Categorical(['a', 'b'])}, index=idx)
df
Out[19]: 
    cat
A 0   a
  1   b

df.unstack()
Out[21]: 
  cat   
    0  1
A   a  b

Categorical dtype is lost:

df.unstack().dtypes
Out[22]: 
cat  0    object
     1    object
dtype: object

But it works ok for a Series:

df['cat'].unstack()
Out[24]: 
   0  1
A  a  b
df['cat'].unstack().dtypes
Out[25]: 
0    category
1    category
dtype: object

Expected Output

I believe df.unstack() should preserve categorical dtypes as df['cat'].unstack() does.

output of pd.show_versions()

pd.show_versions()

INSTALLED VERSIONS
------------------
commit: 5d791cc7d955c0b074ad602eb03fa32bd3e17503
python: 3.5.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.1.20-1
machine: x86_64
processor: Intel(R)_Core(TM)_i5-2520M_CPU_@_2.50GHz
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.18.1+368.g5d791cc
nose: 1.3.7
pip: 8.1.2
setuptools: 21.0.0
Cython: 0.24
numpy: 1.11.0
...

This is related to #13743 (and PR #13854), though a bit more difficult to fix.

[mroeschke]

Looks to be fixed on master. Could use a test

In [26]: df.unstack().dtypes
    ...:
Out[26]:
cat  0    category
     1    category
dtype: object

In [27]: pd.__version__
Out[27]: '1.1.0.dev0+1974.g0159cba6e'

[Abhispy007]

I can take this one? Just need guidance on which test file to edit? First time Git contributor

[mroeschke]

Sure @Abhispy007. pandas/tests/frame/tests_reshape.py would be a good place this test

[Abhispy007]

Afraid I don't understand the problem at hand here? The dtype remains the same?

[mroeschke]

That dtype is the dtype of the resulting Series.

Since Series is holding dtype objects, that should be object

[Abhispy007]

I see, is this what it should look like?

If not, is it possible to set up a 2 minute call? I genuinely want to contribute

[mroeschke]

Sorry don't have time for a call. But the output should be as noted in #14018 (comment)

[Abhispy007]

Came up with this test

def test_unstack_preserves_categorical_values():
    # GH 14018
    idx = pd.MultiIndex.from_product(
        [['A'], [0, 1]]
    )
    df = DataFrame(
        {'cat': pd.Categorical(['a', 'b'])},
        index=idx
    )
    assert np.array(df.unstack()).dtype.name == 'category'

Appropriately fails with AssertionError: assert 'object' == 'category'.
Should I submit a pull request? Forgive me if I'm being pedantic, first time contributor issues

[mroeschke]

You can use tm.assert_frame_equal to create the expected dataframe and compare the result of df.unstack() to the expected dataframe.

No worries, yes please open a pull request. Discussing the test in a pull request will be a lot easier

[Abhispy007]

Turns out my test doesn't work... Fails on df[cat].unstack().dtypes as well.
Biggest problem I'm having is that the terminal output I get is different than my PyCharm Output. Even though both PyCharm and terminal are using the same conda environment/Python version. Not sure what to do
IDE is giving correct output, #14018 comments
Terminal is giving the incorrect output #14018