BUG:Pivot table drops column/index names=nan when dropna=false

[OXPHOS]

• closes pivot_table margins bottom-left total does not correspond to other content when dropna=False #14072
• tests added / passed
• passes git diff upstream/master --name-only -- '*.py' | flake8 --diff
• whatsnew entry
  clean-up of Pivot table drops column/index names=nan when dropna=false #14246

[TomAugspurger]

Do you have some unintentional changes in here? This shouldn't be removed.

[OXPHOS]

I feel like the Index, Index([None, u'A', u'B'], dtype='object'), needs to be passed to Categorical when doing MultiIndex, as when dropna=False, None could also be the index/column name. Or I didn't get this correctly?

[TomAugspurger]

If I remove this if dropna, most of the tests pass (including a fix for this one) other than

        df = pd.DataFrame({'a': [1, 2, 2, 2, 2, np.nan],
                           'b': [3, 3, 4, 4, 4, 4]})
        actual = pd.crosstab(df.a, df.b, margins=True, dropna=False)
        expected = pd.DataFrame([[1, 0, 1], [1, 3, 4], [2, 4, 6]])
        expected.index = Index([1.0, 2.0, 'All'], name='a')
        expected.columns = Index([3, 4, 'All'], name='b')

Here's the result and expected

(Pdb) pp actual
b    3  4  All
a
1.0  1  0    1
2.0  1  3    4
All  2  3    5
(Pdb) pp expected
b    3  4  All
a
1.0  1  0    1
2.0  1  3    4
All  2  4    6

You have more experience with this section of the code than I do, but the margins on the expected look incorrect to me.

[OXPHOS]

You're definitely right. I think it should be (if dropna=False):

b	3	4	All
a
1.0	1	0	1
2.0	1	3	4
np.nan	0	1	1
All	2	4	6

• I need to fix the np.nan as it is still being ignored even with the current fix when dropna=False (i.e. the fix only works for None)
• Actually I didn't get why removing dropna here would help yet. I'll check closer.

[jreback]

this is doing similar things to changes in #12607

[TomAugspurger]

this is doing similar things to changes in #12607

Ah I see. That is a much larger change that the original issue I was looking at :)

[OXPHOS]

I think the change in Cython is definitely required. The problem is how to pass dropna to it without disturbing too many existing structures.
I just reset my developing environment and am trying to use Anaconda Python2.7. Interestingly, numerous tests failed on me even with the master branch. So I just tested pivot and groupby at local. I'll do more research on the weekend.

[OXPHOS]

Some tests will be failing and many are actually different/separate problems. I already located several and will update soon.

[jreback]

can you rebase and update?

[jreback]

needs a rebase. if you'd like to continue, pls comment.