Column Offset Bug with to_html(index=False) with MultiIndex Columns and Index

[simonjayhawkins]

import pandas as pd
from IPython.display import HTML
df = pd.DataFrame({'a': range(10), 'b': range(10,20), 'c': range(10,20), 'd': range(10,20)})
df.columns = pd.MultiIndex.from_product([['a', 'b'], ['c', 'd']])
df.index = pd.MultiIndex.from_product([['a', 'b'], ['c', 'd', 'e', 'f', 'g']])
HTML(df.to_html(index=False))

Output:

		a		b
		c	d	c	d
0	10	10	10
1	11	11	11
2	12	12	12
3	13	13	13
4	14	14	14
5	15	15	15
6	16	16	16
7	17	17	17
8	18	18	18
9	19	19	19

Problem description

The code sample above is taken from #8452 but with a MultiIndex applied to the index as well as the columns. This seems to reproduce the column offset bug as seen in the original issue. The desired behaviour is observed if either one of the axes is a MultiIndex but not both.

INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.0
pytest: 3.5.1
pip: 10.0.1
setuptools: 39.1.0
Cython: 0.28.2
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: 1.7.4
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: 1.2.1
tables: 3.4.3
numexpr: 2.6.5
feather: None
matplotlib: 2.2.2
openpyxl: 2.5.3
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.4
lxml: 4.2.1
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[WillAyd]

Thanks for the report. Just to be clear this doesn't have anything to do with the IPython rendering. AFAICT if both the index and columns are MultiIndex and you use index=False in to_html the column headers of the output seem misaligned.

Investigation and PRs welcome

[simonjayhawkins]

on initial investigation of how the index=False is handled, it appears that the handling of a multiIndex index is not quite right for the to_string method when 'max_cols' is also set:

>>> import pandas as pd
>>> print(pd.__version__)
0.24.0.dev0+562.g996f361f8
>>> df = pd.DataFrame({'a': range(10), 'b': range(10,20), 'c': range(10,20), 'd': range(10,20)})
>>> df.columns = pd.MultiIndex.from_product([['a', 'b'], ['c', 'd']])
>>> df.index = pd.MultiIndex.from_product([['a', 'b'], ['c', 'd', 'e', 'f', 'g']])
>>> df = df.head(2)
>>> print(df.to_string(index=True))
     a       b
     c   d   c   d
a c  0  10  10  10
  d  1  11  11  11
>>>
>>> print(df.to_string(index=False))
a       b
c   d   c   d
0  10  10  10
1  11  11  11
>>>
>>> print(df.to_string(index=True, max_cols=3))
     a ...   b
     c ...   d
a c  0 ...  10
  d  1 ...  11
>>>
>>> print(df.to_string(index=False, max_cols=3))
a   b ...
c   d ...
0  10 ...
1  11 ...
>>>
>>> print(df.to_string(index=True, max_cols=2))
     a ...   b
     c ...   d
a c  0 ...  10
  d  1 ...  11
>>>
>>> print(df.to_string(index=False, max_cols=2))
a   b ...
c   d ...
0  10 ...
1  11 ...
>>>

[simonjayhawkins]

column misalignment also occurs for a standard index when the columns MultiIndex is named:

import numpy as np
import pandas as pd
df = pd.DataFrame(np.zeros((2, 2), dtype=int))
df.columns = pd.MultiIndex.from_product([['a'], ['b', 'c']], names=[
                'columns.name.0', 'columns.name.1'])
df.to_html(index=False)

output:

columns.name.0	a
columns.name.1	b	c
0	0
0	0

expected:

columns.name.0	a
columns.name.1	b	c
	0	0
	0	0