API: concat of frames with multi-index with non-matching names

[jorisvandenbossche]

Suppose you have two dataframes with multi-index with non-matching names. When you concat these, the name of the first index will be used, and the indices are just merged ignoring the names of the levels.

Small example:

In [41]: df1 = pd.DataFrame([[1,2], [3,4]], index=pd.MultiIndex.from_tuples([('a', 1), ('a', 2)], names=['A', 'B']))
In [42]: df2 = pd.DataFrame([[5,6], [7,8]], index=pd.MultiIndex.from_tuples([(1, 'b'), (2, 'b')], names=['B', 'A']))

In [43]: df1
Out[43]:
     0  1
A B
a 1  1  2
  2  3  4

In [44]: df2
Out[44]:
     0  1
B A
1 b  5  6
2 b  7  8

In [45]: pd.concat([df1, df2])
Out[45]:
     0  1
A B
a 1  1  2
  2  3  4
1 b  5  6
2 b  7  8

This of course stems from the fact that index (level) names are not 'real' labels like eg column names, but following the aligning logic in pandas for index labels, this can give a bit unexpected results.

[jreback]

hmm, that is odd. I think this is the expected result.

In [14]: pd.concat([df1.reset_index(),df2.reset_index()]).set_index(['A','B'])
Out[14]: 
     0  1
A B      
a 1  1  2
  2  3  4
b 1  5  6
  2  7  8

[jorisvandenbossche]

Yes, that is indeed the expected result. But I don't think this is really a bug, but rather an API design issue. Or are there other places where we do aligning of index names?
E.g for merging, it is not yet implemented (pd.merge(df1, df2, left_index=True, right_index=True))

[jreback]

yeh, I think they have to exactly match, though they are on different levels, so something is going on.

[shoyer]

I think aligning of index names (only if they are provided) would be a nice API change. Hard to see folks taking issue with this one.