BUG: concat column order behaviors changes after 1.4

[Yikun]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

After 1.4:
>>> import pandas as pd
>>> pdf = pd.DataFrame({"A": [0, 2, 4], "B": [1, 3, 5], "C": [6, 7, 8]})
>>> pd.concat([pdf, pdf["C"], pdf["A"]], ignore_index=True, join='outer', sort=True)
     A    B    C    0
0  0.0  1.0  6.0  NaN
1  2.0  3.0  7.0  NaN
2  4.0  5.0  8.0  NaN
3  NaN  NaN  NaN  6.0
4  NaN  NaN  NaN  7.0
5  NaN  NaN  NaN  8.0
6  NaN  NaN  NaN  0.0
7  NaN  NaN  NaN  2.0
8  NaN  NaN  NaN  4.0


Before 1.4:
>>> import pandas as pd
>>> pdf = pd.DataFrame({"A": [0, 2, 4], "B": [1, 3, 5], "C": [6, 7, 8]})
>>> pd.concat([pdf, pdf["C"], pdf["A"]], ignore_index=True, join='outer', sort=True)
     0    A    B    C
0  NaN  0.0  1.0  6.0
1  NaN  2.0  3.0  7.0
2  NaN  4.0  5.0  8.0
3  6.0  NaN  NaN  NaN
4  7.0  NaN  NaN  NaN
5  8.0  NaN  NaN  NaN
6  0.0  NaN  NaN  NaN
7  2.0  NaN  NaN  NaN
8  4.0  NaN  NaN  NaN

Issue Description

concat column order behaviors changes after 1.4

Expected Behavior

>>> pd.concat([pdf, pdf["C"], pdf["A"]], ignore_index=True, join='outer', sort=False)
     A    B    C    0
0  0.0  1.0  6.0  NaN
1  2.0  3.0  7.0  NaN
2  4.0  5.0  8.0  NaN
3  NaN  NaN  NaN  6.0
4  NaN  NaN  NaN  7.0
5  NaN  NaN  NaN  8.0
6  NaN  NaN  NaN  0.0
7  NaN  NaN  NaN  2.0
8  NaN  NaN  NaN  4.0
>>> pd.concat([pdf, pdf["C"], pdf["A"]], ignore_index=True, join='outer', sort=True)
     0    A    B    C
0  NaN  0.0  1.0  6.0
1  NaN  2.0  3.0  7.0
2  NaN  4.0  5.0  8.0
3  6.0  NaN  NaN  NaN
4  7.0  NaN  NaN  NaN
5  8.0  NaN  NaN  NaN
6  0.0  NaN  NaN  NaN
7  2.0  NaN  NaN  NaN
8  4.0  NaN  NaN  NaN

Installed Versions

1.4+

[Yikun]

Related first commit: 01b8d2a

@CloseChoice @jreback

[simonjayhawkins]

Thanks @Yikun for the report.

Related first commit: 01b8d2a

in _get_combined_index in pandas/core/indexes/api.py in #43833, index = union_indexes(indexes, sort=sort) -> index = union_indexes(indexes, sort=False)

(Pdb) a
indexes = [Index(['A', 'B', 'C'], dtype='object'), Int64Index([0], dtype='int64'), Int64Index([0], dtype='int64')]
intersect = False
sort = True
copy = True
(Pdb) union_indexes(indexes, sort=False)
Index(['A', 'B', 'C', 0], dtype='object')
(Pdb) union_indexes(indexes, sort=True)
Index([0, 'A', 'B', 'C'], dtype='object')
(Pdb) 

and the code a few lines later

    if sort:
        try:
            index = index.sort_values()
        except TypeError:
            pass

index.sort_values() raises TypeError: '<' not supported between instances of 'int' and 'str' as _sort_mixed from pandas/core/algorithms.py is not used nor is the same sort logic as implemented in union_indexes(indexes, sort=True)