REGR/DOC: pd.concat(axis=1) should special case DatetimeIndex to sort even when sort=False

[torext]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

For a very long time now it used to be the default behaviour that calling pd.concat(..., axis=1) on DataFrames with DateTimeIndex would result in a DataFrame with sorted index. This is very reasonable default behaviour and pretty much what one wants in this case the vast majority of times.

With #54769 this was considered a bug simply because it wasn't documented correctly, and now the default behaviour, without any warning, was suddenly changed. This means likely code breakages across many users' libraries that were written based on the (up until now true) assumption that concatenating timeseries-like DataFrames/Series would give another timeseries-like object, i.e. something with a clearly sorted DateTimeIndex. Making sort=False work in the DateTimeIndex case is of course something that should be ensured, as it is something that might come in handy in some rare cases, but having that the default I don't think is a good idea at all, given the opposite has been the default for a long time and frankly makes more sense.

Feature Description

Change the default back to sort=True in the case of DataFrames/Series with DateTimeIndex and update the documentation to more accurately reflect this original default behaviour.

Alternative Solutions

Many users suddenly (and without clear warning) need to put sort=True in all the places they call pd.concat on timeseries-like Pandas objects.

Additional Context

No response

[enzbus]

Thank you, this just caused my unit tests to fail. I agree on giving a deprecation warning and/or reverting to the previous behavior.

[lithomas1]

This seems reasonable to me.

[lukemanley]

It sounds like #54769 should be reverted in 2.2.1 and documented instead due to long-standing behavior.

The long-standing behavior does have a few inconsistencies that motivated #54769 that may be worth noting:

1. pd.concat(axis=1, how="outer") special cases DatetimeIndex to always sort (ignores sort parameter). All other dtypes and index types do NOT sort when sort=False (default) including other time-like dtypes such as pyarrow timestamp/date/time and PeriodIndex.

2. pd.concat(axis=1, how="outer") always sorts for DatetimeIndex, but pd.concat(axis=1, how="inner") only sorts if sort=True

3. I can see the argument for outer joining two sorted (monotonic) indexes via pd.concat(axis=1) and expecting a monotonic output by default. I dont know the full history, but that may be the reason pd.merge and DataFrame.join ALWAYS sort outer joins (for all dtypes, not just DatetimeIndex). The behavior of pd.concat does not necessarily need to align with merge/join but it might be more consistent and less confusing than maintaining/documenting these kind of exception cases.

[torext]

3. I can see the argument for outer joining two sorted (monotonic) indexes via pd.concat(axis=1) and expecting a monotonic output by default. I dont know the full history, but that may be the reason `pd.merge` and `DataFrame.join` ALWAYS sort outer joins (for all dtypes, not just DatetimeIndex). The behavior of `pd.concat` does not necessarily need to align with merge/join but it might be more consistent and less confusing than maintaining/documenting these kind of exception cases.

I like this suggestion a lot. I primarily work with and write libraries for timeseries-like data, whether that be DataFrames or Series, and I can't actually imagine a single scenario where I wouldn't wan't a monotonic output from an outer join induced on the other axis by pd.concat(axis=1). In fact, even for an inner join I can't think of such a scenario, given the input indices were monotonic.

A general argument for why maybe having the indices sorted by default could go as follows: if one cares about the sorting of indices in a given context, then the input indices in that context will presumably be monotonic and it will be desirable to keep the output monotonic as well. If instead one is in a situation where one doesn't care about the sorting (since e.g. it's categorical "labeling"-type of data that's stored in the index), then one won't be too fussed if the output ends up sorted by default; the context and code around it presumably is written in an order-agnostic way in this case and having the additional sorting step thus at most has a performance impact (which can be mitigated with sort=False).

Ironically enough, .groupby(), which I feel is more often used with unsorted categorical-type data in the index/grouper, has sort=True by default, which I often have to end up overriding since I'm grouping by some categories for display-purposes and I want to keep the implicit original order intact. Cases where .groupby() is used with a DateTimeIndex-like grouper are rare, since for those cases often there are more adapted approaches like .resample()etc. already implemented. And to top off the irony, having sort=False in those cases would probably anyways give the desired output assuming the input indices were monotonic (think grouping a monotonic daily frequency DateTimeIndex by month with sort=Falseand applying some aggregation function).