BUG: DataFrame.sort_index() ignores the values returned by the key function

[n-splv]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

def sort_function(idx: pd.Index) -> pd.Index:
    print(f"Called with {idx=}")
    return sorted(idx)


df = pd.DataFrame(
    {"col": [1, 9, 4]},
    index=[1, 3, 2]
)
df.sort_index(key=sort_function)

Issue Description

Result:

Called with idx=Index([1, 3, 2], dtype='int64')
    col
1	1
3	9
2	4

Expected Behavior

Must be:

Called with idx=Index([1, 3, 2], dtype='int64')
    col
1	1
2	4
3	9

Installed Versions

>>> pd.__version__
'2.1.3'

[paulreece]

This isn't a bug. It's failing because you're sorting it before it has a chance to sort and it exits the sort function early in order to avoid doing unnecessary work. So it thinks that you have sorted it even though it hasn't. I'm not sure I see the use case here of altering this function in order to get this to work.

pandas/pandas/core/sorting.py

Lines 101 to 106 in ae6d279

	elif (np.all(ascending) and target.is_monotonic_increasing) or (
	not np.any(ascending) and target.is_monotonic_decreasing
	):
	# Check monotonic-ness before sort an index (GH 11080)
	return None
	elif isinstance(target, ABCMultiIndex):

[rhshadrach]

I think this might be a misunderstanding of what key does. Here is an example using just sorted:

mylist = [(1, 3), (2, 2), (3, 1)]

print(sorted(mylist, key=lambda x: x[0]))
# [(1, 3), (2, 2), (3, 1)]

print(sorted(mylist, key=lambda x: x[1]))
# [(3, 1), (2, 2), (1, 3)]

key takes the input values, and then the original list is sorted based on key's output - where the output goes determines the resulting order of the input. You want to think of pandas as doing the same, but vectorized. So it gives you an index [1, 3, 2] and you return an index [1, 2, 3]. This tells the sort function to put the first element first, the 2nd element 2nd, and the 3rd element 3rd - in other words to do nothing. What you should return is the positions you want the elements to go:

def sort_function(idx: pd.Index) -> pd.Index:
    print(f"Called with {idx=} giving {idx.argsort()=}")
    return idx.argsort()


df = pd.DataFrame(
    {"col": [1, 9, 4]},
    index=[1, 3, 2]
)
print(df.sort_index(key=sort_function))
# Called with idx=Index([1, 3, 2], dtype='int64') giving idx.argsort()=array([0, 2, 1])
#    col
# 1    1
# 2    4
# 3    9

[rhshadrach]

Closing for now - @nick-maykr if you think we've missed something please comment here and can reopen.

[n-splv]

@rhshadrach @paulreece You are absolutely right, I misunderstood how the key function works, although the documentation is pretty straightforward:
key
callable, optional
If not None, apply the key function to the index values before sorting.

Sorry for my oversight.

[paulreece]

@nick-maykr no worries we are here to help 😎