New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MultiIndex is_monotonic_decreasing is incorrect #16554

Closed
TomAugspurger opened this Issue May 31, 2017 · 4 comments

Comments

Projects
None yet
4 participants
@TomAugspurger
Contributor

TomAugspurger commented May 31, 2017

Or just isn't implemented properly yet?
This MultiIndex is identical on the second level, so its is_monotonic_decreasing should be the same as get_level_values(0).is_monotonic_decreasing (True)

In [27]: idx = pd.MultiIndex([['baz', 'bar'], ['a', 'b']], labels=[[0, 1], [0, 0]])

In [28]: idx
Out[28]:
MultiIndex(levels=[['baz', 'bar'], ['a', 'b']],
           labels=[[0, 1], [0, 0]])

In [29]: idx.is_monotonic_decreasing
Out[29]: False

In [30]: idx.get_level_values(0).is_monotonic_decreasing
Out[30]: True

They should both return True.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback May 31, 2017

Contributor

only implemented is_monotonic_increasing. It needs the same impl. (but its not used anywhere ATM).

Contributor

jreback commented May 31, 2017

only implemented is_monotonic_increasing. It needs the same impl. (but its not used anywhere ATM).

@Tafkas

This comment has been minimized.

Show comment
Hide comment
@Tafkas

Tafkas Jun 1, 2017

Contributor

is_monotonic_increasing is based on numpy.lexsortwhich seems not to support ordering in descending order.

Contributor

Tafkas commented Jun 1, 2017

is_monotonic_increasing is based on numpy.lexsortwhich seems not to support ordering in descending order.

@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger Jun 1, 2017

Contributor

Can we define is_monotonic_decreasing in terms of increasing? Something like

def is_monotonic_decreasing(self):
    return not self.is_strictly_monotonic_increasing or idx.nunique() == 1

not sure if that will work. Looks like nunique may not be implemented on MI.

Contributor

TomAugspurger commented Jun 1, 2017

Can we define is_monotonic_decreasing in terms of increasing? Something like

def is_monotonic_decreasing(self):
    return not self.is_strictly_monotonic_increasing or idx.nunique() == 1

not sure if that will work. Looks like nunique may not be implemented on MI.

@jschendel

This comment has been minimized.

Show comment
Hide comment
@jschendel

jschendel Jun 5, 2017

Member

I don't think that will work in general. You can't get monotonic decreasing from monotonic increasing in all scenarios, unless there's an assumption that the data is monotonic/sorted in some regard. For example, a MultiIndex generated by [(3, 3), (1, 1), (2, 2)] is not monotonic decreasing, but will return True.

I think you can implement this similar to is_monotonic_increasing. It's true that numpy.lexsort only seems to support ordering in descending order, but you can get around this. My initial thought was to just reverse the output of numpy.lexsort, but that messes up the order if you have non-unique indices, e.g. [(3, 3), (2, 2), (2, 2), (1, 1)]. The fix would be to add a fake unique decreasing level to the input of numpy.lexsort to force uniqueness. Looking at the source code for is_monotonic_increasing, I think you'd just need to make changes along the lines of values = [np.arange(len(idx) - 1, -1, -1)] + [existing comprehension] followed by sort_order = np.lexsort(values)[::-1]. And in the except clause pd.Index(idx.values).is_monotonic_decreasing.

Member

jschendel commented Jun 5, 2017

I don't think that will work in general. You can't get monotonic decreasing from monotonic increasing in all scenarios, unless there's an assumption that the data is monotonic/sorted in some regard. For example, a MultiIndex generated by [(3, 3), (1, 1), (2, 2)] is not monotonic decreasing, but will return True.

I think you can implement this similar to is_monotonic_increasing. It's true that numpy.lexsort only seems to support ordering in descending order, but you can get around this. My initial thought was to just reverse the output of numpy.lexsort, but that messes up the order if you have non-unique indices, e.g. [(3, 3), (2, 2), (2, 2), (1, 1)]. The fix would be to add a fake unique decreasing level to the input of numpy.lexsort to force uniqueness. Looking at the source code for is_monotonic_increasing, I think you'd just need to make changes along the lines of values = [np.arange(len(idx) - 1, -1, -1)] + [existing comprehension] followed by sort_order = np.lexsort(values)[::-1]. And in the except clause pd.Index(idx.values).is_monotonic_decreasing.

@jreback jreback modified the milestones: 0.21.0, Next Major Release Sep 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment