PERF/CLN: Improve datetime-like index ops perf

[sinhrks]

Add _isnan to DatetimeIndexOpsMixin to cache NaT mask.

This leads to some perf improvement which is noticeable in larger data.

after fix

import pandas as pd
import numpy as np

np.random.seed(1)
idx = pd.DatetimeIndex(np.append(np.array([pd.tslib.iNaT]),
                       np.random.randint(500, 1000, size=100000000)))
%timeit idx.max()
1 loops, best of 3: 599 ms per loop

%timeit idx.min()
1 loops, best of 3: 608 ms per loop

before fix:

%timeit idx.max()
1 loops, best of 3: 940 ms per loop

%timeit idx.min()
1 loops, best of 3: 883 ms per loop

[jreback]

ohh nice!

do we have sufficient benches for this?

[sinhrks]

Updated top description. It is harder to notice in small size of Index. Shoud I add vbench for this?

[jreback]

yes I would add a vbench for this.

btw, I think its actually useful to use .hasnans to skip some of the masking operations

min_stamp = self[~self._isnan].asi8.min()
can be

if self._hasnans:
    self = self[~self._isnan]
min_stamp = self.asi8.min()

on a NO nan series that would be a big win (assume that this is not the first operation, e.g. it obviously needs caching).

here's another thing. I think you can actually do a lot of this not in the constructor but in certain operations
(this may require you to keep another attribute)

e.g.

if you construct from a date_range you KNOW that you don't have any NaT already.
On insert/append you can see if you have any nans (in the appended one and the existing), so you don't actually have to do the check.

So it can still be lazy but in some cases you can set it a-priori w/o actually doing the check.

[sinhrks]

@jreback Exactly! As a first step, added hasnans conditions before mask (_isnan)

I think you can actually do a lot of this not in the constructor but in certain operations
(this may require you to keep another attribute)

I assume it means adding new property to _attributes to show whether data has NaT or not. From my understanding, implementation should be:

• Adding a flag which shows whether the data MAY contains NaT, like _maybe_hasnans.
• Update the value to False when we can confirm the value DOESN'T contains NaT
• When _maybe_hasnans is False, hasnans returns False without checking _isnan.
• When _maybe_hasnans is True, hasnans checks _isnan and return the result, cache itself and change _maybe_hasnans.

[jreback]

@sinhrks yep, exactly. For instance, Int64Index, will always be False for this.

[jreback]

status?

[jreback]

@sinhrks status?

[sinhrks]

I couldn't undertake to add an attribute which indicates `NaN`` possibility.

Because this is PR improve a perf somewhat, can I do it separately?

[jreback]

I think this needs to wait for Index.fillna yes?

[sinhrks]

Correct, will finish #11343 first.

[jreback]

@sinhrks I belive we can put this in now? move whatsnew note to 0.18.0

[jreback]

@sinhrks status?

[sinhrks]

@jreback Rebased, and ready for review.

I misunderstood about fillna. It can't be used because the target is np.ndarray.

[jreback]

thanks!