PERF: pd.Index.is_all_dates doesn't require full-scan inference

[immerrr]

While working on #6328 I've stumbled upon this piece that deserves optimization.

Current implementation Index.is_all_dates invokes lib.infer_dtype which does a full-scan dtype inference which is not not always necessary. If I read cython sources correctly, new code is equivalent to the old one and the performance increase comes from short-circuiting the check on the first non-datetime element.

This change cuts roughly 50% of runtime of the following snippet:

import numpy as np
import pandas as pd

NUM_ELEMENTS = 1e7
s = pd.Series(np.arange(NUM_ELEMENTS),
              index=['a%s' % i for i in np.arange(NUM_ELEMENTS)])

print("pd.version", pd.__version__)
%timeit s[:int(NUM_ELEMENTS/2)]

On recent master:

$ ipython test_index_is_all_dates.ipy 
pd.version 0.13.1-108-g87b4308
10 loops, best of 3: 55.5 ms per loop

On this branch:

$ ipython test_index_is_all_dates.ipy 
pd.version 0.13.1-109-g3b43e2b
10 loops, best of 3: 28 ms per loop

[jreback]

fyi...this is a cached attribute so only hit once, but it is valid

the infer_dtype routines does do this optimization however

[immerrr]

fyi...this is a cached attribute so only hit once, but it is valid

I know that. Unfortunately, the cache doesn't help in case of slicing: the result is always a new object (unless the slice was itself cached), the cache is not calculated yet and the value is required during construction unconditionally (fastpath=True doesn't help).

the infer_dtype routines does do this optimization however

I'm not sure what you mean. lib.infer_dtype needs to know the exact type and thus is forced to scan all the elements. is_all_dates is only interested if all of the series elements are datetimes and the first element which doesn't fit is enough to have the answer.

[jreback]

look at the impl of infer_dtype it fails fast. (as does is_datetime_array)

[immerrr]

Hmmm... can it be that you confuse these two cases:

In [50]: arr_s = np.array(map(str, range(10000)))

In [51]: arr_o = np.array(map(str, range(10000)), dtype=object)

In [52]: timeit pd.lib.infer_dtype(arr_s)
1000000 loops, best of 3: 351 ns per loop

In [53]: timeit pd.lib.infer_dtype(arr_o)
10000 loops, best of 3: 42 µs per loop

In [54]: pd.lib.infer_dtype(arr_s)
Out[54]: 'string'

In [55]: pd.lib.infer_dtype(arr_o)
Out[55]: 'string'

Because, I cannot see how infer_dtype can return 'string' for arr_o without scanning the whole array at line 124 and returning the value at line 125:

    elif PyString_Check(val):
        if is_string_array(values):
            return 'string'

[jreback]

no that is correct. The is_string_array like all of the array functions fails fast. Of course they DO scan the entire array; but some cases are very optimized, e.g. if its already dtype = datetime64[ns] then it doesn't even need to do anything.

Pls run a perf-check with the change and see if anything changes. and post the results

test_perf.sh

[immerrr]

Here's the result. Is there a benchmark for slice indexing?

[jreback]

look at vb_suite/indexing.py

so you prob need to create a benchmark which tests this change....nothing is obviously improved (but its a worthwhile change though)

you can use the example from above that you did

[immerrr]

Done and updated the gist.

[jreback]

nice!

ok just need an entry in release notes (ref this pr as their is no associated issue) and good 2 go

[immerrr]

Done.

[jreback]

@immerrr thanks.....

sorry about being pendantic....but expect a lot of contributions from you!