PERF: additonal GIL releasing

[jreback]

after #10199 (all of the groupbys)
these require slightly different templates as the object cannot have the with nogil:. These are all in pandas/src/generate_code.py

- [ ] groupby_count
- [ ] is_monotonic

• take_1d
• take_2d
• take_2d_multi
• pad
• backfill
• diff
• join_indexer (outer/inner/left)

in algos.pyx

• kthsmallest/kthlargest
• group_median
• roll_*
• drop_duplicates machinery
• nancorr

[mrocklin]

I've run up against drop_duplicates a bit recently. I'd bump that slightly in priority here. No specific rush though.

[jreback]

ints

In [41]: np.random.seed(1234)

In [42]: N = 1000000

In [43]: df = DataFrame({'A' : np.random.randint(0,100,size=N)})

In [44]: def f():
   ....:     for i in range(2):
   ....:         df.drop_duplicates()
   ....:         

In [45]: @test_parallel(2)
   ....: def g():
   ....:     df.drop_duplicates()
   ....:     

In [46]: %timeit f()
10 loops, best of 3: 86.3 ms per loop

In [47]: %timeit g()
10 loops, best of 3: 54.8 ms per loop

objects

In [48]: idx = tm.makeStringIndex(100)

In [49]: df = DataFrame({'A' : idx.take(np.random.randint(0,100,size=N))})

In [50]: %timeit f()
10 loops, best of 3: 244 ms per loop

In [51]: %timeit g()
10 loops, best of 3: 168 ms per loop

@mrocklin what dtypes are you typically dropping? note that dropping on categoricals is particularly efficient (though the conversion cost can be higher than the time simply to drop, but if its already categorized then its cheap).

[mrocklin]

I was dropping integers and categoricals. Does this benefit increase beyond two cores?

[jreback]

In [10]: N = 10000000

In [11]: df = DataFrame({'A' : np.random.randint(0,1000,size=N)})

In [12]: def f():
   ....:     for i in range(8):
   ....:         df.drop_duplicates()
   ....:         

In [13]: @test_parallel(8)
   ....: def g():
   ....:     df.drop_duplicates()
   ....:     

In [14]: %timeit f()
1 loops, best of 3: 2.49 s per loop

In [15]: %timeit g()
1 loops, best of 3: 1.02 s per loop

[jreback]

#10917

will make this:

In [5]: %timeit f()
1 loops, best of 3: 1.6 s per loop

In [6]: %timeit g()
1 loops, best of 3: 595 ms per loop

[jbrockmendel]

This has likely been pushed as far as it’ll go.

[jreback]

can u update the metrics just to see

[jbrockmendel]

@mroeschke is there any room to release the gil more in window.pyx?

[mroeschke]

We're pretty liberally using with nogil in a lot of the window routines (cursory check).

[mroeschke]

Probably better if we make separate issues where we can identify targeted areas to release the GIL; going to close for now.