Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
PERF: Release GIL on some datetime ops #11263
Conversation
|
ohh nice! can u share some timings? |
jreback
commented on an outdated diff
Oct 8, 2015
| @@ -164,10 +165,11 @@ def periodarr_to_dt64arr(ndarray[int64_t] periodarr, int freq): | ||
| out = np.empty(l, dtype='i8') | ||
| for i in range(l): | ||
| - if periodarr[i] == iNaT: | ||
| - out[i] = iNaT | ||
| - continue | ||
| - out[i] = period_ordinal_to_dt64(periodarr[i], freq) | ||
| + with nogil: | ||
| + if periodarr[i] == NPY_NAT: |
|
|
jreback
commented on an outdated diff
Oct 8, 2015
| cdef: | ||
| pandas_datetimestruct dts | ||
| date_info dinfo | ||
| float subsecond_fraction | ||
| - if ordinal == iNaT: | ||
| + if ordinal == NPY_NAT: #TODO: does this break anything? |
jreback
Contributor
|
|
Here are some timings - getting a pretty nice speedup. In single-threaded case things are looking about flat. In [1]: from pandas.util.testing import test_parallel
In [2]: dti = pd.date_range('1900-1-1', periods=100000)
In [3]: def f():
...: for i in range(4):
...: dti.year
In [4]: @test_parallel(4)
...: def g():
...: dti.year
In [8]: %timeit f()
10 loops, best of 3: 25.8 ms per loop
In [9]: %timeit g()
100 loops, best of 3: 7.71 ms per loop |
jreback
commented on an outdated diff
Oct 8, 2015
| - pandas_datetime_to_datetimestruct(dtindex[i], PANDAS_FR_ns, &dts) | ||
| - out[i] = monthrange(dts.year, dts.month)[1] | ||
| + pandas_datetime_to_datetimestruct(dtindex[i], PANDAS_FR_ns, &dts) | ||
| + out[i] = days_per_month_table[is_leapyear(dts.year)][dts.month-1] |
jreback
Contributor
|
jreback
added Timeseries Performance
labels
Oct 8, 2015
kawochen
commented on the diff
Oct 15, 2015
| @@ -3849,6 +3849,7 @@ def get_time_micros(ndarray[int64_t] dtindex): | ||
| @cython.wraparound(False) | ||
| +@cython.boundscheck(False) | ||
| def get_date_field(ndarray[int64_t] dtindex, object field): |
kawochen
Contributor
|
jreback
added this to the
0.17.1
milestone
Oct 16, 2015
|
@chris-b1 loooks good. can you add a whatsnew note (perf) and squash. |
chris-b1
changed the title from
(WIP) PERF: Release GIL on some datetime ops to PERF: Release GIL on some datetime ops
Oct 16, 2015
|
@jreback - updated |
jreback
added a commit
that referenced
this pull request
Oct 17, 2015
|
|
jreback |
7e5b223
|
jreback
merged commit 7e5b223
into pandas-dev:master
Oct 17, 2015
1 check passed
|
thanks! |
|
@chris-b1 can you add these (clean then make again to see them) |
chris-b1 commentedOct 8, 2015
This is a WIP, but far enough along I thought I'd share and see if the approach was reasonable.
This releases the GIL on most vectorized field accessors (e.g.
dt.year) and conversion to and fromPeriod. May be places it could be done - obviously would be nice for parsing, but I'm not sure that's possible.