New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release the GIL in Cython code #8882

Closed
mrocklin opened this Issue Nov 23, 2014 · 4 comments

Comments

Projects
None yet
4 participants
@mrocklin
Contributor

mrocklin commented Nov 23, 2014

Pandas could release the GIL in parts of the Cython code. This would enable the parallel use of multiple cores with threading.

From off-line conversation with @jreback @cpcloud

http://lbolla.info/blog/2013/12/23/python-threads-cython-gil

@jreback jreback added the Performance label Nov 24, 2014

@jreback jreback added this to the 0.16.0 milestone Nov 24, 2014

@jreback jreback modified the milestones: 0.16.0, Next Major Release Mar 6, 2015

@TomAugspurger

This comment has been minimized.

Show comment
Hide comment
@TomAugspurger

TomAugspurger May 22, 2015

Contributor

FYI: scinkit-image just did this in scikit-image/scikit-image#1519.

Contributor

TomAugspurger commented May 22, 2015

FYI: scinkit-image just did this in scikit-image/scikit-image#1519.

@mrocklin

This comment has been minimized.

Show comment
Hide comment
@mrocklin

mrocklin May 22, 2015

Contributor

It happened surprisingly quickly in their case. I think that it was mostly the result of an afternoon sprint of a couple developers followed by a couple days of cleanup.

Contributor

mrocklin commented May 22, 2015

It happened surprisingly quickly in their case. I think that it was mostly the result of an afternoon sprint of a couple developers followed by a couple days of cleanup.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback May 22, 2015

Contributor

@cpcloud did some prelim work on this. I don't think its too difficult. Just requires some effort.

Contributor

jreback commented May 22, 2015

@cpcloud did some prelim work on this. I don't think its too difficult. Just requires some effort.

jreback added a commit to jreback/pandas that referenced this issue May 22, 2015

@shoyer

This comment has been minimized.

Show comment
Hide comment
@shoyer

shoyer May 22, 2015

Member

Indeed, it mostly just takes a little bit of testing work. Glad to see this being done!

For what it's worth, I looked into releasing the GIL a little bit in the hashtable/index API as part of my IntervalIndex work (#8707). The main stumbling block I encountered was pandas.hashtable.Int64Vector (and the other Vector objects). Hopefully someone more clever than me could figure out how to release the GIL on the append method by moving the resize call to use the C API: https://github.com/pydata/pandas/blob/0aceb38ac1a4659ae298d726bc447b97943163dc/pandas/hashtable.pyx#L68

Member

shoyer commented May 22, 2015

Indeed, it mostly just takes a little bit of testing work. Glad to see this being done!

For what it's worth, I looked into releasing the GIL a little bit in the hashtable/index API as part of my IntervalIndex work (#8707). The main stumbling block I encountered was pandas.hashtable.Int64Vector (and the other Vector objects). Hopefully someone more clever than me could figure out how to release the GIL on the append method by moving the resize call to use the C API: https://github.com/pydata/pandas/blob/0aceb38ac1a4659ae298d726bc447b97943163dc/pandas/hashtable.pyx#L68

jreback added a commit to jreback/pandas that referenced this issue May 26, 2015

jreback added a commit to jreback/pandas that referenced this issue May 26, 2015

jreback added a commit to jreback/pandas that referenced this issue May 28, 2015

jreback added a commit to jreback/pandas that referenced this issue May 28, 2015

jreback added a commit to jreback/pandas that referenced this issue May 28, 2015

jreback added a commit to jreback/pandas that referenced this issue Jun 3, 2015

jreback added a commit to jreback/pandas that referenced this issue Jun 3, 2015

jreback added a commit to jreback/pandas that referenced this issue Jun 26, 2015

jreback added a commit to jreback/pandas that referenced this issue Jun 26, 2015

jreback added a commit that referenced this issue Jun 30, 2015

yarikoptic added a commit to neurodebian/pandas that referenced this issue Jul 2, 2015

Merge commit 'v0.16.2-65-g054821d' into debian
* commit 'v0.16.2-65-g054821d':
  TST: 32-test fixes
  BUG: GH9907 generate_range when start and end have higher resolution than offset
  TST/CLN: remove assert_isinstance
  PERF: releasing the GIL, pandas-dev#8882
  PERF: vbenches for pandas-dev#8882, releasing the GIL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment