New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prange() dead-locks with memoryviews when not freeing the GIL #2798

Open
NicolasHug opened this Issue Jan 14, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@NicolasHug
Copy link

NicolasHug commented Jan 14, 2019

prange seems to hold forever if in the loop we call a function that uses a 1d view of a 2d view. Here is a reproducing example:

import numpy as np
from cython.parallel import prange


def wrapper():  # will hold forever
    a = np.random.uniform(0, 100, size=(100, 100)).astype(np.int32)
    g(a)


cdef int f(int [:] a_i) nogil:  # uses 1d view
    return 3


cdef int g (int [:, :] a) nogil:

    cdef:
        int i

    for i in prange(a.shape[0]):  # Works OK with range
        f(a[i])

As a workaround we can still pass a and i to f and use a[i] instead of a_i.

Thanks for all the work on Cython!

Versions:

Cython 0.29.2,
Python 3.7.2

@scoder

This comment has been minimized.

Copy link
Contributor

scoder commented Feb 2, 2019

Thanks for the report, I can reproduce this. My guess is that it dead-locks somehow. Note that you are not freeing the GIL anywhere. I think it's trying to acquire the GIL while it already holds it, or something like that. Ideally, Cython should be able to detect that somehow, but I can't say how difficult that would be.

@scoder scoder changed the title prange holds forever prange() dead-locks with memoryviews when not freeing the GIL Feb 2, 2019

@NicolasHug

This comment has been minimized.

Copy link
Author

NicolasHug commented Feb 2, 2019

Note that you are not freeing the GIL anywhere

Just to make sure, the GIL is implicitly freed when returning to the Python function (wrapper), right?

@scoder

This comment has been minimized.

Copy link
Contributor

scoder commented Feb 2, 2019

The GIL is a lock. It is held when entering your Python function (which allows you to do Python stuff and call NumPy functions, for example). You declared the cdef functions as nogil, which means that they can be called without holding the GIL, but it does not release the GIL when entering them. You have to do that yourself, explicitly, either using with nogil: somewhere, or by passing nogil=True into the prange() function.

@NicolasHug

This comment has been minimized.

Copy link
Author

NicolasHug commented Feb 2, 2019

I see, thanks a lot for the clarification

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment