New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process hangs when pyfftw used within multiprocessing #135

Open
bwohlberg opened this Issue Sep 5, 2016 · 9 comments

Comments

Projects
None yet
4 participants
@bwohlberg

bwohlberg commented Sep 5, 2016

In certain circumstances, pyfftw causes multiprocessing processes to hang, as in the following minimal example:

from __future__ import print_function
from builtins import input
from builtins import range

import numpy as np
import pyfftw
import multiprocessing as mp
import logging

mpl = mp.log_to_stderr()
mpl.setLevel(logging.INFO)

nthreads=2

X = np.random.randn(512, 512).astype(np.float32)
Z = pyfftw.interfaces.numpy_fft.fftn(X.astype(np.float64), threads=nthreads)
aX = mp.Array('f', X.ravel())

def fn(b, ax=aX, sh=X.shape):

    X = np.array(ax).reshape(sh)
    Y = pyfftw.interfaces.numpy_fft.fftn(X, threads=nthreads)
    return np.linalg.norm(b*Y)

p = mp.Pool(processes=4)
R = p.map(fn, [1e-2, 1e-1, 1e0, 1e1])
print(R)

The example does not hang if nthreads=1 or if X.astype(np.float64) is replaced with X. Setting the number of threads to one is a reasonable workaround, but it would be much better if this were not necessary.

@bwohlberg bwohlberg changed the title from Process hangs when pyfftw used within a multiprocessing process to Process hangs when pyfftw used within multiprocessing Sep 5, 2016

@hgomersall

This comment has been minimized.

Collaborator

hgomersall commented Oct 3, 2016

I thought I'd replied to this, apologies.

There is going to interactions between the memory management by pyFFTW/FFTW and multiprocessing - the problem is I have no idea what multiprocessing is doing behind the scenes (my ignorance is the problem here). FFTW is not thread safe during construction - is the memory implicitly shared?

@bwohlberg

This comment has been minimized.

bwohlberg commented Oct 5, 2016

I should have mentioned that the bug report was for Python 3.4 running on Linux x86_64. My understanding is that multiprocessing on this platform uses a fork call to create new processes, and results in a copy being made of the parent process, with no implicit shared memory. I'm afraid I don't have any suggestions for where to start looking for the cause of this bug, but my first guess was not that it's memory-related.

@bwohlberg

This comment has been minimized.

bwohlberg commented Oct 5, 2016

Single-stepping through the example above in pdb indicates that the program hangs at line 290, at waiter.acquire(), in /usr/lib/python3.4/threading.py. It looks as if the bug is related to thread locking or notification rather than memory issues. Does this help?

@hgomersall

This comment has been minimized.

Collaborator

hgomersall commented Oct 5, 2016

Interesting. I take if you're not enabling the cache? AFAICT the only place a lock is used is in line 1103 of pyfftw/pyfftw.pyx. What happens to this lock between processes? I've no idea. If you comment out that with plan_lock (and possibly the declaration of plan_lock) does it work?

@bwohlberg

This comment has been minimized.

bwohlberg commented Oct 8, 2016

No, the cache isn't enabled in the example, and explicitly enabling or disabling it doesn't make any difference to the process hanging. Commenting out the lock in pyfftw/pyfftw.pyx as suggested also doesn't have any apparent effect.

@jordan-heemskerk

This comment has been minimized.

jordan-heemskerk commented Jan 26, 2017

+1 running into this issue as well

Python 2.7 Linux x86_64

@jakirkham

This comment has been minimized.

Contributor

jakirkham commented Mar 30, 2017

Have you tried using spawn instead of fork with multiprocessing, @bwohlberg?

ref: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

@bwohlberg

This comment has been minimized.

bwohlberg commented Mar 31, 2017

No, I hadn't. After your suggestion I tried modifying the minimal example above by adding mp.set_start_method('spawn') but getting it to work turned out to be non-trivial. I'll report back if I have any success when I find time to try again.

@jakirkham

This comment has been minimized.

Contributor

jakirkham commented Mar 16, 2018

Sorry to be slow updating.

Should just be...

import multiprocessing
mp = multiprocessing.get_context('spawn')

Then mp can be used as you have above as it contains everything from multiprocessing just using spawn instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment