Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Remove multiprocessing code #2472
This PR removes the use of multiprocessing Pool in Gammapy.
The reason for this is that multiprocessing isn't reliable, and acutely we had the issue of crashes on macOS: #2453
It's not really clear if CPython or matplotlib or Apple is to blame for these crashes, but overall I think the situation can be characterised like this: multiprocessing mostly works, but sometimes doesn't on some platforms and Python and library versions, and it is tricky to make it work and get good performance consistently. That's why e.g. scikit-learn has written their on replacement,
@adonath and I discussed whether we should attempt to change to
Note that currently we're using multiprocessing for ring convolution and for TS map computation, where really our most CPU-intensive tasks are data reduction and modeling & fitting. When we re-introduce parallelism in Gammapy, we should introduce it for those tasks where it's most relevant first.
@adonath - Please review.
I'm not familiar with that code - maybe there's a quick way to write it in a clearer or more efficient way, using a for loop?
Performance for the detect tutorials looks acceptable to me:
Maybe just merge as-is and move on?