Added PyForwardFromTo and PyBackwardFromTo to Net, for releasing GIL …

[alessandroferrari]

…in pycaffe. Added ScopedGILRelease for easy GIL release. Modified _caffe.cpp in pycaffe accordingly.

[seanbell]

The tests failed due to style issues -- please run make lint.

[alessandroferrari]

Multithreading in python is doomed by GIL. Many users (as I do) are often using caffe for performing nets prediction by pycaffe, and often they are doing so from multithreaded applications. Pycaffe interface is not releasing GIL, which means that if caffe is doing predictions, other python threads are stuck. Since predictions are time consuming, this can lead to terrible slow down of the application. By means of this pull-request, GIL is released for forward pass and backward pass of the network. Within the pass, it is required temporarily only for coping with python layers, that require to hold GIL.
Thanks to this, within the same process, from the python interface is possible to perform predictions of different networks in parallel on multiple threads, still mantaining complete thread safety. Before everything was locked by GIL (false sharing).

[ajtulloch]

This should be an RAII wrapper around Save/Restore thread for exception safety.

This definitely isn't the right place to put this logic though, FWIW.

[alessandroferrari]

RAII wrapper: Thank you for the suggestion. I will take a look to fix it.

Not right place: I agree. However, where would you suggest to place it? (consider that you have to reacquire GIL in case you forward/backward a python layer within the net). Even though this solution insert "Pythonish" code within the C++ logic, not releasing GIL for the pycaffe is a painful bug for python interface. If there is a cleaner alternative I would love to discuss about it.

[ajtulloch]

Why not just simply:

a) Release the GIL in PyCaffe
b) Reaquire the GIL in PythonLayer before running the function in that layer?

[alessandroferrari]

It would be cleaner and it was my first thought. However, I have been stopped by two reasons:

• PyEval_SaveThread return the thread_state of the current thread, you need somehow to pass from PyCaffe to the PythonLayer this state, without relying on Net (otherwise it would be the same);
• I suspect that in python layer, python-boost does some magic that require GIL for passing the python objects arguments to the C++ code, since I get segfault if I acquire GIL in PythonLayer before calling the inner forward. GIL needs to be acquired before calling forward method for Python layers.
  Thus, even if releasing GIL within the Net is not clean, since pythonish stuffs are inserted in the C++ logic, looked like the only viable solution. But if you have any ideas on how to circumvent those problems, I will be happy to give them a try.

[crowsonkb]

I use the multiprocessing package instead of threading both to work around the GIL and to parallelize PyCaffe over multiple GPUs. My main Python interpreter spawns several other interpreters each of which has one instance of PyCaffe.

[alessandroferrari]

Of course multiprocessing does not have synchronization problems......
Still, multiprocessing come with a interprocess communication cost. There are many applications in production environment where multithreading may be beneficial, such as model ensembling and performance critical apps. Numpy, OpenCv, scikit all release GIL....

[seanbell]

Why is this change necessary?

[seanbell]

Thanks @alessandroferrari for the great PR, and I agree that the GIL should be released during forward/backward. This enables pre-fetching with multi-threading (not multi-processing).

Can you please squash this into a single commit?

@shelhamer @longjon I think this is an important feature when using Python data layers. Do you have any thoughts?

[seanbell]

This should be scoped_gil_release_ to be consistent with other naming.

[cypof]

Each net can run in it's own fork now, there is an example in /python/train.py. #4563

[willyd]

Thanks @alessandroferrari for the great PR, and I agree that the GIL should be released during forward/backward. This enables pre-fetching with multi-threading (not multi-processing).

I think that even with new multiprocessing parallel training releasing the GIL is still relevant for the above use case. I can a get better throughput of data using the threading module and releasing the GIL in Net::ForwardFromTo and Net::BackwardFromTo that by using the multiprocessing module for single GPU training on windows. Of course we have to reacquire the GIL before calling any python layer functions like setup, reshape, etc.

[willyd]

PyEval_SaveThread return the thread_state of the current thread, you need somehow to pass from PyCaffe to the PythonLayer this state, without relying on Net (otherwise it would be the same);
I suspect that in python layer, python-boost does some magic that require GIL for passing the python objects arguments to the C++ code, since I get segfault if I acquire GIL in PythonLayer before calling the inner forward. GIL needs to be acquired before calling forward method for Python layers.

Thus, even if releasing GIL within the Net is not clean, since pythonish stuffs are inserted in the C++ logic, looked like the only viable solution. But if you have any ideas on how to circumvent those problems, I will be happy to give them a try.

@alessandroferrari I am not sure I completely understand what you are saying here. Have you tried something like:

Using:

class ReleasePyGIL {
 public:
  ReleasePyGIL() { state = PyEval_SaveThread(); }
  ~ReleasePyGIL() { PyEval_RestoreThread(state); }
 private:
  PyThreadState* state;

inside ForwardFromTo and BackwardFromTo and

class AcquirePyGIL {
 public:
  AcquirePyGIL() { state = PyGILState_Ensure(); }
  ~AcquirePyGIL() { PyGILState_Release(state); }
 private:
  PyGILState_STATE state;
};

inside PythonLayer::LayerSetup, PythonLayer::Reshape, etc.

This works flawlessly for me on Windows.