scipy.interpolate.RectBivariateSpline construction locks PyQt event loop

[Felix-neko]

Construction of a RectBivariateSpline is time-consuming, and I want to create it in a background PyQt thread and then pass to my GUI. Alas, its construction even in dedicated thread freezes PyQt event loop. Why does it happen? AFAIK, scipy calls do release GIL lock?

Here is my sample code

# -*- coding: utf-8 -*-

import numpy as np
import scipy as sp
import scipy.interpolate

import os
import sys
import time
import pickle
from PyQt4 import QtCore, QtGui


class LongRunningWorker(QtCore.QObject):

    result_ready = QtCore.pyqtSignal(tuple)

    @QtCore.pyqtSlot()
    def generate_uber_mega_data(self):
        print "Oy!"
        # time.sleep(5)

        r_ticks = np.linspace(0, 1000, 5000)
        phi_ticks = np.linspace(0, 2 * np.pi, 5000)

        r_grid, phi_grid = np.meshgrid(r_ticks, phi_ticks)

        print u"Starting interpolation"
        spline = sp.interpolate.RectBivariateSpline(r_ticks, phi_ticks, r_grid * phi_grid)
        print u"Интерполяция всё / Interpolation complete"
        time.sleep(5) # And here (while the sleep is running) GUI starts working
        print "Vey!"
        self.result_ready.emit((7, 40))

@QtCore.pyqtSlot(tuple)
def process_mega_data(mega_data):
    print mega_data

if __name__ == "__main__":
    app = QtGui.QApplication(sys.argv)
    mega_button = QtGui.QPushButton("Press__________________________________________here")
    mega_button.show()

    background_thread = QtCore.QThread()
    worker = LongRunningWorker()
    worker.moveToThread(background_thread)

    background_thread.start()

    # Press here -- and interpolation will start and GUI events on SpinBox will freeze
    mega_button.clicked.connect(worker.generate_uber_mega_data)
    worker.result_ready.connect(process_mega_data)

    # This little scum doesn't change values, when interpolation is running!
    mega_spinbox = QtGui.QSpinBox()
    mega_spinbox.show()

    sys.exit(app.exec_())

[erosennin]

Seems like Fortran functions in interpolate/src/fitpack.pyf are not marked as thread safe. Is there a reason for that?

[Felix-neko]

There is an affirmative reason to make them thread-safe.

For example, this interpolator may be created for several seconds or even more. While it's being created, without GIL release, we have only three ways:

1. Suffer from freezed GUI

2. Do various sodomic perversions, creating the interpolator in other process and them calling it from main process and getting little pieces of data in main process (point by point): its serialization and deserialization is extremely slow.

3. Make it release GIL.

[rgommers]

I don't remember this being asked or looked into before, could be that there's no good reason.

Fitpack code doesn't seem to have save or common blocks, so that should be enough for Fortran77 code to be thread-safe (?). (disclaimer: my Fortran knowledge is poor)

[charris]

Fortran didn't use to have stack variables, and after they were introduced they were't the default. Hmm... See http://stackoverflow.com/questions/2582409/are-local-variables-in-fortran-77-static-or-stack-dynamic

[pv]

Note that LAPACK threadsafety also relies on the fortran compiler having
stack variables. The default behavior in gfortran is IIRC dangerous, as
it puts only "small" arrays on stack. There was a LAPACK bug reported
some years back on threadsafety, don't remember how they addressed it.
.
However, be it as it may, there's no specific need to use the GIL for
this purpose. Each of the fortran codes could as well use their own
locks. This in principle could be implemented inside f2py, but we might
also work around it manually by declaring routines threadsafe or not.
.
The main question here is dealing with potential deadlocks on recursive
calls from Python callbacks.

[erosennin]

There was a LAPACK bug reported some years back on threadsafety, don't remember how they addressed it.

You must be referring to this one: bug112 :: DSYEVR does not seem to be thread-safe.

The problem was solved by compiling LAPACK with-frecursive. Perhaps we should enable it by default for gfortran, in numpy/distutils/fcompiler/gnu.py.

[erosennin]

Anyway, I've looked through FITPACK sources and haven't found any local arrays larger than 160 bytes. The default value for -fmax-stack-var-size is 32768, so the code must be thread safe with gfortran even without -frecursive.