reservoirpy.Model().fit() doesn't support ProcessPoolExecutor

[DS-Liu]

I'm trying to implement committes of ESNs, i.e. multiple ESNs to improve the performance of a classification task.
I use ProcessPoolExecutor to fit these ESNs in parallel. However, the following error occurs.

concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\concurrent\futures\process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "C:\Users\dsliu\Documents\Projects\Quantum Machine Learing\mnist.py", line 54, in individual
    esn_model.fit(X_train, Y_train, warmup=int(.1*len(X_train)))
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\site-packages\reservoirpy\model.py", line 1091, in fit
    self.initialize_buffers()
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\site-packages\reservoirpy\model.py", line 794, in initialize_buffers
    node.initialize_buffers()
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\site-packages\reservoirpy\node.py", line 716, in initialize_buffers
    self._buffers_initializer(self)
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\site-packages\reservoirpy\nodes\readouts\ridge.py", line 89, in initialize_buffers
    readout.create_buffer("XXT", (input_dim, input_dim))
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\site-packages\reservoirpy\node.py", line 618, in create_buffer
    self._buffers[name] = memmap_buffer(self, data=data, shape=shape, name=name)
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\site-packages\reservoirpy\utils\parallel.py", line 75, in memmap_buffer
    memmap = np.memmap(temp, shape=shape, mode=mode, dtype=dtype)
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\site-packages\numpy\core\memmap.py", line 228, in __new__
    f_ctx = open(os_fspath(filename), ('r' if mode == 'c' else mode)+'b')
OSError: [Errno 22] Invalid argument: 'C:\\Users\\dsliu\\AppData\\Local\\Temp\\reservoirpy-temp\\Ridge-0XXT'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\dsliu\Documents\Projects\Quantum Machine Learing\mnist.py", line 93, in <module>
    main()
  File "C:\Users\dsliu\Documents\Projects\Quantum Machine Learing\mnist.py", line 85, in main
    Y_pred, esn_models =committee(21)
  File "C:\Users\dsliu\Documents\Projects\Quantum Machine Learing\mnist.py", line 77, in committee
    Y_pred, esn_model = future.result()
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\concurrent\futures\_base.py", line 451, in result
    return self.__get_result()
  File "C:\Users\dsliu\miniconda3\envs\reservoir\lib\concurrent\futures\_base.py", line 403, in __get_result
    raise self._exception
OSError: [Errno 22] Invalid argument: 'C:\\Users\\dsliu\\AppData\\Local\\Temp\\reservoirpy-temp\\Ridge-0XXT'

The function initialize_buffers() in "reservoirpy\nodes\readouts\ridge.py" is defined as

def initialize_buffers(readout):
    """create memmaped buffers for matrices X.X^T and Y.X^T pre-computed
    in parallel for ridge regression
    ! only memmap can be used ! Impossible to share Numpy arrays with
    different processes in r/w mode otherwise (with proper locking)
    """
    input_dim = readout.input_dim
    output_dim = readout.output_dim

    if readout.input_bias:
        input_dim += 1

    readout.create_buffer("XXT", (input_dim, input_dim))
    readout.create_buffer("YXT", (output_dim, input_dim))

I think the same string "XXT" used by all processes cause the above problem.

[nTrouvain]

Hello @DS-Liu,

I think the problem come from the order you chose for your operations. The parallel execution should happen within the fit method, after initialization of buffers, otherwise it won't work. Maybe moving the initialize_buffers method from outside the parallel loop will solve the problem?