ESN Parameter Effects

[DJIDM]

I am trying to perform system identification using reservoir computing. I have .mat data consisting of the time and values of the input and output to a system. I have attached this data (named "train_trial_60s.xlsx"), as well as a script (named "rpy_RCP.txt") that reads in this data, separates it into training and testing (trial) sets, trains an ESN on the training set, and evaluates it on the testing set. The original data was a .MAT file (train_trial_60s.mat) while the original code was a Python script (rpy_RCP.py)

Originally, the reservoir of my ESN contained 1000 units, a leaking rate 'lr' = 1.0, and a spectral radius 'sr' = 1.0. This resulted in an RMSE of 45.719 and R^2 of -272.420. However, when I repeat the experiment with the leaking rate and spectral radius omitted (keeping the number of units equivalent, the regression improves to an RMSE of 0.738 and R^2 of 0.918.

What does this mean? Why does removing the leaking rate and spectral radius affect the performance to this degree (especially when the documentation states that the default value for the leaking rate = 1, so why does explicitly stating this lead to a worse performance)?

I also noticed that, for the case where the ESN is trained with just the number of units defined, that despite the better performance, the regression result appears very 'noisy'. I am curious as to why this is as well.

[PAUL-BERNARD]

Hello,
Indeed, specifying the leaking rate to 1.0 shouldn't change anything, as this is already the default in ReservoirPy.
The difference in performance comes from the spectral radius :

• if not specified, a random sparse matrix W is generated, and its non-zero values are normally distributed (you can also change the distribution).
• if specified, the same matrix is created, and is then scaled to have a spectral radius of sr

In your case, the spectral radius you have if you don't specify its value is around ~10. You can get its value by using the reservoirpy.observables.spectral_radius method:

from reservoirpy.nodes import Reservoir
from reservoirpy.observables import spectral_radius
import numpy as np

my_reservoir = Reservoir(
    units=1000,
    # sr=1.0
)

my_reservoir.initialize(np.random.normal(size=(12, 1)))

spectral_radius(my_reservoir.W)

The spectral radius of the recurrent connection weights has a significant impact on the performances of the task, so it's not a surprise that you have such bad performances if you don't specify it. You can read more about its impact in the documentation.

[PAUL-BERNARD]

Hello,
Its difficult to tell why you don't have better results with an hyper-parameter exploration. Can you provide more details about it ? What kind of performances does the hyper-parameter exploration gives you around the default parameters ?

It seems you have more units in the default reservoir (500) than in the optimized version (150). That could explain the performance decrease.

For the smoothness of your output, many parameters comes into play, with high inter-dependencies.

[neuronalX]

Hello,

Thank you for testing new tasks with ReservoirPy.
Reservoir Computing rarely works just out-of-the-box, as it is a machine learning tool with few trained parameters (only the output layer), one need to find optimal hyperparameters (parameters not trained) for each kind of task.

Here there are several factors that could influence your performances.

1. Normalize your data if not done. Between -1 and 1 by default, or maybe between 0 and 1 if you have only positive values.
2. Start with the simplest model, i.e. an ESN that do not have a feedback from the read-out layer and the reservoir. This makes the training more complicated and more unstable, in particular if you use offline learning (ridge) and not online learning (RLS, FORCE, ...).
3. Make an extensive hyperparameter search, including changing the ridge parameter (regularization parameter). It is also important to keep fixed the number of units inside the reservoir while doing this search, because results will be less interpretable. It is better to make several searches with fixed number of units for each search, but increase the number of units for different searches instead.
4. Look at the results of hyperparameter search to understand what are the most robust set of hyperparameters, and do not just take the "best" result. If you just want to take the best, then be careful to take also the same seed, to be sure to obtain the same results. You can find several exemples of the influence of hyperparameters and how to look at the hyperparameter search results:
   https://github.com/reservoirpy/reservoirpy/blob/master/tutorials/4-Understand_and_optimize_hyperparameters.ipynb

This kind of plot will help you to understand what are the hyperparameters that give the most robust results.

I hope this helps. If you show us this kind of plot for all the hyperparameters, we could help you interpret them.

[neuronalX]

Additionally, we provide some hints of how to optimise hyperparemeters in this paper:

Which Hype for My New Task? Hints and Random Search for Echo State Networks Hyperparameters. ICANN 2021 HTML HAL PDF