ODEBench, an ODE benchmark for system identification

ODEBench is a benchmark for system identification, consisting of 63 different Ordinary Differential Equations systems (ODE). Each system comes with a reference, parameter values, and two sets of initial conditions. ODEBench uses default values for solving the systems, generating trajectories for the state variables, and sampling the trajectories to obtain data points to be used in system identification. However, default values can be easily changed by the user.

ODEBench has been originally presented in [1,2], along with the ODEFormer transformer architecture for system identification. As ODEBench is a precious resource for benchmarking system identification, we decided to repackage it in a stand-alone repository, to make it easier to access for practitioners.

Installing

ODEBench depends on a few Python packages that you can install with:

pip install -r requirements.txt

Basic usage

In order to obtain the data for the noiseless trajectories of all systems used in [1,2], you can just run:

python generate_trajectories.py

This will create a JSON file solutions.json in the working directory, containing all data for all systems.

Advanced usage

Adding noise/dropping points from the trajectories

The original papers [1,2] showed comparisons of different system identification approaches under conditions which made the tasks more difficult, for example by adding noise to the data points or randomly dropping some of the points.

To add noise to the trajectories, you can just set the noise_amplitude variable inside generate_trajectories.py to the desired amplitude ratio; following the original publications, the noise model is Gaussian multiplicative, and the amplitude values in the papers were $\sigma={0.01, 0.02, 0.03, 0.04, 0.05}$.

To randomly drop a certain ratio of the points in the trajectory, you can likewise set the variable subsample_rate to the desired value. For example, subsample_rate=0.4 would drop 40% of the points. Value used in the paper were $\rho={0.0, 0.5}$.

Changing integration settings

Inside the generate_trajectories.py script, you will find two dictionaries, config and matplotlib_rc. config contains all parameters needed to generate the trajectories. Since the whole config is passed as arguments for the scipy.integrate.solve_ivp function, we recommend reading the documentation of the solve_ivp function. Summarizing:

• t_span is a tuple (t_min, t_max) defining the integration interval
• method is the integration method to be used to generate the trajectories, by default is LSODE
• rtol relative tolerance
• atol absolute tolerance
• first_step initial step size
• t_eval times that will be sampled from the obtained trajectory; this means that even if the original trajectory contains millions of points, only a few will be provided in output. The default value is 150 points uniformely sampled between the beginning and the end of t_span (see above)
• min_step minimum step size, a LSODE-specific parameter

The matplotlib_rc dictionary contains parameters that influence only the style of the plots.

Adding/changing initial conditions

At the moment, initial conditions are part of the odebench.strogatz_equations.equations dictionary. The initial conditions in the dictionary are the same used for the experiments in [1,2].

If you wish to add/change the initial conditions for some systems, you can do it by editing the my_initial_conditions dictionary inside generate_trajectories.py. The initial conditions in that dictionary will override the default values for the specified systems. The dictionary key is the system id (as an integer), while the values are a list of lists of real values (describing potentially multiple sets of initial conditions).

Adding ODE systems

If you wish to add your own ODE system to ODEBench, you will need to modify the dictionary odebench.strogatz_equations.equations, found in the subfolder odebench/strogatz_equations.py. You will need to specify the equations in the system as strings, and provide several other details, such as a set of default initial conditions. An entry in the dictionary should follow the format:

{
    'id': # integer,
    'eq': # equations as strings, separated by | with state variables names x_0, ..., x_n and constants named c_0, ..., c_n; e.g. 'x_1 | c_0 - sin(x_0) - c_1 * x_1'
    'dim': # number of state variables,
    'consts': # list of lists containing the values of the constants, in order,
    'init': # list of lists with initial conditions,
    'init_constraints': # constraints for the initial conditions, e.g. 'x_0 > 1',
    'const_constraints': # constraints for the constants, e.g. 'c_1 > 0, c_2 > 0',
    'eq_description': # string with a description, e.g. 'RC-circuit (charging capacitor)',
    'const_description': # string with a description of the meaning of the constants, e.g. 'c_0: fixed voltage source, c_1: capacitance, c_2: resistance',
    'var_description': # string with a description of the state variables, e.g. 'x_0: charge',
    'source': # string with a source, e.g. 'strogatz p.20'
},

Citing ODEBench

If you use ODEBench, you can cite this paper:

@inproceedings{
    dascoli2024odeformer,
    title={{ODEF}ormer: Symbolic Regression of Dynamical Systems with Transformers},
    author={St{\'e}phane d'Ascoli and S{\"o}ren Becker and Philippe Schwaller and Alexander Mathis and Niki Kilbertus},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=TzoHLiGVMo}
}

References

1. Stéphane d’Ascoli, Sören Becker, Alexander Mathis, Philippe Schwaller, and Niki Kilbertus. 2023. ODEFormer: Symbolic Regression of Dynamical Systems with Transformers. arXiv:2310.05573 [cs.LG] https://arxiv.org/abs/2310.05573
2. Stéphane d’Ascoli, Sören Becker, Philippe Schwaller, Alexander Mathis, and Niki Kilbertus. 2024. ODEFormer: Symbolic Regression of Dynamical Systems with Transformers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, International Conference on Machine Learning, San Diego, CA, 1–28. https://openreview.net/forum?id=TzoHLiGVMo