This paper will apply the idea of sparse regression to perform model discovery on dynamical system to extract mathematical equations that describe the underlying mechanism of complex system. This idea plays a vital role in science applications since it is a data-driven approach to give scientists a hint in understanding the mathematics under high-dimensional data that people may not be able to see through. For instance, this approach can help scientists extract insights and understand more about physical phenomena such as turbulence, fluid flow or complex biological systems such as biological neural networks with a variety of free variables.
This problem framework begins with time-series of the dynamical system with each vector yt representing system at one time step with n variables. Next, we can compute the derivative y'through various difference schemes from scientific computing and this serve as the right hand side b for our regression framework from Formula 1.
Intuitively, this approach works by building a library of functions from scientists’ knowledge about the system that they are considering to be good candidates for the system and use sparse regression to evaluate which equations are more suitable to the given data. = {y, 2y + y3 + y4, log(y), ey, siny, cos(3y),....}
Where is represented as a matrix with each column is the corresponding library function at each timestep ytand this matrix serve as matrix A in our regression framework. By using sparse regression algorithm such as Lasso, the algorithm can extract important coefficient that correspond to the library function that matches with the derivative y' of the system.
This model discovery steps can be repeated to similarly discover partial differential equations with respect to space and time from the dynamical system. This article also provides numerous extensions to this model discovery framework by applying Hankel matrix to perform time-delay embeddings and extract hidden latent variables of the system when we actually do not have resources or not aware of more variables from the system. This paper also further discusses various approaches to evaluate the performance of a machine learning model with Kullback-Leibler divergence (KL divergence), Akaike information criterion (AIC), Bayesian information criterion (BIC). In short, KL divergence measures the similarity of two distributions, the lower the divergence value, the more similar the two distributions. In our scope of supervised learning, this means that we can compare the distribution of our actual labeled data versus the predictions from our model, where the lower KL divergence value essentially signals us that the model can capture the features of the data. In similar usage, AIC and BIC from information theory field, , exploit the relationship between the KL divergence measure and maximum likelihood estimation method to effectively compare the efficiency among models.
AIC = -2logL() + 2k BIC = -2logL() + klogn
Where is the set of model parameters, L()is the likelihood of the model given the data with parameters and kis the number of parameters in the model.Since the likelihood function value is multiplied by a negative number, then the model with the minimum AIC or BIC is the one with the highest value for the likelihood function. Furthermore, AIC and BIC give penalty for the number of parameters, which means that it provides a trade-off to evaluate models: the better fit, created by a complex models with many parameters must have penalty compared simpler model with similar accuracy. As a result, we can choose algorithm both based on its accuracy and its simplicity. AIC value is not used isolatedly, but in fact, the AIC value calculated for each model can be further used to calculate delta AIC and Akaike weights to relatively compare models. The interpretation of Akaike weights is the measure of strength of each candidate model given the data, or the probability that the candidate mode is the best among the set of candidate models.