Written by Ben Moseley, April 2025.
In this workshop we will train deep neural networks to carry out symbolic regression. This workshop is inspired by these papers: Deep Learning for Symbolic Mathematics, Lample and Charton, 2019 and End-to-end symbolic regression with transformers, Kamienny et al, 2022.
The workshop is split into 3 parts:
- Part 1: Understanding mathematical expressions and how to generate them
- Part 2: Generating a training dataset of expressions and tokenization
- Part 3: Training a deep neural network to carry out symbolic regression
The notebook includes coding tasks. Each task is labelled like this:
Task: do this!
The notebook is designed for you to go at your own pace, and also contains extension exercises to stretch those with previous experience.
If you get stuck, please ask the instructors for help, and the completed notebooks are available in this repository for your reference.
The goal of this workshop is to train a network to identify a mathematical expression given example data points from the expression. More precisely, given a dataset,
Note:
- We are not just doing function fitting! In function fitting, we want to predict the value of
$f$ given$\mathscr{D}$ . In symbolic regression we want to discover the underlying mathematical expression of$f$ given$\mathscr{D}$ . This means the network must predict an expression. - For simplicity, we only consider scalar (1D) functions with a single input variable (
$x$ ) and integer constant coefficients. It is a fun task to extend the methods in this notebook to higher dimensions with floating point constants!
We will use a deep neural network to directly predict
In Part 2 we will explain how to enable the network to directly predict symbolic expressions by using tokenization and autoregressive prediction.