Mathematical Definitions

Definitions of mathematical functions used in Neural Network Notions. Partial derivative notation is used to represent gradient functions unless overly complex.

Element-Wise Tensor Operations

All variables are either values at identical spatial positions in the involved tensors or scalar values applied to all values in tensor arguments.

Addition

Forward Function:

$r = a + b$

Gradient Function:

$\frac{\partial r}{\partial a} = 1; \frac{\partial r}{\partial b} = 1$

Subtraction

Forward Function:

$r = a - b$

Gradient Function:

$\frac{\partial r}{\partial a} = 1; \frac{\partial r}{\partial b} = -1$

Multiplication

Forward Function:

$r=a*b$

Gradient Function:

$\frac{\partial r}{\partial a} = b; \frac{\partial r}{\partial b} = a$

Division

Forward Function:

$r = \frac{a}{b}$

Gradient Function:

$\frac{\partial r}{\partial a} = \frac{1}{b}; \frac{\partial r}{\partial b} = \frac{a}{b ^ 2}$

Exponentiation (Pow)

Forward Function:

$r = a ^ {exp}$

Gradient Function:

$\frac{\partial r}{\partial a} = exp * a ^ {exp - 1}; \frac{\partial r}{\partial exp} = a ^ {exp} * \ln(a)$

Natural Exponentiation (Exp)

Forward Function:

$r = e ^ t$

Gradient Function:

$\frac{\partial r}{\partial t} = e ^ t$

Logarithm

Forward Function:

$r = \log_{log\textunderscore base}(arg)$

Gradient Function:

$\frac{\partial r}{\partial arg} = \frac{1}{arg * \ln(log\textunderscore base)};  \frac{\partial r}{\partial log\textunderscore base} = \frac{\ln(arg)}{log\textunderscore base * \ln^2(log\textunderscore base)}$

Natural Logarithm

Forward Function:

$r = \ln(t)$

Gradient Function:

$\frac{\partial r}{\partial t} = \frac{1}{t}$

Complex Tensor Operations

Matrix Multiplication (matmul)

Implemented via transposing matrix $b$ to optimize memory layout and avoid cache misses.

Forward Function:

$r = a @ b$

Gradient Function:

$\nabla a = \nabla r @ b ^ T; \nabla b = a ^ T @ \nabla r$ Implemented via transposing matrices $a$ and $b$ to reduce to matrix multiplication.

Convolution (convolve)

Implemented as im2col-style convolution.

Forward Function:

$r = convolve(input,\ kernels)$

Gradient Function:

$\nabla input = convolve(\nabla r,\ rotated\ kernels); \nabla kernels = convolve(input,\ rotated\ \nabla r)$

Activation Functions

All variables are values at identical spatial positions in the involved tensors.

Linear

Forward Function:

$r = t$

Gradient Function:

$\frac{\partial r}{\partial t} = 1$

Sigmoid

Forward Function:

$r = \frac{1}{1 + e ^ {-t}}$

Gradient Function:

$\frac{\partial r}{\partial t} = Sigmoid(t) * (1 - Sigmoid(t))$

Tanh (hyperbolic tangent)

Forward Function:

$r = \frac{e ^ {2t} - 1}{e ^ {2t} + 1}$

Gradient Function:

$\frac{\partial r}{\partial t} = 1 - {Tanh}^2(t)$

ReLU (Rectified Linear Unit)

Forward Function:

$r = \begin{cases} t & \text{if } t \gt 0 \\ 0 & \text{if } t \le 0 \end{cases}$

Gradient Function:

$\frac{\partial r}{\partial t} = \begin{cases} 1 & \text{if } t \gt 0 \\ 0 & \text{if } t \le 0 \end{cases}$

LeakyReLU (Leaky Rectified Linear Unit)

Forward Function:

$r = \begin{cases} t & \text{if } t \gt 0 \\ \tau * t & \text{if } t \le 0 \end{cases}$ where $\tau$ is the constant used by the Leaky ReLU function

Gradient Function:

$\frac{\partial r}{\partial t} = \begin{cases} 1 & \text{if } t \gt 0 \\ \tau & \text{if } t \le 0 \end{cases}$

Softmax

Forward Function:

$r_i = \frac{e^{t_i}}{\sum_{j=1}^K e^{t_j}}$ where $t_i$ is a given score in a given batch and $K$ is the number of scores in the given batch

Gradient Function:

$\nabla t = r_i * (\nabla r_i - \sum_j \nabla r_j * r_j)$

Cost Functions

All variables are values at identical spatial positions in the involved tensors. $t$ is used to denote the tensor representing the predicted values of the neural network model and $target$ is used to denote the target values.

MSE (Mean Squared Error)

Forward Function:

$L = (t - target) ^ 2$

Gradient Function:

$\frac{\partial L}{\partial t} = 2 * (t - target)$

Huber (pseudo-Huber)

Forward Function:

$L = \delta ^ 2 * (\sqrt{1 + (\frac{target - t}{\delta})^2} - 1)$

Gradient Function:

$\frac{\partial L}{\partial t} = -\frac{target - t}{\sqrt{1 + (\frac{target - t}{\delta})^2}}$

SoftmaxCrossEntropy

Forward Function:

$L = -\sum_{i = 1}^{c} target_i * \ln(p_i)$ where $p_i = \frac{e ^ {t_i}}{\sum_{c} e ^ {t_c}}$, $c$ is the number of classes in the batch

Gradient Function:

$\frac{\partial L}{\partial t_i} = p_i - target_i$

Optimizer Functions

$\theta_n$ denotes the value of a given parameter following the optimizer step, $\theta_{n-1}$ denotes the value of a given parameter prior to the optimizer step, $\nabla \theta$ denotes the current stored gradient of a given parameter, and $\eta$ denotes the learning rate factor being used.

SGD (Stochastic Gradient Descent)

Step Function:

$\theta_n = \theta_{n-1} - \nabla \theta * \eta$

Adam (Adaptive Moment Estimation)

$m$ and $v$ denote the first and second moment estimates respectively for a given parameter, and $step$ denotes the number of times the parameter has been updated.

Moment Estimate Updates:

$m_n = \beta_1 * m_{n-1} + (1 - \beta_1) * \nabla \theta$
$v_n = \beta_2 * v_{n-1} + (1 - \beta_2) * \nabla \theta^2$

Bias Correction:

$\hat{m} = \frac{m}{1 - \beta_1^{step}}$
$\hat{v} = \frac{v}{1 - \beta_2^{step}}$

Step Function:

$\theta_n = \theta_{n-1} - (\frac{\eta}{\sqrt{\hat{v}} + \epsilon})\hat{m}$

AdamW Weight Decay:

$\theta_{n\ w} = \theta_n - \eta * \lambda * \theta_n - \eta * (\frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon})$ where $\lambda$ denotes the weight decay coefficient

Miscellaneous Functions

Clip (Tensor)

$t$ and $r$ denote values at identical spatial positions in the input and result tensors respectively.

Forward Function:

$r = \begin{cases} max & \text{if } t \gt max \\ t & \text{if } min \le t \le max \\ min & \text{if } t \lt min \end{cases}$

Gradient Function:

$\frac{\partial r}{\partial t} = \begin{cases} 1 & \text{if } min \le t \le max \\ 0 & \text{if } t \lt min \text{ or } t \gt max \end{cases}$

ClipGradients (Model)

$\nabla \theta_n$ denotes the gradient of the parameter following the clip, $\nabla \theta_{n-1}$ denotes the gradient of the parameter prior to the clip, and $\Vert \nabla M \Vert_2$ denotes the total Euclidean norm of the gradients of all parameters in the neural network model.

Euclidean Norm Calculation:

$\Vert \nabla M \Vert_2 = \sum_{\theta} {\sum_{i} \nabla \theta_i^2}$ where $\theta$ denotes a given parameter in the neural network model

Gradient Clipping:

$\nabla \theta_n = \begin{cases} \nabla \theta_{n-1} * \frac{\Vert \nabla M \Vert_{2\ max}}{\Vert \nabla M \Vert_2 + \epsilon} & \text{if } \Vert \nabla M \Vert_2 \gt \Vert \nabla M \Vert_{2\ max} \\ \nabla \theta_{n-1} & \text{if } \Vert \nabla M \Vert_2 \le \Vert \nabla M \Vert_{2\ max} \end{cases}$

SoftUpdate (DQN Target Model)

$\theta_{t\ n}$ denotes the given parameter of the target model following the update, $\theta_{t\ n-1}$ denotes the given parameter of the target model prior to the update, and $\theta_a$ denotes the corresponding parameter of the training agent.

Update Function:

$\theta_{t\ n} = \tau * \theta_a + (1 - \tau) * \theta_{t\ n-1}$