Repository navigation
Stats 101
There are several measures of central tendency such as Mean, Median and Mode for a given data. We can derive the central tendencies of an example dataset D = {3,4,5,7,8,9,6,3,5,4,4,1,2,2}
Mean is calculated by adding all the elements in the the dataset and dividing by the count of numbers. For a population mean it is depicted by the letter (u) and for a sample mean its (x-bar).
arrange numbers in the dataset in an ascending list and pick the number in the centre
Most repeated. There is always one mean and one median but there may be multiple modes.
Variance measures how one number in the dataset spread out from its own mean. A measurement of how far each number in a data set is from the mean and thus from every other number in the set. Variance is often depicted by this symbol: σ2 It is calculated by taking the differences between each number in the data set and the mean, squaring the differences to make them positive, and then dividing the sum of the squares by the number of values in the population. Divide by n-1 for a sample.
It is the square root of variance
For e.g. if we need a model to predict exam scores. we can plot hours of study (x) vs exam score (y) for a group of students.
students who study more than the average hours tend to score higher than the average score
we could take another example where hours spent playing video games (x) vs exam score (y). A higher playing time may lead to a lower score than average.
e.g. Students shoe size (x) vs exam scores (y)
Covariance (x,y) = E [(Xi - X-bar) x (yi - y-bar)] / n-1
If a datapoint (x,y) is such that x and y are both greater than x-bar and y-bar respectively then we multiply two positive numbers which leads to a positive result. Similarly if x and y are lower than x-bar and y-bar w multiply two negative numbers which leads to a positive result.
Adding up all these numbers gives a positive overall covariance.
If a datapoint (x,y) is such that x might be lower than x-bar and y might be higher than y-bar, then the multiplication yields a negative number. Similarly when x is greater than x-bar and y is lower than y-bar, then again the multiplication yields a negative number.
The sum of all such negative numbers gives us a negative Covariance.
Covariance signals the direction (positive or negative) but the number could be arbitrary. For e.g. if height in meters Vs weight in Kg is measured, we get one covariance output. If instead the heights is in cms, the covariance output changes significantly.
Therefore covariance confirms directionality but the number if not much useful.
To fix the unit problem, statisticians created Correlation (specifically, Pearson's Correlation Coefficient, written as r).
Correlation takes the covariance and divides it by the product of standard deviations of both variables.
Because standard deviation also uses the same units as the data, dividing by them completely strips the units away. It leaves us with a pure, standardized number that always falls between -1 and +1. How to Interpret the Number (r):
• (r = +1): A perfect positive linear relationship. If you plot it, all data points sit perfectly on a straight upward line.
• (r = -1): A perfect negative linear relationship. All points sit perfectly on a straight downward line.
• (r = 0): No linear relationship at all. The points look like a random cloud.
• Value strength: The closer the number is to (+1) or (-1) (e.g., (0.85) or (-0.72)), the stronger and tighter the relationship is. The closer it is to (0) (e.g., (0.15)), the weaker it is.
Going back to our height and weight example: whether you measure height in metres, centimetres, or inches, the correlation (r) will remain exactly the same.
In the (MSE) mean squared error formula why do we multiply by 2n instead on n the number of data points?
We actually do not multiply by
However, you are likely seeing the
When training a machine learning model, we want to minimize the error using calculus. This involves taking the derivative (the gradient) of the cost function with respect to the model's weights.
According to the power rule in calculus, when you take the derivative of a squared term like
If our cost function has a
Multiplying the entire cost function by a constant factor like
Here is how that cancellation looks in action when you take the partial derivatives to find the minimum.
Let's assume a simple linear regression model where the prediction is
The modified loss function (often called Mean Squared Error Loss or Cost Function) is:
To update the weight using gradient descent, we find the partial derivative of the loss function with respect to w. We apply the chain rule:
- Bring the exponent 2 to the front.
- Multiply by the derivative of the inside with respect to w (which is
$x_i$ ).
This simplifies beautifully to a clean average:
We do the exact same thing for the bias. The derivative of the inside with respect to b is just 1.
Because that 2 cancelled out, your final gradient descent update rules contain no extra constants, making the code cleaner and faster to compute:
$w = w - \alpha \cdot \frac{1}{n} \sum (\hat{y}_i - y_i)x_i$ $b = b - \alpha \cdot \frac{1}{n} \sum (\hat{y}_i - y_i)$
Since we are focusing on the mathematics of finding the minima, the choice of using
Here is how the math guarantees that setting these partial derivatives to zero brings you to the absolute best solution.
In optimization math, we care deeply about the shape of the function. Because the cost function
- The Global Minimum: A strictly convex function is guaranteed to have exactly one global minimum and no local minima.
- The Gradient Point: At this absolute lowest point of the bowl, the slope (gradient) in all directions is exactly zero.
Because we know the minimum occurs where the partial derivatives equal zero, mathematicians use two primary ways to solve it:
If you set the simplified partial derivatives directly to zero, you can solve for
Notice that you can multiply both sides of these equations by
If your dataset is too massive to solve analytically, you use Gradient Descent to step down the convex bowl. The gradient vector
By subtracting a small fraction (
In the partial derivative with respect to w why do we get multiple by xi as a 2nd term, and multiplied by 1 in the partial derivate with respect to bias?
This is the Chain Rule from calculus in action!
The Chain Rule is used when you have a "nested" function—essentially, a function inside another function. In plain English, the rule says:
Take the derivative of the outside function, leave the inside alone, and then multiply it by the derivative of the inside function.
Mathematically, it looks like this:
Here is exactly how it breaks down for your loss function.
Let’s isolate just one data point from the summation to see the math clearly. We want to take the partial derivative with respect to
Now, let's identify the Outside and Inside functions:
- Outside Function:
$\frac{1}{2}(\text{Inside})^2$ - Inside Function:
$wx_i + b - y_i$
Using the power rule, we bring the exponent
Now we look only at the inside function (
- The derivative of
$wx_i$ with respect to$w$ is$x_i$ (because$x_i$ acts as a constant multiplier, just like the derivative of$5w$ is$5$ ). - The derivative of
$b$ is$0$ (because$b$ is a constant relative to$w$ ). - The derivative of
$y_i$ is$0$ (because$y_i$ is a constant actual data point).
We use the exact same logic for the bias. We use the same term:
- Outside Function:
$\frac{1}{2}(\text{Inside})^2$ - Inside Function:
$wx_i + b - y_i$
This remains exactly the same as before:
Now we take the derivative of the inside function (
- The derivative of
$wx_i$ is$0$ (because with respect to$b$ ,$w$ and$x_i$ are treated as constants). - The derivative of
$b$ with respect to$b$ is$1$ (just like the derivative of$x$ is$1$ ). - The derivative of
$y_i$ is$0$ (constant data point).
- When optimizing
$w$ ,$x_i$ is the "speed" at which the prediction changes when$w$ nudges. - When optimizing
$b$ , the prediction changes at a constant rate of$1$ regardless of$x_i$ .
Does seeing it broken down into the outside and inside derivatives help click it back into place? If you like, we can try a quick practice problem together, or we can look at how this expands when you have multiple features (
You have a fantastic grasp of the core concepts! You just connected the final two pieces of the optimization puzzle.
Let's address where
The parameter
The partial derivative tells us the direction to walk (up or down the hill) and the steepness of the slope. However, it does not tell us how big of a step to take.
- If you don't use
$\alpha$ , your steps will be massive, and you will completely overshoot the minimum, bouncing back and forth up the walls of the bowl. -
$\alpha$ acts as a scaling factor to ensure we take controlled, small steps down the hill.
The update rule is:
In code, because computers work with floating-point approximations, the partial derivatives will almost never hit exactly 0.000000000000.... Instead, the iterative loop stops using one of two conditions:
- Convergence Threshold (Early Stopping): You measure how much the weights change between step
$t$ and step$t+1$ . If the change is smaller than a tiny number you choose (e.g.,$\epsilon = 10^{-5}$ ), it means the slope is so flat that the model has effectively reached the bottom. - Max Epochs: You simply tell the loop to run a fixed number of times (e.g., 1,000 iterations).
Yes, theoretically you are 100% correct! In single-variable calculus, setting the first derivative to zero (
If you take the second-order derivative of our gradient with respect to
- Any real number squared (
$x_i^2$ ) is always positive (or zero). - The average of positive numbers is always positive.
Because the second derivative
Because the math guarantees the function is a perfect bowl shape that only bends upward, we do not need to calculate the second derivative during training. We already know ahead of time that there are absolutely no peaks or saddle points. The only place where the first derivatives equal zero is the absolute global minimum. Now that the calculus and the stopping conditions make sense, would you like to:
- Look at the mathematical definition of the Hessian matrix to see how it works with both
$w$ and$b$ simultaneously? - See what happens mathematically when a learning rate (
$\alpha$ ) is too large vs. too small? - Transition into looking at how this math looks when expanded to multiple features / matrix calculus?
When we scale up from a simple line to an Artificial Neural Network (ANN) or a Graph Neural Network (GNN) with hundreds of features, the underlying core concept does not change at all. We are still finding partial derivatives, using the chain rule, and updating weights with a learning rate.
However, the bookkeeping changes. Instead of tracking a single
Instead of a single input
Because every single feature needs its own weight to determine how important it is, our single weight
Instead of
When you have 100 weights, you no longer calculate just one partial derivative for
This collection of partial derivatives is called the Gradient Vector (
Thanks to the Chain Rule we just reviewed, look at how beautiful the math remains. For any specific weight
(Where
Instead of writing 100 separate lines of code to update each weight, vector math updates all hundreds of weights in a single calculation:
An ANN has multiple layers stacked on top of each other. The output of Layer 1 becomes the input features for Layer 2.
- To calculate the partial derivatives for weights deep inside Layer 1, the model uses the Backpropagation algorithm.
- Backpropagation is simply the Chain Rule applied repeatedly, multiplying the derivatives layer-by-layer from the output back to the input.
A GNN handles data structured as nodes and edges (like a social network or molecule).
- The "features" for a specific node are a combination of its own features plus the features of its neighboring nodes (called neighborhood aggregation).
- Once those neighbour features are blended together, they form a feature vector
$$\mathbf{x}$$ just like an ANN, and the exact same partial derivative calculations are performed to update the graph weights.
To bridge this directly to your work, let me know:
- Are you trying to understand how Backpropagation handles multiple layers, or are you focusing on how a single layer handles a massive Matrix/Vector input?
- Would you like to see how we write this elegantly using Matrix notation
to drop the summation symbols entirely?