Linear Regression with Gradient Descent

A collection of notebooks and scripts demonstrating the fundamentals of a Simple Single-Neuron Neural Network, focusing on Linear Regression and Gradient Descent from scratch.

No deep learning frameworks — just NumPy and a clean implementation of forward pass and gradient descent.

Followed on by MLP Digits Classifier, extending gradient descent from a single-output linear model to a multi-class neural network.

📚 Contents

• Gradient Descent from Scratch - Jupyter Notebook exploring the implementation step-by-step.

• Linear Regression Explanation - Detailed explanation of the underlying gradient descent code and math.

• Linear Regression Gradient Descent - Basic Python implementation using raw inputs.

• Gradient Descent Animation - Animated visualisation of the regression line converging over training epochs.

• Linear Regression Normalised - Implementation with normalized inputs for more stable and faster convergence.

• Linear Regression - Ames Housing - Real-life example applying linear regression with gradient descent on the Ames Housing dataset (predicting sale price from living area).

How It Works

1. Data — Generates 30 points following y = 2x + 50 using np.arange.
2. Initialisation — Weight W and bias b start at 0; learning rate learning_rate = 0.001.
3. Forward pass — Computes predictions: y_pred = W * X + b.
4. Gradients — Calculates partial derivatives of Mean Squared Error (MSE) loss with respect to W and b.
5. Update — Subtracts learning_rate * gradient from each parameter, nudging them toward lower loss.
6. Result — After 10,000 epochs, the model recovers w ≈ 2 and b ≈ 50.

See linreg_notes.md for a detailed breakdown of the math and code.

Note 1: The 30 data points were found empirically to be enough for the model to reliably recover w ≈ 2 and b ≈ 50.

Note 2: Removing the learning rate (learning_rate) causes dW and db to explode, resulting in y_pred becoming NaN and training failing completely.

Note 3: For the Linear Regression Gradient Descent example, themv Gradient_Descent.gif linreg_gd_anim.gifmv Gradient_Descent.gif linreg_gd_anim.gif learning rate (0.001) and number of epochs (10,000) were found empirically — small changes to either can cause the model to converge too slowly or diverge entirely.

Note 4: The Linear Regression Normalised example normalises the inputs for more stable convergence, reducing sensitivity to the choice of learning rate. This allows a much lower learning rate of 0.1 and only 20 epochs to converge.

🛠️ Key Libraries

• matplotlib - For histogram and scatter plot visualizations.

🔗 References

• Boston Housing - Practical real-life example applying linear regression with gradient descent on a well-known dataset.
• Regression from Scratch Example (Medium)
• Gradient Descent in Python (GeeksforGeeks)
• Gradient Derivative Calculation (Towards Data Science)