Gradient-Based Learning: From Linear Regression to a One-Hidden-Layer Neural Network
This project implements two supervised learning algorithms completely from scratch using only NumPy and Matplotlib. The objective is to demonstrate the mathematical foundations of gradient-based learning by manually deriving and implementing gradient descent and backpropagation without relying on machine learning frameworks or automatic differentiation.
The project was developed as part of the practical assignment:
Deriving and Implementing Gradient-Based Learning: From Linear Regression to a One-Hidden-Layer Neural Network
- Derive gradients for Linear Regression using the chain rule.
- Implement Linear Regression from scratch.
- Derive backpropagation equations for a one-hidden-layer neural network.
- Implement forward propagation and manual backpropagation.
- Compare model performance on a nonlinear regression dataset.
- Investigate the effect of different learning rates on convergence.
- Linear Regression implemented from scratch
- One-Hidden-Layer Neural Network implemented from scratch
- Manual Gradient Descent optimization
- Manual Backpropagation (no autograd)
- Xavier weight initialization
- Hyperbolic Tangent (tanh) activation function
- Mean Squared Error (MSE) loss
- Synthetic nonlinear dataset generation
- Learning rate comparison
- Multiple visualization plots
gradient_learning.ipynb
README.md
requirements.txt
- Python 3
- NumPy
- Matplotlib
- Jupyter Notebook / Google Colab
A synthetic nonlinear regression dataset is generated using:
[ y = \sin(x) + \epsilon ]
where
- (x \in [-3,3])
- (\epsilon) is Gaussian noise.
This dataset was selected to clearly demonstrate the limitations of Linear Regression and the ability of neural networks to learn nonlinear relationships.
Implemented completely from scratch.
Includes:
- Prediction function
- Mean Squared Error Loss
- Manual gradient computation
- Gradient Descent optimization
- Training loop
Prediction equation
[ \hat y = wx+b ]
2. One-Hidden-Layer Neural Network
Architecture
Input
│
Hidden Layer (16 neurons)
│
tanh Activation
│
Output Layer
│
Prediction
Implemented components
- Xavier Initialization
- Forward Propagation
- Manual Backpropagation
- Gradient Descent
- MSE Loss
No machine learning libraries were used.
The neural network was trained using three different learning rates.
| Learning Rate | Purpose |
|---|---|
| 0.001 | Slow convergence |
| 0.01 | Stable convergence |
| 0.1 | Faster updates with potential instability |
The convergence behavior of each configuration is visualized and compared.
The notebook includes:
- Synthetic dataset visualization
- Linear Regression fit
- Neural Network fit
- Linear Regression loss curve
- Neural Network loss curve
- Learning rate comparison
- Linear Regression vs Neural Network prediction comparison
- Training loss comparison
The experiments demonstrate that:
- Linear Regression performs well only for linear relationships.
- A one-hidden-layer neural network successfully learns nonlinear patterns.
- Learning rate significantly influences convergence speed and training stability.
- Manual backpropagation produces effective parameter updates using the derived gradients.
Example final performance:
| Model | Mean Squared Error |
|---|---|
| Linear Regression | 0.183468 |
| Neural Network | 0.034625 |
The neural network achieves substantially lower prediction error on the nonlinear dataset.
- Gradient-Based Learning
- Gradient Descent
- Mean Squared Error
- Chain Rule
- Multivariate Chain Rule
- Forward Propagation
- Backpropagation
- Weight Initialization
- Activation Functions
- Learning Rate Analysis
Clone the repository
git clone https://github.com/Yeabebe/gradient-based-learning.gitNavigate to the project directory
cd gradient-based-learningInstall dependencies
pip install -r requirements.txtLaunch Jupyter Notebook
jupyter notebookor open the notebook using Visual Studio Code or Google Colab.
numpy
matplotlib
jupyter
ipykernel
This project demonstrates how modern neural networks are built from the fundamental principles of calculus and linear algebra.
By implementing every component manually, the project provides a deeper understanding of:
- Gradient computation
- Parameter optimization
- Backpropagation
- Neural network training
- The relationship between mathematical derivations and practical machine learning implementation
Yeabsera Abebe
AI Engineer
This project is intended for educational and academic purposes.