gmm-from-scratch

A Gaussian Mixture Model fit with the Expectation-Maximization (EM) algorithm, implemented from scratch in NumPy. Demonstrated on a 3D toy dataset (3,500 points clustered around the corners of a cube).

What it does

Fits a $J$-component Gaussian mixture to 3D data using the EM algorithm. Every component of the math (log-Gaussian density, log-sum-exp for numerical stability, full covariance updates with diagonal regularization) is implemented by hand without any ML library.

The dataset (dataGMM.txt) has 3,500 points clustered near the corners of a cube. A quick visualization suggests $J=8$, but counting points per octant reveals one octant is empty - so the correct mixture order is $J=7$. The notebook walks through this reasoning, fits the mixture with EM in 6 iterations, and tests on 5 hand-picked points.

Why it matters

GMMs are the canonical example of a latent-variable generative model. The EM algorithm derived for GMMs is the same algorithmic skeleton you use for HMMs, factor analysis, and probabilistic PCA. Knowing the recipe by heart - and spotting numerical issues like covariance underflow before they bite - is a baseline literacy for everyone working with probabilistic models.

How it works

• E-step: compute responsibilities $\gamma_{nj} = P(z_n = j \mid x_n)$ using log-densities and log-sum-exp.
• M-step: closed-form ML updates for $\pi_j, \mu_j, \Sigma_j$, plus a small diagonal regularization $\varepsilon I$ on each covariance to keep them positive definite.
• Initialization: per-octant sample means and proportions (geometry-aware, not k-means).
• Stopping: relative change in log-likelihood below tolerance (default 1e-8).
• Inference: hard cluster id = $\arg\max_j \gamma_{nj}$, soft probs = the full $\gamma$ vector.

Quickstart

git clone https://github.com/Mathos34/gmm-from-scratch
cd gmm-from-scratch
python -m venv .venv && source .venv/bin/activate   # or .venv\Scripts\activate on Windows
pip install -r requirements.txt
jupyter notebook lab_gmm.ipynb

The notebook runs end-to-end in a few seconds.

Results

Quantity	Value
Number of points $N$	3,500
Dimension $d$	3
Visual guess for $J$	8 (one per cube corner)
Refined $J$ (data-driven)	7 (one octant is empty)
EM iterations to converge	6
Final log-likelihood	-21,626
Recovered $\mu_j$	within 0.1 of $(\pm 20, \pm 20, \pm 20)$ for the 7 active corners
Mixing proportions $\pi_j$	uniform 1/7

The 5 test points are all classified into the matching corner cluster with responsibility ~1.

About

Lab from the Advanced Machine Learning course at ECE Paris (4th-year engineering, Major Data & AI).

Built by Mathis Lacombe, AI Maker at the Intelligence Lab, ECE Paris. LinkedIn · Hugging Face