Bare Metal ML

Every algorithm here is implemented from raw NumPy and proven equivalent to scikit-learn by automated parity tests.

Quickstart

python -m pip install -e ".[dev]"
pytest -q

import numpy as np
from bareml import LogisticRegression
X = np.array([[-2.0, -1.0], [-1.0, -2.0], [1.0, 2.0], [2.0, 1.0]])
y = np.array([0, 0, 1, 1])
model = LogisticRegression().fit(X, y)
predictions = model.predict(X)

The executed heart disease notebook compares bareml and scikit-learn models on the UCI dataset using the same split, preprocessing, and bareml metrics.

Parity

Algorithm	File	Reference implementation	Parity tolerance
Scalar reverse-mode autodiff	src/bareml/autograd.py	Central finite differences	Absolute gradient error ≤ 1e-4
Multilayer perceptron	src/bareml/nn.py	NumPy two-moons learning check	Training accuracy ≥ 95%
Linear regression, closed form	src/bareml/linear.py	sklearn.linear_model.LinearRegression	Coefficients and intercept atol=1e-6
Linear regression, gradient descent	src/bareml/linear.py	sklearn.linear_model.LinearRegression	R² difference ≤ 0.01
Logistic regression	src/bareml/logistic.py	sklearn.linear_model.LogisticRegression	Prediction agreement ≥ 98%; ROC-AUC difference ≤ 0.02
K-nearest neighbours	src/bareml/knn.py	sklearn.neighbors.KNeighborsClassifier	Exact prediction match
K-means	src/bareml/kmeans.py	sklearn.cluster.KMeans	Inertia difference ≤ 5%
Accuracy	src/bareml/metrics.py	sklearn.metrics.accuracy_score	atol=1e-12
Mean squared error	src/bareml/metrics.py	sklearn.metrics.mean_squared_error	atol=1e-12
Binary log-loss	src/bareml/metrics.py	sklearn.metrics.log_loss	atol=1e-9
ROC-AUC	src/bareml/metrics.py	sklearn.metrics.roc_auc_score	atol=1e-12, including tied scores

The maths

Reverse-mode autodiff

For a composition (L=f(g(x))), the chain rule gives (\mathrm{d}L/\mathrm{d}x=(\mathrm{d}L/\mathrm{d}g)(\mathrm{d}g/\mathrm{d}x)). Value.backward() visits the computation graph in reverse topological order, multiplying each incoming gradient by the node's local derivative. A value can feed several later operations, so each path contributes a term and gradients use +=; this is the multivariable chain rule as a sum over paths, and it is also why gradients must be cleared between optimisation steps.

Logistic loss

For a design matrix (X), weights (w), binary targets (y), and (n) samples:

[ z = Xw,\qquad p=\sigma(z) ] [ L(w)=-\frac{1}{n}\sum_i\left[y_i\log p_i+(1-y_i)\log(1-p_i)\right] ] [ \frac{\partial L}{\partial z}=\frac{\sigma(z)-y}{n} ] [ \nabla_w L=\frac{X^\mathsf{T}(\sigma(Xw)-y)}{n} ]

Normal equation

Ordinary least squares minimises (\lVert Xw-y\rVert_2^2). Setting its gradient (2X^\mathsf{T}(Xw-y)) to zero gives (X^\mathsf{T}Xw=X^\mathsf{T}y), hence the normal equation (w=(X^\mathsf{T}X)^{-1}X^\mathsf{T}y). The implementation calls np.linalg.solve(X.T @ X, X.T @ y) because solving the linear system avoids explicitly forming an inverse, uses less work, and generally introduces less numerical error.

Lloyd's algorithm

With centres fixed, assigning every sample to its nearest centre cannot increase the sum of squared distances, or inertia. With assignments fixed, replacing each centre by its cluster mean minimises that cluster's squared distances, so alternating the two steps makes inertia decrease monotonically until convergence.

ROC-AUC

After assigning average ranks to tied scores, the rank-sum formula is (\mathrm{AUC}=[R_+-n_+(n_++1)/2]/(n_+n_-)), where (R_+) is the sum of positive ranks. This is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative, with a tied pair counting as one half.

Limitations

The scalar autograd engine is orders of magnitude slower than vectorised frameworks; optimisation is limited to batch gradient descent, and the library is educational rather than production-oriented. KNN also stores the full training set and forms a query-by-training distance matrix, so its memory use grows with the training data.