A from-scratch, end-to-end reimplementation of the core of Data 100 — Principles and Techniques of Data Science (UC Berkeley, Spring 2023 offering), part of a csdiy.wiki full-catalog build.
Data 100 is Berkeley's upper-division data-science course: the full pipeline from
pandas wrangling and EDA through visualization, SQL, regular expressions,
least-squares modeling, gradient descent, regularization, the bias–variance
tradeoff and cross-validation, logistic regression & classification metrics, PCA,
and clustering — capped by two projects (housing-price regression and spam/ham
email classification).
This repo implements a representative, running set that spans every one of
those topics. The estimators are written from scratch in NumPy (OLS via the
normal equations, batch gradient descent, ridge, coordinate-descent LASSO,
logistic regression, PCA via SVD, k-means) in a small tested library
(src/data100/), then cross-checked against scikit-learn. Every lab and project
runs on real data downloaded at runtime — the Ames housing dataset, the
Apache SpamAssassin email corpus, and the handwritten-digits dataset — and writes
its measured metrics and figures to results/.
Cross-check test-suite: 16/16 pytest passing — every from-scratch estimator
matches its scikit-learn reference (OLS coef to 1e-6, ridge to 1e-15, PCA EVR to
3e-16, ROC-AUC / precision / recall exact, k-means ARI ≥ sklearn).
| Project | Data | Model | Held-out result |
|---|---|---|---|
| A — Housing regression | Ames, 1460 homes, 93 engineered features | from-scratch Ridge (α tuned on validation) | RMSE = $22,652, RMSE(log) = 0.1243, R² = 0.892 (predict-mean baseline $70,301) |
| B — Spam/ham classification | SpamAssassin, 4,395 emails (43% spam), 120 word + 7 engineered features | from-scratch logistic regression (GD) | acc = 0.973, precision = 0.976, recall = 0.960, F1 = 0.968, ROC-AUC = 0.997 |
Project A's from-scratch ridge matches scikit-learn's Ridge to 1.4e-15;
Project B's from-scratch logistic regression is within 0.003 accuracy of
scikit-learn's.
| Lab | Topic | Result (measured) |
|---|---|---|
| 2 | Pandas | groupby/pivot/merge on Ames; priciest neighborhood NoRidge $335,295; median $120/sqft |
| 3 | Data cleaning, EDA, regex | spam averages 4.27 URLs vs 1.72 for ham, 2.40 vs 0.11 money mentions; PoolQC 99.5% missing |
| 4 | Visualization & transforms | SalePrice skew 1.88 → 0.12 under log; log-log Pearson r = 0.730 |
| 6 | OLS | from-scratch normal equations = sklearn to 8e-4; test RMSE $39,487, R² 0.797 |
| 7 | Gradient descent | batch GD → OLS optimum; relative MSE gap 1.7e-16, coef gap 3e-7 (lr set from Hessian λ_max=9.9) |
| 8 | Model selection | 5-fold CV picks polynomial degree 3; LASSO drives 4/10 coefficients to exactly 0 |
| 9 | Probability & inference | bootstrap median 95% CI [$158,896, $167,700]; permutation test p = 1e-4 (A/C ⇒ +$80,923); CLT SE 11,057 ≈ theory 11,231 |
| 11 | SQL | sqlite3 GROUP BY/HAVING, correlated subquery, RANK() OVER window; 636 homes above their neighborhood average |
| 12 | Logistic regression | digits 3-vs-8: acc 0.991, precision 0.981, recall 1.000, ROC-AUC 1.000 (= sklearn) |
| 13 | PCA | from-scratch SVD; explained-variance ratios = sklearn to 3e-16; 21 PCs for ≥90% variance |
| 14 | Clustering | k-means (from scratch) on digits: ARI 0.663, majority-vote purity 0.789 |
projectA_pred_vs_actual.png · projectB_spam_roc_confusion.png ·
lab04_visualization.png · lab07_gd_convergence.png ·
lab08_bias_variance.png · lab08_regularization_paths.png ·
lab09_inference.png · lab12_roc_curve.png · lab13_pca.png ·
lab14_elbow.png · lab14_centroids.png (13 figures + 13 metric JSONs total).
Labs (spanning the full Data 100 syllabus):
- Lab 2 — Pandas manipulation
- Lab 3 — Data cleaning, EDA & regular expressions
- Lab 4 — Visualization, transformations & KDEs
- Lab 5/6 — Modeling, loss functions & OLS
- Lab 7 — Gradient descent
- Lab 8 — Model selection, bias–variance & regularization (ridge/LASSO/CV)
- Lab 9 — Probability, bootstrap & the permutation test
- Lab 11 — SQL
- Lab 12 — Logistic regression & classification metrics
- Lab 13 — PCA
- Lab 14 — Clustering (k-means)
Projects:
- Project A — Housing-price regression (Ames)
- Project B — Spam/ham email classification (SpamAssassin)
The homework topics (permutation test, EDA/"Food Safety"-style cleaning, regression, probability, SQL) are covered by the correspondingly-themed labs above.
ucb-data100/
├── src/data100/ # from-scratch, tested library
│ ├── linear_models.py # OLS (normal eq / QR), gradient descent, ridge, LASSO
│ ├── logistic.py # logistic regression via gradient descent
│ ├── decomposition.py # PCA via SVD
│ ├── cluster.py # k-means (++ init)
│ ├── metrics.py # RMSE/R², precision/recall/F1, ROC curve & AUC
│ ├── datasets.py # runtime download + caching of real datasets
│ └── report.py # results/ helpers (headless figures + JSON)
├── labs/ # lab02 … lab14, one runnable script each
├── projects/
│ ├── project_a_housing/housing_regression.py
│ └── project_b_spam/spam_classifier.py
├── scripts/
│ ├── download_data.py # fetch all datasets into data/ (git-ignored)
│ └── run_all.py # run the whole repo end-to-end
├── tests/test_data100.py # 16 cross-checks vs scikit-learn
└── results/ # measured metrics (JSON) + figures (PNG)
# Python 3.11. Use the shared csdiy env or a fresh venv:
# D:\Project\_csdiy\.venv-ml\Scripts\python.exe
python -m pip install -r requirements.txt
# Fetch the real datasets (Ames / SpamAssassin / digits) into ./data (git-ignored):
python scripts/download_data.py
# Run everything (all labs + both projects), writing figures & metrics to results/:
python scripts/run_all.py
# Or run one piece:
python labs/lab08_model_selection_cv.py
python projects/project_a_housing/housing_regression.py
python projects/project_b_spam/spam_classifier.py
# Cross-check the from-scratch library against scikit-learn:
python -m pytest tests/ -q # 16 passedpytest tests/→ 16 passed. Each from-scratch estimator is asserted equal to its scikit-learn counterpart on synthetic data: OLS coefficients (atol 1e-6), gradient descent converging to the OLS optimum, ridge (1e-6), LASSO sparsity, logistic-regression accuracy & coefficient direction (cosine > 0.99), ROC-AUC / precision / recall (exact), PCA explained-variance ratios, and k-means recovering separable blobs (ARI > 0.95).- Full run (
scripts/run_all.py) completes all 14 stages in ~40 s on CPU and regenerates every file inresults/. - The numbers in the tables above are read straight from the JSON files in
results/produced by that run.
Python 3.11 · NumPy · pandas · scikit-learn (reference cross-checks only) · Matplotlib · SciPy · SQLite (stdlib) · pytest.
- The normal equations, three ways. OLS as
(XᵀX)⁻¹Xᵀy, as a QR/lstsqsolve, and as the fixed point of gradient descent — and why the learning rate must respect the loss Hessian's largest eigenvalue (Ames' correlated features make naïve step sizes diverge). - Regularization geometry. Ridge shrinks coefficients smoothly; LASSO's ℓ¹ penalty (coordinate descent + soft-thresholding) drives them to exactly zero, giving feature selection for free.
- Bias–variance & cross-validation as the principled way to choose model complexity, shown on polynomial degree with k-fold CV.
- Classification is thresholding a score. Precision/recall trade off with the decision threshold; the ROC curve and its AUC summarize the whole sweep.
- Feature engineering is where the accuracy comes from — log-transforming a skewed target, building aggregate features, and selecting discriminative words (no leakage) lift the housing R² to 0.89 and spam ROC-AUC to 0.997.
- PCA is just the SVD of the centered data matrix; k-means is Lloyd's alternating minimization.
Based on the assignments of Data 100 — Principles and Techniques of Data Science by the UC Berkeley EECS/Statistics departments (Spring 2023, ds100.org). This repository is an independent educational reimplementation; all course materials belong to their original authors.
Datasets are downloaded at runtime and not committed — sources: Ames housing (De Cock, 2011; via OpenML), the Apache SpamAssassin public corpus, and scikit-learn's bundled handwritten-digits dataset.
Original code in this repo is released under the MIT License.