Ensemble Tabular Benchmark

A benchmark harness that trains and compares 7 classical ML algorithms — Logistic Regression, Decision Tree, Random Forest, AdaBoost, XGBoost, LightGBM, and CatBoost — across 3 tabular datasets of varying size, class balance, and feature mix, then recommends a production model per dataset based on an accuracy/latency tradeoff.

Architecture

config.yaml → data_loader.py → benchmark.py → results/benchmark_results.csv
                                                        ↓
                                              model_selector.py (recommend)
                                                        ↓
                                              train_production_model.py (train + save)
                                                        ↓
                                              app.py (FastAPI serving)

Results are also visualized via visualize.py (F1 by dataset, accuracy vs. latency, AUPRC on imbalanced data) with charts saved to results/.

Datasets

Dataset	Task	Notes
Wine Quality	Multi-class classification	Small, noisy, imbalanced classes
Credit Fraud	Binary classification	Heavily imbalanced (fraud = rare class)
Adult Income	Binary classification	Larger, mixed categorical/numeric features

Results

Wine Quality (multi-class)

Algorithm	Accuracy	F1
Logistic Regression	0.5938	0.5655
Decision Tree	0.5875	0.5872
Random Forest	0.6875	0.6719
AdaBoost	0.5531	0.5430
XGBoost	0.6594	0.6460
LightGBM	0.6844	0.6734
CatBoost	0.6687	0.6537

Credit Fraud (binary, imbalanced)

Algorithm	F1	AUPRC	ROC-AUC
Logistic Regression	0.9992	0.7117	0.9486
Decision Tree	0.9993	0.6406	0.7993
Random Forest	0.9996	0.8697	0.9764
AdaBoost	0.9990	0.7243	0.9792
XGBoost	0.9995	0.8485	0.9628
LightGBM	0.9981	0.1694	0.6528
CatBoost	0.9996	0.8756	0.9812

⚠️ Key finding: LightGBM posts a strong F1 (0.9981) on Credit Fraud but a near-random AUPRC (0.1694) and ROC-AUC (0.6528). On heavily imbalanced data, F1/accuracy alone would have hidden this failure — AUPRC is the metric that actually surfaces it.

Adult Income (binary, larger + mixed features)

Algorithm	Accuracy	F1
Logistic Regression	0.8072	0.7845
Decision Tree	0.8551	0.8486
Random Forest	0.8657	0.8585
AdaBoost	0.8595	0.8532
XGBoost	0.8746	0.8716
LightGBM	0.8739	0.8710
CatBoost	0.8730	0.8692

Production Recommendations (model_selector.py, balanced priority)

Dataset	Recommended	F1	Train time (s)	Inference (ms/sample)
Wine Quality	LightGBM	0.6734	3.00	0.0311
Credit Fraud	CatBoost	0.9996	3.90	0.0003
Adult Income	CatBoost	0.8692	0.89	0.0005

Takeaways

• Boosting algorithms consistently beat the classical baselines, but the margin over Random Forest shrinks on small/noisy data (Wine Quality) and widens on larger, mixed-feature data (Adult Income). "Just use XGBoost" is directionally right but imprecise — the actual winner depends on dataset size, feature mix, and class balance.
• On imbalanced data, the metric you optimize for matters as much as the algorithm. LightGBM's near-invisible failure on AUPRC despite a strong F1 score shows that accuracy-style metrics alone can silently pick the wrong production model.
• CatBoost was the most consistently strong performer across both imbalanced (Credit Fraud) and larger mixed-feature (Adult Income) datasets, while LightGBM led on the small/noisy multi-class dataset.

Known Limitations / Gotchas Hit During Development

• SSL cert errors on macOS/Python 3.14: UCI dataset downloads failed with CERTIFICATE_VERIFY_FAILED. Fixed by pointing SSL_CERT_FILE at certifi's bundle: export SSL_CERT_FILE=$(python3 -c "import certifi; print(certifi.where())").
• XGBoost label encoding: XGBoost requires contiguous 0..n-1 class labels. Wine Quality's raw labels (3–8) caused Invalid classes inferred from unique values of y. Fixed by always remapping targets through LabelEncoder before the classification split.
• Adult Income target column: The UCI Adult dataset ships headerless, so target='income' wasn't found until column names were explicitly passed to pd.read_csv via header=None, names=ADULT_COLUMNS.
• Logistic Regression throws a ConvergenceWarning at max_iter=1000 on all three datasets; results are still reported, but accuracy would likely improve with feature scaling or a higher iteration cap.

Stack

Python 3.14, scikit-learn, XGBoost, LightGBM, CatBoost, pandas, MLflow (experiment tracking), FastAPI (serving), matplotlib/seaborn (visualization), pytest.

Run the benchmark

pip install -r requirements.txt
python3 -m src.benchmark

This trains every configured algorithm on every configured dataset, logs to MLflow, and writes results/benchmark_results.csv.

View experiment tracking:

mlflow ui

Get a production recommendation

python3 -m src.model_selector wine_quality balanced

{
  "dataset": "wine_quality",
  "priority": "balanced",
  "recommended_algorithm": "lightgbm",
  "f1": 0.87,
  "inference_ms_per_sample": 0.031,
  "reasoning": "lightgbm offers the best accuracy/latency tradeoff (F1=0.87, 0.031 ms/sample)."
}

Train and deploy the recommended model

python3 -m src.train_production_model wine_quality
uvicorn app:app --reload

curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"dataset": "wine_quality", "features": {"alcohol": 10.5, "pH": 3.3}}'

Docker

docker build -t ensemble-benchmark .
docker run -p 8000:8000 ensemble-benchmark

Testing

pytest tests/ -v

Findings (fill in after running)

Dataset	Best Algorithm	F1	Inference (ms/sample)	Notes
wine_quality	—	—	—	small, clean
credit_fraud	—	—	—	imbalanced
adult_income	—	—	—	mixed categorical

Run python3 -m src.benchmark and fill this table in from results/benchmark_results.csv.

CI/CD

.github/workflows/ci.yml runs tests, compile-checks all source files, and builds the Docker image on every push — a green check confirms the whole pipeline actually works, not just that it looks right.