Repository navigation
Model Comparison and Evaluation Methodology
The test set (n=1,300, stratified 80/20 split) has severe class imbalance: quality 3 has 6 samples, quality 4 has 43, quality 9 has just 1, while qualities 5 and 6 together account for 995 of 1,300 (77%). A model that always predicts "6" would already score ~44% accuracy without learning anything about the tails. Macro-averaged F1 (unweighted mean of per-class F1) is reported alongside accuracy throughout precisely because it does not let performance on the majority classes mask complete failure on the minority classes.
| Model | Baseline config | Tuned grid (GridSearchCV) |
|---|---|---|
| KNN | n_neighbors=5 |
n_neighbors ∈ {3,5,7,9}, weights ∈ {uniform, distance}
|
| SVM |
kernel='rbf', default C/gamma |
C ∈ {0.1,1,10}, gamma ∈ {scale, auto}
|
| Decision Tree | default (unbounded) |
max_depth ∈ {5,10,20,None}, min_samples_leaf ∈ {1,5,10}
|
| Random Forest | (tuned only — no untuned baseline reported) |
n_estimators ∈ {100,200}, max_depth ∈ {None,10,20}
|
All features are StandardScaler-normalized before fitting (required for KNN/SVM distance/margin computations; harmless for tree-based models). random_state=42 throughout for reproducibility.
| Quality | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| 3 | 0.00 | 0.00 | 0.00 | 6 |
| 4 | 0.57 | 0.09 | 0.16 | 43 |
| 5 | 0.74 | 0.72 | 0.73 | 428 |
| 6 | 0.67 | 0.79 | 0.72 | 567 |
| 7 | 0.66 | 0.58 | 0.62 | 216 |
| 8 | 0.93 | 0.36 | 0.52 | 39 |
| 9 | 0.00 | 0.00 | 0.00 | 1 |
Interpretation: the model is genuinely competent on the well-populated middle classes (5, 6, 7 — F1 0.62–0.73) but cannot classify quality 3 or 9 at all (0 support-adequate training signal — 6 and 1 samples respectively is far too few for any classifier to learn from), and quality 8's high precision (0.93) with low recall (0.36) means it's conservative — only calling "8" when very confident, missing most true 8s.
The baseline (unbounded) and tuned (grid-searched max_depth/min_samples_leaf) decision trees produced identical results. This means the default unbounded tree was already selecting effectively the same structure as the best grid point — a legitimate finding, not a bug, and a useful contrast to KNN/SVM/RF, where tuning materially helped.
| Model | Baseline accuracy | Tuned accuracy | Δ | Baseline macro F1 | Tuned macro F1 | Δ |
|---|---|---|---|---|---|---|
| KNN | 0.559 | 0.670 | +0.111 | 0.28 | 0.39 | +0.11 |
| SVM | 0.572 | 0.585 | +0.013 | 0.24 | 0.29 | +0.05 |
| Decision Tree | 0.595 | 0.595 | 0.000 | 0.33 | 0.33 | 0.000 |
KNN benefits most from tuning (distance weighting + neighbor count both matter a great deal for this feature space); SVM improves only marginally, suggesting the RBF kernel's default scale parameter was already close to optimal for this data; Decision Tree shows no improvement, as noted above.
Per the project's own statistical-rigour convention: accuracy/F1 point estimates are reported without confidence intervals or formal significance testing between models (e.g. no paired test confirming Tuned Random Forest significantly outperforms Tuned KNN rather than differing by chance on this particular 1,300-sample test split). A rigorous follow-up would bootstrap the test set or use repeated stratified k-fold cross-validation to attach uncertainty to the accuracy/F1 gaps reported above, particularly given how few samples back the quality-3/4/8/9 estimates.
Software: scikit-learn (KNeighborsClassifier, SVC, DecisionTreeClassifier, RandomForestClassifier, GridSearchCV, StandardScaler). Full pipeline: main.py. Full classification reports: Tables/classification_report_*.txt.