🎓 Student Health Risk Engine

Predicting at-risk / unhealthy / fit students from 690,088 records — optimized for balanced accuracy on a brutally imbalanced target

A leakage-safe, seed-bagged LightGBM · XGBoost · CatBoost ensemble with Optuna-tuned hyperparameters, per-fold target encoding, KMeans clustering, a self-selecting stacking meta-learner, probability-threshold optimisation for a brutally imbalanced target, and confidence-gated pseudo-labeling — evaluated entirely on 5-fold stratified CV so the noisy public board never got to lie to me.

---

⏱️ 60 seconds

690,088 students, 13 features, 3 classes split 85.9% / 8.4% / 5.8% — and the metric is balanced accuracy, not plain accuracy. That imbalance-plus-metric combination is the entire game; every technique below exists because of it.

Best local CV (balanced accuracy)	0.9521
Public leaderboard (20% slice)	0.9500
Base models per round	5 folds × 3 families × 3 seeds = 45 fits
Train / test rows	690,088 / 295,753
Total training jobs, start to finish	~280

The five folds hold inside a 0.003 band — that tightness is the result. A model that doesn't collapse on any single held-out fold is a model that survives the private 80% shakeup. The public leaderboard is a blurry 20% mirror; it even reordered my own two submissions.

---

⚖️ Why balanced accuracy — the imbalance is the whole game

A model that always guesses at-risk is right 85.9% of the time on plain accuracy and looks brilliant — until the metric asks "how many fit and unhealthy students did you actually catch?" and the answer is zero.

Balanced accuracy is the average of per-class recall:

balanced_accuracy = mean(recall_at-risk, recall_unhealthy, recall_fit)

The naive always-at-risk model scores (1.0 + 0 + 0) / 3 = 0.333 — garbage. So the metric forces the model to fight for the two thin minority classes. Class weights, threshold tuning, the blend threshold — nearly every design choice below traces back to this one fact.

Class	Share of data
at-risk	85.9%
unhealthy	8.4%
fit	5.8%

---

🏆 The four engines, head to head

Model	Role	Tuned CV
LightGBM	leaf-wise growth, fast, the numeric workhorse	0.9490
XGBoost	level-wise growth, strong regularisation, tree_method="hist"	0.9480
CatBoost	native categorical handling, ordered boosting	0.9475
Ensemble	9-model blend + stack + threshold tuning	0.9521

Three gradient-boosted families, each seed-bagged three times, then averaged and stacked into the final ensemble. CatBoost earns its seat on the six categorical columns; LightGBM and XGBoost carry the numerics. They're kept together because they're decorrelated — each makes different mistakes, so averaging beats any one alone.

---

📈 The climb — and the honest dip at the end

Step	Technique	Local CV
Baseline	single LightGBM, argmax	0.876
+Threshold	Nelder-Mead multipliers on probabilities	0.949
+Ensemble	LGB + XGB + CatBoost weighted blend	0.9510
+SeedBag	3 random seeds per family	0.9515
+Pseudo	retrain on top-2% confident test rows	0.9521 ★
Mega·Stack	+ target-encoding + KMeans + LR/LGB meta-learner	0.9515

Every rung was kept only after it beat the previous one on local CV. The last row is the one most write-ups would quietly omit: the fancier mega-pipeline scored below the leaner pseudo-labeled ensemble. More machinery isn't automatically more signal — sometimes it just adds noise that cancels its own gain. Left in on purpose.

---

🧬 The pipeline

%%{init: {'theme':'base', 'themeVariables': {
  'primaryColor':'#12303D','primaryTextColor':'#EAF2F1','primaryBorderColor':'#38C6C0',
  'lineColor':'#38C6C0','secondaryColor':'#0E2733','tertiaryColor':'#1E4150',
  'fontSize':'15px','fontFamily':'Segoe UI, sans-serif'}}}%%
flowchart LR
    A[(train · test<br/>690k / 296k)] --> B[Feature engineering<br/>ratios · squares · missingness]
    B --> C{per fold}
    C --> D[Target encoding<br/>3 cols × 6 cats]
    C --> E[KMeans cluster<br/>impute → scale → k=3]
    D --> F[9 base models<br/>LGB·XGB·CB × 3 seeds]
    E --> F
    F --> G[Seed + family average]
    G --> H{meta-learner race}
    H -->|higher OOF| I[Stack: LR vs shallow LGB]
    I --> J[Threshold optim<br/>Nelder-Mead, multi-start]
    J --> K{confident ≥ 0.9997?}
    K -->|yes| L[Pseudo-label round 2]
    K -->|no| M[submission.csv]
    L --> N{round2 OOF > round1?}
    N -->|yes| M
    N -->|no| M

    classDef io fill:#0E2733,stroke:#F2B134,color:#EAF2F1;
    classDef proc fill:#12303D,stroke:#38C6C0,color:#EAF2F1;
    classDef dec fill:#1E4150,stroke:#E8604C,color:#EAF2F1;
    class A,M io;
    class B,D,E,F,G,I,J,L proc;
    class C,H,K,N dec;

Loading

Raw rows get engineered features → every fold builds its own target-encoding and KMeans clusters (so nothing leaks) → nine base models train on that fold → their probabilities average, then feed a meta-learner chosen by measurement, not assumption → a threshold optimiser re-balances the decision for an imbalanced metric → the most confident test rows get pseudo-labelled back in for a second round, kept only if it actually helps.

---

🧠 Why gradient-boosted trees, specifically

LightGBM (leaf-wise, fast, the numeric workhorse) · XGBoost (level-wise, strongly regularised, tree_method="hist" + class-balanced weights) · CatBoost (native categorical handling, ordered boosting, slowest but earns it on the six categorical columns).

Gradient boosting means: grow one shallow tree, look at what it got wrong, grow the next tree to fix specifically that, repeat for hundreds of rounds, sum the votes. For mixed-type tabular data at this size, that's still the sweet spot — neural nets typically lose to trees on tables like this one.

---

🎛️ Hyperparameters vs. "parameters" — the honest answer

The knobs we set (hyperparameters). Found by Optuna — a Bayesian search that tries configs, scores them on a cheap 3-fold proxy, and intelligently proposes the next one (20 trials per model):

Model	Key tuned values
LightGBM	learning_rate 0.0104 · num_leaves 37 · min_child_samples 12 · subsample 0.74 · colsample_bytree 0.52 · reg_alpha 0.001 · reg_lambda 0.036
XGBoost	learning_rate 0.0329 · max_depth 4 · min_child_weight 4 · subsample 0.67 · colsample_bytree 0.85 · reg_alpha 1.01 · reg_lambda 0.003
CatBoost	learning_rate 0.0585 · depth 7 · l2_leaf_reg 2.76

Plus structural choices fixed by hand: up to 3000 trees with early stopping (patience 150), balanced class weights, 3 seeds, 5 folds, target-encoding alpha=10, KMeans k=3, pseudo top-2%. Roughly 17 tuned + ~10 hand-set knobs.

The neural-net notion of "parameters." Trees don't have a meaningful parameter count the way a neural net does — a net's parameters are millions of learned weights; a tree's are its split thresholds and leaf values. The whole ensemble sits in the low millions of stored split/leaf values, but that number isn't what characterizes the model — the hyperparameter table above is the actual spec.

---

🧰 Techniques, in plain English

Leakage-safe per-fold target encoding — For each categorical, the smoothed probability of each of the three classes inside that category is computed only on the training fold, then mapped onto val/test. Six categories × three classes = 18 high-signal columns, without letting the validation target leak back in. Smoothing (alpha=10) pulls rare categories toward the global average.

Fold-local KMeans on scaled numerics — Impute (median) → StandardScaler → KMeans(k=3), fit on the train fold only. Gives the trees a "which cluster of students is this?" shortcut.

Self-selecting stacking meta-learner — The 9 models' probabilities become features for a second-level model. Both a Logistic Regression and a shallow LightGBM are trained on the OOF split; whichever scores higher wins — it chose LightGBM both rounds.

Threshold optimisation for an imbalanced metric — Instead of argmax, each class's probability is multiplied by a tunable factor before the max is taken, searched via Nelder-Mead to maximise balanced accuracy. Final multipliers were roughly [1.16, 0.96]. This single trick took the baseline from 0.876 → ~0.949 — the highest-impact-per-line-of-code change in the whole project.

Confidence-gated pseudo-labeling — Only the top-2% most confident test predictions (probability ≥ 0.9997, 5,916 rows) are added as labelled rows and the whole pipeline is retrained. Round 2 is kept only if its OOF strictly beats Round 1 (it did: 0.9515 → 0.9521).

Two data-hygiene bugs that cost real time — CatBoost hard-crashes on a single NaN inside a categorical column; filling with the string "missing" turns the absence into its own learnable category. All numeric columns were downcast to float32, halving RAM with no accuracy cost.

---

🗂️ Repo layout

kaggle/
├── run.py                 # lean ensemble → 0.9521 OOF  (the public-tested file)
├── run_local.py           # mega pipeline (Optuna + stack + TE + KMeans) → 0.9515 OOF
├── blend.py                # probability blend of the two pipelines (weight + threshold search)
├── train.csv / test.csv
├── submission_pseudo.csv  # 0.9521 local  ·  0.94976 public
├── submission_final.csv   # 0.9515 local  ·  0.95000 public
└── feature_importance.xls

---

▶️ Run it yourself

pip install pandas numpy scikit-learn lightgbm xgboost catboost optuna

# fast re-run (tuning skipped — reuses the params Optuna already found)
# in run_local.py set:  SKIP_TUNING = True
python run_local.py        # writes submission_final.csv  +  *.npy probs for blending

python run.py              # writes submission_pseudo.csv

# blend the two pipelines' saved probabilities (no retraining)
python blend.py            # writes submission_blend.csv

Tuning from scratch is ~280 full model fits on 690k rows — hours, not minutes. SKIP_TUNING=True reuses the discovered hyperparameters and skips the warm-up entirely.

Stage	Model fits
Optuna tuning	20 trials × 3 folds × 3 models = 180
Main pipeline, Round 1	5 folds × 9 models = 45
Stacking meta-learner	~10
Pseudo-label retrain, Round 2	5 folds × 9 models = 45
Total	~280 full trainings on 690k rows

---

📊 Honest results ledger

Pipeline	Round-1 OOF	Round-2 OOF	Public (20%)	Note
run.py (lean + pseudo)	0.9496	0.9521	0.94976	best local CV
run_local.py (mega)	0.9499	0.9515	0.95000	best public of the two
Leaderboard #1	—	—	0.95316	top-10 span = 0.00028

The local-CV ordering and the public ordering disagree — that's the 20% slice being noisy, not a flaw. The top ten on that board are separated by less than the noise floor, which is exactly the regime where a robust, un-overfit model climbs on the private 80% while the public-board probers fall.

---

🧠 What this project actually taught me

• axis=0 vs axis=1 on a (samples, models, classes) array isn't a typo — it's the difference between a working ensemble and a shape-mismatch crash on fold 1.
• CatBoost will hard-crash on a single NaN in a categorical column. LightGBM and XGBoost forgive you; CatBoost does not.
• KMeans needs imputed + scaled numerics, or the largest-magnitude column silently owns the distance metric.
• Local CV is the only number you can trust. The public leaderboard is a 20% mirror and it will reorder your own models out of pure noise.
• Fancier ≠ better. The mega-stack lost to the lean pseudo ensemble. The discipline of "only keep it if CV improves" is the whole job.
• Blending needs probabilities, not labels. A majority vote of two CSVs can't break ties; the gain lives in averaging the confidence and re-tuning the threshold.
• The metric shapes the model, not the other way around. Balanced accuracy on an 85.9%-majority target is the reason threshold tuning, class weights, and the whole minority-rescue strategy exist in the first place.

_{Built over one very long night · tuned on free Kaggle CPUs · no leaderboard was harmed in the making of this README.}