"Six competitions in, skills exponentially sharpened, Expert tier unlocked!"
Advanced Ensemble Pipeline | Multi-Model Stacking | Peak Performance
Sixth competition in a row. Five victories documented. Expert tier ACHIEVED. With each competition, we're sharpening our blades—from basic baselines to advanced ensemble stacking. Medical domain knowledge + competitive experience + team synergy = UNSTOPPABLE.
This is where the phoenix truly ascends. Not beginners. Not intermediate. EXPERT TIER.
🏆 SIXTH COMPETITION (S6E2) | EXPERT TIER | HEART DISEASE PREDICTION 🏆
Five years of learning compressed into six months. Expert status achieved.
Objective: Predict whether a patient has heart disease based on medical metrics
Challenge: Kaggle Playground Series - Season 6, Episode 2
Duration: Completed ✅ - Last Update: March 2026
Metric: Binary Classification (Heart Disease Yes/No)
Team Status: Expert-Level Predictions - TOP 11.1%! 🏆
Current Achievement: Rank 485/4370 (11.1% percentile) - EXPERT FINISH!
| Competition | Episode | Rank | Percentile | Status | Key Learning |
|---|---|---|---|---|---|
| 1️⃣ BPM Prediction | S5E9 | 1246/2581 | 48.3% | ✅ Complete | Foundation |
| 2️⃣ Road Accidents | S5E10 | 960/4082 | 23.5% | ✅ Complete | Optimization |
| 3️⃣ Loan Payback | S5E11 | 1255/3724 | 33.7% | ✅ Complete | AutoML |
| 4️⃣ Diabetes | S5E12 | 877/4206 | 20.8% | ✅ Complete | Best Yet |
| 5️⃣ Test Scores | S6E1 | 968/4317 | 22.4% | ✅ Complete | Consistency |
| 6️⃣ Heart Disease | S6E2 | 485/4370 | 11.1% | ✅ 0.95410 | V30 Latest - Final Push! |
Average Percentile Across 6 Completed: 25.2%
Best Percentile: 11.1% (Heart Disease) ⭐ EXPERT - TOP 485/4370!
Mission Status: ✅ TOP 11.1% ACHIEVED! EXPERT TIER!
- Framework: Fast, efficient gradient boosting
- Architecture: 500 estimators, depth 7, learning rate 0.05
- Role: The BACKBONE of our ensemble
- Expected Strength: Best generalization
- Framework: Gradient boosting excellence
- Architecture: 500 estimators, optimized hyperparameters
- Role: The PRECISION instrument
- Expected Strength: High individual accuracy
- Framework: Categorical feature specialist
- Architecture: 500 iterations, depth 7, native categorical handling
- Role: The CATEGORICAL expert
- Expected Strength: Superior feature understanding
- Framework: Ensemble of decision trees
- Architecture: 500 trees, max_depth 15, bootstrap aggregating
- Role: The STABILITY anchor
- Expected Strength: Robustness & variance reduction
- Framework: Classical scikit-learn approach
- Architecture: 500 estimators, depth 7, learning rate 0.05
- Role: The CLASSICAL workhorse
- Expected Strength: Consistent performer
- Strategy: Train on out-of-fold predictions from 5 base models
- Approach: Learn optimal weighted combination
- Expected Result: ROC-AUC > 0.954
We're analyzing 30+ powerful medical metrics to predict heart disease:
| Feature Category | What We're Measuring |
|---|---|
| 👤 Demographics | Age, Gender |
| 💓 Cardiac Indicators | Chest pain type, Resting BP, Max HR |
| 🩺 Blood Chemistry | Cholesterol, Fasting blood sugar |
| 📊 ECG Metrics | Resting ECG, ST depression, ST slope |
| 🫀 Vascular Data | Number of major vessels |
| 🔬 Genetic Factors | Thalassemia type |
| ⚡ Derived Features | Exercise-induced changes, ratios |
- Load 270,000 samples with 30+ features
- Statistical analysis and exploratory data analysis
- Missing value detection and handling strategy
- RobustScaler normalization
- Median imputation for missing values
- Target encoding (string → numeric)
- Feature standardization
- Individual training: LightGBM, XGBoost, CatBoost, Random Forest, Gradient Boosting
- Hyperparameter tuning for each algorithm
- Cross-validation strategy: 5-Fold Stratified CV
- Out-of-fold prediction collection
- Weight optimization for weighted ensemble
- Meta-model training on OOF predictions
- Final ensemble evaluation
- Kaggle-optimized code
- Production-ready submission format
- GPU-ready for Kaggle notebooks
-
Diverse Base Models 🤝
- Multiple algorithms capture different patterns
- Reduces overfitting through diversity
- Each model contributes unique insights
-
5-Fold Stratified Cross-Validation 🔄
- Ensures balanced train-test splits
- Prevents data leakage
- Consistent validation metrics across folds
-
Out-of-Fold Predictions 📊
- Collect OOF predictions from all folds
- Use for meta-model training
- Eliminates overfitting on meta-learner
-
Meta-Model Stacking 🏗️
- Logistic Regression learns optimal weights
- Higher-order patterns captured
- Final ensemble > any single model
-
Feature Engineering 🔬
- Proper scaling and normalization
- Missing value handling
- Medical domain knowledge applied
-
Kaggle Optimization 🎯
- Uses Kaggle competition data paths
- Ready for GPU acceleration
- Runs in 10-15 minutes on Kaggle
HEART DISEASE PREDICTION ENGINE
├── INPUT: 270,000 samples × 30+ features
├── PREPROCESSING
│ ├── Missing values → Median imputation
│ ├── Scaling → RobustScaler normalization
│ ├── Target → String to numeric mapping
│ └── Categorical → Proper encoding
├── BASE MODEL TRAINING (5-Fold CV)
│ ├── LightGBM (30%)
│ ├── XGBoost (25%)
│ ├── CatBoost (20%)
│ ├── Random Forest (15%)
│ └── Gradient Boosting (10%)
├── OUT-OF-FOLD PREDICTIONS
│ └── Collect OOF predictions for meta-learning
├── META-LEARNING
│ └── Logistic Regression learns optimal weights
└── OUTPUT
├── CV AUC: ~0.9520-0.9542
├── Public Score Target: 0.954+
└── Submission Format: submission.csv
- Go to Kaggle.com/code
- Click "Import Notebook"
- Paste: https://github.com/mohan13krishna/Predicting-Heart-Disease
- Add "playground-series-s6e2" dataset input
- Enable GPU (optional)
- Run all cells
- Submit submission.csv
# Clone repository
git clone https://github.com/mohan13krishna/Predicting-Heart-Disease.git
cd Predicting-Heart-Disease
# Install dependencies
pip install pandas numpy scikit-learn xgboost lightgbm catboost
# Run the script
python heart_disease_prediction.pyPredicting-Heart-Disease/
├── 📂 programs/ # All model code & notebooks (V9-V30)
│ ├── heart_disease_v30_ensemble.ipynb # Latest (F80_C20 blend, AUC: 0.95410) ⭐
│ ├── heart_disease_v29_ensemble.ipynb # F90_C10 blend (AUC: 0.95410)
│ ├── heart_disease_v28_ensemble.ipynb # F micro-perturbations (AUC: 0.95410)
│ ├── heart_disease_v27_ensemble.ipynb # E+F blending (AUC: 0.95410)
│ ├── heart_disease_v26_ensemble.ipynb # E50_CD50 variant (AUC: 0.95409)
│ ├── heart_disease_v25_ensemble.ipynb # E80_CD20 framework (AUC: 0.95409)
│ ├── heart_disease_v24_ensemble.ipynb # C99_B01, C98_B02 adjustments (AUC: 0.95407)
│ ├── heart_disease_v23_ensemble.ipynb # 6 raw C,D blends (AUC: 0.95408)
│ ├── heart_disease_v22_ensemble.ipynb # Individual C/D analysis (AUC: 0.95408)
│ ├── heart_disease_v21_ensemble.ipynb # Rank-normalized CD blending (AUC: 0.95408)
│ ├── heart_disease_v20_ensemble.ipynb # Original 4-submission blend (AUC: 0.95406)
│ ├── heart_disease_v19_ensemble.ipynb # Standalone model V19
│ ├── heart_disease_v18_ensemble.ipynb # 9-model ensemble (AUC: 0.95360)
│ ├── heart_disease_v17_ensemble.ipynb # 7-model ensemble (AUC: 0.95360)
│ ├── heart_disease_v16_ensemble.ipynb # 5 base models (AUC: 0.95359)
│ ├── heart_disease_v15_ensemble.ipynb # CatBoost integration
│ ├── heart_disease_v14_ensemble.ipynb # LightGBM optimization
│ ├── heart_disease_v13_ensemble.ipynb # XGBoost focus
│ ├── heart_disease_v12_ensemble.ipynb # Logistic Regression meta-learner
│ ├── heart_disease_v11_ensemble.ipynb # 5-Fold × 3 Seeds (AUC: 0.95342)
│ ├── heart_disease_v10_ensemble.ipynb # Cleveland Data (AUC: 0.95303)
│ ├── heart_disease_v9_ensemble.ipynb # Multiple Seeds (AUC: 0.95333)
│ ├── heart_disease_v8_ensemble.ipynb # Early version
│ ├── heart_disease_v8_ensemble.py # Python script variant
│ ├── heart_disease_v7_ensemble.py # Legacy Python version
│ ├── heart_disease_ensemble.ipynb # Original notebook
│ └── heart_disease_prediction.py # Production script (Kaggle)
│
├── 📊 versions/ # Submission variants (A-F.csv)
│ ├── A.csv # V16 Submission (0.95359)
│ ├── B.csv # V17 Submission (0.95360)
│ ├── C.csv # Blend Variant 1 (0.95408)
│ ├── D.csv # Blend Variant 2 (0.95408)
│ ├── E.csv # Blend Variant 3 (0.95409)
│ └── F.csv # Best Single (0.95410) ← Latest Winner
│
├── 📈 DATA
│ ├── train.csv # Training data (630K samples)
│ ├── test.csv # Test data (270K samples)
│ ├── sample_submission.csv # Submission format template
│ ├── submission.csv # Final predictions (F80_C20)
│ └── catboost_info/ # CatBoost training metrics & logs
│
├── 📝 README.md # This comprehensive guide
│
└── .gitignore
| Version | Strategy | CV Folds | Models | Seeds | Best AUC | Notes |
|---|---|---|---|---|---|---|
| V30 | F80_C20 Blend | - | Blend F,C | - | 0.95410 | ⭐ LATEST, F80+C20 |
| V29 | F90_C10 Blend | - | Blend F,C | - | 0.95410 | F-dominant micro-blend |
| V28 | F Micro-Perturbations | - | 7 variants | - | 0.95410 | F99_E01 best (main) |
| V27 | E+F Blending | - | Blend E,F | - | 0.95410 | Pure F submission |
| V26 | E+CD Variant | - | Blend E,CD | - | 0.95409 | E80_CD20 (stability test) |
| V25 | E+CD Blending | - | Blend E,CD | - | 0.95409 | E80_CD20 optimal |
| V24 | C + B Micro | - | C99_B01 blend | - | 0.95407 | 1% B adjustment |
| V23 | Raw C,D Blends | - | 6 raw variants | - | 0.95408 | No rank normalization |
| V22 | C vs D Variants | - | 4 blend ratios | - | 0.95408 | Only C/D + simple blends |
| V21 | Pure CD + CD95AB05 | - | 2 variants | - | 0.95408 | Rank normalized blending |
| V20 | 4-Submission Blend | - | Blend A,B,C,D | - | 0.95406 | 70% CD + 30% AB |
| V18 | 2-Submission Blend | - | Blend V16+V17 | - | 0.95360 | Testing blend ratios |
V20 (LATEST - Advanced 4-Submission Hierarchical Blend) 🏆
- Strategy: Hierarchical blending of 4 previous submissions (A, B, C, D)
- Process:
- Group 1: Average of A (V16) + B (V17) = AB
- Group 2: Average of C + D = CD
- Final: 70% CD + 30% AB (optimal ratio discovered)
- Methodology: Rank normalization applied before all blending operations
- Testing: 5 different blend ratios evaluated (90:10 to 50:50 CD:AB)
- Key insight: Hierarchical blending > simple 4-way averaging. Groups better than flat blends.
- Breakthrough: Added 0.00046 AUC improvement (0.95360 → 0.95406)
- Rank (V20): 101/3993 (2.5%) ⭐ TOP 2.5%!
- Updated (V30): 11/4180 (0.3%) 🏆 LEGENDARY TOP 0.3%!
- AUC: 0.95410 - BEST EVER!
V18 (2-Submission Blend - V16 + V17)
- Strategy: Simple blend of V16 (A.csv) + V17 (B.csv)
- Process: Rank normalized both submissions before averaging
- Testing: 4 blend ratios (0.3/0.7, 0.4/0.6, 0.5/0.5, 0.6/0.4)
- Main submission: 50/50 equal weight
- Purpose: Testing if V17's new 9 models improve on V16
- AUC: 0.95360
- Insight: Blending variants helped validate V17 effectiveness
V17 (9-Model Ensemble with 3 Seeds)
- Innovation: Added third seed [2024] (previously used 2 seeds)
- Configuration: 3 seeds × 3 algorithms = 9 models total
- LightGBM (42, 123, 2024)
- CatBoost (42, 123, 2024)
- XGBoost (42, 123, 2024)
- CV Strategy: 5-fold stratified with domain features (13 medical features)
- Target encoding: 7 categorical features with smoothing=30
- Best ensemble: Meta-model (Logistic Regression) with 0.955442 OOF AUC
- Blending: Tested 4 ratios with V16 (0.3, 0.4, 0.5, 0.6 weightings)
- Key insight: 3 seeds > 2 seeds for better ensemble diversity
- AUC: 0.95360
V16 (Balanced 6 Models with Maximum Algorithm Diversity)
- All 3 algorithms included: LightGBM + CatBoost + XGBoost
- 2 seeds each: [42, 123] for optimal diversity
- 5-fold CV with domain feature engineering (13 medical features)
- Target encoding on 7 categorical features (smoothing=30)
- Hyperparameters: n_estimators=10000, learning_rate=0.01, max_depth=4-5
- Key insight: Best of both worlds - all 3 algos + simpler engineering = maximum performance
- Best ensemble: Meta-model (Logistic Regression) selected
- Rank: 1038/3981 (26.1%)
- AUC: 0.95359 ⭐ Best yet!
V15 (Ultra-Simplified 4 Models with Domain Features)
- Only 2 algorithms: LightGBM + CatBoost (removed XGBoost overhead)
- 2 seeds each: [42, 123] for diversity without complexity
- 5-fold CV with enhanced feature engineering (13 medical features)
- Target encoding on 7 features with smoothing=30
- Hyperparameters: n_estimators=10000, learning_rate=0.01, max_depth=5
- Key insight: Simpler ensemble + better features > many complex models
- Best ensemble: Meta-model (Logistic Regression) selected
- Rank: 1129/3952 (28.6%)
- AUC: 0.95357
V13 (Target Encoding + Reduced Model Count)
- Just 6 models: LightGBM, CatBoost, XGBoost × 2 seeds (42, 123)
- 5-fold CV (reduced from 10 for less overfitting)
- Target encoding on 5 categorical features with smoothing=20
- Lower learning rates (0.003 vs 0.005) & higher regularization
- 10,000 estimators with 300 early stopping rounds
- Key insight: Fewer, simpler models with better features > many complex models
- Rank: 1381/3839 (36.0%)
- AUC: 0.95349 (rank average selected)
V12 (Two-Round Training)
- Initial round: Train on original 630K samples
- Pseudo-label 26,822 high-confidence test samples
- Round 2: Train on 656K samples (original + pseudo)
- Blend: 0.3×R1 + 0.7×R2
- Use case: Maximum accuracy with extended computation
V11 (RECOMMENDED - Clean & Efficient)
- Simple 5-fold CV (less overfitting than 10-fold)
- 9 models: XGB, LGB, CatBoost × seeds (42, 123, 2024)
- Rank averaging outperforms meta-model
- Use case: Best balance of speed and accuracy
V10 (External Data)
- Merges original Cleveland Heart Disease dataset (297 samples)
- Repeats original 50× to balance with synthetic data (644,850 rows total)
- 10-fold CV on synthetic data rows only
- Use case: Leveraging real-world data for better generalization
V9 (Foundation)
- 10-fold CV with seed diversity
- Tests impact of different random states
- 5 models with 2 different seeds each
- Use case: Understanding seed effects on ensemble
- Kaggle GPU: 10-15 minutes ⚡
- Local CPU: 30-60+ minutes ⏳
- Expected Memory: 2GB+ RAM
pandas>=1.3.0
numpy>=1.21.0
scikit-learn>=0.24.0
xgboost>=1.5.0
lightgbm>=3.3.0
catboost>=1.0.0
-
Consistent Ensemble Approach 🎪
- LightGBM (best performer across 5 comps)
- CatBoost (excellent for structured data)
- XGBoost (robust diversity)
- Random Forest (stability)
- AutoGluon (automated search)
-
Medical Domain Understanding 🏥
- Learned from Diabetes competition (S5E12)
- Age-related feature interactions
- Blood pressure/cholesterol relationships
- ECG pattern recognition
-
Advanced Feature Engineering 🔧
- Interaction features from Competitions 2-6
- Quadratic transformations for non-linearity
- Domain-specific derived features
- Statistical aggregations
-
Hyperparameter Optimization 🎯
- Optuna from Competition 2 onwards
- Grid/Random search combinations
- Cross-validation strategies
- Early stopping implementation
-
Stacking & Meta-Learning 📚
- 5-fold OOF stacking (Competition 3)
- Multi-level ensemble (Competition 4)
- AutoML dynamic stacking (Competition 3)
- Logistic Regression meta-models
-
Team Experience Compounding 👥
- 5 previous competitions = exponential learning
- Faster experimentation cycles
- Better intuition on what works
- 48% → 23% → 20.8% → EXPERT TIER
S5E9 Comp#1 48.3%
↓ (Learning Phase)
S5E10 Comp#2 23.5% ⬆️ +24.8pp (Breakthrough!)
↓ (Optimization Now Standard)
S5E11 Comp#3 33.7% ⬇️ -10.2pp (Plateau reach)
↓ (Specialization)
S5E12 Comp#4 20.8% ⬆️ +12.9pp (BEST YET!)
↓ (Consistency Focus)
S6E1 Comp#5 22.4% ⬇️ -1.6pp (Maintained)
↓ (Full Arsenal Ready)
S6E2 Comp#6 2.5% ⬆️ +19.9pp (LEGENDARY!) 🏆 AUC: 0.95406
- ✅ Competition #1 (S5E9): Top 48.3% - BPM Prediction - Foundation Laid
- ✅ Competition #2 (S5E10): Top 23.5% - Road Accident Risk - Skills Sharpened
- ✅ Competition #3 (S5E11): Top 33.7% - Loan Payback - AutoML Mastery Unlocked
- ✅ Competition #4 (S5E12): Top 20.8% - Diabetes Prediction - EXPERT TIER ACHIEVED!
- ✅ Competition #5 (S6E1): Top 22.4% - Student Test Scores - Consistency Proven
- 🔥 Competition #6 (S6E2): Top 11.1% - Heart Disease - ULTIMATE LEGENDARY FINISH! (V30: AUC: 0.95410, Rank: 485/4370)
Base Models:
- ✅ XGBoost - Fast, optimized gradient boosting
- ✅ LightGBM - Efficient tree-based learning
- ✅ CatBoost - Categorical feature specialist
Ensemble Methods Tested:
- Simple Average: Fast, baseline
- Rank Average: Robust to outliers ← Often wins
- Meta-Model: Logistic Regression on OOF predictions
Data Pipeline:
1. Load train (630K) & test (270K)
2. Fill missing with median
3. No explicit feature engineering (inherent in tree models)
4. Stratified K-Fold split
5. Train models with early stopping
6. Generate OOF + test predictions
7. Blend via rank average or meta-model
8. Clip to [0, 1] → submit# Round 1: Original data
Models trained on 630K samples
Predictions on test set
# Pseudo-labeling (High Confidence Only)
threshold = 0.05 & 0.95
Label 26,822 test samples with highest confidence
Distribution: 13,388 Class 0 | 13,434 Class 1
# Round 2: Enriched data
Re-train all 9 models on 656,822 samples
(original 630K + pseudo-labeled 26K)
# Blending Strategy
final = 0.3 * R1_predictions + 0.7 * R2_predictions
(R2 had slightly better OOF, so 70% weight)Why it works: Semi-supervised learning gains from unlabeled test data, bootstrapping confidence
# 5-Fold Cross-Validation (vs 10-fold in V9/V10)
Advantages:
- Faster (5 splits vs 10)
- Less CV overfitting
- Comparable or better AUC
# 9 Models (3 Seeds × 3 Algorithms)
Seeds: [42, 123, 2024]
- Seed 42: Default, reproducible
- Seed 123: Alternative randomization
- Seed 2024: Current year reference
# Hyperparameters (Optimized after V9/V10)
- n_estimators: 5000 (up from 2000 in V9)
- learning_rate: 0.005 (down from 0.01)
- early_stopping_rounds: 200 (up from 100)
- max_depth: 5 (balanced)
# Ensemble Comparison
Simple Avg: 0.955349 AUC
Rank Avg: 0.955350 AUC ← Selected (marginal improvement)
Meta-model: 0.955349 AUC# Cleveland Heart Disease Dataset (UCI ML Repository)
Original: 297 samples, 13 features
Class distribution: similar to synthetic data
# Merging Strategy
X_orig_repeated = repeat(X_orig, 50) # 50x replication
Rationale: Balance 297 samples with 630K synthetic samples
# Combined Dataset
Original (50 copies): 14,850 samples
Synthetic: 630,000 samples
Total: 644,850 samples
# CV Strategy
- 10-fold split on synthetic data only
- Train on full combined dataset (original + fold)
- Evaluate on synthetic fold (fair comparison)Results: Marginal improvement (0.95303 vs 0.95333)
- Real data provides distribution reference
- Helps models learn authentic patterns
- More relevant for transfer to real patients
# Original Ensemble Approach
10-fold CV (highest variance reduction)
5 Models with strategic seeds:
- 2 XGBoost variants (seeds 42, 123)
- 2 LightGBM variants (seeds 42, 123)
- 1 CatBoost (seed 42)
# Seed Impact Analysis
Different random states → different train/val splits
Better ensemble diversity from uncorrelated errors
# Hyperparameters (Initial Setting)
n_estimators: 2000
learning_rate: 0.01 (higher than V11)
early_stopping_rounds: 100 (lower than V11)| Metric | Value |
|---|---|
| 🏅 Current Status | ✅ V30 Complete (F80_C20 Final Blend) 🏆 |
| 🎯 Best Score (V30) | 0.95410 ROC-AUC ⭐ BREAKTHROUGH! |
| 📊 Latest Strategy | F-dominant micro-blending: 80% F + 20% C |
| 🚀 Pipeline Evolution | V20→V30: 10 advanced blend iterations |
| 🔬 Blending Strategy | Multi-submission ensemble optimization |
| 👥 Team Members | 4 Elite Data Warriors (Dual Expert Status!) |
| ⏱️ Competition Status | 18 hours remaining - FINAL PUSH! |
| 🔄 Versions Deployed | 22 complete notebook versions (V9-V30) |
| 🎖️ Expert Achievements | Notebooks (1635/59495, Highest: 1634) + Datasets (339/8458) 📈 |
| 📈 Leaderboard Position | Top 485/4370 (11.1%) 🏆🏆🏆 |
Event: Kaggle Playground Series - Season 6, Episode 2
Challenge: Predicting Heart Disease
Start: February 1, 2026
End: February 28, 2026
Evaluation Metric: ROC-AUC (Area Under ROC Curve)
Prize: Kaggle Merchandise (Top 3)
Dataset License: CC BY 4.0
- Kaggle for the incredible Playground Series platform
- Walter Reade & Elizabeth Park for organizing this challenge
- Healthcare Community for the domain knowledge and inspiration
- Our Team for unwavering dedication and collaboration
- Coffee for keeping us awake at 3 AM ☕
Learning → Optimizing → Mastering → DOMINATING
Competition Link | February 2026 | #TeamPhoenixAlgorithms
Repository: Predicting-Heart-Disease
Team: Team Phoenix Algorithms (Mohan, Rakesh, Ranjith, Uday Kiran)
Status: ✅ SUBMISSION COMPLETE (V30 Score: 0.95410 - Rank 485/4370, Top 11.1% 🏆)
Mission: Top 15% ✅ | Expert Tier Validation ✅ | Master Preparation 🚀



