Stacking by feature diversity and model diversity

Four leaf clover 🍀

🍀 code

We added model and feature diversity and used stacking to combine the results.

Feature Extraction

We refactored the feature engineering so that it extracts all the features from train/valid/test in one go and later the features are divided by idx.

Application data -> eda-application.ipynb 📝

Installment Payments data -> eda-installments.ipynb 📝

POS Cash Balance application data -> eda-pos_cash_balance.ipynb 📝

Model

• Logistic Regression CV 0.749 neptune experiment
• Neural Network CV 0.762 neptune experiment
• LightGBM on various feature subsets

Then we used stacking on all the out of fold predictions we had:

• out of fold predictions stacking with LightGBM got CV 0.7972 neptune experiment
• out of fold predictions + features (specified in the neptune_stacking.yaml CV 0.7975 neptune experiment

Pipeline diagram

Since the diagram below is quite wide (it uses multiple input files), here is a link to the larger version.