Difficulty estimator with categorical dtype support

[FlyingCurryMonster]

Hi, I'm learning about CPD and wanted to try it for a xgboost model on a kaggle dataset. My current model uses categorical features without one hot encoding. I'm wondering if anyone has a solution on applying the DifficultyEstimator when the training data has categorical features, so that it can be used as sigma for calibrating a WrapRegressor. Should I just only include numeric features when fitting the difficulty estimator?

Thanks!

[henrikbostrom]

Hi,

This is a good question, as the construction of good difficulty estimators is a central challenge. In addition to what you propose, one possible (but not necessarily good) option is to employ one-hot encoding, which would allow you to use the k-NN approach for the difficulty estimator, which relies on the Euclidean distance. This may however not scale very well with the number of unique categorical values. Another option would be to compute the difficulty outside the DifficultyEstimator and implement a tailored distance function, e.g., which allows for combining categorical and numerical features, such as Gower's distance.

Best regards,
Henrik

[FlyingCurryMonster]

Hi Henrik,

I think I found a quick and dirty fix, that I'm not sure is appropriate. Following guidance from the crepes extra modules:
documentation https://crepes.readthedocs.io/en/latest/crepes.extras.html :

It ocurred the me that the xgb booster is an ensemble of weak trees like in RF. I created a new class variable estimators_ and passed that in as a learner argument for fitting the difficulty estimator. Some example code looks like this:

booster = xgb_model.learner.get_booster()

xgb_model.learner.estimators_ = xgb_model.learner.get_booster()
xgb_model.learner.estimators_
de_var = DifficultyEstimator()
de_var.fit(learner=xgb_model.learner)

I'm not exactly sure how XGBoost and other boosted tree libraries support categorical data types, hence my suspicion what I'm doing is not exactly kosher.