New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GreedyRuleListClassifier has wildly varying performance and sometimes crashes #145
Comments
Thanks for raising this issue! Will look into it shortly... |
Hi @davidefiocco, just looked into it. I fixed the KeyError issue and just pushed/bumped the imodels version, so if you upgrade with The accuracy does indeed fluctuate quite a lot for this dataset....GRL is a good algorithm when you are trying to identify a clear subgroup that has high probability of being in a single class, but does poorly with finding interactions since it only ever identifies samples from class 1 and the remaining samples after all rules are predicted as class 0. If you want to look into it farther, you can visualize some of the models and see how they are overfitting (just need to add the line |
Thanks so much @csinva and of course absolutely no worries and kudos for your great work on |
The performance of the model is not "wildly varying" anymore after @mcschmitz fix of the behavior in #167, released with
|
When running a certain number of experiments with different splits of a given dataset, I see that
GreedyRuleListClassifier
's accuracy wildly varies, and sometimes the code (see for loop below) crashes.So, for example running 10 experiments like this, with different random splits of the same set:
Will give as output something along the lines of
Is this intended behavior? While my test dataset is smallish, the variation in accuracy is still surprising for me and so is the throwing of a
KeyError
. I'm usingscikit-learn==1.0.2
andimodels=1.3.6
and can edit the issue here to add more details.Incidentally, the same behaviour was observed in https://datascience.stackexchange.com/a/116283/50519, noticed by @jonnor.
Thanks!
The text was updated successfully, but these errors were encountered: