Feature selection based on permutation importance

[qinhanmin2014]

We now support permutation importance, maybe it's reasonable to support feature selection method based on it (e.g., remove features whose premutation importance < 0).

[amueller]

Yes, I think so. There is discussion about this in #9606.
I'm not aware of any literature on this, though?

[qinhanmin2014]

Ahh, I saw the comment from @thomasjpfan #9606 (comment)

I'm not aware of any literature on this, though?

No, but
(1) It's straightforward and reasonable IMO
(2) eli5 talks about it in their doc https://eli5.readthedocs.io/en/latest/blackbox/permutation_importance.html#feature-selection
They make use of SelectFromModel by introducing a class PermutationImportance. Permutation importance is stored as feature_importances_ in that class. I think this implementation is awkward and we can do better.
(3) I don't think it's difficult to find some related posts from Kaggle, e.g., https://www.kaggle.com/c/ieee-fraud-detection/discussion/108575#latest-636647

[cailurus]

Feature selection by permutation is common in Kaggle and other tabular data competition. This
method has been introduced on 《ESL 2nd》page 593. https://web.stanford.edu/~hastie/Papers/ESLII.pdf

[qinhanmin2014]

This method has been introduced on 《ESL 2nd》page 593.

I don't think ESL mention feature selection based on permutation importance? It only mentions permutation importance based on RF (and that'a actually different from the permutation importance we've implemented.)

[qinhanmin2014]

Maybe we can go ahead to discuss how to implement it.
Thomas prefer to integrate it into SelectFromModel, but I think that will be difficult, because sometimes we fit the model using the traning set, and calculate the permutation importance using the validation set.
Perhaps we can introduce a new class (e.g., SelectFromPermutationImportance, or a shorter name).

[ysunmi0427]

How's it going? If no one is there, I would like to start working on this issue.

I agree with @qinhanmin2014 that we need to create new class. In my opinion, the new class will derive SelectFromModel to reuse other parts of original class except calculating importances.

[jnothman]

Is that to handle a validation set? What API are you proposing for that?

…

On Mon., 27 Jan. 2020, 1:06 am Sunmi Yoon, ***@***.***> wrote: How's it going? If no one is there, I would like to start working on this issue. I agree with @qinhanmin2014 <https://github.com/qinhanmin2014> that we need to create new class. In my opinion, the new class will derive SelectFromModel to reuse other parts of original class except calculating importances. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#15075?email_source=notifications&email_token=AAATH22VQ5TABJ4Z4PNT4X3Q7WKHNA5CNFSM4IZ5RWKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ5US7Y#issuecomment-578505087>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAATH275K6XJ4BOSBE2AM5LQ7WKHNANCNFSM4IZ5RWKA> .

[MaxwellLZH]

Hi, I've opened a PR which adds another parameter importance_type to feature_selection.SelectFromModel, which could be set to "permutation" to use permutation importance for feature selection. Any advice or feedback would be really appreciated, thank you !