feature importance [enhancement]

[mglowacki100]

It'd be nice to have 'feature importance' exposed it the same way as in sklearn.

[mglowacki100]

It can be done with rfpimp library i monkey-patching AutoML with scikit (quick temp fix)
Here is sample code:

#https://github.com/parrt/random-forest-importances
#!pip install rfpimp 
import rfpimp

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from supervised.automl import AutoML
from sklearn.metrics import accuracy_score
        

df = pd.read_csv(...)

df_train, df_test = train_test_split(df, test_size=0.20)

X_train, y_train = df_train.drop('Target',axis=1), df_train['Target']
X_test, y_test = df_test.drop('Target',axis=1), df_test['Target']
X_train['random'] = np.random.random(size=len(X_train))
X_test['random'] = np.random.random(size=len(X_test))


print('training')
automl = AutoML(...
)

automl.fit(X_train, y_train)

#monkey-patching of AutoML
def score(self, X, y, sample_weight=None):
    return accuracy_score(y, self.predict(X)['label'], sample_weight=sample_weight)
setattr(AutoML, 'score', score)


print('feature importance')
imp = rfpimp.importances(automl, X_test, y_test) # permutation
viz = rfpimp.plot_importances(imp)
viz.view()

[pplonski]

sklearn supports feature importance: https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html

I will use their implementation. (I need to add predict_proba in algorithms interface just to be compatible with sklearn interface)

[pplonski]

For each fold, the feature importance is computed based on permutation. In the plot, there are displayed top-25 best features. All importance values are saved to the CSV file.

The example of the plot:

[Tonywhitemin]

Hi pplonski,
Sorry to keep bothering you...
I tried to understand the "features_scores_threshold_2.5.csv" file as following table.
And some questions list below, could you help?

1. Do the numbers under the learnerX indicate the importance of the feature? How are they calculated?
2. How to compare which feature corresponds to each column?
3. How to calculate the "counter" value?
4. It seems that the number of features in this csv file may contain the number of golden features, right?

[pplonski]

@Tonywhitemin the feature importance is computed for each learner. The importance is computed with a permutation method. Each learner has a vector with importance for each feature. The columns in CSV are joined based on features. The first row is the first feature from the dataset.

When computing importance there is injected random feature to the dataset. The counter keeps information of how many times a feature has lower importance than a random feature.

[Tonywhitemin]

@pplonski Thanks for your reply!
As you said, "The counter keeps information of how many times a feature has lower importance than a random feature."
But can we know the information about the random feature's value in this csv file? (or which row should be the random feature's value?)

By the way, it seems that the number of features in this csv file may contain some of golden features, is that correct?
Thank you!

[pplonski]

@Tonywhitemin you will need to check that in code ... I dont remember all details, sorry!

[Tonywhitemin]

Hi @pplonski,
Please don't say sorry, you've helped a lot, I really appreciate your help!
Have a nice day!