Change default parameters for feature selectors - #3110
Conversation
| hyperparameter_ranges = { | ||
| "percent_features": Real(0.01, 1), | ||
| "threshold": ["mean", -np.inf], | ||
| "threshold": ["mean"], |
There was a problem hiding this comment.
This PR proposes using the mean as the threshold but experiments show that median performs similarly as well. median will select exactly half of the features and mean depending on the distribution. Happy to discuss which one to choose but I chose mean in the... meantime..
There was a problem hiding this comment.
On second thought: using median might be the move as the mean of feature importances will bound to be dragged down by low signal features (which are inevitable). However, performance results show similar model quality between median and mean but median having slightly longer fit times due to median selecting more features.
There was a problem hiding this comment.
Why not allow both 'mean' and 'median' to be in the hyperparameter ranges? We can default to one, but allowing both in the ranges seems to be ideal for our automlsearch algo
There was a problem hiding this comment.
I like that suggestion. Right now the DefaultAlgorithm only runs one feature selector, so I'm down to set median as the default. Yesterday we discussed letting the algorithm tune the parameters of the selector in which case broadening the threshold search space (like parametrizing it as a quantile of the observed feature importance distribution) will be in play.
There was a problem hiding this comment.
sounds good, I'll add them both as hyperparameter ranges! My main concern is what Freddy brought up about the default parameter and having only 1 FS batch but I'm still on the fence on chosing mean or median. Either way it'll be a quick fix so I'm not too worried!
Codecov Report
@@ Coverage Diff @@
## main #3110 +/- ##
=======================================
+ Coverage 99.8% 99.8% +0.1%
=======================================
Files 313 313
Lines 30579 30585 +6
=======================================
+ Hits 30489 30495 +6
Misses 90 90
Continue to review full report at Codecov.
|
bchen1116
left a comment
bchen1116
left a comment
There was a problem hiding this comment.
Looks great! So this change will then allow the RF...FeatureSelector to now choose the top features that fall above the mean importance weight?
Looking at the perf tests, the fit times and performance changes seem reasonable. In all three tests, though, LightGBM seems to drop very significantly in performance:
Do you know why this is? This seems to be something that we should figure out and resolve before moving ahead with the change, especially if it's a potential bug somewhere in the code.
| hyperparameter_ranges = { | ||
| "percent_features": Real(0.01, 1), | ||
| "threshold": ["mean", -np.inf], | ||
| "threshold": ["mean"], |
There was a problem hiding this comment.
Why not allow both 'mean' and 'median' to be in the hyperparameter ranges? We can default to one, but allowing both in the ranges seems to be ideal for our automlsearch algo
| hyperparameter_ranges = { | ||
| "percent_features": Real(0.01, 1), | ||
| "threshold": ["mean", -np.inf], | ||
| "threshold": ["mean"], |
There was a problem hiding this comment.
Also here, seems to be ideal to allow median as a possible value in the hyperparams.
| elif isinstance(component, RFRegressorSelectFromModel): | ||
| assert transform_output.shape == (X.shape[0], 2) | ||
| elif isinstance(component, RFClassifierSelectFromModel): | ||
| assert transform_output.shape == (X.shape[0], 5) |
There was a problem hiding this comment.
What's the 2 or 5 values coming from here?
There was a problem hiding this comment.
it's the number of columns selected by the FS component using the default parameters on X_y_binary and X_y_regression.
freddyaboulton
left a comment
freddyaboulton
left a comment
There was a problem hiding this comment.
Looks good to me @jeremyliweishih ! Agree with @bchen1116 that setting median and mean as hyperparameter values makes sense.
| hyperparameter_ranges = { | ||
| "percent_features": Real(0.01, 1), | ||
| "threshold": ["mean", -np.inf], | ||
| "threshold": ["mean"], |
There was a problem hiding this comment.
I like that suggestion. Right now the DefaultAlgorithm only runs one feature selector, so I'm down to set median as the default. Yesterday we discussed letting the algorithm tune the parameters of the selector in which case broadening the threshold search space (like parametrizing it as a quantile of the observed feature importance distribution) will be in play.
|
LightGBM drops significantly in terms of percentage change (since the log loss is < 0.1) but it's not that big in absolute terms and likewise with the best pipeline validation score for that dataset so I'm not too worried about it. I guess the columns selected by the FS doesn't play nicely with LightGBM but I don't have enough knowledge about how LightGBM works to make any concrete statements about the change. |
bchen1116
left a comment
bchen1116
left a comment
There was a problem hiding this comment.
Changes look good to me!
3 participants
This PR proposes changing the default parameters for
RFRegressorSelectFromModelandRFClassifierSelectFromModel. The current, incorrect, behavior of these components is as follows:number_features=None,percent_features=0.5, andthreshold=-np.infmax_features = Nonemax_features == None andthreshold=-np.inf, the component will select every feature with importance above-np.inf` which is every feature available.Performance tests using default algorithm:
fs_parameters_tests.zip