EHN: implementation of SMOTE-NC for continuous and categorical mixed types #412
Conversation
|
Hello @ddudnik! Thanks for updating the PR.
Comment last updated on October 12, 2018 at 10:07 Hours UTC |
Codecov Report
@@ Coverage Diff @@
## master #412 +/- ##
==========================================
+ Coverage 98.66% 98.69% +0.03%
==========================================
Files 79 80 +1
Lines 4803 4999 +196
==========================================
+ Hits 4739 4934 +195
- Misses 64 65 +1
Continue to review full report at Codecov.
|
|
I'll check it when I get some time. Thanks for the contribution. |
|
Hi |
|
Not yet. We need to make the implementation to accept both continuous and categorical. |
|
Hi |
|
@jijo7 Feel free to take over the PR and finish the implementation. |
imblearn/base.py
Outdated
| @@ -63,7 +63,7 @@ def sample(self, X, y): | |||
|
|
|||
| return self._sample(X, y) | |||
|
|
|||
| def fit_sample(self, X, y): | |||
| def fit_sample(self, X, y, **fit_params): | |||
There was a problem hiding this comment.
I think that I would avoid to pass a keyword here.
We need to pass the keyword when creating the instance.
In addition, I would propose to have an argument categorical_features='auto'orcategorical_features=[1, ...]to inform for the name of the index. With'auto', think that we can check the column which are binary {0, 1} previously encoded by a OneHotEncoder`.
It will be slower and we can directly pass the index of the column then (or even the column name if we want to support pandas dataframe).
There was a problem hiding this comment.
Hi there, thanks for looking into this PR!
-
fit_params keyword (or, more specifically, feature_indices keyword parameter in SMOTENC class): I also considered the option of making it a init parameter to be set when SMOTENC instance is being created. However, in the end I opted for a keyword argument. Here's my motivation: feature_indices indicates which columns of input represent the nominal features after they were one-hot-encoded. Thus, it is directly connected to the provided input X. So if you want to call fit method on the different X, you might have to provide different feature_indices. Speaking more generally, I assumed that init should accept parameters which are not input-dependent and fit can take parameters which are dependent on given input X.
-
categorical_features=auto: I like this idea! (I guess you mean aforementioned feature_indices parameter?). Just one question: would it maybe make sense to implement it as an improvement/follow-up ticket after this PR is merged? Asking since I'm not sure if I will have enough time in near future to work on it.
As for you question regarding OneHotEncoding you're right, it is not mentioned in the paper. Also, strictly speaking, it is not part of SMOTENC implementation either. It's just expects the data to be one-hot encoded because: a) SMOTENC re-uses a lot of code from SMOTE (nearest-neighbors, initial sample generation etc.) which will not work if input contains non-numerical data ; b) OneHotEncoding is a go-to tool when dealing with nominal features in sklearn; c) when applied, it conveniently returns feature_indices array to be used by SMOTENC
Regards,
Denis
There was a problem hiding this comment.
-
My concern is that we don't follow the current scikit-learn API. The
ColumnTransformerwill for instance get the same kind of parameters at initialization. Even if I agree that it would make sense to have it at fit time, I am not aware of mechanism to be able to forward the information of theOneHotEncodertoSMOTEwithin aPipeline. It is something that at the long term should be done withsample_propswhich is actually something pass infitsimilarly to what you are proposing. However, in the meantime I would be more conservative. -
I prefer the term
categorical_featuresinstead offeatures_indicessince it is explicit what are we targeting. My proposal regarding the'auto'is actually to partially solve the problem spotted in 1. Regarding the extra work, I could implement it and push it in your PR, if you are OK with it. -
I agree with your last point. However, since that we are actually not implemented the original
SMOTE-NC, I would propose to rename the class to something different to not have any issue with other implementation which might differ. I don't have a good naming yetCatSMOTEmaybe. Any proposal would be nice.
There was a problem hiding this comment.
- Agreed (I'll address it during re-base)
- Agreed
- I still think that we could use SMOTE-NC as a name. Even though OneHotEncoding is not mentioned in the paper, I think using it does not contradict the paper. All the steps 1-3 (median computation, nearest neighbors, sample generation) behave as described in the paper. I would call the usage of OneHotEncoder just an "implementation detail". But maybe I'm missing something so please let me know if you have a different idea.
(for reference https://arxiv.org/pdf/1106.1813.pdf, section 6.1)
There was a problem hiding this comment.
@here I just wanted to ask if a method has been implemented to inverse_transform the OneHotEncoded values after applying the SMOTE-NC.
Also, I did test it with some sample datasets having categorical values and nominal, but it results in a way that the one-hot-encoded values (binary) are also getting incremented with the 'step' parameter provided.It results in the values such as (0.1,0.12,0.6..etc).
How to convert them back to the original categorical dataset with these type of values?
There was a problem hiding this comment.
I just wanted to ask if a method has been implemented to inverse_transform the OneHotEncoded values after applying the SMOTE-NC.
Since I integrated the OneHotEncode within SMOTENC, the feature will be reverted.
How to convert them back to the original categorical dataset with these type of values?
Now you will get the original categories so it should not be an issue. I still have to implement more tests. to be sure that it behave as exptected in all situation.
There was a problem hiding this comment.
@glemaitre There is some issue with OHE (when using normally).
Recent version provides functionality of converting it without LabelEncoding it. I tried the process with a dataset of shape ((284807, 30)) and it was giving me memory error.
I hope it might be of some use.
There was a problem hiding this comment.
Recent version provides functionality of converting it without LabelEncoding it.
Yes you should not use the LabelEncoding for such usage. OneHotEncoding has been fixed in 0.20
I tried the process with a dataset of shape ((284807, 30)) and it was giving me memory error.
It depends how many categories you have by column. You can have a really huge matrix and depending of your RAM, you will get this error. One possibility is to convert the array to a sparse matrix.
We could think to always use the OHE internally with sparse matrix to avoid as much as possible such issue.
|
We recently refactored the SMOTE class which will require also a rebase. WDYT? |
|
@jnothman similarly to Considering the following case: a ColumnTransformer with a OneHotEncoder for categorical data and a StandardScaler for the numerical, SMOTE-NC would benefit if the OneHotEncoder is outputting an array with the column names which could be passed at Do we have something in mind regarding this case and is it a "generic" enough case such that other estimators in scikit-learn can benefit from it? |
|
That conversation is not happening in any straightforward way, though I have noted that they are sides of the same coin in the Draft Roadmap (see "Passing around information that is not (X, y)"). I do think we have substantial related problems in the limitations of We have not yet considered combined/holistic solutions for sample, feature and target properties, but perhaps we should... |
|
PR is rebased |
imblearn/over_sampling/_smote_nc.py
Outdated
| """ | ||
| sample = super(SMOTENC, self)._generate_sample(X, nn_data, nn_num, | ||
| row, col, step) | ||
| if not hasattr(self, "categorical_feature_indices_"): |
There was a problem hiding this comment.
I would remove this part. We will have raised an error much earlier.
|
Regarding I gave it another thought and looked through the code one more time (it was a long time since I initially implemented it) and it seems that 'auto' mode is not possible here. Thing is that we need to distinguish between two different categorical features from the original dataset. I'll try to explain via an example. Let's say, we have a dataset with 4 columns with features A, B, C and D. Features C and D are two separate categorical ones. Let's say, feature C has 2 possible values and feature D has 3 possible values. After one-hot encoding we'll end up with a new dataset with 7 columns: features A and B are left intact, but instead of feature C we have two columns (C1, C2) and instead of feature D we have 3 columns (D1, D2, D3). These new columns are of course binary. For this code to work properly, it has to know where feature C columns end and where D start. I.e. we need to know that columns 2-3 represent feature C and columns 4-6 represent feature D. This is exactly what we say with I hope this explains why 'auto' is not really an option here. Unless I miss something. |
imblearn/over_sampling/_smote_nc.py
Outdated
| See also | ||
| -------- | ||
| SMOTE : Over-sample using SMOTE. | ||
|
|
There was a problem hiding this comment.
We should add the BorderlineSMOTE
imblearn/over_sampling/_smote_nc.py
Outdated
| from . import SMOTE | ||
|
|
||
|
|
||
| class SMOTENC(SMOTE): |
There was a problem hiding this comment.
We still need to move this class in smote.py
imblearn/over_sampling/_smote_nc.py
Outdated
|
|
||
| {random_state} | ||
|
|
||
| {k_neighbors} |
There was a problem hiding this comment.
We should keep the full documentation in the docstring for those parameters.
We use it for the sampling_strategy and random_state only.
The other parameters can ease development if they are written in the documentation.
Long story short, people try to do that in scikit-learn and it failed, so we might want to avoid this :)
imblearn/over_sampling/_smote_nc.py
Outdated
| sampling_strategy='auto', | ||
| random_state=None, | ||
| k_neighbors=5, | ||
| m_neighbors='deprecated', |
There was a problem hiding this comment.
Let's remove all parameters which are deprecated. We don't need them here.
We will not support the old style version using kind.
imblearn/over_sampling/_smote_nc.py
Outdated
| self.categorical_feature_indices = categorical_feature_indices | ||
|
|
||
| def _fit_resample(self, X, y): | ||
| if self.categorical_feature_indices is None: |
There was a problem hiding this comment.
I would prefer:
if self.categorical_feature_indices == 'auto':
self.categorical_feature_indices_ = [i for i in range(n_features)
if len(_encode(X[:, i]) < 2]
else:
categorical_feature_indices = check_array(...)
if categorical_feature_indices not in np.arange(n_features):
raise ValueError(...)
self.categorical_feature_indices_ = categorical_feature_indices
imblearn/over_sampling/_smote_nc.py
Outdated
| else: | ||
| sample[start:end] = new_nominal_value | ||
|
|
||
| return sample.tocsr() if is_sparse else sample |
There was a problem hiding this comment.
only return dense. The magic should occur in fit_resample in the sparse branch which should concatenate matrices and eliminate zeros
imblearn/over_sampling/_smote_nc.py
Outdated
| between two feature vectors, the difference of two different categorical | ||
| features will always equal to median standard deviation. | ||
| """ | ||
| if not hasattr(self, "categorical_feature_indices_"): |
There was a problem hiding this comment.
Remove this part. We will not accept it for the moment.
imblearn/over_sampling/_smote_nc.py
Outdated
| 'to normal SMOTE', RuntimeWarning) | ||
| return X | ||
|
|
||
| X_copy = X.copy().tolil() if sparse.issparse(X) else X.copy() |
There was a problem hiding this comment.
Actually it should be better to create a temporary CSC matrix.
Go column of the feature and change the value inplace and return X.tocsr()
There was a problem hiding this comment.
X = X.tocsc()
for feature_idx in self.categorical_feature_indices:
nnz_idx = X[:, feature_idx].nonzero()[0]
X[nnz_idx, feature_idx] = self.std_median_ / 2
return X.to_csr()| X_indices = [2, 5, 7] | ||
|
|
||
|
|
||
| def test_sample_regular(): |
There was a problem hiding this comment.
We will remove those steps with numbers. There is a common test that we will try the over-sampling. We will try to test the corner case and thing which we can properly control.
| smote.fit_resample(X, Y) | ||
|
|
||
|
|
||
| def test_fit_feature_indices_out_of_bounds(): |
There was a problem hiding this comment.
rename feature_indices to categorical_...
There was a problem hiding this comment.
I would concatenate all the error in a single test using pytest parametrize
@pytest.mark.parametrize(
"categorical_feature_indices, msg_error",
[([1], 'minimum of 2 is required'),
([3, 5, 8], 'Indices ...'),
(...)]
)
def test_smote_nc_errors(categorical_feature_indices, msg_err):
smote = SMOTENC(random_state=RND_SEED,
categorical_feature_indices=categorical_feature_indices)
with pytest.raises(ValueError, match=msg_err):
smote.fit_resample(X, y)
|
For the common test you need to add SMOTENC to the |
|
Thinking of which that we need to provide |
So categorical_feature_indices is actually the starting index of a given feature. I would prefer then to include the OneHotEncoder in SMOTE-NC. We also ensure that the columns are encoded for sure (a user could forget the OneHotEncoder) if they use it without care. |
| 'For this method to work X should have ' | ||
| 'at least 1 continuous feature.') | ||
|
|
||
| if sparse.issparse(X): |
There was a problem hiding this comment.
Actually used the following function mean_variance_axis to compute the std on the sparse matrix.
https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/sparsefuncs.py#L64
There was a problem hiding this comment.
Basically you can factorize:
X_continous = X[:, self.continuous_feature_indices_]
stds = (np.sqrt(mean_variance_axis(X_continous, axis=0)[1])
if sparse.issparse(X) else np.std(X_continous, axis=0))
self.std_median_ = np.median(stds)
glemaitre
force-pushed
the
master
branch
2 times, most recently
from
bbf2b12 to
513203c
Compare
|
I will try to finish up this PR. To simplify the implementation without changing as much code:
Perfectly, we could have use the ColumnTransformer but it does not have the |
glemaitre
changed the title
|
Merging since it seems ready. @ddudnik Thanks for the implementation. |
|
Glad to see it merged! Thank you for taking over and finishing this thing. I wish I would have had more time in the past month to be of more help. |
|
@glemaitre I've looked through the code briefly and I have one remark regarding replacing the '1' entries of the categorical features with the median of the standard deviation. Shouldn't we replace '1' entries with median of standard deviation divided by two? Otherwise the distance between two different categorical values is |
|
Did I forgot to divide it? Ups ... It seems that we will need a bug fix.
Thanks to noticing it
…On Fri, 19 Oct 2018 at 20:51, Denis Dudnik ***@***.***> wrote:
@glemaitre <https://github.com/glemaitre> I've looked through the code
briefly and I have one remark regarding replacing the '1' entries of the
categorical features with the median of the standard deviation. Shouldn't
we replace '1' entries with median of standard deviation divided by two?
Otherwise the distance between two different categorical values is 2 *
med_std.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#412 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AHG9P2nQSDN2RjpnpQHl5-vgGCtyv2kDks5umh6ogaJpZM4SchbN>
.
--
Guillaume Lemaitre
INRIA Saclay - Parietal team
Center for Data Science Paris-Saclay
https://glemaitre.github.io/
|
Reference Issue
#401
What does this implement/fix? Explain your changes.
Implements SMOTE-NC as per paragraph 6.1 from original SMOTE paper by Chawla, K. W. Bowyer, L. O.Hall, W. P. Kegelmeyer
Any other comments?
Some parts are missing to make it ready to merge, but I would like to get an opinion on implementation first, especially on the part which deals with sparse matrices as I do not have much experience with them.
Points to pay attention to: