CategoricalImputer most_frequent strategy

[omar-araboghli]

The CategoricalImputer replaces all missing values with the most frequent value of the whole passed data frame.

The desired behaviour is to replace each missing value with the most frequent value of the missing value's column.

[impiyush]

Interesting! I always thought that's what it does. @omar-arab-oghli Do you happen to have an example of what you're experiencing? I can take a look at this.

[omar-araboghli]

I can't reproduce the issue right now since i am not working on the project anymore. But, if you take a look on the following code snippet you can see that the mode() function returns the most frequent value of the whole data frame instead of doing this column-wise

# the fit function
mask = _get_mask(X, self.missing_values)
X = X[~mask]
if self.strategy == 'most_frequent':
     modes = pd.Series(X).mode()
self.fill_ = modes[0]

# the transform function
mask = _get_mask(X, self.missing_values)
X[mask] = self.fill_

[impiyush]

First I believed it was the intention to use a pandas series or np.ndarray as input as the imputer is meant to be used along with a DataFrameMapper, so I gave it a quick shot and was surprised -

Something definitely does not looks right here and maybe as well in the base classes for the Transformers. We'll probably need help from the core developers to have a better idea first and then I can create a pull request to fix.

@dukebody @paulgb what's your thoughts here?