How to deal with OneHotEncoder() in pipeline?

[DataTerminatorX]

I find a bug when I use sklearn_pandas.DataFrameMapper. I've already read the source code of DataFrameMapper while fail to fix it. Could someone help with it?

Firstly, I get a categorical variable as below

from sklearn.preprocessing import LabelBinarizer
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import OneHotEncoder
from sklearn_pandas import DataFrameMapper
import pandas as pd

testdata = pd.DataFrame({'pet':    ['cat', 'dog', 'dog', 'fish', 'cat', 'dog', 'cat', 'fish']})

When I implement transformer LabelBinarizer() through DataFrameMapper, I got what I expect

In[11]
mapper1 = DataFrameMapper([('pet',[LabelBinarizer()])],sparse=False)
mapper1.fit_transform(testdata)
Out[11]:
array([[1, 0, 0],
       [0, 1, 0],
       [0, 1, 0],
       [0, 0, 1],
       [1, 0, 0],
       [0, 1, 0],
       [1, 0, 0],
       [0, 0, 1]])

However, when I use cascaded transformer LabelEncoder() and OneHotEncoder(), it just output wrong result with some warnings

In[14]
mapper1 = DataFrameMapper([('pet',[LabelEncoder(),OneHotEncoder()])],sparse=False)
mapper1.fit_transform(testdata)
Out[14]:
array([[ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.]])

The problem lies on OneHotEncoder. Seems that it is not compatible with DataFrameMapper, also not compatible with sklearn.pipeline.Pipeline

---

updates:

I found this bug the same as #60 . The reason is that in scikit-learn 0.17, 1-D array input to OneHotEncoder is deprecated. LabelEncoder() output 1-D array, thus things go wrong when cascading with OneHotEncoder()

[dukebody]

Thanks for the report.

This is a long standing issue with scikit-learn and transformers APIs. LabelEncoder was designed to be used only with 1-d array class labels, and OneHotEncoder with 2-d arrays, but the fact that OneHotEncoder only accepts integer valued inputs forced many people to chain both of them. See for example scikit-learn/scikit-learn#4920

To make things worse, LabelEncoder derives inconsistent results when using fit+transform and fit_transform:

>>> le.fit_transform(testdata[['pet']])
/home/.../.virtualenvs/sklearn-pandas/lib/python3.5/site-packages/sklearn/preprocessing/label.py:125: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)
array([0, 1, 1, 2, 0, 1, 0, 2])

>>> le.fit(testdata[['pet']]).transform(testdata[['pet']])
/home/.../.virtualenvs/sklearn-pandas/lib/python3.5/site-packages/sklearn/preprocessing/label.py:108: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)
Out[10]: 
array([[0],
       [1],
       [1],
       [2],
       [0],
       [1],
       [0],
       [2]])

DataFrameMapper uses the transformer fit_transform method when possible, so the input given to OneHotEncoder will have the wrong shape even using the [['pet']] selector.

Possible workarounds:

• Use LabelBinarizer - that's what I do.
• Use LabelEncoder first in the mapper, then OneHotEncoder in a separate step of a pipeline where the mapper is the first step. The problem with this approach is that you need to keep track of the categorical features indexes.

The definitive solution would be to make OneHotEncoder accept string features, or perhaps better, make it work with pandas categorical dtypes.

I'm closing the ticket since it's not really an issue, but I encourage you to post this to Stack Overflow so we can write the answer there to be better indexed.