Interaction terms between categorical and numerical features

[lorentzenchr]

Description

While it is possible to create feature interactions with the same individual preprocessing in ColumnTransformer via PolynomialFeatures, I find no (convincing) solution for interactions of features with different individual preprocessing, e.g. categorical column with a continuous numerical column.
Such interactions might improve models from sklearn.linear_model.

Code Example

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, PolynomialFeatures


df = pd.DataFrame({'a': ['red', 'red', 'blue', 'blue'],
                   'b': ['high', 'low', 'high', 'low'],
                   'x': [1, 1, 1, 2],
                   'y': [2, 3, 4, 2]
                  })

# interactions for features with no individual preprocessing works fine,
# i.e. numerical ones
column_trans = ColumnTransformer(
    [('xy_num',
      PolynomialFeatures(degree=2, interaction_only=True, include_bias=False),
      ['x', 'y'])],
     remainder='drop')
column_trans.fit_transform(df)

# interactions for ohe encoded also works with helper function
cat_cat = make_pipeline(
    OneHotEncoder(),
    PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
)
column_trans = ColumnTransformer(
    [('ab_cat', cat_cat, ['a', 'b'])],
     remainder='drop')
column_trans.fit_transform(df)

Expected Results

# no clue for interactions between one-hot-encoded 'a' and 'x' 
column_trans = ColumnTransformer(
    [('a_x',
      magic_pipeline(OneHotEncoder(), 'passthrough'),
      ['a', 'x'])],
     remainder='drop')

Versions

sklearn version 0.21

[jnothman]

Why can't you Pipeline a ColumnTransformer into a PolynomialFeatures? Or do you only want the interaction terms?

[lorentzenchr]

@jnothman Yes, one often wants only certain interactions. Passing everything from ColumnTransformer into PolynomialFeatures can generate a lot of features. Fist, this may blow up memory. Second, a feature selection method like L1 penalty might be needed, which slows down fitting (or is not available). Third, sometimes, one really only wants some specific interactions.

Based on your suggestion, I managed to pass a ColumnTransformer into a pipeline for ColumnTransformer like this:

cat_num = make_pipeline(
    ColumnTransformer(
        [('a_cat', OneHotEncoder(), ['a']),
         ('x_num', 'passthrough', ['x'])],
        remainder='drop'),
    PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
)

column_trans = ColumnTransformer(
    [('a_x_inter', cat_num, ['a', 'x'])],
    remainder='drop')

column_trans.fit_transform(df)

Result:

array([[0., 1., 1., 0., 0., 1.],
       [0., 1., 1., 0., 0., 1.],
       [1., 0., 1., 0., 1., 0.],
       [1., 0., 2., 0., 2., 0.]])

There are several issues with this:

• One has to specify the feature names twice, once in cat_num and then again in column_trans.
• It generates also interactions of OHE of 'a', which, of course, results in a column of zeros (4th column in the result). Would be nice, not to create this in the first place instead of having to call VarianceThreshold afterwards.
• It would also be nice, if PolynomialFeatures had an option to output only the real interactions, like [ab, ac, bc] and not [a, b, c, ab, ac, bc] despite interaction_only=True. solved by ENH add minimal degree support to PolynomialFeatures #20250.

[mayer79]

@lorentzenchr : Any progress on this?

@jnothman : What is the recommended way to generate interaction terms between one numeric feature and the dummy variables of a OHE categorical?

[ciberger]

Solution for second-order interaction and categorical features with strictly two categories.

preprocessor = make_pipeline(
  make_column_transformer(
      (StandardScaler(), num_columns),
      (OneHotEncoder(drop='if_binary'), cat_columns),
    remainder='drop',
    n_jobs=-1
    ), 
  PolynomialFeatures(interaction_only=True, include_bias = False))

pipe = preprocessor.fit(X_train)

tested on scikit-learn 0.24

[lorentzenchr]

@ciberger This is a bit more concise as the code snippet before, but has the same drawback as to create columns of zeros which I'd like to avoid.