MRG Feature stacker #1173

Merged
merged 11 commits into from Sep 30, 2012

Projects

None yet

7 participants

@amueller
scikit-learn member

This estimator provides a Y piece for the pipeline.
I used it to combine word ngrams and char ngrams into a single transformer.
Basically it just concatenates the output of several transformers into one large feature.

If you think this is helpful, I'll add some docs and an example.
With this, together with Pipeline, one can build arbitrary complex graphs (with one source and one sink) of estimators in sklearn :)

TODO

  • tests
  • narrative documentation
  • example

Thanks to the awesome implementation of the BaseEstimator, grid search simply works - though with complicated graphs you get parameter names like feature_stacker__first_feature__feature_selection__percentile (more or less from my code ^^).

@ogrisel ogrisel and 1 other commented on an outdated diff Sep 20, 2012
sklearn/linear_model/tests/test_randomized_l1.py
+
+ # center here because sparse matrices are usually not centered
+ X, y, _, _, _ = center_data(X, y, True, True)
+
+ X_sp = sparse.csr_matrix(X)
+
+ F, _ = f_classif(X, y)
+
+ scaling = 0.3
+ clf = RandomizedLogisticRegression(verbose=False, C=1., random_state=42,
+ scaling=scaling, n_resampling=50, tol=1e-3)
+ feature_scores = clf.fit(X, y).scores_
+ clf = RandomizedLogisticRegression(verbose=False, C=1., random_state=42,
+ scaling=scaling, n_resampling=50, tol=1e-3)
+ feature_scores_sp = clf.fit(X_sp, y).scores_
+ assert_equal(feature_scores, feature_scores_sp)
@ogrisel
ogrisel Sep 20, 2012

This hunk seems to be unrelated to this PR.

@amueller
amueller Sep 20, 2012

whoops sorry, forked from wrong branch. just a sec.

@ogrisel
scikit-learn member

Very interesting. I want an example first! (then documentation and tests :)

@amueller
scikit-learn member

on it :)

@ogrisel
scikit-learn member

@amueller to avoid forking from non-master branches you should use something such as http://volnitsky.com/project/git-prompt/

@ogrisel ogrisel and 2 others commented on an outdated diff Sep 20, 2012
sklearn/pipeline.py
+
+ def get_feature_names(self):
+ pass
+
+ def fit(self, X, y=None):
+ for name, trans in self.transformer_list:
+ trans.fit(X, y)
+ return self
+
+ def transform(self, X):
+ features = []
+ for name, trans in self.transformer_list:
+ features.append(trans.transform(X))
+ issparse = [sparse.issparse(f) for f in features]
+ if np.any(issparse):
+ features = sparse.hstack(features).tocsr()
@ogrisel
ogrisel Sep 20, 2012

Maybe the tocsr() can be avoided. For instance the downstream model might prefer CSC such as ElasticNet for instance.

@larsmans
larsmans Sep 21, 2012

Then again, bugs crop up every now and then where estimators that are supposed to handle any sparse format turn out to only handle CSR. It's a good defensive strategy to produce CSR by default (and it's unfortunate that sparse.hstack doesn't do this already).

@amueller
amueller Sep 21, 2012

I wrote this thing in the heat of the battle and I don't remember if there was a reason or if it was just a precaution. I'm inclined to think that I put it there because something, somewhere, broke.

@amueller
scikit-learn member

Yes, it should derive from transformer mixin.
@larsmans can I interpret your comments such that you think this is a good thing to have?

@amueller
scikit-learn member

Added a toy example.

@ogrisel
scikit-learn member

I think such a feature stack should provide some way to do feature group normalization in one way or another. But this probably require some experiments to know which normalization pattern is useful on such beast in practice.

Anybody has practical experience or insight to share on this?

@larsmans
scikit-learn member

GREAT idea! However, I don't like the name FeatureStacker much, as stacking implies putting things on top of each other, while this class concatenates things side-by-side.

I tried to find a "plumbing equivalent" of this class to keep with the pipeline metaphor, but I can't seem to find it. It's not quite a tee as it connects the various streams back together in the end. Maybe one of the other devs is more experienced with plumbing? :)

@ogrisel
scikit-learn member

BTW I think the example could be improved my using a less trivial example (e.g. using the digits dataset) and showing that the cross validate score best grid searched parameter set of the pipeline with stacked features is better than the pipeline with individual feature transformers separately.

@ogrisel
scikit-learn member

@larsmans maybe FeatureConcatenator?

@ogrisel
scikit-learn member

FeatureUnion?

@ogrisel ogrisel closed this Sep 20, 2012
@larsmans larsmans reopened this Sep 20, 2012
@larsmans
scikit-learn member

MultiTransformer?

@amueller
scikit-learn member
@GaelVaroquaux
scikit-learn member
@amueller
scikit-learn member

I also like FeatureUnion.
Other possibilities: FeatureBinder, FeatureAssembler, FeatureCombiner.
Or maybe go away from feature? TransformerUnion, TransformBinder, TransformerBundle?

Hm i think I like TransformerBundle

@ogrisel
scikit-learn member

+1 for FeatureAssembler or FeatureUnion or TransformerBundle

@larsmans
scikit-learn member

+1 for TransformerBundle.

@amueller
scikit-learn member

In my application, I found the get_feature_names very helpful - I was using text data and some handcrafted features.
I fear in general this is hard to do. I thought about doing hasattr("get_feature_names") and otherwise just return estimator_name_0, estimator_name_1,.... This might be a bit problematic, though, as I don't think there is a reliable method to get the output dimensionality of a transformer :-/

Oh and @ogrisel for the normalization, each feature should be normalized separately, right?
This is "easily" possible but feeding the object pipelines of preprocessing and transformers. As normalization might be quite application specific, I think this solution is ok for the moment.
The code doesn't actually get too messy doing this.

@amueller
scikit-learn member

ugh I just tried to work on the example and noticed that #1034 wasn't in master yet.
Without a good way to look at the grid search results, this PR is a lot less useful I think.
Have to work on #1034 more :-/

@larsmans
scikit-learn member

We might introduce an n_features_out_ attribute/property on all transformers that work on feature vectors. For now, only supporting get_feature_names only when all underlying transformers do would be good enough, IMHO.

@amueller
scikit-learn member

@larsmans ok, will do that. Should be easy enough.

@amueller
scikit-learn member

Having a bit of a hard time creating a good example :-/

@ogrisel
scikit-learn member

Have you been able to use this kind of tool successfully for your kaggle contest? If so then we can stick to a simplistic toy example and tell in the narrative documentation which kind of feature bundle was proven useful in practice on which kind of problem (e.g. PCA feature + raw TF-IDF for text classification for instance).

@amueller
scikit-learn member

I can tell you how successful I was tomorrow ;)
It was definitely helpful to combine handcrafted features with word n-grams. Doing it using this estimator, I was still able to grid-seach for count-vectorize parameters such as min_df, ngram_range, etc. So that definitely helped.

@mblondel mblondel and 1 other commented on an outdated diff Sep 21, 2012
sklearn/pipeline.py
@@ -199,3 +202,81 @@ def score(self, X, y=None):
def _pairwise(self):
# check if first estimator expects pairwise input
return getattr(self.steps[0][1], '_pairwise', False)
+
+
+class FeatureStacker(BaseEstimator, TransformerMixin):
+ """Concatenates results of multiple transformer objects.
+
+ This estimator applies a list of transformer objects in parallel to the
+ input data, then concatenates the results. This is useful to combine
+ several feature extraction mechanisms into a single estimator.
@mblondel
mblondel Sep 21, 2012

single feature representation?

@amueller
amueller Sep 21, 2012

I prefer it the way it is, as getting the features out is not the important part, the important part is formulating it as an estimator.

@mblondel
mblondel Sep 21, 2012

I misunderstood what you meant. Since you're talking about extraction mechanisms, it may be clearer to say "in a single transformer".

@mblondel mblondel and 2 others commented on an outdated diff Sep 21, 2012
sklearn/pipeline.py
+ for name, trans in self.transformer_list:
+ if not hasattr(trans, 'get_feature_names'):
+ raise AttributeError("Transformer %s does not provide"
+ " get_feature_names." % str(name))
+ feature_names.extend([name + "__" + f for f in trans.get_feature_names()])
+ return feature_names
+
+ def fit(self, X, y=None):
+ """Fit all transformers using X.
+
+ Parameters
+ ----------
+ X : array-like or sparse matrix, shape (n_samples, n_features)
+ Input data, used to fit transformers.
+ """
+ for name, trans in self.transformer_list:
@mblondel
mblondel Sep 21, 2012

supporting n_jobs would be nice :)

@amueller
amueller Sep 21, 2012

In principle +1. Are there any transformers that use n_jobs? I am always afraid of having it on the wrong abstraction level....

@mblondel
mblondel Sep 21, 2012

Since it is embarrassingly parallel and each transformer can take time to fit, I think supporting n_jobs would make sense.

Are there any transformers that use n_jobs?

Not that I know of but I hope that users have enough common sense to not enable n_jobs at two different levels :)

@amueller
amueller Sep 21, 2012

So I see you are of the optimist persuasion ;)
I'll add it.

@GaelVaroquaux
GaelVaroquaux Sep 21, 2012
@mblondel mblondel and 1 other commented on an outdated diff Sep 21, 2012
sklearn/pipeline.py
+
+
+class FeatureStacker(BaseEstimator, TransformerMixin):
+ """Concatenates results of multiple transformer objects.
+
+ This estimator applies a list of transformer objects in parallel to the
+ input data, then concatenates the results. This is useful to combine
+ several feature extraction mechanisms into a single estimator.
+
+ Parameters
+ ----------
+ transformers: list of (name, transformer)
+ List of transformer objects to be applied to the data.
+
+ """
+ def __init__(self, transformer_list):
@mblondel
mblondel Sep 21, 2012

a transformer_weight option to give more importance to some transformers could be useful!

@amueller
amueller Sep 21, 2012

hm ok, why not.

@mblondel
scikit-learn member

Nice idea indeed!

@amueller
scikit-learn member

@mblondel any votes on the name?

@mblondel
scikit-learn member

Some I like include FeatureAssembler, FeatureCombiner and FeatureUnion.

@amueller
scikit-learn member

Name votes:
FeatureAssembler II
FeatureCombiner I
FeatureUnion IIII
TransformerBundle III

(If I counted correctly, which is unlikely given my degree in math)
If no-one objects I'll rename to FeatureUnion and change the state of the PR to MRG.

@agramfort
scikit-learn member
@amueller
scikit-learn member

Renamed, think this is good to go.

@amueller
scikit-learn member

Any more comments? (github claims this can not be merged but I just rebased, so it should be a fast-forward merge).

@ogrisel
scikit-learn member

This cannot be merged in master currently but appart from that +1 for merging :)

@GaelVaroquaux
scikit-learn member

LGTM. 👍 for merge. Thanks @amueller !

@amueller amueller merged commit d087830 into scikit-learn:master Sep 30, 2012
@vene
scikit-learn member

Thank you for this convenient transformer. In my application I had to hack it a bit, and I wonder whether the feature I wanted could be more generally useful.

Basically, sometimes you want to concatenate the same feature extractor multiple times, and have some of the parameters tied when grid searching.

In my case, I was learning a hyphenator, so my data points consist of 2 strings: the one to the left of the current position and the one to the right of the current position. For this I defined a ProjectionVectorizer that has a column attribute that just says "I only work on X[:, column]" and concatenated two of these. Now, when grid searching, it is common sense to use the same n-gram range for both transformers, so the cleanest way to do this was this quick hack (no error handling):

class HomogeneousFeatureUnion(FeatureUnion):
    def set_params(self, **params):
        for key, value in params.iteritems():
            for _, transf in self.transformer_list:
                transf.set_params(**{key: value})

This can be easily extended to support both tied params and specific params. I'm not sure whether I overengineered this, but I still have the feeling that this might pop up in other people's applications, so I wanted to raise the question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment