Skip to content

Add built-in column-specific transformers to TableVectorizer - #583

Merged
Vincent-Maladiere merged 41 commits into
skrub-data:mainfrom
LilianBoulard:add_column_specific_transformers_tv
Aug 31, 2023
Merged

Add built-in column-specific transformers to TableVectorizer#583
Vincent-Maladiere merged 41 commits into
skrub-data:mainfrom
LilianBoulard:add_column_specific_transformers_tv

Conversation

@LilianBoulard

@LilianBoulard LilianBoulard commented Jun 9, 2023

Copy link
Copy Markdown
Member

Fixes part of #554

As discussed in #580, adds the column_specific_transformers parameter to the TableVectorizer.

This allows this kind of functionality:

from skrub import TableVectorizer, MinHashEncoder
from sklearn.compose import make_column_transformer

make_column_transformer([
    (
        MinHashEncoder(),
        ["PRODUCTTYPENAME"],
    ),
    remainder=TableVectorizer(),
])

from the TableVectorizer directly, with this syntax:

TableVectorizer(
    column_specific_transformers=[
        (MinHashEncoder(), ["PRODUCTTYPENAME"])
    ],
)

When the assignements need to be named (e.g. for a grid-search), the user can specify a name (same syntax as the ColumnTransformer):

TableVectorizer(
    column_specific_transformers=[
        ("mh_product_type", MinHashEncoder(), ["PRODUCTTYPENAME"])
    ],
)

@LilianBoulard LilianBoulard added the enhancement New feature or request label Jun 9, 2023
@LilianBoulard LilianBoulard self-assigned this Jun 9, 2023
@LilianBoulard LilianBoulard changed the title [Add column_specific_transformers to TableVectorizer Add built-in column-specific transformers to TableVectorizer Jun 9, 2023
@LilianBoulard
LilianBoulard marked this pull request as draft June 9, 2023 14:56
@LilianBoulard
LilianBoulard marked this pull request as ready for review June 30, 2023 18:05
Comment thread skrub/_table_vectorizer.py Outdated

@GaelVaroquaux GaelVaroquaux left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! A few minor comments.

One major one: we should use this in an example. Maybe by modifying an existing one

Comment thread CHANGES.rst Outdated
Comment thread skrub/_table_vectorizer.py Outdated
Comment thread skrub/_table_vectorizer.py Outdated
@GaelVaroquaux

Copy link
Copy Markdown
Member

You have failing tests. Can you address them please

@LilianBoulard

Copy link
Copy Markdown
Member Author

Thanks for reminding me :)
The PR should be ready to be reviewed/merged now. The implementation is complete and the tests are done.

Comment thread examples/07_grid_searching_with_the_tablevectorizer.py
Comment thread examples/07_grid_searching_with_the_tablevectorizer.py Outdated
Comment thread examples/07_grid_searching_with_the_tablevectorizer.py
Comment thread examples/07_grid_searching_with_the_tablevectorizer.py
@LilianBoulard
LilianBoulard dismissed GaelVaroquaux’s stale review July 28, 2023 14:19

Concerns were addressed

@jovan-stojanovic jovan-stojanovic left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @LilianBoulard, nice to see this implemented :)
A few comments for the example

@Vincent-Maladiere Vincent-Maladiere left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Lilian, here is my first round of reviews! The example looks neat, let's add the plots from the results of the grid search :)

Comment thread examples/07_grid_searching_with_the_tablevectorizer.py
Comment thread examples/07_grid_searching_with_the_tablevectorizer.py Outdated
Comment thread examples/07_grid_searching_with_the_tablevectorizer.py Outdated
Comment thread examples/07_grid_searching_with_the_tablevectorizer.py
Comment thread skrub/_table_vectorizer.py Outdated
Comment thread skrub/_table_vectorizer.py
Comment thread skrub/tests/test_table_vectorizer.py
Comment thread skrub/tests/test_table_vectorizer.py Outdated
LilianBoulard and others added 4 commits July 31, 2023 16:31
Co-authored-by: Vincent M <maladiere.vincent@yahoo.fr>
Co-authored-by: Jovan Stojanovic <62058944+jovan-stojanovic@users.noreply.github.com>
@LilianBoulard

LilianBoulard commented Jul 31, 2023

Copy link
Copy Markdown
Member Author

The grid-search doesn't work as expected, there is something different between the ColumnTransformer and the TableVectorizer (in the sense, it works with one but not the other), but I don't know what yet. I think you had a concern about set_params @glemaitre?

@LilianBoulard

Copy link
Copy Markdown
Member Author

So the feature has been ready for a while, but this example is blocking. Since it doesn't work and needs some more work, I've put it into a temporary directory, so it's not accessible on the website, but so we don't lose the code. I'll open an issue in a few minutes to fix that.

@jovan-stojanovic jovan-stojanovic left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we need to move on, until we find a good solution for the grid search.
Add one more small test before approving this :)

Comment thread skrub/_table_vectorizer.py

@Vincent-Maladiere Vincent-Maladiere left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @LilianBoulard! After adding the missing test, I'm happy with this PR as-is. Let's investigate the grid-search on a different PR :)

Comment thread CHANGES.rst
Comment thread skrub/tests/test_table_vectorizer.py Outdated
Comment thread skrub/tests/test_table_vectorizer.py Outdated
Co-authored-by: Vincent M <maladiere.vincent@yahoo.fr>

@Vincent-Maladiere Vincent-Maladiere left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's merge once the CI is green

@Vincent-Maladiere
Vincent-Maladiere merged commit 9f4ca19 into skrub-data:main Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants