Skip to content

Adding the SimpleCleaner object#1266

Merged
jeromedockes merged 33 commits into
skrub-data:mainfrom
rcap107:add-skrubber
Apr 1, 2025
Merged

Adding the SimpleCleaner object#1266
jeromedockes merged 33 commits into
skrub-data:mainfrom
rcap107:add-skrubber

Conversation

@rcap107

@rcap107 rcap107 commented Mar 31, 2025

Copy link
Copy Markdown
Member

Addresses #925

This PR adds moves the code used to build the pre-processors used by the TableVectorizer to a private function, and implements the Skrubber object. The Skrubber is a lightweight object that executes some of the transformations done by the TableVectorizer, mainly to do a "first pass" over datasets, prior to the actual transformation.

The Skrubber is equivalent to a TableVectorizer with all encoders set to "passthrough", with the major distinction of not converting numeric columns to float32.

We might want to reconsider the name, because calling it "Skrubber" may give it more importance in the context of the library than I think it deserves (this is really just a lightweight object that does something very simple).

@rcap107 rcap107 marked this pull request as ready for review March 31, 2025 15:11
@jeromedockes

Copy link
Copy Markdown
Member

thanks a lot @rcap107 this is really great and something we've wanted for a while! I think the code part is good to go so it is mostly a question of getting the docstring past the linter, so we can probably have this in the upcoming release 🚀

@Vincent-Maladiere

Copy link
Copy Markdown
Member

Very nice! Regarding the name, what about something like SimpleCleaner?

@jeromedockes

Copy link
Copy Markdown
Member

you will also need to add it to doc/reference/index.rst

@Vincent-Maladiere

Copy link
Copy Markdown
Member

Also, to better illustrate the capabilities of this transformer, could you list its features in the docstring, in a bulletpoint way? Some transformations seem missing in the docstring, like CleanCategories.

@jeromedockes jeromedockes changed the title Adding the skrubber object Adding the SimpleCleaner object Apr 1, 2025

@jeromedockes jeromedockes left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a few nitpicks (+ the formatting of the docstring to fix).

looks great otherwise, thanks! 🚀

Comment thread skrub/_table_vectorizer.py Outdated
Comment thread skrub/_table_vectorizer.py Outdated
Comment thread skrub/_table_vectorizer.py Outdated
Comment thread skrub/_table_vectorizer.py Outdated
Comment thread skrub/_table_vectorizer.py Outdated
Comment on lines +189 to +191
data. This is because the ``SimpleCleaner`` does not convert null values to a
consistent representation, which can lead to issues when the data is passed to
downstream models.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that part is a bit hard to understand. Maybe something like:

the simplecleaner does not perform any transformations on numeric columns. the tablevectorizer on the other hand converts them all to float32 and ensures null values are represented with nans which can be handled correctly by downstream scikit-learn estimators.

rcap107 and others added 7 commits April 1, 2025 11:25
Co-authored-by: Jérôme Dockès <jerome@dockes.org>
Co-authored-by: Jérôme Dockès <jerome@dockes.org>
Co-authored-by: Jérôme Dockès <jerome@dockes.org>
Co-authored-by: Jérôme Dockès <jerome@dockes.org>

@jeromedockes jeromedockes left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome, thanks a lot @rcap107 . This is super useful when you want more control over the encoding than what the tablevectorizer allows, but still want to benefit from its preprocessing

@jeromedockes jeromedockes added this to the 0.5.2 milestone Apr 1, 2025
@jeromedockes jeromedockes merged commit df109af into skrub-data:main Apr 1, 2025
@rcap107 rcap107 deleted the add-skrubber branch April 1, 2025 13:22
rcap107 added a commit to rcap107/skrub that referenced this pull request Apr 2, 2025
rcap107 added a commit that referenced this pull request Apr 2, 2025
@GaelVaroquaux

GaelVaroquaux commented Apr 2, 2025 via email

Copy link
Copy Markdown
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants