Adding the SimpleCleaner object#1266
Conversation
|
thanks a lot @rcap107 this is really great and something we've wanted for a while! I think the code part is good to go so it is mostly a question of getting the docstring past the linter, so we can probably have this in the upcoming release 🚀 |
|
Very nice! Regarding the name, what about something like |
|
you will also need to add it to |
|
Also, to better illustrate the capabilities of this transformer, could you list its features in the docstring, in a bulletpoint way? Some transformations seem missing in the docstring, like |
jeromedockes
left a comment
There was a problem hiding this comment.
just a few nitpicks (+ the formatting of the docstring to fix).
looks great otherwise, thanks! 🚀
| data. This is because the ``SimpleCleaner`` does not convert null values to a | ||
| consistent representation, which can lead to issues when the data is passed to | ||
| downstream models. |
There was a problem hiding this comment.
I think that part is a bit hard to understand. Maybe something like:
the simplecleaner does not perform any transformations on numeric columns. the tablevectorizer on the other hand converts them all to float32 and ensures null values are represented with nans which can be handled correctly by downstream scikit-learn estimators.
Co-authored-by: Jérôme Dockès <jerome@dockes.org>
Co-authored-by: Jérôme Dockès <jerome@dockes.org>
Co-authored-by: Jérôme Dockès <jerome@dockes.org>
Co-authored-by: Jérôme Dockès <jerome@dockes.org>
jeromedockes
left a comment
There was a problem hiding this comment.
awesome, thanks a lot @rcap107 . This is super useful when you want more control over the encoding than what the tablevectorizer allows, but still want to benefit from its preprocessing
|
Sorry, I was too slow on this, but regarding name, I would like to strongly suggest that "SimpleCleaner" is not a good name: the "Simple" is too much. It's not informative, and should not be there. "Cleaner" is enough, and if we ever do a sophisticated cleaner, it will come with a longer name.
Maybe that was merged a bit too fast, because we now have a released version and we need to change the name
|
Addresses #925
This PR adds moves the code used to build the pre-processors used by the TableVectorizer to a private function, and implements the Skrubber object. The Skrubber is a lightweight object that executes some of the transformations done by the TableVectorizer, mainly to do a "first pass" over datasets, prior to the actual transformation.
The Skrubber is equivalent to a TableVectorizer with all encoders set to "passthrough", with the major distinction of not converting numeric columns to float32.
We might want to reconsider the name, because calling it "Skrubber" may give it more importance in the context of the library than I think it deserves (this is really just a lightweight object that does something very simple).