Change auto_cast in SuperVectorizer - #238
Merged
GaelVaroquaux merged 15 commits intoMar 17, 2022
Merged
Conversation
…hich fixed the bug which made the parser convert a column to numeric if it had failed to convert it as date). We now use pd.to_datetime after the parsing.
LilianBoulard
left a comment
Member
There was a problem hiding this comment.
Thanks for the PR, it looks nice !
Co-authored-by: Lilian <lilian@boulard.fr>
Co-authored-by: Lilian <lilian@boulard.fr>
…irty_cat into dates-super-vectorizer merge
Contributor
Author
|
In trying to take into account @GaelVaroquaux's comment, I've actually realized that instead of using TextParser, it would be simpler to go back to @LilianBoulard's version of autocast, and:
|
Member
|
Very interesting!! You know these questions better than us, now, so your input is super useful. Tell us when you want a second review. |
Member
|
Darn, the tests errors are showing us that the strategy here is probably quite sensitive to the version of pandas. |
Co-authored-by: Lilian <lilian@boulard.fr>
GaelVaroquaux
left a comment
Member
There was a problem hiding this comment.
A few thoughts (I'm too tired to finish tonight).
GaelVaroquaux
approved these changes
Mar 17, 2022
GaelVaroquaux
left a comment
Member
There was a problem hiding this comment.
LGTM. Merging. Thanks a lot!!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uses pandas
TextParser(used inread_csv, as per @GaelVaroquaux suggestion in #234) instead ofconvert_dtypesin the SuperVectorizer_auto_castmethod. This has several advantages:["3", "2", "1"]would get detected as integer. Solves Handle string in auto_cast (SuperVectorizer) #234[1, 2, "", 4]is detected as integers. (solves Replace strings which indicates missing values by NaNs (for SuperVectorizer) #231). Which strings get detected as Nans is easy to change (see below).Some comments:
STR_NA_VALUES, and I've added[None, " ", "?", "..."]. I think we could find a more principled way to choose which strings to include.