Recently dirty_cat dropped support for edit distances and left only ngram similarity, so I think it's time to add encoder like this.
The whole concept is pretty simple: instead of encoding categories if there is an exact match like OneHotEncoder, we encode categories based on their string similarity. It's very useful if dataset has 'dirty' (poorly-defined) categories (for example, professions). Since only very few libraries have something like that, it could be beneficial to add it here.
Proposition:
Add StringSimilarityEncoder with only basic functionality for now, i.e. support one similarity measure (gestalt pattern matching will do, it's from standard library so it will be fast and no additional dependencies required) and only 'most_frequent' categories, i.e. just like OneHotEncoder we encode all or top-k categories in the variable, no kmeans method for finding prototypes for now.
dirty_cat similarity encoder, twds article
Recently dirty_cat dropped support for edit distances and left only ngram similarity, so I think it's time to add encoder like this.
The whole concept is pretty simple: instead of encoding categories if there is an exact match like OneHotEncoder, we encode categories based on their string similarity. It's very useful if dataset has 'dirty' (poorly-defined) categories (for example, professions). Since only very few libraries have something like that, it could be beneficial to add it here.
Proposition:
Add StringSimilarityEncoder with only basic functionality for now, i.e. support one similarity measure (gestalt pattern matching will do, it's from standard library so it will be fast and no additional dependencies required) and only 'most_frequent' categories, i.e. just like OneHotEncoder we encode all or top-k categories in the variable, no kmeans method for finding prototypes for now.
dirty_cat similarity encoder, twds article