Skip to content

Optional stop_words_ attribute for CountVectorizer and TfidfVectorizer #20178

@xiangb

Description

@xiangb

Describe the workflow you want to enable

Hi,

Refering to this message found for sklearn.feature_extraction.text.CountVectorizer and sklearn.feature_extraction.text.TfidfVectorizer:

The stop_words_ attribute can get large and increase the model size when pickling. This attribute is provided only for introspection and can be safely removed using delattr or set to None before pickling.

This attribute can indeed be very large when the dataset is big (leading to out of memory error sometimes). Besides, what it contains is not used by everyone (not always).
So, it would be nice to have this attribute as optional.

Describe your proposed solution

  • Add the possibility to indicate that we don't want to build this attribute when initializing CountVectorizer and TfidfVectorizer

Describe alternatives you've considered, if relevant

Additional context

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions