Stop words in corpus of large texts

[DKanarsky]

1. Should stop words be removed from corpus beforehand? My topic_model generates clusters with most frequent words like "the", "and", "to" and etc.

2. Is there any model to process long text without truncating them to maximum available for transformers?

[MaartenGr]

It is typically not necessary to remove stopwords if enough topics are generated (> 40). However, if you want to remove stopwords because they appear for some reason still in the topic representation, I would advise using a CountVectorizer in BERTopic to remove them as described here.

Transformers typically have a token limit which makes it difficult to process long documents. You can, however, either split them into paragraphs or sentences and use those in BERTopic or use an entirely different model, such as TF-IDF to process the documents. You can find more about different models here and here.

[aph61]

I used an alternative approach that worked quite well.

The c-tfidf step is to characterize a topic; it does not take part in the clustering. So you can use any method you wish to characterize a sentence. After topic assignment I tokenized the full corpus and kept nouns, pronouns and adjectives only for topic characterization. After that you collect all tokenized sentences and characterize that set using (c-)tfidf or any other method you deem suitable.
Someday in the future I may try word2vec on the reduced corpus; see where that leads to

But the bottom line is, once your sentences are assigned to a cluster you can do anything you wish to characterize the cluster. That can be c-tfidf on the raw text, on the filtered text, you name it

[DKanarsky]

Thanks!!!