Data Input (vs. LDA & NMF)

[ZhiliWang]

Hey this is an awesome project. My question is that how exactly do you pre-process long texts. I notice that the metadata you demo with are all units of short texts (most of them are one sentences each). I tried to imitate that by segmented my texts into sentences (split long texts by period and semicolons, and validated their lengths) while cleaning all the punctuations, but getting only 1 cluster. Any suggestion would be much appreciated. I understand how transformer models are different from topic models like LDA and NMF, but do you think it's possible for BERT and transformer models to do something similar, which is inputting several long text files and simply generate models without a limitation of text length. Thank you.

[MaartenGr]

I am not sure as to why you would get only 1 cluster. It could be that your data is relatively small or that the minimum cluster size is too large. It might even be that your sentences are too short or overcleaned after preprocessing.

Another reason might be due to Umap, it is stochastic by nature which results in different results every time you run it. It might be strange advice, but perhaps try running it again?

When it comes to long text files, there are several methods for extracting the correct embeddings. Currently, you only input documents into BERTopic. However, I am planning on creating an option for using your own embeddings. This would allow you to embed paragraphs and apply mean pooling to merge them to a single embedding for the long text. Similarly, it would also allow you to choose any other transformer model that might be better suited for embedding long texts.