Maximum length of token length

[ginward]

In SentenceTransformer (and most transformer-based models), the maximum length of a document is 512 tokens.

I was wondering if there is a similar document length restrictions in BERTopic?

Thanks!

[ginward]

This is related to #114

[MaartenGr]

The token restrictions only apply to the embedding technique that you use. In your case, that would depend on the SentenceTransformer model that you are using. Moreover, if you were to use Flair or Huggingface Transformer embeddings, then you might run into token limits if you are not careful.

In general, it should not be an issue as long as you are not going too far over the limit. The thing is with going too far over the limit is that the document representation gets vague. In those cases, I would advise splitting your documents into sentences.