v0.9.4

[MaartenGr]

A number of fixes, documentation updates, and small features:

• Expose diversity parameter
  • Use diversity=0.1 to change how diverse the words in a topic representation are (ranges from 0 to 1)
• Improve stability of topic reduction by only computing the cosine similarity within c-TF-IDF and not the topic embeddings
  • This also means that the topic embeddings could be improved or do not work properly with certain language models
  • Perhaps this could be improved by averaging out the exemplars but that still would not fully represent a topic
  • c-TF-IDF seems to hold the most information with respect to the topics
• Drop python 3.6
  • numpy >=1.20.0 doesn't support Python3.6 causing BERTopic installation failure #333
• Major documentation overhaul (mkdocs, tutorials, FAQ, images, etc. )
  • Customize similarity of topics reduction #330
• Add property to c-TF-IDF that all IDF values should be positive (has no influence on performance!)
  • Negative values in c-TF-IDF matrix #351
  • Note that this should have no influence on performance, it just adds a nice property to have 🙂
• Relax plotly dependency
  • No topic words in visualization #88
• Additional logging for .transform
  • Feature Request: Add logging to transform() #356

[eyang9002]

After dropping Python 3.7 support, do you suggest we stay on 3.6? Or > 3.7 moving forward? I've had to do a lot of work to maintain my dev environments and retraining of models using updated versions of BERTopic to get the newest enhancements.

[MaartenGr]

I would suggest going with a more recent version of python, such as 3.9. The thing is, security updates for numpy have stopped for python 3.6 so it was necessary to drop that version as it was not supported anymore.