Topic maining

[dvmorozov]

Solution

1. Implement script collecting dictionary. Represent document as "bag-of-words". Save dictionary into file. ✔️
2. Implement iterator class over files in directory. ✔️
3. Implement model and use iterator class. ✔️
4. Save corpus into file (every text should be converted into single line) for processing with META. ✔️
5. Remove Greek letters from the list of special characters. ❓
6. Output topics into JSON. ✔️
7. Add lemmatization. Add reference to the main page. ✔️
8. Set encoding in reading and writing files as script parameter. ✔️

Related

1. Graph displaying topics vs. time #111.
2. Save article word vectors into files for subsequent comparison #112.
3. Support adding new files to the corpus (merge dictionaries) #113.

References

https://www.qblocks.cloud/blog/best-nlp-libraries-python

https://pypi.org/project/gensim/ ✔️
https://radimrehurek.com/gensim/
https://radimrehurek.com/gensim/auto_examples/core/run_topics_and_transformations.html (model)
https://radimrehurek.com/gensim/auto_examples/core/run_corpora_and_vector_spaces.html (corpus iteration)
https://radimrehurek.com/gensim/auto_examples/tutorials/run_lda.html#pre-process-and-vectorize-the-documents (lemmatization)

https://www.nltk.org/ ✔️ (pip install --user -U nltk; after that recreate virtual environment inheriting packages)

https://github.com/clips/pattern

https://www.machinelearningplus.com/nlp/gensim-tutorial/
https://www.machinelearningplus.com/nlp/lemmatization-examples-python/#stanfordcorenlplemmatization

https://stackoverflow.com/questions/8884188/how-to-read-and-write-ini-file-with-python3