Graph displaying topics vs. time

[dvmorozov]

Solution

1. Implement Python-script to split corpus into a set of sub-spaces with granularity of one month. Use metadata to do that. ✔️
2. Implement Python script to mine topics sequentially and fill mining data to JavaScript-file (which is used as graph data). ✔️
3. Count and display not existing, skipped (by version) and copied files. Write not existing files into CSV-file together with article id. and version.
4. Print estimated time of processing set of all month.
5. Implement Python script saving article identifiers as set of JSON-files for parallel mining. 💡
6. Implement Python script for mining topics for month given as script parameter for parallel mining. 💡
7. Print estimated time of processing metadata. ❌ Impossible to implement with ijson because it doesn't get number of articles in advance.
8. Implement Python script for parallel mining topics on partitioned corpus. 💡
9. Mine topics for the last year. 💡
10. Group articles by metadata topics (abbreviations). 💡
11. Compute number of articles by topic from metadata vs. time. Use this for graph data. 💡
12. Number of topics and topic items should be adjustable.
13. Use stream graph.

Related

1. Topic maining #109.
2. Graph displaying rate of adding articles by topic #83.
3. Graph displaying article topics #74.