Schedule automatic execution of wikidata query and switching between visualizations

[GonzaFP]

Hello @fernandouval @EneHacheCe @SannitaSSJ @wikicurricula-uy , this PR solves the folllowing issue as suggested by @EneHacheCe :

"I think that the Wikidata query and the excecution of bot.py should be done automatically but scheduled (monthly or weekly or daily). This should give us 2 voci files, one for Uruguay and one for Ghana, which are periodically updated.

Then, when the user clicks on the link, the visualization will switch between ghana's voci file and uruguay's voci file."

I have created an automation.batch file which can be scheduled (monthly. weekly or daily) to execute the bot.py and translate.py for all countries' curricula. When run, this file will make the wikidata query, write to query.csv, resultati.csv and then create a voci_file for each country. Behind the scenes, the batch file depends on bot.py and translate.py which I modified as follows:

I have modified bot.py so that it can execute the wikidata query and then write to query.csv. If run manually, a user will type "bot.py en 117" which will fetch Ghana's curricula in English wikipedia. "bot.py es 77" will fetch Uruguay's curricula.in spanish. Further, I modified translate.py so that it creates a voci file for each country so If the script is run manually, it will be like so "translate.py Ghana" which will create Ghana_voci_2023.tsv or "translate.py Uruguay " which will create Uruguay_voci_2023.tsv

Then, I added a link on Uruguay's visualization. This link allows a user to switch between Uruguay and Ghana's curricula. To do so, the visualization has to switch between Ghana's voci_file and Uruguay's voci_file hence I modified dv1.js and utilities.js so that they can read voci files dynamically.

I created a temporary config file to aid with dynamically executing bot.py but this PR #46 (comment) has a config file which can be used instead.

Here is a demo

Wikicurricula.Uy.-.Google.Chrome.2023-10-16.12-55-10.mp4

Here are some issues I faced:
Dynamically switching between languages in the tooltip and the legend. How can we do that?
Extracting subjects from the Wikidata query. In Ghana's case, I split the programLabel at the word "curriculum" But it feels hacky. For Uruguay, I used the hard coded subjects in the boilerplate.

I look forward to hearing from you.

[wikicurricula-uy]

@GonzaFP can you please fix the conflicts? Thanks for your work, it looks very interesting!

[GonzaFP]

@wikicurricula-uy Let me resolve them. Thank you for your feedback

[GonzaFP]

@wikicurricula-uy I have resolved the conflicts. Thanks

[SannitaSSJ]

@GonzaFP please resolve the new conflicts, so that I can merge your PR, thanks (I could've done it myself, but I have doubts about the first file)

[GonzaFP]

@SannitaSSJ , Let me resolve them. Thank you,

[GonzaFP]

@SannitaSSJ , I have resolved the conflicts. Thank you,

[SannitaSSJ]

@GonzaFP don't kill me please, but I recently merged another PR and... there are new conflicts. :( Please solve them, so that I can move on and finally merge your PR.

[GonzaFP]

@SannitaSSJ Haha, let me resolve them.

[GonzaFP]

@SannitaSSJ I have resolved them. Thanks

[GonzaFP]

@SannitaSSJ Please wait to merge, I have over-written my changes.

[GonzaFP]

@SannitaSSJ , It's Okay now. Thanks

[GonzaFP]

@SannitaSSJ Thank you.

[wikicurricula-uy]

Hello, @GonzaFP ! Thanks for your work here
I have some questions regarding your contribution, specifically because we have this pull request #76 by @Bhbee. I think some parts of the contributions are overlapping, and some parts not.

So, I've got these questions for you @GonzaFP

1. Can you provide instructions to set up the automation?
2. Does your automation also bring data from Wikidata (make the query) or does it read files like query.csv?
3. Is the visualization really loading the voci files dinamically when you change from one contry to another? Or does it just link to index.html and ghana.html?

Please, do not make any changes stemming from my questions. I just want to understand how this works, I'm not implying we should make changes.

[GonzaFP]

Hello @wikicurricula-uy, thanks for the feedback. Here are my answers.

1. the automation.batch file can be scheduled using windows task scheduler (for now). The detailed instructions can be found here https://www.geeksforgeeks.org/schedule-a-python-script-to-run-daily/
2. The automation brings data from Wikdata and then stores the article names in query,csv, then creates a voci file for each country.
3. The visualization loads the voci files dynamically,. The dv1.js file detects which html file has loaded and then loads its voci file.

Please let me know whether I have answered your questions.