Integrates Wortschatz frequencies #122
Conversation
lfashby
changed the title
| import tarfile | ||
| import time | ||
|
|
||
| with open("wortschatz_languages.json", "r") as langs: |
There was a problem hiding this comment.
why is this at global scope?
|
Basically looks good to me. I don't understand the *.tar.gz in the .gitignore though. I get why you want to keep the tarballs around, but maybe just add the directory as a .gitignore, not all filetypes like that? |
|
I experimented with different ways of grabbing the Wortschatz data (including with cURL) to see if I could resolve the 404 response problem, but generally got the same behavior. The solution in |
| @@ -51,6 +48,9 @@ def unpack(): | |||
|
|
|||
|
|
|||
| def main(): | |||
| with open("wortschatz_languages.json", "r") as langs: | |||
There was a problem hiding this comment.
make this string a global variable at the top?
There was a problem hiding this comment.
Or perhaps I should add it to the other path constants in src/codes.py?
|
That'd be fine too, whatever you think is more consistent.
…On Sun, Dec 15, 2019 at 6:28 PM Lucas Ashby ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In languages/wikipron/src/frequencies/grab_wortschatz_data.py
<https://github.com/kylebgorman/wikipron/pull/122#discussion_r358014869>:
> @@ -51,6 +48,9 @@ def unpack():
def main():
+ with open("wortschatz_languages.json", "r") as langs:
Or perhaps I should add it to the other path constants in src/codes.py?
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
<https://github.com/kylebgorman/wikipron/pull/122?email_source=notifications&email_token=AABG4OOKL37LWXNIPAXX6P3QY24P7A5CNFSM4J3CKGC2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCPHG4XA#discussion_r358014869>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABG4ON7ZX4I6ZO2U6THPCDQY24P7ANCNFSM4J3CKGCQ>
.
|
This PR adds two scripts and a json file to a new directory
src/frequencies.grab_wortschatz_data.pyautomatically downloads and unpacks Wortschatz tars andmerge.pymerges in the Wortschatz word frequency counts with our TSVs such that we end up with three column TSVs (our old TSVs are replaced):This is currently done for 61 languages, I'll see if I can add more before our next big scrape.
For words that we scrape and that Wortschatz doesn't have a frequency count for, the third column is set to 0.
I added the
tarsandfreq_tsvsdirectories (both created when runninggrab_wortschatz_data.py) to.gitignoreas opposed to automatically deleting them oncemerge.pycompletes because we download about 10 gigabytes worth of Wortschatz stuff and it's a pain to have to re-download it all every time we want to test these two scripts.