Pending removal of download links to "postprocessed" wiktextract data; these links are deprecated

[kristian-clausal]

The kaikki.org website has some links to "postprocessed" wiktextract data, and in the near future links to these files will be removed because they're causing problems when users mistakenly download them instead of the raw wiktextract dump data.

Some of these issues are related to the fact that the data is used specifically in the generation of the kaikki.org website, which means modifying it by frex. sorting words alphabetically instead of in the order of generation (which is generally the same as the order they have on the wiktionary page they were extracted from). Other issues are old projects by Tatu trying to create identifier numbers, and trying to create a system to disambiguate words from certain senses or meanings; these things will still remain in the output of the kaikki.org website, but they are unlikely to progress any further and they're probably not useful to users wanting to do their own processing of the data.

• Remove links to language-specific word data that is post-processed; this is the stuff that's been causing the most confusion. It will not be replaced by anything else, and instead users are encouraged to download the complete raw data file; all you need to do is filter each line by the language_code entry to get the entries that would appear in the old file.
• Remove links to postprocessed word-specific entries on word pages (might already have been done earlier)
• Remove link to the big postprocessed file for the whole dictionary.

The raw extract file is the more complete of the datasets. Most of the things done to the postprocessed data is specific to generating the kaikki website, like the sorting, and some of it will lose information in the process.

[kristian-clausal]

Added deprecation warnings to Kaikki.org download links, should show up in a few days when the website regenerates.

[MLNW]

It might be helpful to include a direct link to this issue in the deprecation warning. I eventually found it myself, but it wasn’t straightforward. Right now the warning doesn’t explain why it’s deprecated or how to work around it. Linking here would make both the rationale and the migration steps clear.

[ginxxxxx]

I use and need the postprocessed files. I do not have enough ram to process the raw files

[xxyzz]

How could process JSONL file consume too much RAM, if read one line at a time?

Can't help think of this: https://xkcd.com/1172/

[kristian-clausal]

I use and need the postprocessed files. I do not have enough ram to process the raw files

Because the raw data is a .jsonl file with each word entry in its own json object separate from others on a new line, you can filter out the appropriate data line by line in your program or script (which shouldn't use up RAM except for that one line at a time), or if you're using some third-party program that would otherwise load the whole file as one clump you can preprocess the files in your flow by using something like the command-line utility jq.

If you're handling json data in general, learning to use jq is really, really handy, and I recommend it, even though it can be a bit difficult to reason about... but for the purposes of filtering out json objects from our raw data it's pretty simple, it only gets a bit tricky when you want to handle nesting or array syntax, or manipulating the data.

[kristian-clausal]

Forgot to mention it, but what you want to filter for is the lang_code field; if you want all English language word entries, pick out all objects with lang_code: "en", etc.

[MLNW]

@ginxxxxx here is a concrete example. This filters out all Spanish verbs, renames word to verb and removes some redundant information:

jq -c 'select(.lang_code=="es" and .pos=="verb") | {verb: .word} + del(.word, .pos, .lang_code, .lang)' \
  resources/kaikki.org-dictionary-es-2025-05-20-2025-06-01.jsonl \
  > resources/kaikki.org-dictionary-es-2025-05-20-2025-06-01.cleaned.jsonl

[jx2zhou]

I, and I assume many others, use this invaluable add-on for Anki (github) which consumes the postprocessed language-specific json files to add dictionary info to flashcards for language learning purposes. Seeing that these json files still need to be generated for kaikki website, it seems hiding access would cause a waste of effort for each user of this add-on to reduplicate the postprocessing individually.

If the issue is user confusion, can we keep access to these files and put a link to them somewhere else? maybe more obscure, where users won't mistakenly download these instead of the raw dump?

[non-bin]

/dictionary/English/ says

if you wish to process the data yourself use the data found on /dictionary/rawdata

but /dictionary/rawdata says

For post-processed data, please look at the download links at the end of the main page for each language (or the page for all languages combined) under https://kaikki.org/dictionary/.

I'm confused

And the file sizes are all wrong. On the pages it says 2.2 and 19.2Gb respectively, and my browser currently says 2.5Gb/400mb and 19.3Gb/2.2Gb respectively. Some weird shenanigans be afoot

[xxyzz]

Only download .jsonl files from https://kaikki.org/dictionary/rawdata.html

Don't overthink...

[non-bin]

thanks, but I don't think I am overthinking it...

[kristian-clausal]

, and I assume many others, use this invaluable add-on for Anki (github) which consumes the postprocessed language-specific json files to add dictionary info to flashcards for language learning purposes.

If Anki is using the postprocessed data, they shouldn't. There is no point in using the post-pro data, it is the same data as the raw data, except with irrelevant added bits that don't mean anything to anyone else. This is why we have a deprecation warning. We cannot inform everyone who uses the data by hand, so we put a warning on the website and assume people will stumble upon it sooner or later.

If the Anki developers haven't checked out the website in a while (and are just downloading old links or scraping the data), then they'll find out when the links die. There's not much we can do about that.

but /dictionary/rawdata says

For post-processed data, please look at the download links at the end of the main page for each language (or the page for all languages combined) under https://kaikki.org/dictionary/.

There is a big capitalized "DEPRECATED" before that text. Do not use post-processed data.