Indexing wikipedia zim - once and for all

[ghost]

Previously discussed: #546 #680 #763

I'd like to see wikipedia zim files work out of the box with goldendict and I'm willing to do the work.
The solution (1) I have now is to:

• disable FTS for zim files above ? GB
• Modify the addWord() logic to index only the full (folded) title and not to generate additional entries on word boundaries - if the article count is large enough.

how about it, @Abs62 ? I can send a PR for (1) soon if you're interested.

[darnn]

If you do move forward on this, please consider adding the option to create a Wikipedia-based dictionary from dumps, i.e. something that would enable you to input, say, "The Terminator" and have GD output "Терминатор (фильм)" (if you've selected English->Russian).

[Abs62]

wikipedia_en_all_nopic_2017-08.zim, 64-bit Qt5-based GD - 8 minutes for indexing, 720 MB index file. Up to 7 GB memory consumption while indexing.

What criterion do you suggest for reduce headwords handling?

[data-man]

IMO, the best solution would be to use the libzim library, which has built-in support for the Xapian library with advanced full-text search capabilities.

[Abs62]

I have add parameter to config file.

[ghost]

Thanks. The problem is still there because:

• FTS indexing is on by default for zim -> OOM for fresh users (FTS should have size limit too? some smaller zim files could benefit from FTS)
• default value set for maxHeadwordsToExpand is 0, so headword expansion it now turned off for ALL dicts (IIUC). default should probably be large enough to only apply to wikipedia-scale files.

[Abs62]

FTS indexing is on by default for zim -> OOM for fresh users (FTS should have size limit too? some smaller zim files could benefit from FTS)

default value set for maxHeadwordsToExpand is 0, so headword expansion it now turned off for ALL dicts (IIUC). default should probably be large enough to only apply to wikipedia-scale files.

No. Limit should be turned on only if GD can't index dictionary wthout one. 64-bit GD and 8+ GB RAM is not rare configuration nowadays.

[ghost]

Fair enough. Thanks for fixing! 👍

Future Visitors:
If your machine doesn't have enough memory to complete the indexing of wikipedia_nopic_201?.zim, use the following workaround:

• set the limit indicated in the image above. somewhere between 2000000 and 10000000 should work. This eliminates FTS indexing for very large files.
• open up your ~/.goldendict/config file, and add the key:

 <maxHeadwordsToExpand>2000000</maxHeadwordsToExpand>

anything between 2M and 10M should work. This reduces title indexing to be less extensive for files with more then 2M articles. So for example, with reduced indexing, if you search for "Alice" you'll have "Alice in wonderland" turn up, but it won't appear if you searching for "wonderland". With full-indexing, "wonderland" would also match the "Alice in wonderland" article.

[jjzz]

@darnn
I just put wikipair online https://github.com/jjzz/wikipair
I guess that's exactly what you want.

[darnn]

@jjzz That is, in fact, exactly what I want! I'm clueless when it comes to Linux, but I managed to run it as far as getting the txt and the dsl file, after which it gave an error on the dos2unix command. I copied over the files and changed the linebreak style to Windows, and turned on BOM, and Windows Goldendict now reads it! So, thank you so much!

[xiaoyifang]

xiaoyifang/goldendict-ng@7a2ec4f