utf-8 search support

[imehdihosseini]

Hey, we are working on a Persian version of a mkdocs theme, everything works fine but we have a problem on search contents with Persian contents, when i search the expression nothing found, i also test it with Arabic, Chinese and Hebrew and all of theme have problem with this case. any solution?

[d0ugal]

Unfortunately search wont work well in non-latin character sets at the moment. We use lunr.js to provide the search functionality, I believe it can work with them but I am not sure what is involved in making that happen.

It is possible for themes to overwrite and completely replace the search implementation. To do this, themes would need to at least replace this file (or modify the theme so that file isn't loaded) by including a javascript file in the theme with the path mkdocs/js/search.js. However, as I said at the top, I have no experience in making search work for different character sets, so I don't know how difficult it will be.

We could also consider adding a setting to disable search. While this isn't ideal, it would be better to turn search off than have it in a broken state.

I don't think there is a good, easy solution for this at the moment. 😞

[waylan]

There is also some related discussion in #859. Apparently, non-English search indexes are extremely large as lunr.js only "excludes" insignificant English words from the index by default. MihaiValentin/lunr-languages appears to be an add-on to lunr.js which adds support for various other languages including Japanese and Russian (both use non-latin alphabets). Unfortunately, at this time, Persian, Arabic, Chinese and Hebrew are not on the list of supported languages. Although there are currently outstanding requests for Arabic and Chinese to be added. If you would like to have support added for a certain language, I would suggest starting there. If/when, mkdocs adds language support, then your language will work if its supported by lunr.js.

https://github.com/mkdocs/mkdocs/blob/master/mkdocs/commands/build.py#L223

json dump to utf-8?