Fix an error when the lang of node prebuild_index is ja or jp.

[m15o]

With the following configuration, node prebuild_index raise an error.

site_name: prebuild_index_ja
plugins:
  - search:
      lang: [ja]
      prebuild_index: node

The log is as follows. Formatted for readability:

WARNING -  Failed to pre-build search index. Error: b'
/Users/m15o/.pyenv/versions/3.8.5/envs/mkdocs/lib/python3.8/site-packages/mkdocs/contrib/search/lunr-language/lunr.ja.js:86
    var segmenter = new lunr.TinySegmenter(); // \xe3\x82\xa4\xe3\x83\xb3\xe3\x82\xb9\xe3\x82\xbf\xe3\x83\xb3\xe3\x82\xb9\xe7\x94\x9f\xe6\x88\x90
                    ^

TypeError: lunr.TinySegmenter is not a constructor
    at /Users/m15o/.pyenv/versions/3.8.5/envs/mkdocs/lib/python3.8/site-packages/mkdocs/contrib/search/lunr-language/lunr.ja.js:86:21
    at Socket.<anonymous> (/Users/m15o/.pyenv/versions/3.8.5/envs/mkdocs/lib/python3.8/site-packages/mkdocs/contrib/search/prebuild-index.js:27:55)
    at Socket.emit (events.js:322:22)
    at endReadableNT (_stream_readable.js:1187:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
'

This PR fixes the issue by adding lunr.TinySegmenter to the prebuild-index dependency.

[fu-sen]

I'm experiencing a problem where I can't search Japanese when using MkDocs.
I will issue a separate issue about this.

I tested this, but it doesn't improve Japanese search.
It seems that worker.js support is lacking

[waylan]

If tinyseg.js is needed for search to work, then it also needs to be loaded in worker.js here:

mkdocs/mkdocs/contrib/search/templates/search/worker.js

Lines 36 to 54 in dd903d3

	function onJSONLoaded () {
	data = JSON.parse(this.responseText);
	var scriptsToLoad = ['lunr.js'];
	if (data.config && data.config.lang && data.config.lang.length) {
	lang = data.config.lang;
	}
	if (lang.length > 1 || lang[0] !== "en") {
	scriptsToLoad.push('lunr.stemmer.support.js');
	if (lang.length > 1) {
	scriptsToLoad.push('lunr.multi.js');
	}
	for (var i=0; i < lang.length; i++) {
	if (lang[i] != 'en') {
	scriptsToLoad.push(['lunr', lang[i], 'js'].join('.'));
	}
	}
	}
	loadScripts(scriptsToLoad, onScriptsLoaded);
	}

Ideally, this lib would only be loaded if it is needed, so only when lang includes ja or jp.

In other words, this "fix" is incomplete.

[waylan]

And I just realized that this PR is 5 months old and I had set the needs further review label on it. Apparently, I forgot about it.

[fu-sen]

I was able to confirm the close of the issue #2333, so I will pay attention to it as well.
I've come to see MkDocs in Japanese a lot lately, With this solution, we can expect a further increase in Japanese users.

To solve this, it seems that we need the help of someone who is familiar with the implementation of lunr-languages.

[fu-sen]

@m15o 対応ありがとうございます。早速試してみます。 I will try this change immediately.

[fu-sen]

@m15o 残念ながらまだコードが不完全で動作していません。
Unfortunately the code is incomplete and not working.
作ったコードを実際にインストールして動作を確認してみて下さい。
Please actually install the created code and check the operation.

pip install git+https://github.com/m15o/mkdocs.git

[m15o]

@fu-sen
I cannot understand whats the exact problem is.
Could you please specify what did you do and what kind of error did you get?

And you need to specify the branch name to test the PR:

pip install git+https://github.com/m15o/mkdocs.git@fix-prebuild-index-ja 

[fu-sen]

Oh, it looks like my pip command was a bit missing. 😫

prebuild_index: node still returns an error with mkdocs build.
If you lose this, you will not be able to build, but you will not be able to continue searching in Japanese.

Do you know developer tools such as Chrome? You will get a detailed answer using it.

[fu-sen]

Also note that even if this is resolved, @waylan and I are aware of the problem in multiple languages and have a debate that cannot be merged.

[waylan]

This looks good. It just needs a note added to the release notes (under Other Changes and Additions to Version 1.2).

[fu-sen]

@m15o ソースの対応ありがとうございます。 Thank you for supporting the source.
これで日本の利用者はテーマを選択できるようになりました。It's great that we are now able to choose a theme.

@waylan Thank you. There will be more and more MkDocs users in Japan.