search in Japanese doesn't work

[waka4g]

Contribution guidelines

• I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the problem doesn't occur with the mkdocs or readthedocs themes
• ... the problem persists when all overrides are removed, i.e. custom_dir, extra_javascript and extra_css
• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Search in Japanese doesn't work. When I type characters in search box, error message "invalid query" displays in browser console.

Expected behaviour

Search in Japanese works propery as well as other language as attached (theme language "en").

Actual behaviour

When I type characters in search box, error message "invalid query" displays in browser console, and nothing found.

Steps to reproduce

1. Install mkdocs-material.
2. mkdocs serve

Package versions

• Python: 3.8.12
• MkDocs: version 1.2.3
• Material: Version: 7.3.3

Configuration

site_name: My Docs
theme:
  name: material
  language: ja

nav:
  - Home: index.md

plugins:
  - search:
      lang: ja

System information

• Operating system: Windows 10
• Browser: Chrome 94.0.4606.8

[squidfunk]

Thanks for reporting. Fixed in 7725e555f.

The problem stems from the incompatibility of the tokenizer provided by lunr-languages and the new tokenizer of Material for MkDocs. I think I managed to workaround the issue for now, moving the segmenting process into the tokenizer, but this definitely needs some work. I'll revisit the integration of the segmenting process into the pipeline at a later time. The search now seems to work quite well I think.

[squidfunk]

Released as part of 7.3.4+insiders-3.1.4 – if the issue persists, please reopen.