Search function multi-language support

[netcmcc]

Other aspects of this theme are good for multi-language support, but the search function currently only supports English, and I hope to be able to extend the search function to support beyond English, such as Chinese support.

[netcmcc]

The title and directory support Chinese, but searching for Chinese is not working properly.

[alex-shpak]

Hi!
Looks like lunr.js, which is used for search, does not have support for Chinese olivernn/lunr.js#173

I will need to check what can be done.
This might be an alternative https://github.com/nextapps-de/flexsearch

[alex-shpak]

I tried flexsearch and looks like it works with this configuration
https://github.com/nextapps-de/flexsearch#cjk-word-break-chinese-japanese-korean

Unfortunately it's either chinese or english (or other language) not both at the same time.
Or I didn't find correct way to have both.

[oshliaer]

Russian search is not supported too.

@alex-shpak Any recommendations. How can I help?

[alex-shpak]

Hi!
I pushed changes to master. It introduces https://github.com/nextapps-de/flexsearch in replace for lunr.js. FlexSearch has more configuration options for multi-language support.

There is now BookSearchConfig parameter which is flexsearch configuration object.
So for example for chinese, accroding to this https://github.com/nextapps-de/flexsearch#cjk-word-break-chinese-japanese-korean

BookSearchConfig = '''{
  encode: false,
  tokenize: function(str){
    return str.replace(/[\x00-\x7F]/g, "").split("");
  }
}'''

For russian I think stemmer needs to be set
https://github.com/nextapps-de/flexsearch#add-language-specific-stemmer-andor-filter

Future work will include integration with multi-lang mode, having different configuration for indexing per language.
Unfortunately there is no easy way to make support for multiple languages in same index.

[alex-shpak]

I think this config should work for russian, filter is optional.

BookSearchConfig = '''{
  split: /[^a-zа-яё0-9]/gi,
  filter: [ 
    "в", "на", "и", "не", "о", "от", "с"
  ]
}'''

[alex-shpak]

Changes has been merged to master

[kevinclcn]

Hi!
I pushed changes to master. It introduces https://github.com/nextapps-de/flexsearch in replace for lunr.js. FlexSearch has more configuration options for multi-language support.

There is now BookSearchConfig parameter which is flexsearch configuration object.
So for example for chinese, accroding to this https://github.com/nextapps-de/flexsearch#cjk-word-break-chinese-japanese-korean

BookSearchConfig = '''{
  encode: false,
  tokenize: function(str){
    return str.replace(/[\x00-\x7F]/g, "").split("");
  }
}'''

For russian I think stemmer needs to be set
https://github.com/nextapps-de/flexsearch#add-language-specific-stemmer-andor-filter

Future work will include integration with multi-lang mode, having different configuration for indexing per language.
Unfortunately there is no easy way to make support for multiple languages in same index.

I worked around this issue by below config:

    {
      encode: false,
      tokenize: function(str) {
        return str.split(/\W+/).concat(str.replace(/[\x00-\x7F]/g, '').split('')).filter(e => !!e)
      }
    }

[peter-liu]

never mind, it's wrong （only works when you search upper case)