Elasticsearch issue when search Thai language

[herdianabdillah]

Describe the bug
unable to search with some Thai word. this is example sentence i have in my FAQ body ปรับการปฏิบัติงานโดยรวม
it can find the FAQ with first 5 character, but when i try with last 5 character the FAQ not found.

To Reproduce
Steps to reproduce the behavior:

1. Go to Home page -> change to Thai Language
2. paste the last 5 character in search column and the FAQ not found

Expected behavior
FAQ should be found even with first 5 character or last five character as long as it contain the word

Screenshots
this is the FAQ

this when i search with first 5 character

this when i search with last 5 character

phpMyFAQ (please complete the following information):

• phpMyFAQ version : 3.0.4
• PHP version : 7
• Database : MySQL
• Elasticsearch yes/no - Yes
• Elasticsearch version : 6.8.12

[thorsten]

Does the Thai language have spaces between the words?

[thorsten]

Looks like you need a different Tokenizer for the Thai language: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-thai-tokenizer.html

[herdianabdillah]

Does the Thai language have spaces between the words?

Seems like Thai language not have spaces between the words

[herdianabdillah]

Looks like you need a different Tokenizer for the Thai language: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-thai-tokenizer.html

so Should i drop the current index?
and then in this function, should i change tokenizer to array? and add thai inside that? because i still want able to search other language like china, japan and etc

private function getParams()
{
global $PMF_ELASTICSEARCH_STEMMING_LANGUAGE;

    return [
        'index' => $this->esConfig['index'],
        'body' => [
            'settings' => [
                'number_of_shards' => PMF_ELASTICSEARCH_NUMBER_SHARDS,
                'number_of_replicas' => PMF_ELASTICSEARCH_NUMBER_REPLICAS,
                'analysis' => [
                    'filter' => [
                        'autocomplete_filter' => [
                            'type' => 'edge_ngram',
                            'min_gram' => 1,
                            'max_gram' => 20
                        ],
                        'Language_stemmer' => [
                            'type' => 'stemmer',
                            'name' => $PMF_ELASTICSEARCH_STEMMING_LANGUAGE[$this->config->getDefaultLanguage()]
                        ]
                    ],
                    'analyzer' => [
                        'autocomplete' => [
                            'type' => 'custom',
                            'tokenizer' => 'standard',
                            'filter' => [
                                'lowercase',
                                'autocomplete_filter',
                                'Language_stemmer'
                            ]
                        ]
                    ]
                ]
            ]
        ]
    ];
}

[thorsten]

I think this Tokenizer would be the best: https://www.elastic.co/guide/en/elasticsearch/plugins/7.10/analysis-icu-tokenizer.html

[thorsten]

So, it would be cool if you could test it. Use the ICU Tokenizer, drop the index and re-create it and re-index the content.

[herdianabdillah]

I think this Tokenizer would be the best: https://www.elastic.co/guide/en/elasticsearch/plugins/7.10/analysis-icu-tokenizer.html

Okay so i just need change that "standard" to "icu_tokenizer" right? let me try it

[thorsten]

Yes, seems so :-)

[herdianabdillah]

Hi Thorsten,

i got this error when i click create index


Fatal error: Uncaught Elasticsearch\Common\Exceptions\BadRequest400Exception: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Custom Analyzer [autocomplete] failed to find tokenizer under name [icu_tokenizer]"}],"type":"illegal_argument_exception","reason":"Custom Analyzer [autocomplete] failed to find tokenizer under name [icu_tokenizer]"},"status":400} in /var/www/html/phpMyFAQ/src/libs/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:632
Stack trace:
#0 /var/www/html/phpMyFAQ/src/libs/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(317): Elasticsearch\Connections\Connection->process4xxError(Array, Array, Array)
#1 /var/www/html/phpMyFAQ/src/libs/react/promise/src/FulfilledPromise.php(28): Elasticsearch\Connections\Connection->Elasticsearch\Connections{closure}(Array)
#2 /var/www/html/phpMyFAQ/src/libs/ezimuel/ringphp/src/Future/CompletedFutureValue.php(55): React\Promise\FulfilledPromise->then(Object(Closure), NULL, NULL)
#3 /var/www/html/phpMyFAQ in /var/www/html/phpMyFAQ/src/libs/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php on line 632

[thorsten]

Ah, sorry, looks like, you have to install the ICU support via plugins:

sudo bin/elasticsearch-plugin install analysis-icu

[herdianabdillah]

Done install the ICU support, success create index -> full import
trying search again. but it still not working

[thorsten]

Could you please try this example?

https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-thai-tokenizer.html

[herdianabdillah]

change the tokenizer to "thai" the condition still same. can search the 5 first character but cant find with the last 5 character

[thorsten]

Did you try the example mentioned in the Elasticsearch documentation?

[herdianabdillah]

Hi Thorsten, in the elasticsearch documentaion for thai tokenizer, there is no example configuration but when check another tokenizer configuration the point is change the tokenizer value to the what we want to use right? correct me if imwrong because im not really good at elasticsearch

[thorsten]

I'll try to reproduce it on my test installation.

[thorsten]

I tried it with the ICU Tokenizer on my local v3.1 installation using Elasticsearch 7.10. The search worked for

• the full string "ปรับการปฏิบัติงานโดยรวม"
• the string "รับการปฏิบัติงานโดยรวม"
• the string "บการปฏิบัติงานโดยรวม"

Removing one more character results in an empty search result.

[herdianabdillah]

can you try with 5 - 8 character only? from the last character. so in latest development version and latest elasticsearch its working fine with full string, and remove 1 or 2 character from the full string. and it will also working like that in 3.0.8 release later?

[thorsten]

I moved it to the next version as I need the possibility for the users to configure which tokenizer will be used. So we need a new configuration to handle this.

[thorsten]

The v3.1 release will be a drop in replacement for 3.0, so no need to change the templates.

[herdianabdillah]

Thank you, cant wait for the new version :)

also i found another issue with DB MSSQL, but i already fix it in the code and from data type in the table, also i dont know if thats because the collation or not. because in SQL Server i use we dont have any suffix _utf8. i just use the default collation.

and this is example issue, we cant insert chinese, japan, thai character in MSSQL, because data type is "varchar". i change it into "nvarchar" also in the query i add N before string. should i open new issue for this?

[thorsten]

@herdianabdillah a new issue would be awesome. I would change the CREATE TABLE statement for MS SQL.