Weird words in index

[janroz]

Hello,

we have some issue with search in Czech language. We have characters like ěščřžýáíéěůú etc., and some words cannot be found.

Today I have some free time to look deeper on tntsearch and I see some weird words in our index.

For example - on web (in database) we have word "ergonomicky" but in search index there is "ergonomicki". Look at picture. There are more changes "y" => "i"

I can help with some issues.

[nticaric]

The weird words that you're experiencing are called stems, take a look here

[nticaric]

I think the solution to your problem would be to write a custom tokenizer that would replace

ěščřžýáíéěůú to escrzyaieeuu

[janroz]

I know what stems are, I have not Czech stemmer (yet). Stemming is off now. However, stem for word "ergonomiky" is not "ergonomicki" but "ergonomick".

I made custom tokenizer:

• replace special chars to ascii (č => c, ď => d etc.)
• replace special symbols ($#@*{ etc.)
• strip tags
• translate entities back to chars (í => í)

The problem was solved in this way.

Now it looks very good, works like a charm. I think relevance is better also.