RussianLightStemmer incorrectly handles the words ending with 'ее' [LUCENE-6137]

[asfimport]

Consider the forms of Russian word "синий" and the result returned by RussianLightStemmer:

синий -> син
синяя -> син
синее -> сине
синие -> син

I think the correct result should be:

синий -> син
синяя -> син
синее -> син
синие -> син

---

Migrated from LUCENE-6137 by Alexander Sofronov, 1 vote, updated Jan 01 2015
Attachments: LUCENE-6137.patch

[asfimport]

Alexander Sofronov (migrated from JIRA)

Patch attached

[asfimport]

Alexander Sofronov (migrated from JIRA)

Also, I think it should be reasonable to include "ие" ending to the list of 2-chars endings. With current implementation it is processed in 2 steps: first "е" is removed in removeCase() and then "и" removed in normalize().

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Can you propose your changes to http://members.unine.ch/jacques.savoy/clef/index.html?

Like snowball, these are just implementations of those algorithms. They have done tests and written papers and so on, and can better evaluate these changes.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Hi,
I also agree you should raise this issue at the CLEF people who invented that stemmer!

I talked with my wife (she has russian as mother language) and she can confirm your problem with some neutral adjective forms, but - as expected - she can confirm that removing "-ee" is too risky, because this would change also comparative form (also using -ee), too, which is not intended to be done by a "light stemmer". I think this might be the reason not to remove "-ee" by default (this changes meaning).

[asfimport]

Alexander Sofronov (migrated from JIRA)

OK, I sent email to Ljiljana Doalmic and Jacques Savoy. BTW, I found that Perl version of Russian stemmer (http://members.unine.ch/jacques.savoy/clef/russianStemmerPerl.txt) handles "ее" and "ие" endings properly.

[asfimport]

Alexander Sofronov (migrated from JIRA)

Just received confirmation from Jacques Savoy that this is a bug in Java version of stemmer. Waiting for the fix...

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Hi, thanks for the response!

In any case, did you send a reference to this issue? I am not sure if the problems with comparative forms should be handled.

[asfimport]

Alexander Sofronov (migrated from JIRA)

Yes, I sent link to this issue. My opinion is that comparative forms should be stemmed also. Please note that comparative forms in Russian can have not only "ее" endings, but also "ий", "ей" and some other that listed in RussianLightStemmer.

Also, please take a look and Perl version of Russian light stemmer (http://members.unine.ch/jacques.savoy/clef/russianStemmerPerl.txt) which handles "ее" and "ие" endings properly.