Hello,

I was trying to create czech lemmatisation analyzer using stemmer override filter and czech dictionary from aspell.

This dictionary contains around 300 000 words in base form and some suffix/prefix rules. After expansion format file looks like this

Aakjaer Aakjaerech Aakjaery Aakjaerům Aakjaerů Aakjaerem Aakjaere Aakjaerovi Aakjaeru Aakjaera Aakjaerové
Aakjaerová Aakjaerovými Aakjaerovým Aakjaerových Aakjaerovou Aakjaerové
Aakjaerův Aakjaerovýma Aakjaerovými Aakjaerových Aakjaerovou Aakjaerovo Aakjaerovy Aakjaerovi Aakjaerovým 

each line is one word with its forms.

Because of rules format form => lemma the final rule set is expanded from 300 000 to 4 364 674 lines.

When I was trying on my local machine to index czech wikipedia pages (around 400 000 documents) java.lang.OutOfMemoryError: Java heap space error occured after approx 10 minutes of indexing (log file here)

I'm using snapshot build of elasticsearch (54e7e309a5d407b2fb1123a79e6af9d62e41ea1e), JAVA_OPTS -Xss200000 -Xms2g -Xmx2g with no other indices.

Index settings/mapping and river settings are in separate gist

I was trying to achieve this functionality using synonym token filter, because of better format of synonym rules - form1, form2, form3, form4 => lemma (so number of rules are only about 300 000).

But it's not the same. In the case of using stemmer override filter, when token was not found in rule set, stemmer was used. I probably can do the same by adding keyword marker and stemmer in the filter chain, but I don't think it is the right way to do that.

Please, is there some better 'compressed' format of stemmer override filter rules? Any thoughts how to avoid java.lang.OutOfMemoryError: Java heap space error?