5.0.2 plugin analysis-stempel Array index out of range: -2 #21911

crash2k · 2016-12-01T13:44:29Z

Elasticsearch version: 5.0.2
Plugins installed: analysis-stempel 5.0,2
JVM version: 1.8.0_111
OS version: Ubuntu 14.04.1 LTS

I have problem with indexing data when analysis-stempel is set as fields analyser.

While indexing documents at random receives an error
Array index out of range: -1
Array index out of range: -2

Trace from log:

java.lang.ArrayIndexOutOfBoundsException: Array index out of range: -2
        at java.util.Arrays.rangeCheck(Arrays.java:117) ~[?:1.8.0_111]
        at java.util.Arrays.fill(Arrays.java:2973) ~[?:1.8.0_111]
        at java.lang.AbstractStringBuilder.setLength(AbstractStringBuilder.java:211) ~[?:1.8.0_111]
        at java.lang.StringBuilder.setLength(StringBuilder.java:76) ~[?:1.8.0_111]
        at org.apache.lucene.analysis.stempel.StempelStemmer.stem(StempelStemmer.java:91) ~[?:?]
        at org.apache.lucene.analysis.stempel.StempelFilter.incrementToken(StempelFilter.java:74) ~[?:?]
        at org.apache.lucene.analysis.shingle.ShingleFilter.getNextToken(ShingleFilter.java:390) ~[lucene-analyzers-common-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:51]
        at org.apache.lucene.analysis.shingle.ShingleFilter.shiftInputWindow(ShingleFilter.java:467) ~[lucene-analyzers-common-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:51]
        at org.apache.lucene.analysis.shingle.ShingleFilter.incrementToken(ShingleFilter.java:308) ~[lucene-analyzers-common-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:51]
        at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:714) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1562) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1307) ~[lucene-core-6.2.1.jar:6.2.1 43ab70147eb494324a1410f7a9f16a896a59bc6f - shalin - 2016-09-15 05:15:20]
        at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:558) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:520) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:409) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:191) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:351) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.index(TransportShardBulkAction.java:158) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:137) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:123) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:74) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:78) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:50) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:903) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:873) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:319) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:254) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:839) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:836) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1655) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:848) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:271) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:250) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:242) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:550) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.2.jar:5.0.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

The text was updated successfully, but these errors were encountered:

MaineC · 2016-12-01T14:01:58Z

It would be really helpful if you could try and identify exactly which of your documents is causing this issue.

crash2k · 2016-12-01T14:32:54Z

I get this error for random documents, each time is a different content :/

sample:

[2016-12-01T14:04:23,782][DEBUG][o.e.a.b.TransportShardBulkAction] [es5] [666.pb1][0] failed to execute bulk item (index) index {[666.pb1][document][224], source[{"author":{"id":580,"name":"Waldemar G"},"indexes":[{"id":19,"name":"K
si▒.gi rachunkowe","parent_name":"Rachunkowo▒.▒. i Finanse","ancestors":[{"id":1,"name":"www"},{"id":17,"name":"Rachunkowo▒.▒. i Finanse"}]}],"keywords":[{"name":"inwentaryzacja ▒.rodków trwa▒.ych","id":420},{"name":"cz▒.stotliwo▒.▒.
inwentaryzacji ▒.rodków trwa▒.ych","id":516}],"content":" Pytanie: Zak▒.ad posiada monitoring. Czy z tego wzgl▒.du spis inwentarzowy ▒.rodków trwa▒.ych i pozosta▒.ych ▒.rodków można przeprowadzi▒. raz na 4 lata, czy co roku? Cz
y musi by▒. zarz▒.dzenie dyrektora, np. że inwentaryzacj▒. si▒. przeprowadza raz na 4 lata? Odpowiedź: W opisanej sytuacji do kierownika jednostki należy podj▒.cie decyzji w sprawie cz▒.stotliwo▒.ci przeprowadzania inwentaryza
cji ▒.rodków trwa▒.ych. Odpowiadaj▒.c na pytanie należy podkre▒.li▒., że nie ma przepisów definiuj▒.cych poj▒.cie ▒..teren strzeżony▒... Rozdzia▒. 3 uor przedstawia jedynie ogólne uregulowania dotycz▒.ce inwentaryzacji. W zwi▒.
zku z tym podj▒.cie decyzji w sprawie cz▒.stotliwo▒.ci przeprowadzania inwentaryzacji ▒.rodków trwa▒.ych znajduj▒.cych si▒. na terenie strzeżonym należy do kierownika jednostki, co powinno by▒. okre▒.lone w Pa▒.stwa instrukcji inwe
ntaryzacyjnej lub w zarz▒.dzeniu dyrektora (kierownika jednostki). Brak okre▒.lenia wewn▒.trznych przepisów w zakresie inwentaryzacji narusza wymagania art. 10 uor. Wi▒.cej odno▒.nie inwentaryzacji przeczyta Pan w ABC Ksi▒.goweg
o » Inwentaryzacja Podstawa prawna: - rozdzia▒. 3, art. 10 ustawy z 29 wrze▒.nia 1994 roku o rachunkowo▒.ci (Dz.U. z 2009 r. nr 152, poz. 1223 ze zm.) - uor. Tekst opublikowany: 26 marca 2012 r. ","lead":" Pytanie: Za
k▒.ad posiada monitoring. Czy z tego wzgl▒.du spis inwentarzowy ▒.rodków trwa▒.ych i pozosta▒.ych ▒.rodków można przeprowadzi▒. raz na 4 lata, czy co roku? Czy musi by▒. zarz▒.dzenie dyrektora, np. że inwentaryzacj▒. si▒. przepr9,"na
owadza raz na 4 lata? ","seoDescription":"Rozdzia▒. 3 uor przedstawia jedynie ogólne uregulowania dotycz▒.ce inwentaryzacji. Jak cz▒.sto zatem przeprowadza▒. inwentaryzacj▒.?","seoTitle":"Cz▒.stotliwo▒.▒. inwentaryzacji","title":"Dec:
yzja w sprawie cz▒.stotliwo▒.ci przeprowadzania inwentaryzacji ▒.rodków trwa▒.ych","createdDate":"2012-04-30","publicationDate":"2015-03-26","validFrom":"2012-05-07","validTo":"2021-11-30","docType":"2","idLanguage":"1","isPaid":"0"}
]}
java.lang.ArrayIndexOutOfBoundsException: Array index out of range: -2

clintongormley · 2016-12-01T14:59:53Z

@mikemccand does this look like a bug in Lucene?

modmac · 2017-01-13T06:10:51Z

This problem occurs only with BULK operations. Single update, index requests work fine.

Another exception is StringIndexOutOfBoundsException:

java.lang.StringIndexOutOfBoundsException: String index out of range: 4
        at java.lang.AbstractStringBuilder.getChars(AbstractStringBuilder.java:385) ~[?:1.8.0_111]
        at java.lang.StringBuilder.getChars(StringBuilder.java:76) ~[?:1.8.0_111]
        at org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl.append(CharTermAttributeImpl.java:183) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
        at org.apache.lucene.analysis.stempel.StempelFilter.incrementToken(StempelFilter.java:76) ~[?:?]
        at org.apache.lucene.analysis.synonym.SynonymFilter.parse(SynonymFilter.java:358) ~[lucene-analyzers-common-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:43]
        at org.apache.lucene.analysis.synonym.SynonymFilter.incrementToken(SynonymFilter.java:624) ~[lucene-analyzers-common-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:43]
        at org.apache.lucene.analysis.miscellaneous.UniqueTokenFilter.incrementToken(UniqueTokenFilter.java:53) ~[elasticsearch-5.1.1.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:43]
        at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:712) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
        at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:417) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
        at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:373) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:272) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
        at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:433) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
        at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1375) ~[lucene-core-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:47:11]
        at org.elasticsearch.index.engine.InternalEngine.update(InternalEngine.java:564) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.index.engine.InternalEngine.innerIndex(InternalEngine.java:522) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:409) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:556) ~[elasticsearch-5.1.1.jar:5.1.1]

        at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:546) ~[elasticsearch-5.1.1.jar:5.1.1]

        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:191) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:348) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardUpdateOperation(TransportShardBulkAction.java:411) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.update(TransportShardBulkAction.java:233) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.handleItem(TransportShardBulkAction.java:138) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:120) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.onPrimaryShard(TransportShardBulkAction.java:73) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:75) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportWriteAction.shardOperationOnPrimary(TransportWriteAction.java:48) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:905) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:875) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:113) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:323) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:258) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:855) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:852) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:142) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1655) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:864) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction.access$400(TransportReplicationAction.java:90) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:275) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:254) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:246) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$1(SecurityServerTransportInterceptor.java:208) ~[?:?]
        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$2(SecurityServerTransportInterceptor.java:246) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:53) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$null$0(ServerTransportFilter.java:135) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:131) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:125) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:113) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:137) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:53) ~[elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:180) ~[x-pack-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$2(AuthenticationService.java:199) ~[x-pack-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:211) [x-pack-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:178) [x-pack-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:140) [x-pack-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:112) [x-pack-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:131) [x-pack-5.1.1.jar:5.1.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:253) [x-pack-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:577) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.1.1.jar:5.1.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

modmac · 2017-01-13T09:40:20Z

The problem still occurs on:

Elasticsearch version: 5.1.2
Plugins installed: analysis-stempel 5.1.2

mikemccand · 2017-01-13T10:15:06Z

I'll try to get to the bottom of this. Can you share your full analysis chain? From the stack traces I see at least shingle and synonym filter after the stempel stemmer, but what tokenizers/filters come before it?

mikemccand · 2017-01-13T10:47:03Z

OK, nevermind on the full analysis chain @Mod3st, I see the issue...

There is a thread safety bug in the TokenFilterFactory: https://github.com/elastic/elasticsearch/blob/master/plugins/analysis-stempel/src/main/java/org/elasticsearch/index/analysis/pl/PolishStemTokenFilterFactory.java#L48

That stemmer cannot be shared across threads, but it is here. I'll open a PR to fix it to make a new stemmer in each create call like Lucene's factory.

modmac · 2017-01-13T11:17:19Z

I understand that in the previous version of ES every thread was creating a new instance of PolishStemTokenFilterFactory and now each thread uses the same object?

(the problem doesn't occur on ES 2)

mikemccand · 2017-01-13T11:26:09Z

Hmm, @Mod3st I still see the bug in ES 2.4.x's latest sources (the same stemmer is shared for all calls to PolishStemTokenFilterFactory.create, so as far as I can tell the bug should exist in ES 2.4 as well. Do you not see that?

https://github.com/elastic/elasticsearch/blob/2.4/plugins/analysis-stempel/src/main/java/org/elasticsearch/index/analysis/pl/PolishStemTokenFilterFactory.java#L48

modmac · 2017-01-13T11:32:12Z

I understand, but I don't get any exceptions in ES 2.4.3 and I wonder why.

Closes #21911

clintongormley added :Search/Analysis How text is split into tokens :Plugin Analysis Stempel help wanted adoptme >bug labels Dec 1, 2016

mikemccand self-assigned this Jan 13, 2017

mikemccand removed the help wanted adoptme label Jan 13, 2017

mikemccand mentioned this issue Jan 13, 2017

Fix thread safety of Stempel's token filter factory #22610

Merged

mikemccand pushed a commit that referenced this issue Jan 16, 2017

Fix thread safety of Stempel's token filter factory (#22610)

e28c678

Closes #21911

mikemccand pushed a commit that referenced this issue Jan 16, 2017

Fix thread safety of Stempel's token filter factory (#22610)

ef61ec0

Closes #21911

mikemccand closed this as completed in eea4db5 Jan 16, 2017

lcawl added :Search/Analysis How text is split into tokens and removed :Plugin Analysis Stempel labels Feb 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

5.0.2 plugin analysis-stempel Array index out of range: -2 #21911

5.0.2 plugin analysis-stempel Array index out of range: -2 #21911

crash2k commented Dec 1, 2016 •

edited by clintongormley

MaineC commented Dec 1, 2016

crash2k commented Dec 1, 2016

clintongormley commented Dec 1, 2016

modmac commented Jan 13, 2017 •

edited

modmac commented Jan 13, 2017 •

edited

mikemccand commented Jan 13, 2017

mikemccand commented Jan 13, 2017

modmac commented Jan 13, 2017 •

edited

mikemccand commented Jan 13, 2017

modmac commented Jan 13, 2017

5.0.2 plugin analysis-stempel Array index out of range: -2 #21911

5.0.2 plugin analysis-stempel Array index out of range: -2 #21911

Comments

crash2k commented Dec 1, 2016 • edited by clintongormley

MaineC commented Dec 1, 2016

crash2k commented Dec 1, 2016

clintongormley commented Dec 1, 2016

modmac commented Jan 13, 2017 • edited

modmac commented Jan 13, 2017 • edited

mikemccand commented Jan 13, 2017

mikemccand commented Jan 13, 2017

modmac commented Jan 13, 2017 • edited

mikemccand commented Jan 13, 2017

modmac commented Jan 13, 2017

crash2k commented Dec 1, 2016 •

edited by clintongormley

modmac commented Jan 13, 2017 •

edited

modmac commented Jan 13, 2017 •

edited

modmac commented Jan 13, 2017 •

edited