New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IndexOutOfBoundsException while call suggest api #3196

Closed
piaolingxue opened this Issue Jun 17, 2013 · 14 comments

Comments

Projects
None yet
5 participants
@piaolingxue

piaolingxue commented Jun 17, 2013

java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at java.util.ArrayList.rangeCheck(ArrayList.java:571)
at java.util.ArrayList.get(ArrayList.java:349)
at org.elasticsearch.search.suggest.Suggest$Suggestion.reduce(Suggest.java:251)
at org.elasticsearch.search.suggest.Suggest.reduce(Suggest.java:179)
at org.elasticsearch.action.suggest.TransportSuggestAction.newResponse(TransportSuggestAction.java:145)
at org.elasticsearch.action.suggest.TransportSuggestAction.newResponse(TransportSuggestAction.java:60)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.finishHim(TransportBroadcastOperationAction.java:369)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.onOperation(TransportBroadcastOperationAction.java:306)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:265)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:242)
at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:218)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Jun 17, 2013

Contributor

can you provide an example request?

Contributor

s1monw commented Jun 17, 2013

can you provide an example request?

@piaolingxue

This comment has been minimized.

Show comment
Hide comment
@piaolingxue

piaolingxue Jun 20, 2013

On 06/17/2013 04:51 PM, Simon Willnauer wrote:

can you provide an example request?


Reply to this email directly or view it on GitHub
#3196 (comment).

for example
run this script will throw Exception
like:java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
but if I compile elasticsearch locally, then run the es server and this
script, it will not throw exception.
/#!/bin/bash//
//
//echo "delete index[products]"//
//curl -X DELETE 0:9200/products//
//echo "create index[products]"//
//curl -X PUT 0:9200/products -d '//
//{//
// "settings" : {//
// "analysis" : {//
// "tokenizer" : {//
// "ngram_min_1" : {//
// "type" : "nGram",//
// "min_gram" : "1",//
// "max_gram" : "10"//
// }//
// },//
// "analyzer" : {//
// "ngram_min_1" : {//
// "type" : "custom",//
// "tokenizer" : "ngram_min_1"//
// }//
// }//
// }//
// },//
// "mappings" : {//
// "product" : {//
// "properties" : {//
// "ProductId": { "type": "string", "index":
"not_analyzed" },//
// "ProductName" : {//
// "type": "string", //
// "index_analyzer": "whitespace",//
// "search_analyzer" : "ngram_min_1"//
// }//
// }//
// }//
// }//
//}'//
//echo//
//echo "add docs"//
//for i in 1 2 3 4 5 6 7 8 9 10 100 101 1000; do//
// json=$(printf '{"ProductId": "%s", "ProductName": "%s" }', $i, "My
Product $i")//
// echo $json//
// curl -X PUT 0:9200/products/product/$i -d "$json"//
// echo//
//done//
//
//echo//
//echo 'seggestion with:prodct'//
//curl -XPOST '0:9200/_suggest' -d '{//
// "text" : "prodct",//
// "my-seggestion1" : {//
// "term" : {//
// "field" : "ProductName",//
// "suggest_mode" : "always"//
// }//
// }//
//}'/

piaolingxue commented Jun 20, 2013

On 06/17/2013 04:51 PM, Simon Willnauer wrote:

can you provide an example request?


Reply to this email directly or view it on GitHub
#3196 (comment).

for example
run this script will throw Exception
like:java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
but if I compile elasticsearch locally, then run the es server and this
script, it will not throw exception.
/#!/bin/bash//
//
//echo "delete index[products]"//
//curl -X DELETE 0:9200/products//
//echo "create index[products]"//
//curl -X PUT 0:9200/products -d '//
//{//
// "settings" : {//
// "analysis" : {//
// "tokenizer" : {//
// "ngram_min_1" : {//
// "type" : "nGram",//
// "min_gram" : "1",//
// "max_gram" : "10"//
// }//
// },//
// "analyzer" : {//
// "ngram_min_1" : {//
// "type" : "custom",//
// "tokenizer" : "ngram_min_1"//
// }//
// }//
// }//
// },//
// "mappings" : {//
// "product" : {//
// "properties" : {//
// "ProductId": { "type": "string", "index":
"not_analyzed" },//
// "ProductName" : {//
// "type": "string", //
// "index_analyzer": "whitespace",//
// "search_analyzer" : "ngram_min_1"//
// }//
// }//
// }//
// }//
//}'//
//echo//
//echo "add docs"//
//for i in 1 2 3 4 5 6 7 8 9 10 100 101 1000; do//
// json=$(printf '{"ProductId": "%s", "ProductName": "%s" }', $i, "My
Product $i")//
// echo $json//
// curl -X PUT 0:9200/products/product/$i -d "$json"//
// echo//
//done//
//
//echo//
//echo 'seggestion with:prodct'//
//curl -XPOST '0:9200/_suggest' -d '{//
// "text" : "prodct",//
// "my-seggestion1" : {//
// "term" : {//
// "field" : "ProductName",//
// "suggest_mode" : "always"//
// }//
// }//
//}'/

@martijnvg

This comment has been minimized.

Show comment
Hide comment
@martijnvg

martijnvg Jun 20, 2013

Member

So you build ES from source based on master or based on the 0.90 branch? What Elasticsearch version did initially use when you ran the shell script?

Member

martijnvg commented Jun 20, 2013

So you build ES from source based on master or based on the 0.90 branch? What Elasticsearch version did initially use when you ran the shell script?

@piaolingxue

This comment has been minimized.

Show comment
Hide comment
@piaolingxue

piaolingxue Jun 21, 2013

On 06/20/2013 08:05 PM, Martijn van Groningen wrote:

So you build ES from source based on master or based on the 0.90
branch? What Elasticsearch version did initially use when you ran the
shell script?


Reply to this email directly or view it on GitHub
#3196 (comment).

I build ES based on the 0.90.1 branch (refs/tags/v0.90.1)
and I firstly ran the shell script use the 0.90.1 version.

My OS is Ubuntu 12.04,have you met this problem?

piaolingxue commented Jun 21, 2013

On 06/20/2013 08:05 PM, Martijn van Groningen wrote:

So you build ES from source based on master or based on the 0.90
branch? What Elasticsearch version did initially use when you ran the
shell script?


Reply to this email directly or view it on GitHub
#3196 (comment).

I build ES based on the 0.90.1 branch (refs/tags/v0.90.1)
and I firstly ran the shell script use the 0.90.1 version.

My OS is Ubuntu 12.04,have you met this problem?

@piaolingxue

This comment has been minimized.

Show comment
Hide comment
@piaolingxue

piaolingxue Jun 21, 2013

On 06/20/2013 08:05 PM, Martijn van Groningen wrote:

So you build ES from source based on master or based on the 0.90
branch? What Elasticsearch version did initially use when you ran the
shell script?


Reply to this email directly or view it on GitHub
#3196 (comment).

and my JVM info as follows:
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

piaolingxue commented Jun 21, 2013

On 06/20/2013 08:05 PM, Martijn van Groningen wrote:

So you build ES from source based on master or based on the 0.90
branch? What Elasticsearch version did initially use when you ran the
shell script?


Reply to this email directly or view it on GitHub
#3196 (comment).

and my JVM info as follows:
java version "1.6.0_27"
OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1~12.04.1)
OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

@martijnvg

This comment has been minimized.

Show comment
Hide comment
@martijnvg

martijnvg Jun 24, 2013

Member

@piaolingxue I tried to run your script, but I don't run into the IndexOutOfBoundsException error.

Member

martijnvg commented Jun 24, 2013

@piaolingxue I tried to run your script, but I don't run into the IndexOutOfBoundsException error.

@gromgull

This comment has been minimized.

Show comment
Hide comment
@gromgull

gromgull Jul 24, 2013

I've also had this exception (v. 0.90.2), it is unfortunately not consistently reproducable, the same query will sometimes work ok, and sometimes break.

I can offer two hints though:

  • I think it only happens when there are at least two terms in the suggest text
  • I would guess it's related to using a different field/analyzer for the query and for the suggestions? My content field is a multi_field, once with a snowball stemmer (this one is queried), and once with a standard analyzer for the suggestions.

I will keep trying to make a contained test that breaks at least sometimes :)

gromgull commented Jul 24, 2013

I've also had this exception (v. 0.90.2), it is unfortunately not consistently reproducable, the same query will sometimes work ok, and sometimes break.

I can offer two hints though:

  • I think it only happens when there are at least two terms in the suggest text
  • I would guess it's related to using a different field/analyzer for the query and for the suggestions? My content field is a multi_field, once with a snowball stemmer (this one is queried), and once with a standard analyzer for the suggestions.

I will keep trying to make a contained test that breaks at least sometimes :)

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Jul 24, 2013

Contributor

@gromgull is it the same stacktrace as above?

Contributor

s1monw commented Jul 24, 2013

@gromgull is it the same stacktrace as above?

@gromgull

This comment has been minimized.

Show comment
Hide comment
@gromgull

gromgull Jul 25, 2013

Close, the underlying exception is nearly identical:

2013-07-24 11:30:03,532][DEBUG][action.search.type       ] [Ghazikhanian, Carter] failed to reduce search
org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase [fetch], [reduce]
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:177)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:155)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:149)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:346)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:149)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
        at java.util.ArrayList.RangeCheck(ArrayList.java:547)
        at java.util.ArrayList.get(ArrayList.java:322)
        at org.elasticsearch.search.suggest.Suggest$Suggestion.reduce(Suggest.java:246)
        at org.elasticsearch.search.suggest.Suggest.reduce(Suggest.java:174)
        at org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:404)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:190)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:175)
        ... 8 more

gromgull commented Jul 25, 2013

Close, the underlying exception is nearly identical:

2013-07-24 11:30:03,532][DEBUG][action.search.type       ] [Ghazikhanian, Carter] failed to reduce search
org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase [fetch], [reduce]
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:177)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:155)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:149)
        at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:346)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:149)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
        at java.util.ArrayList.RangeCheck(ArrayList.java:547)
        at java.util.ArrayList.get(ArrayList.java:322)
        at org.elasticsearch.search.suggest.Suggest$Suggestion.reduce(Suggest.java:246)
        at org.elasticsearch.search.suggest.Suggest.reduce(Suggest.java:174)
        at org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:404)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:190)
        at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:175)
        ... 8 more

@ghost ghost assigned s1monw Dec 15, 2013

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Dec 15, 2013

Contributor

Eventually I was able to reproduce this! The reason here seems to be that there is more than one index and the indices share a field with the same name but different analysis chains that produce different number of tokens. It would be great if anybody could confirm that their setup also has more than one index were at least 2 indices share the same field with different analysis chains. I added a commit to a branch with a test that reproduces this issue.

Contributor

s1monw commented Dec 15, 2013

Eventually I was able to reproduce this! The reason here seems to be that there is more than one index and the indices share a field with the same name but different analysis chains that produce different number of tokens. It would be great if anybody could confirm that their setup also has more than one index were at least 2 indices share the same field with different analysis chains. I added a commit to a branch with a test that reproduces this issue.

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Dec 15, 2013

Contributor

Added a fix for this in the commit above. IMO we should throw an exception and fail the reduce if the size is not the same. @martijnvg @imotov what do you think?

Contributor

s1monw commented Dec 15, 2013

Added a fix for this in the commit above. IMO we should throw an exception and fail the reduce if the size is not the same. @martijnvg @imotov what do you think?

@imotov

This comment has been minimized.

Show comment
Hide comment
@imotov

imotov Dec 15, 2013

Member

Looks good to me.

Member

imotov commented Dec 15, 2013

Looks good to me.

@martijnvg

This comment has been minimized.

Show comment
Hide comment
@martijnvg

martijnvg Dec 16, 2013

Member

LGTM as well.

Member

martijnvg commented Dec 16, 2013

LGTM as well.

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw Dec 16, 2013

Contributor

@martijnvg @imotov I force pushed another change that also checks the entry keys since we also need to make sure we can merge them. There is also a test that checks it. Can you take another look?

Contributor

s1monw commented Dec 16, 2013

@martijnvg @imotov I force pushed another change that also checks the entry keys since we also need to make sure we can merge them. There is also a test that checks it. Can you take another look?

s1monw added a commit to s1monw/elasticsearch that referenced this issue Dec 16, 2013

Throw IAE if suggest results return differently sized results.
If the term suggester is used the results are merged depending on
the number of terms produced by the tokenizer / tokenfilter. If a
term suggester is executed across multiple indices that share the
same field but with different analysis chains we can't merge the
result anymore sicne tokens are our of order or have a different size.

This commit throws ESIllegalArgumentException if the number of entries
are not the same across all results.

Closes #3196

s1monw added a commit that referenced this issue Dec 16, 2013

Throw IAE if suggest results return differently sized results.
If the term suggester is used the results are merged depending on
the number of terms produced by the tokenizer / tokenfilter. If a
term suggester is executed across multiple indices that share the
same field but with different analysis chains we can't merge the
result anymore sicne tokens are our of order or have a different size.

This commit throws ESIllegalArgumentException if the number of entries
are not the same across all results.

Closes #3196

@s1monw s1monw closed this in 3e32197 Dec 16, 2013

brusic added a commit to brusic/elasticsearch that referenced this issue Jan 19, 2014

Throw IAE if suggest results return differently sized results.
If the term suggester is used the results are merged depending on
the number of terms produced by the tokenizer / tokenfilter. If a
term suggester is executed across multiple indices that share the
same field but with different analysis chains we can't merge the
result anymore sicne tokens are our of order or have a different size.

This commit throws ESIllegalArgumentException if the number of entries
are not the same across all results.

Closes #3196

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Throw IAE if suggest results return differently sized results.
If the term suggester is used the results are merged depending on
the number of terms produced by the tokenizer / tokenfilter. If a
term suggester is executed across multiple indices that share the
same field but with different analysis chains we can't merge the
result anymore sicne tokens are our of order or have a different size.

This commit throws ESIllegalArgumentException if the number of entries
are not the same across all results.

Closes #3196
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment