Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch hangs using phrase suggester with collate option #9377

Closed
silvestrelosada opened this issue Jan 21, 2015 · 9 comments

Comments

@silvestrelosada
Copy link

commented Jan 21, 2015

Hi I have elasticsearch installation single node with 100 documents, making simultaneously requests to phrase suggester with collate option blocks completely elastic search. And does gives the option to do any thing else.

Here is my query

{  
   "query":{  
      "bool":{  
         "should":[  
            {  
               "prefix":{  
                  "label":"michael"
               }
            },
            {  
               "prefix":{  
                  "synonym":"michael"
               }
            }
         ]
      }
   },
   "explain":false,
   "suggest":{  
      "spell_ngram":{  
         "text":"michael",
         "phrase":{  
            "field":"spell_ngram",
            "size":1,
            "confidence":0.0,
            "max_errors":0.9,
            "gram_size":5,
            "collate":{  
               "query":{ "match" : {
                   "spell_ngram" : {
                        "query" : "{{suggestion}}", "type" : "phrase"
                            }  
                    }
                }
            }
         }
      }
   }
}

And here is my mappings.

{
"concept":{
    "properties" :{
            "_id":{ "type" : "string", "index" : "not_analyzed"},
            "_type":{ "type" : "string", "index" : "not_analyzed"},
            "label": {
            "type" : "multi_field",
                "fields" : {
                    "label" : {"type" : "string", "index" : "not_analyzed","copy_to" : ["spell", "spell_ngram"],"store" : true},
                    "label_analysis" : {"type" : "string", "index" : "analyzed","store" : true,"term_vector": "with_positions_offsets_payloads"},
                    "phonetic_label" : { "type" : "string", "index" : "analyzed", "index_analyzer":"phonetic_analyzer_index", "search_analyzer":"phonetic_analyzer_search","store" : true,"term_vector": "with_positions_offsets_payloads" }
                }
             },
            "synonym" : {
                "type" : "multi_field",
                "fields" : {
                    "synonym" : {"type" : "string", "index" : "not_analyzed","copy_to" : ["spell", "spell_ngram"],"store" : true,"term_vector": "with_positions_offsets_payloads"},
                    "synonym_analysis" : {"type" : "string", "index" : "analyzed" ,"store" : true},
                    "phonetic_synonym" : { "type" : "string", "index" : "analyzed",  "index_analyzer":"phonetic_analyzer_index", "search_analyzer":"phonetic_analyzer_search","store":true,"term_vector": "with_positions_offsets_payloads"  }
                },
           "spell_ngram" : { "type" : "string", "index" : "analyzed", "analyzer":"shingle_analyzer","store":true, "term_vector": "with_positions_offsets_payloads" }
        }
     }
   }
}
@clintongormley

This comment has been minimized.

Copy link
Member

commented Jan 21, 2015

Hi @silvestrelosada

Please could you provide the simplest complete recreation of the problem, so that we can try it out:

  • you're only running suggestions on one field, but you've provided the mapping for all fields
  • you haven't provided the index settings (including the analysis section)
  • you haven't provided an example document

thanks

@silvestrelosada

This comment has been minimized.

Copy link
Author

commented Jan 21, 2015

Hi here is the data needed to reproduce the error, mapping file, elasticsarch.yml, and data

https://gist.github.com/silvestrelosada/eb32afdc8c971504e45c

To reproduce it you have to send seval concurrent queries, Im using jmeter.

Best

@silvestrelosada

This comment has been minimized.

Copy link
Author

commented Jan 22, 2015

If it is helps the issue takes place on multi shard enviroment in fetchMatchingDocCountResponses method when executing the search.

@seyart

This comment has been minimized.

Copy link

commented Jan 26, 2015

Hi,

I think I have the same issue. ES hangs when I send multiple simultaneous suggest requests with collate option to the search API. I tested the version 1.37 and 1.42 of ES. ES is running with one node and the default settings (5 shards per index, but 0 replica). There are about 19 000 documents in the target index.

To make ES hang, I created a script which run a lot of suggest requests with random text. When I launch 2 or more instances of this script, depending on the hardware configuration of my testing environment, ES hangs and my ES client (elasticsearch-py) raises timeout exceptions. Then, I'm unable to make curl request to the search API (it's keep waiting for a response). Index API and bulk API are still running fine.
Using BigDesk to monitor, there is no memory issue. The search thread pools are also fine but the queue size increases after it hangs.

ES log gives nothing.

Here is an exemple of a request :

{
    "aggregations": {
        "brand": {
            "terms": {
                "field": "brand.untouched",
                "size": 0
            }
        },
        "categories": {
            "terms": {
                "field": "categories_parents_catalog_codes",
                "size": 0
            }
        },
        "offers": {
            "terms": {
                "field": "offers.untouched",
                "size": 0
            }
        },
        "stores": {
            "terms": {
                "field": "categories_parents_catalog_codes",
                "size": 0
            }
        }
    },
    "fields": [
        "full_title"
    ],
    "from": 0,
    "query": {
        "filtered": {
            "filter": {
                "term": {
                    "categories_parents_catalog_codes": "root"
                }
            },
            "query": {
                "match": {
                    "full_title.autocomplete": {
                        "operator": "and",
                        "query": "B"
                    }
                }
            }
        }
    },
    "size": 20,
    "suggest": {
        "suggestion_phrase": {
            "phrase": {
                "collate": {
                    "preference": "_primary",
                    "query": {
                        "filtered": {
                            "filter": {
                                "term": {
                                    "categories_parents_catalog_codes": "root"
                                }
                            },
                            "query": {
                                "match": {
                                    "full_title.autocomplete": {
                                        "operator": "and",
                                        "query": "{{suggestion}}"
                                    }
                                }
                            }
                        }
                    }
                },
                "field": "full_title.suggestion"
            }
        },
        "text": "B"
    }
}

Here is the mapping of the suggest field :

            "full_title": {
                "type": "string",
                "fields": {
                    "french": {
                        "type": "string",
                        "analyzer": "custom_french"
                    },
                    "suggestion": {
                        "type": "string",
                        "analyzer": "custom_suggestion"
                    },
                    "autocomplete": {
                        "type": "string",
                        "index_analyzer": "nGram_analyzer",
                        "search_analyzer": "whitespace_analyzer"
                    }
                }
            }

I remain available for any further information.

Best Regards,

@clintongormley

This comment has been minimized.

Copy link
Member

commented Jan 26, 2015

@areek could you take a look at this please

@jpountz jpountz removed the discuss label Feb 6, 2015
@s1monw

This comment has been minimized.

Copy link
Contributor

commented Feb 6, 2015

@areek can you try to write a test to reproduce this?

areek added a commit to areek/elasticsearch that referenced this issue Apr 21, 2015
@s1monw

This comment has been minimized.

Copy link
Contributor

commented May 4, 2015

one idea that I have here is that it seem it only happens if we specify the suggest together with a search and that means the suggestion as well as the collation is executed on the search threadpool @areek can you try to write a test that does the same and try to forcefully reduce the number of threads in the search threadpool to a small number ie 1 or 2? I guess that could trigger the issue...

@areek

This comment has been minimized.

Copy link
Contributor

commented May 13, 2015

I have been able to reproduce this issue.
Collate option internally fires off a search request, when used from the _search API, the internal request uses the same search threadpool used by the search request, hence if all threads are busy, it causes a deadlock.

One solution to avoid this issue would be to execute the collate query on only the local shard from which the suggestions are generated instead of collating on non-local shards. As suggestions are generated from the terms of the local shard, in most cases a generated suggestion which does not yield a hit for the collate query on the local shard would not yield a hit for collate query on non-local shards. Thoughts? I will create a PR for this.

@s1monw

This comment has been minimized.

Copy link
Contributor

commented May 13, 2015

One solution to avoid this issue would be to execute the collate query on only the local shard from which the suggestions are generated instead of collating on non-local shards. As suggestions are generated from the terms of the local shard, in most cases a generated suggestion which does not yield a hit for the collate query on the local shard would not yield a hit for collate query on non-local shards. Thoughts? I will create a PR for this.

I like this a lot - i think it's the right tradeoff

areek added a commit to areek/elasticsearch that referenced this issue May 13, 2015
…al shard.

Previously, collate feature would be executed on all shards of an index using the client,
this leads to a deadlock when concurrent collate requests are run from the _search API,
due to the fact that both the external request and internal collate requests use the
same search threadpool.

As phrase suggestions are generated from the terms of the local shard, in most cases the
generated suggestion, which does not yield a hit for the collate query on the local shard
would not yield a hit for collate query on non-local shards.

This commit removes the ability to specify a `preference` for a collate query, as the
collate query is only run on the local shard.

closes elastic#9377
areek added a commit to areek/elasticsearch that referenced this issue May 14, 2015
…al shard.

Previously, collate feature would be executed on all shards of an index using the client,
this leads to a deadlock when concurrent collate requests are run from the _search API,
due to the fact that both the external request and internal collate requests use the
same search threadpool.

As phrase suggestions are generated from the terms of the local shard, in most cases the
generated suggestion, which does not yield a hit for the collate query on the local shard
would not yield a hit for collate query on non-local shards.

Instead of using the client for collating suggestions, collate query is executed against
the ContextIndexSearcher. This PR removes the ability to specify a preference for a collate
query, as the collate query is only run on the local shard.

closes elastic#9377
areek added a commit to areek/elasticsearch that referenced this issue May 14, 2015
…al shard.

Previously, collate feature would be executed on all shards of an index using the client,
this leads to a deadlock when concurrent collate requests are run from the _search API,
due to the fact that both the external request and internal collate requests use the
same search threadpool.

As phrase suggestions are generated from the terms of the local shard, in most cases the
generated suggestion, which does not yield a hit for the collate query on the local shard
would not yield a hit for collate query on non-local shards.

Instead of using the client for collating suggestions, collate query is executed against
the ContextIndexSearcher. This PR removes the ability to specify a preference for a collate
query, as the collate query is only run on the local shard.

closes elastic#9377
areek added a commit to areek/elasticsearch that referenced this issue May 14, 2015
…al shard.

Previously, collate feature would be executed on all shards of an index using the client,
this leads to a deadlock when concurrent collate requests are run from the _search API,
due to the fact that both the external request and internal collate requests use the
same search threadpool.

As phrase suggestions are generated from the terms of the local shard, in most cases the
generated suggestion, which does not yield a hit for the collate query on the local shard
would not yield a hit for collate query on non-local shards.

Instead of using the client for collating suggestions, collate query is executed against
the ContextIndexSearcher. This PR removes the ability to specify a preference for a collate
query, as the collate query is only run on the local shard.

closes elastic#9377
areek added a commit to areek/elasticsearch that referenced this issue May 14, 2015
…al shard.

Previously, collate feature would be executed on all shards of an index using the client,
this leads to a deadlock when concurrent collate requests are run from the _search API,
due to the fact that both the external request and internal collate requests use the
same search threadpool.

As phrase suggestions are generated from the terms of the local shard, in most cases the
generated suggestion, which does not yield a hit for the collate query on the local shard
would not yield a hit for collate query on non-local shards.

Instead of using the client for collating suggestions, collate query is executed against
the ContextIndexSearcher. This PR removes the ability to specify a preference for a collate
query, as the collate query is only run on the local shard.

closes elastic#9377
@areek areek closed this in 7efc43d May 14, 2015
@kevinkluge kevinkluge removed the in progress label May 14, 2015
areek added a commit that referenced this issue May 14, 2015
…al shard.

Previously, collate feature would be executed on all shards of an index using the client,
this leads to a deadlock when concurrent collate requests are run from the _search API,
due to the fact that both the external request and internal collate requests use the
same search threadpool.

As phrase suggestions are generated from the terms of the local shard, in most cases the
generated suggestion, which does not yield a hit for the collate query on the local shard
would not yield a hit for collate query on non-local shards.

Instead of using the client for collating suggestions, collate query is executed against
the ContextIndexSearcher. This PR removes the ability to specify a preference for a collate
query, as the collate query is only run on the local shard.

closes #9377
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.