Elasticsearch hangs using phrase suggester with collate option

[silvestrelosada]

Hi I have elasticsearch installation single node with 100 documents, making simultaneously requests to phrase suggester with collate option blocks completely elastic search. And does gives the option to do any thing else.

Here is my query

{  
   "query":{  
      "bool":{  
         "should":[  
            {  
               "prefix":{  
                  "label":"michael"
               }
            },
            {  
               "prefix":{  
                  "synonym":"michael"
               }
            }
         ]
      }
   },
   "explain":false,
   "suggest":{  
      "spell_ngram":{  
         "text":"michael",
         "phrase":{  
            "field":"spell_ngram",
            "size":1,
            "confidence":0.0,
            "max_errors":0.9,
            "gram_size":5,
            "collate":{  
               "query":{ "match" : {
                   "spell_ngram" : {
                        "query" : "{{suggestion}}", "type" : "phrase"
                            }  
                    }
                }
            }
         }
      }
   }
}

And here is my mappings.

{
"concept":{
    "properties" :{
            "_id":{ "type" : "string", "index" : "not_analyzed"},
            "_type":{ "type" : "string", "index" : "not_analyzed"},
            "label": {
            "type" : "multi_field",
                "fields" : {
                    "label" : {"type" : "string", "index" : "not_analyzed","copy_to" : ["spell", "spell_ngram"],"store" : true},
                    "label_analysis" : {"type" : "string", "index" : "analyzed","store" : true,"term_vector": "with_positions_offsets_payloads"},
                    "phonetic_label" : { "type" : "string", "index" : "analyzed", "index_analyzer":"phonetic_analyzer_index", "search_analyzer":"phonetic_analyzer_search","store" : true,"term_vector": "with_positions_offsets_payloads" }
                }
             },
            "synonym" : {
                "type" : "multi_field",
                "fields" : {
                    "synonym" : {"type" : "string", "index" : "not_analyzed","copy_to" : ["spell", "spell_ngram"],"store" : true,"term_vector": "with_positions_offsets_payloads"},
                    "synonym_analysis" : {"type" : "string", "index" : "analyzed" ,"store" : true},
                    "phonetic_synonym" : { "type" : "string", "index" : "analyzed",  "index_analyzer":"phonetic_analyzer_index", "search_analyzer":"phonetic_analyzer_search","store":true,"term_vector": "with_positions_offsets_payloads"  }
                },
           "spell_ngram" : { "type" : "string", "index" : "analyzed", "analyzer":"shingle_analyzer","store":true, "term_vector": "with_positions_offsets_payloads" }
        }
     }
   }
}

[clintongormley]

Hi @silvestrelosada

Please could you provide the simplest complete recreation of the problem, so that we can try it out:

• you're only running suggestions on one field, but you've provided the mapping for all fields
• you haven't provided the index settings (including the analysis section)
• you haven't provided an example document

thanks

[silvestrelosada]

Hi here is the data needed to reproduce the error, mapping file, elasticsarch.yml, and data

https://gist.github.com/silvestrelosada/eb32afdc8c971504e45c

To reproduce it you have to send seval concurrent queries, Im using jmeter.

Best

[silvestrelosada]

If it is helps the issue takes place on multi shard enviroment in fetchMatchingDocCountResponses method when executing the search.

[seyart]

Hi,

I think I have the same issue. ES hangs when I send multiple simultaneous suggest requests with collate option to the search API. I tested the version 1.37 and 1.42 of ES. ES is running with one node and the default settings (5 shards per index, but 0 replica). There are about 19 000 documents in the target index.

To make ES hang, I created a script which run a lot of suggest requests with random text. When I launch 2 or more instances of this script, depending on the hardware configuration of my testing environment, ES hangs and my ES client (elasticsearch-py) raises timeout exceptions. Then, I'm unable to make curl request to the search API (it's keep waiting for a response). Index API and bulk API are still running fine.
Using BigDesk to monitor, there is no memory issue. The search thread pools are also fine but the queue size increases after it hangs.

ES log gives nothing.

Here is an exemple of a request :

{
    "aggregations": {
        "brand": {
            "terms": {
                "field": "brand.untouched",
                "size": 0
            }
        },
        "categories": {
            "terms": {
                "field": "categories_parents_catalog_codes",
                "size": 0
            }
        },
        "offers": {
            "terms": {
                "field": "offers.untouched",
                "size": 0
            }
        },
        "stores": {
            "terms": {
                "field": "categories_parents_catalog_codes",
                "size": 0
            }
        }
    },
    "fields": [
        "full_title"
    ],
    "from": 0,
    "query": {
        "filtered": {
            "filter": {
                "term": {
                    "categories_parents_catalog_codes": "root"
                }
            },
            "query": {
                "match": {
                    "full_title.autocomplete": {
                        "operator": "and",
                        "query": "B"
                    }
                }
            }
        }
    },
    "size": 20,
    "suggest": {
        "suggestion_phrase": {
            "phrase": {
                "collate": {
                    "preference": "_primary",
                    "query": {
                        "filtered": {
                            "filter": {
                                "term": {
                                    "categories_parents_catalog_codes": "root"
                                }
                            },
                            "query": {
                                "match": {
                                    "full_title.autocomplete": {
                                        "operator": "and",
                                        "query": "{{suggestion}}"
                                    }
                                }
                            }
                        }
                    }
                },
                "field": "full_title.suggestion"
            }
        },
        "text": "B"
    }
}

Here is the mapping of the suggest field :

            "full_title": {
                "type": "string",
                "fields": {
                    "french": {
                        "type": "string",
                        "analyzer": "custom_french"
                    },
                    "suggestion": {
                        "type": "string",
                        "analyzer": "custom_suggestion"
                    },
                    "autocomplete": {
                        "type": "string",
                        "index_analyzer": "nGram_analyzer",
                        "search_analyzer": "whitespace_analyzer"
                    }
                }
            }

I remain available for any further information.

Best Regards,

[clintongormley]

@areek could you take a look at this please

[s1monw]

@areek can you try to write a test to reproduce this?

[s1monw]

one idea that I have here is that it seem it only happens if we specify the suggest together with a search and that means the suggestion as well as the collation is executed on the search threadpool @areek can you try to write a test that does the same and try to forcefully reduce the number of threads in the search threadpool to a small number ie 1 or 2? I guess that could trigger the issue...

[areek]

I have been able to reproduce this issue.
Collate option internally fires off a search request, when used from the _search API, the internal request uses the same search threadpool used by the search request, hence if all threads are busy, it causes a deadlock.

One solution to avoid this issue would be to execute the collate query on only the local shard from which the suggestions are generated instead of collating on non-local shards. As suggestions are generated from the terms of the local shard, in most cases a generated suggestion which does not yield a hit for the collate query on the local shard would not yield a hit for collate query on non-local shards. Thoughts? I will create a PR for this.

[s1monw]

One solution to avoid this issue would be to execute the collate query on only the local shard from which the suggestions are generated instead of collating on non-local shards. As suggestions are generated from the terms of the local shard, in most cases a generated suggestion which does not yield a hit for the collate query on the local shard would not yield a hit for collate query on non-local shards. Thoughts? I will create a PR for this.

I like this a lot - i think it's the right tradeoff