Completion suggester fails when elastiknn is set to true

[emla2805]

Using the completion suggester feature fails when elastiknn is set to true in the index setting.

Steps to replicate issue:

1. Create index with elastiknn set to true and completion

PUT music
{
  "settings": {
    "elastiknn": true
  }, 
  "mappings": {
    "properties": {
      "suggest": {
        "type": "completion",
        "analyzer": "standard"
      },
      "title": {
        "type": "keyword"
      }
    }
  }
}

2. Add a document

PUT music/_doc/1?refresh
{
  "title": "Smells like teen spirit",
  "suggest" : {
    "input": [ "Nevermind", "Nirvana" ],
    "weight" : 34
  }
}

3. Query suggestion

POST music/_search?pretty
{
  "suggest": {
    "song-suggest": {
      "prefix": "nir",        
      "completion": {         
          "field": "suggest"  
      }
    }
  }
}

The query fails with response:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "suggest is not a SuggestField"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "music",
        "node": "_UooGXTrSV66WkdBnJmb8g",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "suggest is not a SuggestField"
        }
      }
    ],
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "suggest is not a SuggestField",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "suggest is not a SuggestField"
      }
    }
  },
  "status": 400
}

The issue goes away if elastiknn is set to false

[alexklibisz]

Thanks for posting. Some initial thoughts:

The main difference when setting elastiknn to true is that it uses an older (faster) codec for doc values. I guess this is having some sort of interaction with whatever storage mechanism the completion suggester uses. I've personally never used the completion suggester myself, so I'll have to dig into the docs and implementation to see why it doesn't like the older codec.

Also, I believe the Lucene folks have merged a fix for the issue that prompted using an older codec in the first place. This newer version of Lucene should be included in an upcoming version of ES.

[alexklibisz]

The configurable binary doc values compression was released in lucene 8.8.0: https://issues.apache.org/jira/browse/LUCENE-9378. This should let me get rid of the elastiknn: true setting. Elasticsearch 7.11.x is still on 8.7.1, but it looks like their master is currently on 8.8.0 which seems promising.

[alexklibisz]

Hey @emla2805 . I realize it has been a long time. :) ES has still not totally solved the issue, even with the improvements in Lucene.

However I figured out a potential intermediate solution: in 7.12.0, setting "index.elastiknn" to "false" and "index.codec" to "lucene_default" should use the new Lucene compression setting and elastiknn won't interfere with other functionality (e.g. suggestions). Let me know if you get a chance to try this out.

[emla2805]

Thanks for the response @alexklibisz, I will try your suggestion when I get the chance. It turned out that for our use-case and configuration we actually got better performance when setting elastiknn: false. Btw, thanks for a great plugin! 🙏

[alexklibisz]

Happy to hear you found it useful! I'll go ahead and close this since the main fix to this issue is being tracked in #254, but feel free to re-open if needed.