Highlighting sometimes returns too much text (orders of magnitude larger than fragment_size)

[scharron]

Index mapping :

{
    "mappings" : {
        "test": {
            "properties": {
                "content" : {
                    "type" : "string",
                    "analyzer" : "french"
                }
            }
        }
    }
}

Document inserted :
https://gist.github.com/scharron/684a4fbab85135c203ee

Query :

{
  "query": {
    "filtered": {
      "query": {
        "query_string": {
          "query": "\"Vlaams Brabant\"",
          "fields": [
            "content"
          ]
        }
      }
    }
  },
  "highlight": {
    "fragment_size": 100,
    "fields": {
      "content": {}
    }
  }
}

I would expect to get about 200 characters in 2 fragments, but instead I get 12KB of data in only one fragment.
I also tried the postings and the fast vector highlighters with the same result.
The behaviour is the same for both an ES 1.4.2 and ES 1.3.4.

It may be related to other curious behaviours : http://stackoverflow.com/questions/28167990/curious-behaviour-of-fragment-size-in-elasticsearch-highlighting?noredirect=1

[strouptl]

I second this issue, as I am randomly getting up to 1100 characters per fragment, as well (fragment_size set to 300). Does anyone have any ideas?

[youxu71]

I got same issue, sometimes got 1000+ chars, although I set fragment_size to 200.
Any ideas?
I am using ES 1.5.2

[monsieur-d]

I second this too. This is really a problem since this occurs unpredictably except that it seems to occur only with search terms consisting of two or more words.
This makes the search results of a website really ugly!
When is this accepted as a bug (using 1.5.2 too)?

[strouptl]

Good point, @monsieur-d - @nik9000 can we get this marked as a bug like issue #12648 while you are working on highlighting issues? I would be happy to provide a curl sample!

[nik9000]

I would be happy to provide a curl sample!

I think I got it:
https://gist.github.com/nik9000/d420c02f625a956e0faf

[nik9000]

And this shows that its actually just a bug in the plain highlighter:
https://gist.github.com/nik9000/e02b854b67de680759de

Our old, cranky friend the fvh does the right thing here.

[monsieur-d]

Very good. I'm looking forward to a fix :-)

[giles-t]

Out of curiosity, are there any further updates on this issue?

[klimchuk]

We're also interested in resolving this issue

[sfcgeorge]

I had the same issue and was not able to work around.

• Plain highlighter has this bug, sometimes returns too much.
• Postings highlighter doesn't support fragment_size.
• FVH doesn't work on nested fields ¯\⁠_(ツ)_/⁠¯

So basically fragment_size is unusable. I had to set number_of_fragments: 0 to get all the text, then implement my own highlight-aware truncating in code.

[nik9000]

FVH doesn't work on nested fields ¯⁠_(ツ)_/⁠¯

You have to escape the \ or it won't show....

The issue you linked looks like it was fixed for 2.4.0 and 5.0.0. Not that this isn't still a problem, but it isn't one I've looked at for a long, long time.

[clintongormley]

This appears to be fixed in 5.0

[olegskl]

This doesn't seem to be fixed (using 5.5.0). I had to specify fragmenter: 'simple' to avoid the issue.

[sipingliu]

Same issue in 5.4.1. Fixed it by specifying the type explicitly:
"myDataField": {
"fragment_size": 100,
"number_of_fragments": 1,
"type":"plain"
}

[Garrett-R]

Specifying 'type': 'plain' didn't work for me in 6.2.3, but following @olegskl's suggestion of putting 'fragmenter': 'simple' did solve the issue.

[choco-love]

I had the same issue and was not able to work around.

• Plain highlighter has this bug, sometimes returns too much.
• Postings highlighter doesn't support fragment_size.
• FVH doesn't work on nested fields ¯\⁠_(ツ)_/⁠¯

So basically fragment_size is unusable. I had to set number_of_fragments: 0 to get all the text, then implement my own highlight-aware truncating in code.

how did you fix your problem???

[sfcgeorge]

"I had to set number_of_fragments: 0 to get all the text, then implement my own highlight-aware truncating in code."

You need to write application code to do the truncation. It's a terrible workaround because you'll be getting far more text over the network than you need. The code will depend on the language you're using but the algorithm is effectively:

• Scan for <em>s, if there are none then just truncate as usual.
• If there are <em>s:
  • Scan up to the first one.
  • Go back a few chars to provide context around the highlight (if there is any text before the highlight).
  • Count truncate but you have to subtract <em> and </em>s that would be included since they're invisible, tricky.
  • After truncation count the number of opening <em> and closing </em>s and if there aren't enough closing because you've chopped one off add it at the end.