MB-31358: Estimation of memory needed for query with highlighting

[abhinavdangeti]

• Check this test ..

func TestEstimate(t *testing.T) {
    q := []byte(`{"from":8996,"highlight":{},"query":{"bool":false,"field":"free_parking"},"size":405}`)
    var sr *SearchRequest
    err := json.Unmarshal(q, &sr)
    if err != nil {
        t.Fatal(err)
    }

    x := MemoryNeededForSearchResult(sr)
    fmt.Println(x)
}

• For the query, the MemoryNeededForSearchResult API claims that it
  needs .. 6363433784 bytes (that's over 6GB!).
• Fixing the size calculation estimate when the query requests for
  "fields" to be returned or "highlighting", basing the number of
  results returned on the collector's PreAllocSizeSkipCap.
• Also removing the factor "numDocMatches" to the document size in
  this scenario.

[mschoch]

The original code is clearly wrong as it seems end up with something close to (size+from)^2 ...

However, I think this is wrong as well. Stored/highlight is only ever done on the final docs returned (size - from), so even numDocMatches will over estimate it.

Further, why are we adding in a loop, can we not just multiply?

I apologize in advance if I'm missing something fundamental here, the original code feels so wrong it's hard to discern the original intent...

[abhinavdangeti]

Haha you're right.
Although stored/highlight would return just size and not (size - from).

[mschoch]

@abhinavdangeti yeah you're right, just size.