Percolator: High memory usage issue when nested query is registered

[anatoly21]

ES version: 5.2.2.
JDK: 1.8.0_65
OS version: Ubuntu 14.04.4

We faced high memory usage issue when using percolator.
Dedicated index is configured for percolator.
It works as expected and JVM heap usage picture is

After registering a nested nested query the picture changes and there is a steady grow of used memory and CPU on the same input document set

Input documents contain up to hundred of nested documents. Caching settings are by default.

Memory dump:

[martijnvg]

@anatoly21 Just double checking: So after registering a single percolator query that has a nested query this memory issue happens? Also I assume that search requests (containing percolate queries) are being executed. Do you have a lot of these search requests or just a few?

[anatoly21]

Yes, this issue occurs after registerering a single nested query. The attached pictures show results after execution of about 300 000 search requests (percolate queries).

[martijnvg]

@anatoly21 Thanks, are the search requests (with percolate query) similar (besides the document being percolated)? If possible can you share the search request (or at least how this query is structured)? I like to figure out if the percolate query itself gets cached.

[anatoly21]

Mapping has a lot of keyword fields, therefore the attachment is a simplified version. Search requests are quite different.
env.zip

Possibly this issue is related to #23859

[martijnvg]

Thanks for reporting @anatoly21 and this is indeed a bad bug.

This problem is quite easily reproducible and affects all 5.x releases.
The cause of the memory issues / OOM is that the index reader for the memory index isn't closed,
which then causes the cache entries in BitsetFilterCache to never get cleaned up.

Prior to 5.0 the percolator had its own search context (PercolateContext), which always closed the readers of the in-memory index.

I'll work on a fix. The percolator should just never use the BitsetFilterCache and it doesn't need since it is querying an in-memory index.

[anatoly21]

Thank you.