New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Percolator: High memory usage issue when nested query is registered #24108

Closed
anatoly21 opened this Issue Apr 14, 2017 · 6 comments

Comments

Projects
None yet
2 participants
@anatoly21

anatoly21 commented Apr 14, 2017

ES version: 5.2.2.
JDK: 1.8.0_65
OS version: Ubuntu 14.04.4

We faced high memory usage issue when using percolator.
Dedicated index is configured for percolator.
It works as expected and JVM heap usage picture is

without_nested_queries

After registering a nested nested query the picture changes and there is a steady grow of used memory and CPU on the same input document set

with_nested_query

Input documents contain up to hundred of nested documents. Caching settings are by default.

Memory dump:
memory1
memory2

@martijnvg

This comment has been minimized.

Member

martijnvg commented Apr 14, 2017

@anatoly21 Just double checking: So after registering a single percolator query that has a nested query this memory issue happens? Also I assume that search requests (containing percolate queries) are being executed. Do you have a lot of these search requests or just a few?

@martijnvg martijnvg self-assigned this Apr 14, 2017

@anatoly21

This comment has been minimized.

anatoly21 commented Apr 14, 2017

Yes, this issue occurs after registerering a single nested query. The attached pictures show results after execution of about 300 000 search requests (percolate queries).

@martijnvg

This comment has been minimized.

Member

martijnvg commented Apr 14, 2017

@anatoly21 Thanks, are the search requests (with percolate query) similar (besides the document being percolated)? If possible can you share the search request (or at least how this query is structured)? I like to figure out if the percolate query itself gets cached.

@anatoly21

This comment has been minimized.

anatoly21 commented Apr 14, 2017

Mapping has a lot of keyword fields, therefore the attachment is a simplified version. Search requests are quite different.
env.zip

Possibly this issue is related to #23859

@martijnvg

This comment has been minimized.

Member

martijnvg commented Apr 14, 2017

Thanks for reporting @anatoly21 and this is indeed a bad bug.

This problem is quite easily reproducible and affects all 5.x releases.
The cause of the memory issues / OOM is that the index reader for the memory index isn't closed,
which then causes the cache entries in BitsetFilterCache to never get cleaned up.

Prior to 5.0 the percolator had its own search context (PercolateContext), which always closed the readers of the in-memory index.

I'll work on a fix. The percolator should just never use the BitsetFilterCache and it doesn't need since it is querying an in-memory index.

@anatoly21

This comment has been minimized.

anatoly21 commented Apr 14, 2017

Thank you.

@martijnvg martijnvg closed this in c17de49 Apr 26, 2017

martijnvg added a commit that referenced this issue Apr 26, 2017

[percolator] Fix memory leak when percolator uses bitset or field dat…
…a cache.

The percolator doesn't close the IndexReader of the memory index any more.
Prior to 2.x the percolator had its own SearchContext (PercolatorContext) that did this,
but that was removed when the percolator was refactored as part of the 5.0 release.

I think an alternative way to fix this is to let percolator not use the bitset and fielddata caches,
that way we prevent the memory leak.

Closes #24108

martijnvg added a commit that referenced this issue Apr 26, 2017

[percolator] Fix memory leak when percolator uses bitset or field dat…
…a cache.

The percolator doesn't close the IndexReader of the memory index any more.
Prior to 2.x the percolator had its own SearchContext (PercolatorContext) that did this,
but that was removed when the percolator was refactored as part of the 5.0 release.

I think an alternative way to fix this is to let percolator not use the bitset and fielddata caches,
that way we prevent the memory leak.

Closes #24108
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment