Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory leak in BitsetFilterCache #15820

Closed
jamesjin opened this issue Jan 7, 2016 · 2 comments

Comments

Projects
None yet
3 participants
@jamesjin
Copy link

commented Jan 7, 2016

When we have many indices and part of them contains nested type, we can find that the heap is cost by BitsetFilterCache very quickly. It can be reproduced on ES2.1 under the default setting. Here is the steps:

  1. create two indices "index1" and "index2"
  2. add type "tweet" with nested type "comments" to "index1"
  3. add a new tweet doc to "index1"
  4. wait until it finishes the refresh
  5. dump the heap, you can find that the "loadedFilters" field of BitsetFilterCache of "index2" contains the query result of "index1".

And the root cause is:

BitSetProducerWarmer does not check the index name in the method BitSetProducerWarmer#warmNewReaders. And IndicesWarmer notifies every BitSetProducerWarmer even the indexShard does not belong to the index of the BitSetProducerWarmer

@jpountz jpountz self-assigned this Jan 7, 2016

@jpountz

This comment has been minimized.

Copy link
Contributor

commented Jan 7, 2016

This is a very good observation, I could reproduce it.

jpountz added a commit to jpountz/elasticsearch that referenced this issue Jan 7, 2016

BitSetFilterCache duplicates its content.
We have a bug that makes all per-index bitset caches store bitsets for all
indices. In the case that you have many indices, which is fairly common with
time-based data, this could translate to a lot of wasted memory.

Closes elastic#15820

jpountz added a commit that referenced this issue Jan 8, 2016

BitSetFilterCache duplicates its content.
We have a bug that makes all per-index bitset caches store bitsets for all
indices. In the case that you have many indices, which is fairly common with
time-based data, this could translate to a lot of wasted memory.

Closes #15820

jpountz added a commit that referenced this issue Jan 8, 2016

BitSetFilterCache duplicates its content.
We have a bug that makes all per-index bitset caches store bitsets for all
indices. In the case that you have many indices, which is fairly common with
time-based data, this could translate to a lot of wasted memory.

Closes #15820

jpountz added a commit that referenced this issue Jan 8, 2016

BitSetFilterCache duplicates its content.
We have a bug that makes all per-index bitset caches store bitsets for all
indices. In the case that you have many indices, which is fairly common with
time-based data, this could translate to a lot of wasted memory.

Closes #15820

jpountz added a commit that referenced this issue Jan 8, 2016

BitSetFilterCache duplicates its content.
We have a bug that makes all per-index bitset caches store bitsets for all
indices. In the case that you have many indices, which is fairly common with
time-based data, this could translate to a lot of wasted memory.

Closes #15820

jpountz added a commit that referenced this issue Jan 8, 2016

BitSetFilterCache duplicates its content.
We have a bug that makes all per-index bitset caches store bitsets for all
indices. In the case that you have many indices, which is fairly common with
time-based data, this could translate to a lot of wasted memory.

Closes #15820
@ycombinator

This comment has been minimized.

Copy link
Contributor

commented Jan 12, 2016

Hi @jamesjin, thanks for reporting this issue! I work in Developer Relations at Elastic. I understand it must've taken you a significant bit of time to reproduce it and figure out the root cause. Thank you for doing that. As a token of our appreciation, we'd like to send you a small thank you gift. Could you send us a quick "hello" email us at developerrelations [at] elastic.co and we'll figure out how best to get you the gift there? Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.