Rocksdb iterator does not release memory ?

[asad-awadia]

Expected behavior

When the rocksdb iterator is closed the memory used should be cleaned up i.e RSS of my app should not jump up every time an iteration scan is done

Profiler shows that heap memory is not growing

Actual behavior

Ever scan via an iterator jumps RSS of the app up by roughly the DB size [100mb]- and it never goes down - eventually getting OOM

Steps to reproduce the behavior

Basic try-with-resources iteration scan from Java

try(RocksIterator iterator = rocksdb.newIterator()) {
  iterator.seekToFirst();
  while(iterator.isValid() {
    iterator.next()
  }
}

Even though the iterator is closed no memory is being free'd up - and the memory is not being used in the subsequent scans - it is always allocating roughly the same amount each scan

Using Rocksdb v6.22.1 using Linux Java Jars

[asad-awadia]

@ajkr @jay-zhuang @pdillinger

[cbi42]

Can you share your full code for repro? RSS increase is expected during scanning as resources are being pinned by the iterator, but they should be freed when the iterator is destructed. Are you using snapshot? unreleased snapshot can cause resources not being freed.

[asad-awadia]

Can you share your full code for repro?

This is literally it - get a rocksdb object, create an iterator, and scan through in a try with resources. We were debugging this and narrowed it down to this code snippet where the issue was 100% reproducible

try(RocksIterator iterator = rocksdb.newIterator()) {
  iterator.seekToFirst();
  while(iterator.isValid() {
    iterator.next()
  }
}

We have an http server that takes in a request, gets a rocksdb instance from our singleton, triggers this full DB scan and returns an empty response

We start at 2g of RSS and then after around 20 requests it goes to 3gb at which point it gets oom - DB size is 105mb and the rss jumps up by ~100mb each request

but they should be freed when the iterator is destructed.

Yeah this is what i thought too - but rss keeps going up even when the iterator is closed
we are closing the iterator [try with resources does it - also did it explicitly ]

Are you using snapshot?

not using a snapshot - even tried release the implicitly created snapshot before leaving the try - still no luck

@cbi42

[asad-awadia]

is it because of MALLOC_ARENA_MAX ?

[asad-awadia]

is it because this is running inside a docker container?

[pavan-traceable]

Any update on this issue ?

[adamretter]

I very much doubt that this is an issue in the Java API of RocksDB as the try-with-resources you are using will ensure that the underlying C++ iterator is freed.

You would need to use a Profiler or compare before and after heap dumps to see where the memory is being used.

[asad-awadia]

profiler just says 'bytes[]' :(

[adamretter]

@asad-awadia So is what you are seeing now confirmed as being expected behaviour?

[pavan-traceable]

Kafka stream application running in my company which uses rocksdb as state store having unaccounted memory. Thinking it might be leak from rocksdb, any known issues about memory leak ?

[adamretter]

@asad-awadia As far as I am aware there are not any known leaks in RocksDB. Of course there could be unknown ones...

In the example code you provided there is no use of Java's byte[] at all. Perhaps there is further code that is needed to reproduce your issue?

[asad-awadia]

In the example code

The example code was a small reproducer - that was not profiled - it had consistent RSS increase

1 http request that did an iteration - that was the entire application - RSS would go up every time and never come back down

there is no use of Java's byte[] at all

This is what we see when we profiled our 'real' application running in full

So is what you are seeing now confirmed as being expected behaviour?

Took too long to get support - we moved onto different high priority work

We also upgraded to 7.4.4 - so if there is an issue it will be on this version now - we do see OOM under load but have not narrowed it down enough to say if it is because of the iterator logic

[adamretter]

@asad-awadia I think we need a real and small reproducible example.