Investigate missing LevelDB compaction

[krasserm]

imported from https://www.assembla.com/spaces/akka/tickets/3962

See https://groups.google.com/forum/#!topic/akka-user/_d9gpIJyKe0

[bantonsson]

So we don't close the snapshots, which leads to resource leakage.

[movermeyer]

I ran the test code for this issue (found here) and was still able to reproduce this issue, even with Akka 3.9 and Scala 2.11.5.

If I left the example code running for 20 minutes, and the journal files had consumed several GB of space.

Here's a screenshot of the disk usage (the x-axis is the number of 2-second intervals since the start of the program):

Note that it is stable for the first ~6 minutes and then begins to creep up. It's still possible that there is a bug in the test code, but if there is, it wasn't noticed before.

I've updated the repo to include the simple bash script I used to collect this data.

[krasserm]

@movermeyer persistent channels are deprecated and will be removed in Akka 2.4. Can you please try running your test with non-deprecated features only and check if the problem still exists? Thanks.

[movermeyer]

@krasserm I took a guess that a modern version would be useful, so I spent some time and made a version using PersistentActors and Akka 2.3.9.

It can be found here.

[movermeyer]

I assume that I must be doing something wrong (given that there are others using this library).

When I get a SaveSnapshotSuccess(metadata) message, I call deleteMessages(metadata.sequenceNr, permanent=true)

There are at least others who suggest doing it this way.

Sorry in advance if this turns out to be a silly syntax issue, but I've looked around and see no other examples of usage that explain what I could be doing wrong.

[krasserm]

@movermeyer thanks for preparing the test case and the data collection script - very useful. I'll try to find some time next week looking into it. Cheers!

[movermeyer]

I suspect that this issue is related to the size of the messages I'm trying to persist.

I've done a bit of playing around in the akka codebase. I've found that if I set the writeBufferSize in LeveldbStore.scala from the default of 4 MiB to 50 MiB,

val leveldbOptions = new Options().createIfMissing(true).writeBufferSize(50 * 1024 * 1024)

that the disk space used still grows indefinitely, but much more slowly.

This graph shows it using roughly 1GiB after roughly 1 hour, instead of the 700MiB after 13 minutes. If it were not leaking at all, the disk used by my sample code should peak at roughly 288 MiB = 6 files * 12 MiB * 4 minutes.

It should also be noted that this disk usage increase is the result of an increase in the number of files, not the individual file sizes.

I guess once this is figured out, I'll submit a pull request that exposes this parameter in the akka config, something like:

val leveldbOptions = new Options().createIfMissing(true).writeBufferSize(config.getInteger("buffer-size"))

[movermeyer]

@krasserm I don't wish to sound ungrateful, but have you found time to take a look into this?

I realize that at a glance, my last post may have made it seem that I had solved the problem.

I have only determined a way to slow the leak somewhat, which might be useful in determining the source of the issue.

[krasserm]

@movermeyer I'm very sorry that I didn't find time looking into it. It would be great if someone from the Akka Team could take a closer look. Thanks for your understanding.

@bantonsson how did you test the resource leak fix when you worked on this ticket?

[movermeyer]

@krasserm

I believe that I now have enough information to show that this is not a bug within the akka-persistence leveldb journal plugin, but a problem with leveldb.

From the journal plugin's perspective, it is doing everything right.
It makes the appropriate delete calls and leveldb receives the delete calls correctly (and then leveldb creates the delete entries correctly).

However, it seems to be a limitation of the way leveldb works. The value entries are not removed from disk until a compaction compacts both the value entry and its corresponding delete entry into a new SST.

What I am seeing is that the large-sized value entries get compacted to high numbered levels, and the tiny-sized delete entries accumulate in level-1. Here is a log of the additions, subtractions, and results of compactions from the perspective of leveldb

As far as I can tell, leveldb is also doing its job correctly, but has a fundamental limitation on the size of values that can be used.

I have asked about this on the level db mailing list, but their community appears dead, so I'm not holding my breath.

It seems that this is a dead end. It would be nice if leveldb and akka-persistence warned of this (soft) limit.

I'd like to hear your thoughts on the subject when you get a chance.

[movermeyer]

Of course as soon as I made that post, I finally came across this.

Apparently other people have hit the same issue in the past and switched away from using leveldb with akka-persistence.

[bantonsson]

Thanks a lot @movermeyer for getting to the bottom of this. Too bad that LevelDB seems to be broken in this regard.

[kongk]

I had this problem too - I upgraded the leveldb dependencies to newer versions, and now compaction actually happens to all the deleted data - so problem seems solved.
I added these maven dependencies:

        <dependency>
            <groupId>org.iq80.leveldb</groupId>
            <artifactId>leveldb</artifactId>
            <version>0.7</version>
        </dependency>
        <dependency>
            <groupId>org.fusesource.leveldbjni</groupId>
            <artifactId>leveldbjni-all</artifactId>
            <version>1.8</version>
        </dependency>

Thanks for investigating, @movermeyer

[patriknw]

Thanks for the update. We have updated the dependency in Akka 2.4-M1. It is actually at provided scope now, so you have to define it yourself, but 1.8 is the version we recommend and test with.