revs_limit hides, but does not actually delete, old document data

[elafargue]

I am using PouchDB for a data logging application, and one of my databases holds a single 'status' document. This document is updated up to twice per minute, so new revisions are generated fairly rapidly.

I have noticed that even when using compaction - either automatic or explicit -, the storage requirements for this document get out of hand very quick: for a 100 byte JSON structure, I end up with several hundred megabytes of "*.ldb" files in the LevelDB storage directory. The server running for a bit more than a week now uses more than 1GB for the status document...

My expectation was that once I reached the hardcoded 1000 revisions limit, the space required for this status document would be large but stop increasing, but that is not the case.

When fetching all revisions of the 'status' document, I correctly only get the last 1000 revisions, all of them "missing" but the latest, since I use compaction. But exploring the ldb files, I can find references to every single revision ever created, which I suppose explains why my storage increases continuously.

Is this the expected behavior, or is this a bug?

PouchDB version is the latest as of 2015.09.28
OS is Linux

[nolanlawson]

Definitely a bug; sounds like when we trim 1000+ revisions, we don't actually remove the revisions; we just update the metadata.

[elafargue]

@nolanlawson thanks for the feedback - let me know how I can help... is this independent from the underlying database system, or is this linked to LevelDB specifically? I can take a look if you point me in the right direction in the source tree...

[nolanlawson]

Yep, so ideally this would need fixes in all three adapters and a new test (to guarantee that the fix worked). For LevelDB you would be looking at the bulkDocs implementation and what happens when metadata gets trimmed, which currently happens in shared utility functions here and here. Note that in one case the code path applies for new_edits=false, whereas for the other one it applies to new_edits=true.

So yeah, it is very very subtle, but if you have the inclination to attack the bug then I encourage you to do so. :) You can verify that your fix works by writing a test that modifies a document >1000 times and then verifying that you can no longer fetch the oldest revision via a get().

[elafargue]

Hmm, not an ideal candidate for a first PR on PouchDB, I see :) Not sure I'll be able to help in a timely manner here, unfortunately - though it's currently kind of a blocker in my project... Well, I'll see what I can do but don't let me hold you back if you want to tackle it first ;-)

[nolanlawson]

As a temporary workaround, you can replicate from new PouchDB('/path/to/first') to new PouchDB('/path/to/second') and the second one will end up being compacted/smaller.

[elafargue]

Good point. I am doing a few more tests - if I do 50'000 sequential updates to a simple { ts: timestamp} document, the levelDB database size can go up to 50MB which is a lot for such a small doc, but actually the size goes up and down all the time during the loop, from as low as 9M to at high as 50M, in just a couple of seconds. I'm a bit surprised...

Another issue: when running the same test program multiple times, each time the test is restarted the database size increases slightly, and ends up taking hundreds of megabytes even with auto compaction...

The test is a simple:

function update() {
   db.get('001').then(function (result) {
       result.ts = new Date().getTime();
      return db.put(result);
   }).then(function (response) {
      if (i++ < 50000)
         update();
    }).catch(function (err) {
        console.log(err);
    });
}

update();

Also, doing a "db.compact()" after each put seems to make matters worse, the database grew up to 500MB in a couple of minutes on this test.

[nolanlawson]

Interesting. It might have something to do with the internals of LevelDB, which I'm not too familiar with.

[christopherreay]

Just out of interest, does this plugin suffer from the same issues?
I would guess not.
They might be able to help:
https://github.com/redgeoff/delta-pouch

[nolanlawson]

No, delta-pouch wouldn't, because every document only ever has one revision.

[elafargue]

By the way: PouchDB 5.1.0 also suffers from this - I have the same issue on my software on this latest version. It also looks like in some instances, some documents disappear after many updates - this is on code that never deletes a document but updates the docs every 30 seconds...

[elafargue]

I'd love to get an update on this issue: is there something I don't understand with auto compaction or non-leaf revisions that explains this behaviour, or is this really a bug?

This is kind of a show stopper for any NodeJS app that relies on PouchDB for long term operations, especially since I am now getting corruption issues on the same code after a couple of days or running...

[nolanlawson]

If you auto-compact, then that should actually fix your problem, because the old revisions will be deleted every time you make an update.

The bug described in this issue is that, even if you have autocompact disabled, PouchDB should "compact" those >1000 old revisions, but it's not doing that.

[elafargue]

I tried with both autocompact and without, the problem is pretty much the
same with the test code I shared above...

On Sat, Nov 21, 2015, 13:10 Nolan Lawson notifications@github.com wrote:

If you auto-compact, then that should actually fix your problem, because
the old revisions will be deleted every time you make an update.

The bug described in this issue is that, even if you have autocompact
disabled, PouchDB should "compact" those >1000 old revisions, but it's
not doing that.

—
Reply to this email directly or view it on GitHub
#4372 (comment).