IndexedDB size in Chrome keeps increasing whenever doc is updated

[vikaskumar2299]

Issue

I create a doc (code is below) on first page reload and then update the doc every time the page is reloaded or manually updated. On fresh page load, the db size is 6-10 kb. But it increases by 3-4 kb on on each update.

I read 2 related issues (that suggested updating .js file and I'm using 6.4.3 version now) and tried exactly what they said. But it's not working at all.

Here's what I'm doing:

var testDb = new PouchDB('testDb', {revs_limit:1});

//following is enabled only for first time when page is loaded. Then I comment it.
 testDb.put({
          _id: "3",
          key: "3",
          key1: "3sdfdsfdsf",
          key2: "3sdfdsfdsf",
          key3: "3sdfsdfdsfsd",
          key4: "3sdfsdfsddddddddD",
          key5: "3SDFDSSSSSSSSSS",
          key6: "3SDFSDFDSFDSF",
          key7: "3SDFSDFSDFSDFSDF",
          key8: '{"revs_limit": 5, "auto_compaction": true}',
          key9: "3SDFDSSSSSSSSSSSSSSSSSSSSSSSSSSSSSFSDFDSFDSFSD",
          key11: "3WERWBDFSBFGHGF",
          key12: "SFASVCXVXCV3",
          key41: "3sdfsdfsddddddddD",
          key52: "3SDFDSSSSSSSSSS",
          key63: "3SDFSDFDSFDSF",
          key71: "3SDFSDFSDFSDFSDF",
          key81: '{"revs_limit": 5, "auto_compaction": true}',
          key91: "3SDFDSSSSSSSSSSSSSSSSSSSSSSSSSSSSSFSDFDSFDSFSD",
          key111: "3WERWBDFSBFGHGF",
          key121: "SFASVCXVXCV3",
          key122: "3SDFSDFSD",
          key142: '{"status":"success","data":[{"id":31,"attributes":{"type":"polygon"},"name":"vishal polygon","description":null,"area":"POLYGON((28.705738945391726 77.15641915798187, 28.70879487661599 77.16119483113289, 28.70331842689559 77.16471891850232, 28.69895952914934 77.15919189155102))","calendarId":0}],"message":"OK"}',
          key172: "{\"status\":\"success\",\"data\":[{\"id\":31,\"attributes\":{\"type\":\"polygon\"},\"name\":\"vishal polygon\",\"description\":null,\"area\":\"POLYGON((28.705738945391726 77.15641915798187, 28.70879487661599 77.16119483113289, 28.70331842689559 77.16471891850232, 28.69895952914934 77.15919189155102))\",\"calendarId\":0}],\"message\":\"OK\"}",
          recId: 9,
          senderId: 4668,
        }, function(err, response) {
          if (err) {
            console.log(err);
            } else {
            console.log(response);
          }
        });

//This causes the issue. The doc is updated successfully. Further I don't see any previous revisions. But still the db size is increasing on every page reload.

 testDb.get("3", function(err, doc) {
         if (err) {
           console.log(err);
           } else {
          
             doc.isOpen = '000';
             doc.aaa = moment().format("YYYY-MM-DD HH:mm:ss");
             doc.ccc = moment().format("YYYY-MM-DD HH:mm:ss");

           testDb.put(doc, function(err, response) {
             if (err) {
               console.log("Err Updated", err);
               } else {
               console.log("Updated", response);
             }
           });
         }
       });

Even testDb.viewCleanup() doesn't work.

Info

• Environment: (browser)
• Platform: (Chrome)
• Adapter: (IndexedDB)

Reproduce

Just update the created doc with some more info and you'll see the DB size is increasing.

[daleharvey]

How about after db.compact()?

[daleharvey]

Just noticed you have revs_limit: 1, compaction shouldnt really be needed, however how are you measuring the disk size here? if you are measuring the leveldb files directly then chrome may be doing some bookeeping that we dont know about, I would confirm in the inspector that there is no more data stored than you expect, with the code you have shown I wouldnt expect a growing database either. If there isnt any extra data stored by pouchdb then I think you may have to start looking at chromes implementation

[vikaskumar2299]

I read PouchDB uses indexedDB by default. And I'm seeing indexedDB size increasing actually in Chrome (You can see it by clicking "Clear storage" option). 1. This is really confusing. Two of your developers say don't use auto_compaction here. You can see them https://github.com/pouchdb/pouchdb/issues/4372. They discuss NOT using auto_compaction. 2. One thing you might want to know: When the size reaches near 4MB, the size automatically reduces to minimum, maybe equal to actual size. But this does actually increase with each update and it's really frustrating!

[vikaskumar2299]

db.compact() makes no difference!

[daleharvey]

You shouldnt need auto_compaction when you have revs_limit, I dont see the comments you are referring to and we do not 'have' 2 developers, this is an OSS project. If the database compacts to its actual size when it hits 4MB then its very likely to be chromes implementation of indexeddb and nothing to do with pouchdb at all, as I said check the inspector and see if we are storing any unexpected data, if so, its our bug, if not its how chrome works

[vikaskumar2299]

Please read your own comment here: #4372 You say "As a workaround it looks like everything is fine with revs_limits: 1 only, if revs_limit is 1 then auto_compaction isnt needed anyway" there. Also, same is mentioned here in comments https://stackoverflow.com/questions/37182802/pouchdb-growing-with-revisions-even-when-revs-limit-1

[vikaskumar2299]

By developer I mean guys who reply the issues.

[vikaskumar2299]

Further can you please tell me how to look that unexpected data that might be there?

[daleharvey]

In chrome bring up the console and there is an IndexedDB inspector, in my chrome its under the "Application" tab but that seems new, it may be "Storage" in other versions, I made https://petite-space.glitch.me/ and confirmed that I cant see PouchDB storing any data that isnt expected

[vikaskumar2299]

I'm not in my office now but I'll try and let you know tomorrow. But one thing, why did you say in your 2nd comment that auto_compaction is needed?

[daleharvey]

I hadnt noticed that you were using revs_limit:1

[vikaskumar2299]

So compaction not needed when revs_limit:1?

[daleharvey]

Nope

[vikaskumar2299]

Thanks.

[daleharvey]

Cheers, gonna close this out since I am pretty convinced it isnt a pouch issue after inspect, but feel free to ask questions in here and can reopen if we find something that pouchdb can do about it

[vikaskumar2299]

I opened the link you gave and did reload it 5-6 times and clicked Run Code every time:

Here's what I saw in storage (approx.): 6 kb, 9kb, 12kb, 16kb etc.

When I inspected the IndexedDB, I saw this (I don't know what else to see, but this seemed relevant to me: There are two tabs having one child each, so I clicked on 1+1+1+1 = 4 tabs and these were the results:

• by_sequence (onClick => 1 row, with Key=5 and value (same object that I put))

  doc_id_rev (onClick => 1 row, key like 3:sdfasdfdsafsdf, Primary Key=5 and same Value object )

• document-store (onClick => 1 row, with Key=3 and Value object (with id, seq, deletetedOrLocal:0, winning rev etc. No original object that I put))

  deletedLorLocal (onClick => 1 row, key:0, Primary Key=3 and same Value object (with id, seq, deletetedOrLocal:0, winning rev etc. No original object that I put))

[vikaskumar2299]

If you need, I can provide you a screenshot.

[vikaskumar2299]

Waiting for your reply. It may not be issue with Chrome.

[daleharvey]

Having 1 row in document store and by seq store is expected

[vikaskumar2299]

So the issue is definitely with PouchDB. It somehow doesn't compact tha data. It waits for 4-5MB then does it. When will you fix it?

[daleharvey]

When will you fix it?

To be clear, I nor anyone else here works for you, I have tried to patiently help you but that has a limit.

So the issue is definitely with PouchDB

The 2 documents you mentioned (I see the same) are entirely expected and indicate the issue is not with PouchDB. Nor does pouchdb have any code that waits for the database to be 4mb before compacting.

[vikaskumar2299]

Okay. So what may be wrong. I can reproduce same issue in Chrome 60+ and Chromium browser. Is there any way to fix it?

[daleharvey]

Seems a lot like https://github.com/google/leveldb/blob/master/doc/impl.md and is expected behaviour

[vikaskumar2299]

This seems complex to understand for me. Now main thing is: I believe more size makes the pouchDB .find method slow. Is it true that if size is more fetching docs would be slow? If it isn't I wont' be having any issue with it as it automatically does reset.

[daleharvey]

The link clearly explains why leveldb databases will grow until 4mb which will not effect find performance.

Please do not post issues unless you really willing to attempt to understand the issue at hand especially when being guided, it is extremely disrespectful of peoples time

[vikaskumar2299]

Ok. Could you please tell one more thing: We do new PouchDB('name").destroy() to remove DB at login and log out. Now, in Safari browser, we see same DB created TWICE! Is there any chance that using new PouchDB everytime (although name is same) to delete or fetch data is messing things up?

[daleharvey]

No there isnt

[vikaskumar2299]

Okay. Last query: Our website becomes too slow when we save data for around 100 devices (the data is similar to the JSON object that I have added in details). Now you may have noticed that I have stored long strings in the key's value for some of the keys. And even nested JSON strings as well. So I assume that more is the size of a doc, more will be time in fetching and updating it. Is it really like this? Further, we update each document using a for loop. So that we can first fetch the docId and _rev to update them. This is another major reason for our site's slow performance. Isn't there any other way to update all docs? I know about bulk update but that too requires _rev of original docs.

[vikaskumar2299]

Wait. I noticed this here: https://petite-space.glitch.me/ db.compact() increases indexedDB size by around 1 kb and each time I run this statment in console, it keeps increasing! So is it possible whenever this command is executed in pouchdb.min.js, db size increases?

And our website stores around 500 kb data. So db.compact() might be increasing a lot of data!

[vikaskumar2299]

Please reply. It's a genuine question.

[arackaf]

For the benefit of anyone else arriving here from Google: it seems Chromes IDB implementation will keep increasing in size as you call objectStore.put until it reaches about 4MB, at which time it does a memory dump right back to where it started. I don't know why it does this, but it was miserable to debug, to confirm that my code was not leaking memory somehow

[laukstein]

Chrome related issue https://bugs.chromium.org/p/chromium/issues/detail?id=795735