Data leak with goleveldb backend?

[mmindenhall]

The application I'm working on runs on IoT gateways with limited resources (~100MB storage free). Therefore, I submitted #373 to be able to proactively reclaim disk space after deleting "expired" documents from indexes.

I wrote a test where I do the following:

1. Create and initialize a new index, take snapshot of size of index folder
2. Do the following 5 times:
3. Add 1000 documents to the index
4. Take snapshot of size of index folder
5. Delete all documents from the index
6. Call the new Compact method I added in Add compact method to goleveldb store #373
7. Take snapshot of size of index folder

Here are the sizes of the index folder:

Size at start: 52K

Iteration	After adding 1K docs	After delete / compact
1	6.7M	876K
2	7.4M	1.6M
3	8.1M	2.2M
4	8.7M	2.9M
5	9.4M	3.5M

Just to be sure the Compact method was actually doing something, I commented out that line and ran again (create 1000, delete 1000, no compact). Without the call to Compact, deleting the 1000 documents actually increases the size of the index:

Iteration	After adding 1K docs	After delete only
1	6.6M	8.5M
2	7.4M	9.3M
3	8.0M	9.9M
4	8.6M	11M
5	9.4M	11M

Is this a bug? I'm wondering if there's some set of keys that gets created when a document is indexed that is not getting deleted when the document is removed?

[mschoch]

Can you run the bleve_dump utility to see what data (if any) is still there?

If bleve_dump doesn't show anything, you can use the lower-level 'leveldbutil' that comes with leveldb to dump just the contents of the leveldb file.

Running those after one of the iterations should give some insight.

[mschoch]

There are a few things that are expected to be left around.

1. The index mapping itself is stored. (frequently this is less than 1k)
2. The field definitions exist independent of any documents being indexed. So there should be 1 row for each field, again in practice even if you had hundreds of these they are just a few bytes each.

[mmindenhall]

I ran the test with just one iteration, and with just 10 documents. After delete and compact, I ran bleve_dump. There are a huge number (1750, to be exact) of Dictionary Term entries:

Dictionary Term: `<^A^T'k^V3` Field: 2 Count: 0
Key:   64 02 00 3c 01 14 27 6b 16 33

Value: 00

Dictionary Term: `cpu2` Field: 4 Count: 0
Key:   64 04 00 63 70 75 32
Value: 00

Some have binary data (like the first), and some have recognizable strings (second). Then at the very bottom, I see the definitions for the fields:

Field: 0 Name: seq
Key:   66 00 00
Value: 73 65 71 ff

Field: 1 Name: id
Key:   66 01 00
Value: 69 64 ff

...

And finally I see the InternalStore which appears to contain the document mapping.

I noticed that all of the Dictionary Term entries have a Count: 0, which made me wonder if every term bleve encounters is being kept in the index, even after documents containing the terms have been removed?

I attached the dump output if that helps.

bleve_dump.txt.gz

[mmindenhall]

@mschoch, any thoughts on this one?

[mschoch]

I don't have anything really to add, what you found is definitely the case.

It has to do with the fact that we are using a "merge operator" to update the dictionary rows. We built our merge operator API to emulate (and fit in with) what RocksDB had. And at the time that didn't seem to allow us to delete a row as a result of the merge. I think there is now a way to get that behavior, but our API doesn't support it.

So, the net effect is that even if a dictionary term count goes to 0, we don't delete it. Even at the time we knew this was undesirable, but we also didn't expect it to be a big deal.

The use case your testing of creating docs then deleting them all isn't one we optimized for. Is this a particularly interesting case to you? Or are you just testing different things out?

My thought is to leave this open so that we eventually circle back to address it, but its not particularly high priority for me right now.

[mmindenhall]

We're building IoT gateway software that runs on small footprint devices. For example, we're running on a device with 128MB of RAM, and 128MB of flash (of which about 65% is available for our app + data). We're receiving and indexing "reports" from attached "things", and the partition runs out of space pretty quickly.

So I wrote code to monitor how much space is free, and delete the oldest reports (with a Compact) to avoid disk full errors. This solution probably extends our ability to run without filling the disk from hours or days to days or weeks. Eventually it would be great to have this fixed, but not urgent.

If you can give me some pointers at what needs to be done, I might be able to convince my boss to let me work on fixing this.

[mschoch]

Well, I don't think its straightforward to fix. Our current design is broken.

I did another quick review and it seems that the result of a RocksDB merge is always another row, not a delete. Even if we defined our merge wrapper to delete rows if the new row is nil, that wouldn't work correctly with RocksDB's native merge operator.

The only alternative I can see is to have some sort of background process cleaning things up, but I'm not excited about that solution either.

[salarali]

I have a scenario where I add a doc, run queries on it and then delete it again because I do not need it anymore. This is done quite frequently.

Using leveldb as the backend and using the in-memory option when creating the index.

[tmm1]

I've observed the behavior noted here (dictionary term not deleted when count goes to 0) on the levigo based leveldb backend as well.