ldb files

[pedrinimm]

Currently there is a bug on levelDB, unfortunately the implication is the error is also present in goqueue.

(google/leveldb#593)

This error consist on levelDB not getting rid off old ldb files. Currently I am working on an application that handles a high amount of incoming data that has to be stored while being processed. After a couple days running I end up with high amount of ldb files (> 2000).

Then once I restart my application and the gopqueue wants to do a recovery, it ends up exceeding the allowed number of open files descriptors on my system.

I would like to fix this issue by using the CompactRange function from leveldb while my program is executing. Nevertheless, due to the lack of exposure to the db struct from goqueue and no Compact/Clean function from the queue struct itself, this is not possible.

It would be nice if either the db struct could be exposed or if there was a Clean/Compact call from the queue itself.

[nbmoa]

We encounter the same issue, but calling the CompactRange from the underlying levelDB doesn't fix the issue completely.
We use the DiskQueue as a persistent FIFO for our messages, so when we restart the service and don't loose data.

Before CompactRange call:
Queue is empty, it has the complete size of 830MB and consists of roughly 500 files. Most of the ldb files are between 1.6MB and 2MB.

After CompactRange call:
Queue is still empty :), it has a complete size of 700MB and consists of roughly 400 files. Now only ldb files with the size between 1.6MB and 2MB are left, all files that had a smaller size vanished.