badger defrag tool

[deepakjois]

Create a tool for purging older versions of a key from the DB.

[manishrjain]

I think badger purge should also literally remove keys from LSM tables which are deleted. The online Purge function would mark keys as deleted. But, this offline version of purge can be a good way to decrease the size of LSM tree by rewriting the LSM tree to get rid of all deleted keys.

As a counter point, there could be two different tools to achieve this -- one to mark keys as deleted and other to remove them from LSM. But, that just adds confusion. A way to decrease the size of LSM tree is more useful, IMO. In fact, we should do this instead of the Purge functions, which can be called online.

[deepakjois]

@manishrjain How about a badger purge subcommand that also takes in a --compact (-c) option. The default behavior of badger purge would be to just perform deletes. When the --compact option is passed, it will perform tree compaction as well.

[manishrjain]

As per discussion on slack, we need a defrag command, which can run the value log GC, and then also run LSM compactions removing the deleted/expired keys, to decrease the size of LSM tree and either push all tables to a single level, and/or rewrite them in place to remove any such keys.

[manishrjain]

Tasks for the defag tool:

• Run compactions through the entire LSM tree to get rid of all invalid keys.
• Then run value log GC through all the log files to get rid of these keys and reclaim space.

All this would be done offline with the aim to reclaim maximum space.

[arctica]

I'll reference my comment from #226 (comment) here because I saw several other tickets mentioning this ticket here.

In my comment, I argue for a way to compact a DB online via API instead of an offline commandline tool as suggested here.

[SwarnimRaj]

@manishrjain Is the Badger defrag tool part of the current code- it seems its absence is causing an issue in Jaeger OpenTracing using Badger as storage jaegertracing/jaeger#760 (comment)