Feature Request: add a hook to customize SST compaction

[eloff]

I keep running into a form of this problem:

I want to expire items in the database based on custom logic. One such use case is when using badger as a disk-based cache, I don't want to expire items on a fixed schedule, but rather when free space gets low I want to start removing the coldest items. This is not very efficient because it requires walking the whole database and deleting items, and then triggering a compaction which will walk the whole database for a second time.

What if the condition (which is probably IsDeletedOrExpired() currently) were changed to a configurable function pointer that defaults to IsDeletedOrExpired(item)? Could something like that make sense?

[jarifibrahim]

Hey @eloff, We support setting the discard flag for entries. You can do entry.WithDiscard() and compaction will take care of removing it automatically
https://github.com/dgraph-io/badger/blob/dd332b04e6e7fe06e4f213e16025128b1989c491/structs.go#L186-L196

I think this would help you accomplish what you want to do.

I don't want to expire items on a fixed schedule, but rather when free space gets low I want to start removing the coldest items

How do you find the coldest items without a DB scan?

I've changed the title from value log compaction to SST compaction. SST are the files that store the keys and the index. value log is the write-ahead log and it optionally stores values.

[eloff]

I think that will work.

1. Scan the SST (key-only iteration) and check the transaction timestamp and estimated value size to determine if a item should be expired or not.
2. Then use discard to save an entry with just the key and the discard bit.
3. After walking everything, trigger a database compaction to reclaim disk space.

[jarifibrahim]

After walking everything, trigger a database compaction to reclaim disk space.

You don't need to trigger compaction. it will get triggered automatically but please be aware that disk space might not be reclaimed instantly. Most of the data is stored in value log and it won't be cleaned up until you call RunValueLogGC.