Slow compaction progress and high memory usage #1370
Description
What version of Badger are you using?
Steps to Reproduce the issue
Run badger benchmark write -m 1000 command.
What did you see instead?
The compactions don't seem to make progress very slow and L0 gets accumulated with data
badger 2020/06/15 16:45:07 INFO: Got compaction priority: {level:0 score:48.4 dropPrefix:[]}
badger 2020/06/15 16:45:07 INFO: Running for level: 0
badger 2020/06/15 16:45:07 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}
badger 2020/06/15 16:27:51 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}
Time elapsed: 01s, bytes written: 67 MB, speed: 67 MB/sec, entries written: 419420, speed: 419420/sec
badger 2020/06/15 16:27:52 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}
Time elapsed: 02s, bytes written: 101 MB, speed: 50 MB/sec, entries written: 629130, speed: 314565/sec
badger 2020/06/15 16:27:53 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}
Time elapsed: 03s, bytes written: 155 MB, speed: 52 MB/sec, entries written: 970782, speed: 323594/sec
badger 2020/06/15 16:27:54 INFO: Got compaction priority: {level:1 score:60.551320023834705 dropPrefix:[]}
A score of 1 means we've filled the level to 100% capacity. The problem here is that because of commit 3747be5 the writes do not get stalled and levels get filled up. My badger directory has 200+ level 0 tables and they were being compacted with the entire L1 (which has 300+ tables).