mv mutex cleanup

Status

• merged to master
• code complete May 4, 2014
• development started April 30, 2014

History / Context

Each database / vnode in leveldb contains a mutex that protects the table file (.sst) management structures. Alll user activity must briefly acquire the mutex before a Read, Write, or Iterator operation. The background compactions must also briefly acquire the mutex before and after a compaction. This branch addresses two scenarios where a background compaction might hold the mutex for an extended time during heavy disk activities. Holding the mutex during compaction blocks user operations impacting throughput and latency.

The first scenario deals with common logging statements in the compaction process. Here is a sample of the target messages:

2014/04/25-18:39:08.496084 7f8d767a2700 Level-0 table #15: started
2014/04/25-18:39:08.707713 7f8d767a2700 Level-0 table #15: 35220571 bytes, 29779 keys OK
2014/04/25-18:39:08.725394 7f8d767a2700 Delete type=0 #12
2014/04/25-18:39:08.727119 7f8d71798700 Compacting 6@0 + 0@1 files
2014/04/25-18:39:10.886134 7f8d71798700 Generated table #16: 178675 keys, 210224702 bytes
2014/04/25-18:39:10.887666 7f8d71798700 Compacted 6@0 + 0@1 files => 210224702 bytes
2014/04/25-18:39:10.889495 7f8d71798700 compacted to: files[ 0 1 0 0 0 0 0 ]

These messages were being generated while the compaction thread held the database mutex. The problem is that each message generated an fwrite() and fflush() call. Both calls, particularly the fflush(), could block for extended periods during heavy disk activity. Blocking during the logging became blocking of all user operations. The targeted log messages were moved and/or mutex unlocked to prevent blocking.

The changes only addressed common, regularly used log messages. Error logging, debug logging, and such were not changed.

The second scenario addressed by this branch deals with the deletion of old .sst files. There is a routine in db/db_impl.cc called DeleteObsoleteFiles().

Branch description