data_block_manager_t::check_and_handle_empty_extent() is slow

[danielmewes]

It takes up to 20% of the server's CPU time during inserts.
What's worse though is that it is called in data_block_manager_t::mark_garbage() which in turn is called heavily in log_serializer_t::index_write() in a section where we cannot yield. This often hogs the thread for something like 100ms, driving latency up.

We either have to make check_and_handle_empty_extent() faster or modify index_write() to be able to yield (probably by adding some lock).

Roughly half of the time in check_and_handle_empty_extent is spent on updating the priority queue (see the line entry->our_pq_entry->update(); at the end of the function). The profiler could not assign the reamining half to any part in particular. My current suspect is entry->all_garbage() which ends up performing a loop each time it is called.

[danielmewes]

Actually the priority queue code uses a comparison operator (gc_entry_less_t) which calls gc_entry_t::garbage_bytes() which similar to entry->all_garbage() has to loop over a list of block entries.
I'm hopeful that optimizing those operations might solve the issue altogether.

[danielmewes]

A fix is in branch daniel_1559 and code review 989.

[danielmewes]

Merged into next e9a0f6a