Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
common/OpHistory: move insert/cleanup into separate thread #20540
Cluster that's flooded with incoming ops (and enabled optracker) is bottlenecked by OpHistory::insert. Reduce that by:
My initial testing has shown this noticeably reduced optracker impact on cluster perfornance:
Using separate thread ("threaded optracker") didn't improve things by much, neither did replacing OpHistorySvc thread mutex with spinlock ("threaded optracker + spin"). Removing the ops_history_lock from the processing path (by either removing it entirely or replacing
Signed-off-by: Piotr Dałek email@example.com
Yeah, I get that, but there are reasons people tell you not to use spinlocks when memory allocation might happen. So I wonder if we can do some trick to allocate the memory and then put it on the end of the list. I'm probably worrying about it too much given the simple case presented here, though.
(Or, probably out of scope here, but did you consider giving a separate queue to each OSD op thread and having the OpTracker one pick off the front of each? That would greatly reduce the sharing...I notice you emphasize the preserved ordering but the OpTracker worker could handle that by walking forward with timestamp comparisons, and I don't think it's that big a deal anyway.)
That's what I'm thinking too. I mean, sure - your concerns are perfectly valid, but it's not like I'm allocating megabytes or even kilobytes of data. Just a few hundred of bytes max, should be little enough to not get slowed down badly by memory allocator.
Yeah, I've been thinking about it too, but at this point I think it's too early for that. Let's see how far this one takes us, and then let's optimize further. I already see few possibilities to optimize it without complicating matters much.