os/bluestore: narrow lock scope for cache trim()

[xiexingguo]

The cache trim() process can become time consuming
when the cache size is huge. Since it can be operated
safely under its own internal mutex, we shall avoid
to hold irrelevant locks when possible, which is good
for performance.

[xiexingguo]

2016-07-23 12:51:26.319710 7ff4dbc42680 1 bluestore(store_test_temp_dir) fsck finish with 0 errors
[ OK ] ObjectStore/StoreTest.TooManyBlobsTest/2 (22819 ms)
[----------] 59 tests from ObjectStore/StoreTest (2114529 ms total)

[----------] Global test environment tear-down
[==========] 59 tests from 1 test case ran. (2114531 ms total)
[ PASSED ] 59 tests.

[somnathr]

@xiexingguo We can't do that cache->trim needs to be within collection lock as it is modifying onode structures..

[xiexingguo]

Negative, cache->trim() does cache cleanup only and is under its own mutex protection already.

We can't do that cache->trim needs to be within collection lock as it is modifying onode structures..

Otherwise shouldn't we promote to WLocker(c->lock) here?

[somnathr]

trim() does clear blob_map and onode_map. protecting against cache mutex is not good enough.
All the onode operations should be protected by collection lock. In this case, you are right, it should be promoted to Wlock at the time of clearing the onode structures..
BTW, I don't think taking write lock in the beginning instead of Rlock will be impacting performance much as the requests are been serialized within a PG (collection).

[liewegas]

The cache->lock protects the BufferSpace and OnodeSpace structures that get trimmed.

[liewegas]

lgtm

[somnathr]

@liewegas we saw a crash that Mark reported because the cache->trim() is not happening within c->lock() from _osr_reap_done(). The pull request #10578 seems to fix that.
onodes are accessed within c->lock() and without cache->lock() (for ex, onode->flush() ) and if from trim() it decrements reference without c->lock() , it may delete onode object while there still be user.

Am I missing anything ?

[liewegas]

trim should operate with only cache->lock, and any place where it affects a structure needs to be protected with cache->lock in the forward path (not the collection lock).