os/bluestore: drop unreferenced spanning blobs

[liewegas]

Otherwise we can "leak" these and keep them around indefinitely.

Signed-off-by: Sage Weil sage@redhat.com

[liewegas]

@somnathr

[somnathr]

Thanks will try...

[somnathr]

@liewegas it is not fixed. I did some digging and found it is never satisfying the if condition above.
The reason is blob->id is always -1 in this part. So, the extents we are deleting is not holding spanning blobs ever (?)
I saw spanning blob destructors are only called from get_spanning_blob() while destructing dummy blob.
Another point, I am seeing even for the blob->id = -1 , the ref count for the blob is not getting to 0 from _wctx_finish() after deletion of extent, is this expected ?

[liewegas]

Hmm, that doesn't make sense to me. FWIW I see this triggering in my workload (ceph_test_objectstore --gtest_filter=*Synthetic/2). Do you have a log capture for a case where the ref_map doesn't become empty? Also, if b->id == -1 then it's not a spanning blob, so it won't be in that map at all..

[somnathr]

Hmm..I can collect logs but before that let me move to latest master to be in sync with you.
Yes, if b->id = -1 is not a spanning blob case, but, I was talking about non-spanning blob case as well.
Isn't the expectation of non-spanning blob case that ref count will be 0 from here ?

[liewegas]

For the non-spanning case, the Extent goes away and drops the BlobRef and it's deallocated. We only need to handle the spanning case explicitly because the spanning_blob_map has a ref to the Blob too.

[somnathr]

Yeah, that's what I thought too , but, I added some log to print b->nref after extent delete in the _wctx_finish loop , it was printing non zero ref.
For spanning blob case, I synced to latest master and cherry-picked your commit again , it's the same..
it is not entering in the if block because b->id is always -1 , I am not seeing any blob of id >=0

[liewegas]

Most of the time nref will be > 0.. it's only when the last extent is dropped that it goes to 0 (or 1, if it's spanning). What is your workload? Maybe i can reproduce

[liewegas]

Or can you generate a log?

[somnathr]

I created 400G image (without exclusive lock etc.) , preconditioned with 1M SW.
After that, from a single client I am running 4K RW (16 QD, 10 jobs).
Once it is going beyonf inline_bl (1200 bytes) and having spanning blobs , I am seeing memory started growing.
I can generate logs as well , debug_bluestore = 20/20 ?

[liewegas]

Yeah, 20/20. I'll try to reproduce it too. Thanks!

[somnathr]

@liewegas debug_bluestore = 20/20 log is in the below location. Log I started capturing only when I started 4K RW IO after preconditioning. Let me know if you are having difficulties on accessing it.

https://drive.google.com/file/d/0B7W-S0z_ymMJR08yOUxiNkh5LW8/view?usp=sharing

[somnathr]

@liewegas Please look at #11223 , it seems to fix the issue for me.