os/bluestore: fix OnodeSizeTracking testing #12684

xiexingguo · 2016-12-27T12:05:12Z

http://tracker.ceph.com/issues/20498

[ FAILED ] 1 test, listed below:
[ FAILED ] ObjectStore/StoreTest.OnodeSizeTracking/2, where GetParam() = "bluestore"

1 FAILED TEST

The above test failure happens as the bluestore mount() process
will try to load all collections and put them into the coll_map,
which will be also tracked as mempool::bluestore_meta_other.
So total_bytes from mempool won't be equal to zero.

Signed-off-by: xie xingguo xie.xingguo@zte.com.cn

ifed01 · 2016-12-27T13:46:54Z

@xiexingguo, I'm wondering what commit introduced collections into mempools? Can't find any...
Another question - does OnodeSizeTracking test case fail when running the whole bunch of test cases or this specific one only or doesn't matter?

xiexingguo · 2016-12-27T14:36:06Z

does OnodeSizeTracking test case fail when running the whole bunch of test cases or this specific one only or doesn't matter?

Both will fail the test in my local testbed.

liewegas · 2016-12-27T14:59:31Z

It's probably the SharedBlobSet in Collection

liewegas · 2016-12-27T15:22:07Z

...it doesn't fail for me, though. I think it needs to be a GE? My guess is that some stl libs preallocate the hash table and others don't? Maybe? If do you do a json dump of the mempool with mempool_debug=true we can see what is being allocated.

xiexingguo · 2016-12-28T00:55:01Z

@ifed01 @liewegas

Then I think the result of this test case is non-deterministic. In my local test environment it can be
stably reproduced and below is the detail.

--- a/src/test/objectstore/store_test.cc
+++ b/src/test/objectstore/store_test.cc
@@ -5583,12 +5583,23 @@ TEST_P(StoreTest, OnodeSizeTracking) {
   g_conf->set_val("bluestore_compression_mode", "none");
   g_conf->set_val("bluestore_csum_type", "none");
   g_conf->set_val("bluestore_min_alloc_size", stringify(block_size));
+  g_conf->set_val("mempool_debug", "true");
   g_ceph_context->_conf->apply_changes(NULL);
   int r = store->umount();
   ASSERT_EQ(r, 0);
   r = store->mount(); //to force min_alloc_size update
   ASSERT_EQ(r, 0);
 
+  {
+    cout <<" mempool dump:\n";
+    JSONFormatter f(true);
+    f.open_object_section("transaction");
+    mempool::dump(&f);
+    f.close_section();
+    f.flush(cout);
+    cout << std::endl;
+  }
+

The result:

 mempool dump:
{
    "buffer_meta": {
        "total": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_char": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_malloc": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_mmap_pages": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_pipe": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_posix_aligned": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_static": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_unshareable": {
            "items": 0,
            "bytes": 0
        }
    },
    "buffer_data": {
        "total": {
            "items": 2105456,
            "bytes": 2105456
        },
        "char": {
            "items": 2105456,
            "bytes": 2105456
        }
    },
    "osd": {
        "total": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_onode": {
        "total": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Onode": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_other": {
        "total": {
            "items": 11,
            "bytes": 88
        },
        "BlueStore::Blob": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Buffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Extent": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::SharedBlob": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_alloc": {
        "total": {
            "items": 41763,
            "bytes": 826320
        },
        "AllocExtent": {
            "items": 0,
            "bytes": 0
        },
        "BitAllocator": {
            "items": 2,
            "bytes": 480
        },
        "BitMapArea": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaIN": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaLeaf": {
            "items": 13,
            "bytes": 1456
        },
        "BitMapZone": {
            "items": 2452,
            "bytes": 196160
        },
        "BmapEntry": {
            "items": 39232,
            "bytes": 627712
        }
    },
    "bluefs": {
        "total": {
            "items": 72,
            "bytes": 4040
        },
        "BlueFS::Dir": {
            "items": 1,
            "bytes": 88
        },
        "BlueFS::File": {
            "items": 11,
            "bytes": 1672
        },
        "BlueFS::FileLock": {
            "items": 1,
            "bytes": 8
        },
        "BlueFS::FileReader": {
            "items": 2,
            "bytes": 240
        },
        "BlueFS::FileReaderBuffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueFS::FileWriter": {
            "items": 3,
            "bytes": 912
        },
        "bluefs_extent_t": {
            "items": 9,
            "bytes": 144
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::Dir> > >": {
            "items": 1,
            "bytes": 48
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::File> > >": {
            "items": 10,
            "bytes": 480
        },
        "std::__detail::_Hash_node<std::pair<unsigned long const, boost::intrusive_ptr<BlueFS::File> >, false>": {
            "items": 11,
            "bytes": 264
        },
        "std::__detail::_Hash_node_base*": {
            "items": 0,
            "bytes": 0
        },
        "std::pair<unsigned long, boost::intrusive_ptr<BlueFS::File> >": {
            "items": 0,
            "bytes": 0
        }
    }
}

onode(0/0) other(88/11)
/home/xxg/dev/ceph-dev/src/test/objectstore/store_test.cc:5610: Failure
      Expected: total_bytes
      Which is: 88
To be equal to: 0u
      Which is: 0

xiexingguo · 2016-12-28T01:24:54Z

Since the above output does not show as much detail as we want, I add a new pool, put the coll_map into there and retest as below:

--- a/src/include/mempool.h
+++ b/src/include/mempool.h
@@ -146,6 +146,7 @@ namespace mempool {
   f(osd)                             \
   f(bluestore_meta_onode)            \
   f(bluestore_meta_other)            \
+  f(bluestore_meta_collmap)          \
   f(bluestore_alloc)                 \
   f(bluefs)
 
diff --git a/src/os/bluestore/BlueStore.h b/src/os/bluestore/BlueStore.h
index 0bfee14..3908c45 100644
--- a/src/os/bluestore/BlueStore.h
+++ b/src/os/bluestore/BlueStore.h
@@ -1523,7 +1523,7 @@ private:
   bool mounted;
 
   RWLock coll_lock;    ///< rwlock to protect coll_map
-  mempool::bluestore_meta_other::unordered_map<coll_t, CollectionRef> coll_map;
+  mempool::bluestore_meta_collmap::unordered_map<coll_t, CollectionRef> coll_map;
 
   vector<Cache*> cache_shards;

The result(again):

 mempool dump:
{
    "buffer_meta": {
        "total": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_char": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_malloc": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_mmap_pages": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_pipe": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_posix_aligned": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_static": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_unshareable": {
            "items": 0,
            "bytes": 0
        }
    },
    "buffer_data": {
        "total": {
            "items": 2105456,
            "bytes": 2105456
        },
        "char": {
            "items": 2105456,
            "bytes": 2105456
        }
    },
    "osd": {
        "total": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_onode": {
        "total": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Onode": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_other": {
        "total": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Blob": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Buffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Extent": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::SharedBlob": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_collmap": {
        "total": {
            "items": 11,
            "bytes": 88
        }
    },
    "bluestore_alloc": {
        "total": {
            "items": 41763,
            "bytes": 826320
        },
        "AllocExtent": {
            "items": 0,
            "bytes": 0
        },
        "BitAllocator": {
            "items": 2,
            "bytes": 480
        },
        "BitMapArea": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaIN": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaLeaf": {
            "items": 13,
            "bytes": 1456
        },
        "BitMapZone": {
            "items": 2452,
            "bytes": 196160
        },
        "BmapEntry": {
            "items": 39232,
            "bytes": 627712
        }
    },
    "bluefs": {
        "total": {
            "items": 72,
            "bytes": 4040
        },
        "BlueFS::Dir": {
            "items": 1,
            "bytes": 88
        },
        "BlueFS::File": {
            "items": 11,
            "bytes": 1672
        },
        "BlueFS::FileLock": {
            "items": 1,
            "bytes": 8
        },
        "BlueFS::FileReader": {
            "items": 2,
            "bytes": 240
        },
        "BlueFS::FileReaderBuffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueFS::FileWriter": {
            "items": 3,
            "bytes": 912
        },
        "bluefs_extent_t": {
            "items": 9,
            "bytes": 144
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::Dir> > >": {
            "items": 1,
            "bytes": 48
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::File> > >": {
            "items": 10,
            "bytes": 480
        },
        "std::__detail::_Hash_node<std::pair<unsigned long const, boost::intrusive_ptr<BlueFS::File> >, false>": {
            "items": 11,
            "bytes": 264
        },
        "std::__detail::_Hash_node_base*": {
            "items": 0,
            "bytes": 0
        },
        "std::pair<unsigned long, boost::intrusive_ptr<BlueFS::File> >": {
            "items": 0,
            "bytes": 0
        }
    }
}

onode(0/0) other(0/0)
onode(608/1) other(3728/55)
onode(608/1) other(442928/4137)

xiexingguo · 2016-12-28T01:49:03Z

...it doesn't fail for me, though. I think it needs to be a GE?

Then this statement has no effect and can be safely removed..

[ FAILED ] 1 test, listed below: [ FAILED ] ObjectStore/StoreTest.OnodeSizeTracking/2, where GetParam() = "bluestore" 1 FAILED TEST The above test failure happens as the bluestore mount() process will try to load all collections and put them into the coll_map, which will be also tracked as mempool::bluestore_meta_other. So total_bytes from mempool won't be equal to zero. Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

liewegas · 2016-12-28T17:20:27Z

retest this please

xiexingguo added the bluestore label Dec 27, 2016

xiexingguo assigned liewegas Dec 27, 2016

xiexingguo force-pushed the xxg-wip-fix-onodesizetracking branch from abb5b57 to 1d97862 Compare December 28, 2016 01:48

liewegas approved these changes Dec 28, 2016

View reviewed changes

liewegas merged commit 5c1c3c0 into ceph:master Dec 29, 2016

xiexingguo deleted the xxg-wip-fix-onodesizetracking branch December 30, 2016 00:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

os/bluestore: fix OnodeSizeTracking testing #12684

os/bluestore: fix OnodeSizeTracking testing #12684

xiexingguo commented Dec 27, 2016 •

edited by smithfarm

ifed01 commented Dec 27, 2016

xiexingguo commented Dec 27, 2016

liewegas commented Dec 27, 2016

liewegas commented Dec 27, 2016

xiexingguo commented Dec 28, 2016

xiexingguo commented Dec 28, 2016

xiexingguo commented Dec 28, 2016

liewegas commented Dec 28, 2016

os/bluestore: fix OnodeSizeTracking testing #12684

os/bluestore: fix OnodeSizeTracking testing #12684

Conversation

xiexingguo commented Dec 27, 2016 • edited by smithfarm

ifed01 commented Dec 27, 2016

xiexingguo commented Dec 27, 2016

liewegas commented Dec 27, 2016

liewegas commented Dec 27, 2016

xiexingguo commented Dec 28, 2016

xiexingguo commented Dec 28, 2016

xiexingguo commented Dec 28, 2016

liewegas commented Dec 28, 2016

xiexingguo commented Dec 27, 2016 •

edited by smithfarm