New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os/bluestore: fix OnodeSizeTracking testing #12684

Merged
merged 1 commit into from Dec 29, 2016

Conversation

Projects
None yet
3 participants
@xiexingguo
Member

xiexingguo commented Dec 27, 2016

http://tracker.ceph.com/issues/20498

[ FAILED ] 1 test, listed below:
[ FAILED ] ObjectStore/StoreTest.OnodeSizeTracking/2, where GetParam() = "bluestore"

1 FAILED TEST

The above test failure happens as the bluestore mount() process
will try to load all collections and put them into the coll_map,
which will be also tracked as mempool::bluestore_meta_other.
So total_bytes from mempool won't be equal to zero.

Signed-off-by: xie xingguo xie.xingguo@zte.com.cn

@ifed01

This comment has been minimized.

Show comment
Hide comment
@ifed01

ifed01 Dec 27, 2016

Contributor

@xiexingguo, I'm wondering what commit introduced collections into mempools? Can't find any...
Another question - does OnodeSizeTracking test case fail when running the whole bunch of test cases or this specific one only or doesn't matter?

Contributor

ifed01 commented Dec 27, 2016

@xiexingguo, I'm wondering what commit introduced collections into mempools? Can't find any...
Another question - does OnodeSizeTracking test case fail when running the whole bunch of test cases or this specific one only or doesn't matter?

@xiexingguo

This comment has been minimized.

Show comment
Hide comment
@xiexingguo

xiexingguo Dec 27, 2016

Member

does OnodeSizeTracking test case fail when running the whole bunch of test cases or this specific one only or doesn't matter?

Both will fail the test in my local testbed.

Member

xiexingguo commented Dec 27, 2016

does OnodeSizeTracking test case fail when running the whole bunch of test cases or this specific one only or doesn't matter?

Both will fail the test in my local testbed.

@liewegas

This comment has been minimized.

Show comment
Hide comment
@liewegas

liewegas Dec 27, 2016

Member

It's probably the SharedBlobSet in Collection

Member

liewegas commented Dec 27, 2016

It's probably the SharedBlobSet in Collection

@liewegas

This comment has been minimized.

Show comment
Hide comment
@liewegas

liewegas Dec 27, 2016

Member

...it doesn't fail for me, though. I think it needs to be a GE? My guess is that some stl libs preallocate the hash table and others don't? Maybe? If do you do a json dump of the mempool with mempool_debug=true we can see what is being allocated.

Member

liewegas commented Dec 27, 2016

...it doesn't fail for me, though. I think it needs to be a GE? My guess is that some stl libs preallocate the hash table and others don't? Maybe? If do you do a json dump of the mempool with mempool_debug=true we can see what is being allocated.

@xiexingguo

This comment has been minimized.

Show comment
Hide comment
@xiexingguo

xiexingguo Dec 28, 2016

Member

@ifed01 @liewegas

Then I think the result of this test case is non-deterministic. In my local test environment it can be
stably reproduced and below is the detail.

--- a/src/test/objectstore/store_test.cc
+++ b/src/test/objectstore/store_test.cc
@@ -5583,12 +5583,23 @@ TEST_P(StoreTest, OnodeSizeTracking) {
   g_conf->set_val("bluestore_compression_mode", "none");
   g_conf->set_val("bluestore_csum_type", "none");
   g_conf->set_val("bluestore_min_alloc_size", stringify(block_size));
+  g_conf->set_val("mempool_debug", "true");
   g_ceph_context->_conf->apply_changes(NULL);
   int r = store->umount();
   ASSERT_EQ(r, 0);
   r = store->mount(); //to force min_alloc_size update
   ASSERT_EQ(r, 0);
 
+  {
+    cout <<" mempool dump:\n";
+    JSONFormatter f(true);
+    f.open_object_section("transaction");
+    mempool::dump(&f);
+    f.close_section();
+    f.flush(cout);
+    cout << std::endl;
+  }
+

The result:

 mempool dump:
{
    "buffer_meta": {
        "total": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_char": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_malloc": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_mmap_pages": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_pipe": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_posix_aligned": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_static": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_unshareable": {
            "items": 0,
            "bytes": 0
        }
    },
    "buffer_data": {
        "total": {
            "items": 2105456,
            "bytes": 2105456
        },
        "char": {
            "items": 2105456,
            "bytes": 2105456
        }
    },
    "osd": {
        "total": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_onode": {
        "total": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Onode": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_other": {
        "total": {
            "items": 11,
            "bytes": 88
        },
        "BlueStore::Blob": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Buffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Extent": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::SharedBlob": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_alloc": {
        "total": {
            "items": 41763,
            "bytes": 826320
        },
        "AllocExtent": {
            "items": 0,
            "bytes": 0
        },
        "BitAllocator": {
            "items": 2,
            "bytes": 480
        },
        "BitMapArea": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaIN": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaLeaf": {
            "items": 13,
            "bytes": 1456
        },
        "BitMapZone": {
            "items": 2452,
            "bytes": 196160
        },
        "BmapEntry": {
            "items": 39232,
            "bytes": 627712
        }
    },
    "bluefs": {
        "total": {
            "items": 72,
            "bytes": 4040
        },
        "BlueFS::Dir": {
            "items": 1,
            "bytes": 88
        },
        "BlueFS::File": {
            "items": 11,
            "bytes": 1672
        },
        "BlueFS::FileLock": {
            "items": 1,
            "bytes": 8
        },
        "BlueFS::FileReader": {
            "items": 2,
            "bytes": 240
        },
        "BlueFS::FileReaderBuffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueFS::FileWriter": {
            "items": 3,
            "bytes": 912
        },
        "bluefs_extent_t": {
            "items": 9,
            "bytes": 144
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::Dir> > >": {
            "items": 1,
            "bytes": 48
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::File> > >": {
            "items": 10,
            "bytes": 480
        },
        "std::__detail::_Hash_node<std::pair<unsigned long const, boost::intrusive_ptr<BlueFS::File> >, false>": {
            "items": 11,
            "bytes": 264
        },
        "std::__detail::_Hash_node_base*": {
            "items": 0,
            "bytes": 0
        },
        "std::pair<unsigned long, boost::intrusive_ptr<BlueFS::File> >": {
            "items": 0,
            "bytes": 0
        }
    }
}

onode(0/0) other(88/11)
/home/xxg/dev/ceph-dev/src/test/objectstore/store_test.cc:5610: Failure
      Expected: total_bytes
      Which is: 88
To be equal to: 0u
      Which is: 0
Member

xiexingguo commented Dec 28, 2016

@ifed01 @liewegas

Then I think the result of this test case is non-deterministic. In my local test environment it can be
stably reproduced and below is the detail.

--- a/src/test/objectstore/store_test.cc
+++ b/src/test/objectstore/store_test.cc
@@ -5583,12 +5583,23 @@ TEST_P(StoreTest, OnodeSizeTracking) {
   g_conf->set_val("bluestore_compression_mode", "none");
   g_conf->set_val("bluestore_csum_type", "none");
   g_conf->set_val("bluestore_min_alloc_size", stringify(block_size));
+  g_conf->set_val("mempool_debug", "true");
   g_ceph_context->_conf->apply_changes(NULL);
   int r = store->umount();
   ASSERT_EQ(r, 0);
   r = store->mount(); //to force min_alloc_size update
   ASSERT_EQ(r, 0);
 
+  {
+    cout <<" mempool dump:\n";
+    JSONFormatter f(true);
+    f.open_object_section("transaction");
+    mempool::dump(&f);
+    f.close_section();
+    f.flush(cout);
+    cout << std::endl;
+  }
+

The result:

 mempool dump:
{
    "buffer_meta": {
        "total": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_char": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_malloc": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_mmap_pages": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_pipe": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_posix_aligned": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_static": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_unshareable": {
            "items": 0,
            "bytes": 0
        }
    },
    "buffer_data": {
        "total": {
            "items": 2105456,
            "bytes": 2105456
        },
        "char": {
            "items": 2105456,
            "bytes": 2105456
        }
    },
    "osd": {
        "total": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_onode": {
        "total": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Onode": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_other": {
        "total": {
            "items": 11,
            "bytes": 88
        },
        "BlueStore::Blob": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Buffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Extent": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::SharedBlob": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_alloc": {
        "total": {
            "items": 41763,
            "bytes": 826320
        },
        "AllocExtent": {
            "items": 0,
            "bytes": 0
        },
        "BitAllocator": {
            "items": 2,
            "bytes": 480
        },
        "BitMapArea": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaIN": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaLeaf": {
            "items": 13,
            "bytes": 1456
        },
        "BitMapZone": {
            "items": 2452,
            "bytes": 196160
        },
        "BmapEntry": {
            "items": 39232,
            "bytes": 627712
        }
    },
    "bluefs": {
        "total": {
            "items": 72,
            "bytes": 4040
        },
        "BlueFS::Dir": {
            "items": 1,
            "bytes": 88
        },
        "BlueFS::File": {
            "items": 11,
            "bytes": 1672
        },
        "BlueFS::FileLock": {
            "items": 1,
            "bytes": 8
        },
        "BlueFS::FileReader": {
            "items": 2,
            "bytes": 240
        },
        "BlueFS::FileReaderBuffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueFS::FileWriter": {
            "items": 3,
            "bytes": 912
        },
        "bluefs_extent_t": {
            "items": 9,
            "bytes": 144
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::Dir> > >": {
            "items": 1,
            "bytes": 48
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::File> > >": {
            "items": 10,
            "bytes": 480
        },
        "std::__detail::_Hash_node<std::pair<unsigned long const, boost::intrusive_ptr<BlueFS::File> >, false>": {
            "items": 11,
            "bytes": 264
        },
        "std::__detail::_Hash_node_base*": {
            "items": 0,
            "bytes": 0
        },
        "std::pair<unsigned long, boost::intrusive_ptr<BlueFS::File> >": {
            "items": 0,
            "bytes": 0
        }
    }
}

onode(0/0) other(88/11)
/home/xxg/dev/ceph-dev/src/test/objectstore/store_test.cc:5610: Failure
      Expected: total_bytes
      Which is: 88
To be equal to: 0u
      Which is: 0
@xiexingguo

This comment has been minimized.

Show comment
Hide comment
@xiexingguo

xiexingguo Dec 28, 2016

Member

Since the above output does not show as much detail as we want, I add a new pool, put the coll_map into there and retest as below:

--- a/src/include/mempool.h
+++ b/src/include/mempool.h
@@ -146,6 +146,7 @@ namespace mempool {
   f(osd)                             \
   f(bluestore_meta_onode)            \
   f(bluestore_meta_other)            \
+  f(bluestore_meta_collmap)          \
   f(bluestore_alloc)                 \
   f(bluefs)
 
diff --git a/src/os/bluestore/BlueStore.h b/src/os/bluestore/BlueStore.h
index 0bfee14..3908c45 100644
--- a/src/os/bluestore/BlueStore.h
+++ b/src/os/bluestore/BlueStore.h
@@ -1523,7 +1523,7 @@ private:
   bool mounted;
 
   RWLock coll_lock;    ///< rwlock to protect coll_map
-  mempool::bluestore_meta_other::unordered_map<coll_t, CollectionRef> coll_map;
+  mempool::bluestore_meta_collmap::unordered_map<coll_t, CollectionRef> coll_map;
 
   vector<Cache*> cache_shards;

The result(again):

 mempool dump:
{
    "buffer_meta": {
        "total": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_char": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_malloc": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_mmap_pages": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_pipe": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_posix_aligned": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_static": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_unshareable": {
            "items": 0,
            "bytes": 0
        }
    },
    "buffer_data": {
        "total": {
            "items": 2105456,
            "bytes": 2105456
        },
        "char": {
            "items": 2105456,
            "bytes": 2105456
        }
    },
    "osd": {
        "total": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_onode": {
        "total": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Onode": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_other": {
        "total": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Blob": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Buffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Extent": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::SharedBlob": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_collmap": {
        "total": {
            "items": 11,
            "bytes": 88
        }
    },
    "bluestore_alloc": {
        "total": {
            "items": 41763,
            "bytes": 826320
        },
        "AllocExtent": {
            "items": 0,
            "bytes": 0
        },
        "BitAllocator": {
            "items": 2,
            "bytes": 480
        },
        "BitMapArea": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaIN": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaLeaf": {
            "items": 13,
            "bytes": 1456
        },
        "BitMapZone": {
            "items": 2452,
            "bytes": 196160
        },
        "BmapEntry": {
            "items": 39232,
            "bytes": 627712
        }
    },
    "bluefs": {
        "total": {
            "items": 72,
            "bytes": 4040
        },
        "BlueFS::Dir": {
            "items": 1,
            "bytes": 88
        },
        "BlueFS::File": {
            "items": 11,
            "bytes": 1672
        },
        "BlueFS::FileLock": {
            "items": 1,
            "bytes": 8
        },
        "BlueFS::FileReader": {
            "items": 2,
            "bytes": 240
        },
        "BlueFS::FileReaderBuffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueFS::FileWriter": {
            "items": 3,
            "bytes": 912
        },
        "bluefs_extent_t": {
            "items": 9,
            "bytes": 144
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::Dir> > >": {
            "items": 1,
            "bytes": 48
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::File> > >": {
            "items": 10,
            "bytes": 480
        },
        "std::__detail::_Hash_node<std::pair<unsigned long const, boost::intrusive_ptr<BlueFS::File> >, false>": {
            "items": 11,
            "bytes": 264
        },
        "std::__detail::_Hash_node_base*": {
            "items": 0,
            "bytes": 0
        },
        "std::pair<unsigned long, boost::intrusive_ptr<BlueFS::File> >": {
            "items": 0,
            "bytes": 0
        }
    }
}

onode(0/0) other(0/0)
onode(608/1) other(3728/55)
onode(608/1) other(442928/4137)
Member

xiexingguo commented Dec 28, 2016

Since the above output does not show as much detail as we want, I add a new pool, put the coll_map into there and retest as below:

--- a/src/include/mempool.h
+++ b/src/include/mempool.h
@@ -146,6 +146,7 @@ namespace mempool {
   f(osd)                             \
   f(bluestore_meta_onode)            \
   f(bluestore_meta_other)            \
+  f(bluestore_meta_collmap)          \
   f(bluestore_alloc)                 \
   f(bluefs)
 
diff --git a/src/os/bluestore/BlueStore.h b/src/os/bluestore/BlueStore.h
index 0bfee14..3908c45 100644
--- a/src/os/bluestore/BlueStore.h
+++ b/src/os/bluestore/BlueStore.h
@@ -1523,7 +1523,7 @@ private:
   bool mounted;
 
   RWLock coll_lock;    ///< rwlock to protect coll_map
-  mempool::bluestore_meta_other::unordered_map<coll_t, CollectionRef> coll_map;
+  mempool::bluestore_meta_collmap::unordered_map<coll_t, CollectionRef> coll_map;
 
   vector<Cache*> cache_shards;

The result(again):

 mempool dump:
{
    "buffer_meta": {
        "total": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_char": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_malloc": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_mmap_pages": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_pipe": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_posix_aligned": {
            "items": 3,
            "bytes": 288
        },
        "ceph::buffer::raw_static": {
            "items": 0,
            "bytes": 0
        },
        "ceph::buffer::raw_unshareable": {
            "items": 0,
            "bytes": 0
        }
    },
    "buffer_data": {
        "total": {
            "items": 2105456,
            "bytes": 2105456
        },
        "char": {
            "items": 2105456,
            "bytes": 2105456
        }
    },
    "osd": {
        "total": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_onode": {
        "total": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Onode": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_other": {
        "total": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Blob": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Buffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::Extent": {
            "items": 0,
            "bytes": 0
        },
        "BlueStore::SharedBlob": {
            "items": 0,
            "bytes": 0
        }
    },
    "bluestore_meta_collmap": {
        "total": {
            "items": 11,
            "bytes": 88
        }
    },
    "bluestore_alloc": {
        "total": {
            "items": 41763,
            "bytes": 826320
        },
        "AllocExtent": {
            "items": 0,
            "bytes": 0
        },
        "BitAllocator": {
            "items": 2,
            "bytes": 480
        },
        "BitMapArea": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaIN": {
            "items": 0,
            "bytes": 0
        },
        "BitMapAreaLeaf": {
            "items": 13,
            "bytes": 1456
        },
        "BitMapZone": {
            "items": 2452,
            "bytes": 196160
        },
        "BmapEntry": {
            "items": 39232,
            "bytes": 627712
        }
    },
    "bluefs": {
        "total": {
            "items": 72,
            "bytes": 4040
        },
        "BlueFS::Dir": {
            "items": 1,
            "bytes": 88
        },
        "BlueFS::File": {
            "items": 11,
            "bytes": 1672
        },
        "BlueFS::FileLock": {
            "items": 1,
            "bytes": 8
        },
        "BlueFS::FileReader": {
            "items": 2,
            "bytes": 240
        },
        "BlueFS::FileReaderBuffer": {
            "items": 0,
            "bytes": 0
        },
        "BlueFS::FileWriter": {
            "items": 3,
            "bytes": 912
        },
        "bluefs_extent_t": {
            "items": 9,
            "bytes": 144
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::Dir> > >": {
            "items": 1,
            "bytes": 48
        },
        "std::_Rb_tree_node<std::pair<std::string const, boost::intrusive_ptr<BlueFS::File> > >": {
            "items": 10,
            "bytes": 480
        },
        "std::__detail::_Hash_node<std::pair<unsigned long const, boost::intrusive_ptr<BlueFS::File> >, false>": {
            "items": 11,
            "bytes": 264
        },
        "std::__detail::_Hash_node_base*": {
            "items": 0,
            "bytes": 0
        },
        "std::pair<unsigned long, boost::intrusive_ptr<BlueFS::File> >": {
            "items": 0,
            "bytes": 0
        }
    }
}

onode(0/0) other(0/0)
onode(608/1) other(3728/55)
onode(608/1) other(442928/4137)
@xiexingguo

This comment has been minimized.

Show comment
Hide comment
@xiexingguo

xiexingguo Dec 28, 2016

Member

...it doesn't fail for me, though. I think it needs to be a GE?

Then this statement has no effect and can be safely removed..

Member

xiexingguo commented Dec 28, 2016

...it doesn't fail for me, though. I think it needs to be a GE?

Then this statement has no effect and can be safely removed..

os/bluestore: fix OnodeSizeTracking testing
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ObjectStore/StoreTest.OnodeSizeTracking/2, where GetParam() = "bluestore"

 1 FAILED TEST

The above test failure happens as the bluestore mount() process
will try to load all collections and put them into the coll_map,
which will be also tracked as mempool::bluestore_meta_other.
So total_bytes from mempool won't be equal to zero.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
@liewegas

This comment has been minimized.

Show comment
Hide comment
@liewegas

liewegas Dec 28, 2016

Member

retest this please

Member

liewegas commented Dec 28, 2016

retest this please

@liewegas liewegas merged commit 5c1c3c0 into ceph:master Dec 29, 2016

3 checks passed

Signed-off-by all commits in this PR are signed
Details
Unmodifed Submodules submodules for project are unmodified
Details
default Build finished.
Details

@xiexingguo xiexingguo deleted the xiexingguo:xxg-wip-fix-onodesizetracking branch Dec 30, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment