Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Add status query for total bytes used by a LevelDB instance #74

Merged
merged 2 commits into from May 29, 2013

Conversation

Projects
None yet
4 participants
Contributor

jtuple commented Mar 19, 2013

This pull-request adds a new LevelDB property "leveldb.total-bytes" that returns the total bytes used by LevelDB for SSTs. This value is equivalent to the sum of the reported per-level sizes shown in the "leveldb.status" output.


The main purpose of this change is for use in Riak through the eleveldb:status API to allow Riak to estimate the total bytes by a LevelDB vnode. The motivation being the new transfer progress work by @jrwest: basho/platform_tasks#23

@jtuple jtuple Add status query for total bytes used by a LevelDB instance
This commit adds a new LevelDB property "leveldb.total-bytes" that
returns the total bytes used by LevelDB for SSTs. This value is
equivalent to the sum of the reported per-level sizes shown in the
"leveldb.status" output.
6ca6e9e
Contributor

jtuple commented Mar 19, 2013

Changes will need to be made to riak_kv_eleveldb_backend to expose this value through the new data_size API that @jrwest has created, see: riak_kv/jrw-handoff-progress

Nevertheless, to test this feature as-is, you just need to grab an open LevelDB reference and use eleveldb:status.

To test on a running Riak console:

%% Useful funs to grab LevelDB reference from within running vnode
GS = fun(Pid) ->
             {status, Pid, _Mod, Status} = sys:get_status(Pid),
             Status2 = lists:flatten(Status),
             Status3 = [L || {data, L} <- Status2],
             Status4 = lists:flatten(Status3),
             State = proplists:get_value("StateData", Status4),
             State
     end.

LR = fun(Idx) ->
             {ok, Pid} = riak_core_vnode_manager:get_vnode_pid(Idx, riak_kv_vnode),
             State = GS(Pid),
             ModState = element(4, State),
             case element(3,ModState) of
                 riak_kv_eleveldb_backend ->
                     LvlState = element(4, ModState),
                     element(2, LvlState);
                 _ ->
                     undefined
             end
     end.

%% Use 0 or some other known vnode index...
Ref = LR(0).

eleveldb:status(Ref, <<"leveldb.total-bytes">>).

Result (size in bytes / use list_to_integer to convert to number):

{ok, <<"3373592">>}

@jonmeredith jonmeredith reopened this Mar 19, 2013

jrwest commented Mar 28, 2013

works as advertised.

@jtuple we discussed last week adding in the memtable stats to "total-bytes" in order to preserve the upper-bound on transfer progress. were you going to add that as part of this PR?

@jrwest jrwest referenced this pull request in basho/riak_kv Apr 1, 2013

Merged

Expose Backend Size to Handoff for Progress Tracking #526

@matthewvon matthewvon commented on an outdated diff Apr 2, 2013

db/db_impl.cc
@@ -1677,6 +1677,18 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
} else if (in == "sstables") {
*value = versions_->current()->DebugString();
return true;
+ } else if (in == "total-bytes") {
+ char buf[50];
+ double total = 0;
@matthewvon

matthewvon Apr 2, 2013

Contributor

why a double and not a uint64_t?

@matthewvon matthewvon commented on an outdated diff Apr 2, 2013

db/db_impl.cc
@@ -1677,6 +1677,18 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
} else if (in == "sstables") {
*value = versions_->current()->DebugString();
return true;
+ } else if (in == "total-bytes") {
+ char buf[50];
+ double total = 0;
+ for (int level = 0; level < config::kNumLevels; level++) {
+ int files = versions_->NumLevelFiles(level);
+ if (stats_[level].micros > 0 || files > 0) {
@matthewvon

matthewvon Apr 2, 2013

Contributor

Why this test of micros? In fact why even test files? If files is zero the NumLevelBytes is going to return really fast anyway.

@matthewvon matthewvon commented on an outdated diff Apr 2, 2013

db/db_impl.cc
@@ -1677,6 +1677,18 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
} else if (in == "sstables") {
*value = versions_->current()->DebugString();
return true;
+ } else if (in == "total-bytes") {
+ char buf[50];
+ double total = 0;
+ for (int level = 0; level < config::kNumLevels; level++) {
+ int files = versions_->NumLevelFiles(level);
+ if (stats_[level].micros > 0 || files > 0) {
+ total += versions_->NumLevelBytes(level);
+ }
+ }
+ snprintf(buf, sizeof(buf), "%.0f", total);
@matthewvon

matthewvon Apr 2, 2013

Contributor

so if total becomes uint64_t, this would need to change also

Contributor

matthewvon commented May 29, 2013

+1 ... updated code from using double to uint64_t. Executed manual test script from above. All is good.

@matthewvon matthewvon pushed a commit that referenced this pull request May 29, 2013

Matthew Von-Maszewski Merge pull request #74 from basho/jdb-total-bytes-stat
Add status query for total bytes used by a LevelDB instance
75b76a2

@matthewvon matthewvon merged commit 75b76a2 into master May 29, 2013

@matthewvon matthewvon deleted the jdb-total-bytes-stat branch May 5, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment