Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add status query for total bytes used by a LevelDB instance #74

Merged
merged 2 commits into from
May 29, 2013

Conversation

jtuple
Copy link
Contributor

@jtuple jtuple commented Mar 19, 2013

This pull-request adds a new LevelDB property "leveldb.total-bytes" that returns the total bytes used by LevelDB for SSTs. This value is equivalent to the sum of the reported per-level sizes shown in the "leveldb.status" output.


The main purpose of this change is for use in Riak through the eleveldb:status API to allow Riak to estimate the total bytes by a LevelDB vnode. The motivation being the new transfer progress work by @jrwest: basho/platform_tasks#23

This commit adds a new LevelDB property "leveldb.total-bytes" that
returns the total bytes used by LevelDB for SSTs. This value is
equivalent to the sum of the reported per-level sizes shown in the
"leveldb.status" output.
@jtuple
Copy link
Contributor Author

jtuple commented Mar 19, 2013

Changes will need to be made to riak_kv_eleveldb_backend to expose this value through the new data_size API that @jrwest has created, see: riak_kv/jrw-handoff-progress

Nevertheless, to test this feature as-is, you just need to grab an open LevelDB reference and use eleveldb:status.

To test on a running Riak console:

%% Useful funs to grab LevelDB reference from within running vnode
GS = fun(Pid) ->
             {status, Pid, _Mod, Status} = sys:get_status(Pid),
             Status2 = lists:flatten(Status),
             Status3 = [L || {data, L} <- Status2],
             Status4 = lists:flatten(Status3),
             State = proplists:get_value("StateData", Status4),
             State
     end.

LR = fun(Idx) ->
             {ok, Pid} = riak_core_vnode_manager:get_vnode_pid(Idx, riak_kv_vnode),
             State = GS(Pid),
             ModState = element(4, State),
             case element(3,ModState) of
                 riak_kv_eleveldb_backend ->
                     LvlState = element(4, ModState),
                     element(2, LvlState);
                 _ ->
                     undefined
             end
     end.

%% Use 0 or some other known vnode index...
Ref = LR(0).

eleveldb:status(Ref, <<"leveldb.total-bytes">>).

Result (size in bytes / use list_to_integer to convert to number):

{ok, <<"3373592">>}

@jonmeredith jonmeredith reopened this Mar 19, 2013
@jrwest
Copy link

jrwest commented Mar 28, 2013

works as advertised.

@jtuple we discussed last week adding in the memtable stats to "total-bytes" in order to preserve the upper-bound on transfer progress. were you going to add that as part of this PR?

@@ -1677,6 +1677,18 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
} else if (in == "sstables") {
*value = versions_->current()->DebugString();
return true;
} else if (in == "total-bytes") {
char buf[50];
double total = 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why a double and not a uint64_t?

@matthewvon
Copy link
Contributor

+1 ... updated code from using double to uint64_t. Executed manual test script from above. All is good.

matthewvon pushed a commit that referenced this pull request May 29, 2013
Add status query for total bytes used by a LevelDB instance
@matthewvon matthewvon merged commit 75b76a2 into master May 29, 2013
@matthewvon matthewvon deleted the jdb-total-bytes-stat branch May 5, 2016 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants