Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add status query for total bytes used by a LevelDB instance #74

Merged
merged 2 commits into from

4 participants

@jtuple
Owner

This pull-request adds a new LevelDB property "leveldb.total-bytes" that returns the total bytes used by LevelDB for SSTs. This value is equivalent to the sum of the reported per-level sizes shown in the "leveldb.status" output.


The main purpose of this change is for use in Riak through the eleveldb:status API to allow Riak to estimate the total bytes by a LevelDB vnode. The motivation being the new transfer progress work by @jrwest: basho/platform_tasks#23

@jtuple jtuple Add status query for total bytes used by a LevelDB instance
This commit adds a new LevelDB property "leveldb.total-bytes" that
returns the total bytes used by LevelDB for SSTs. This value is
equivalent to the sum of the reported per-level sizes shown in the
"leveldb.status" output.
6ca6e9e
@jtuple
Owner

Changes will need to be made to riak_kv_eleveldb_backend to expose this value through the new data_size API that @jrwest has created, see: riak_kv/jrw-handoff-progress

Nevertheless, to test this feature as-is, you just need to grab an open LevelDB reference and use eleveldb:status.

To test on a running Riak console:

%% Useful funs to grab LevelDB reference from within running vnode
GS = fun(Pid) ->
             {status, Pid, _Mod, Status} = sys:get_status(Pid),
             Status2 = lists:flatten(Status),
             Status3 = [L || {data, L} <- Status2],
             Status4 = lists:flatten(Status3),
             State = proplists:get_value("StateData", Status4),
             State
     end.

LR = fun(Idx) ->
             {ok, Pid} = riak_core_vnode_manager:get_vnode_pid(Idx, riak_kv_vnode),
             State = GS(Pid),
             ModState = element(4, State),
             case element(3,ModState) of
                 riak_kv_eleveldb_backend ->
                     LvlState = element(4, ModState),
                     element(2, LvlState);
                 _ ->
                     undefined
             end
     end.

%% Use 0 or some other known vnode index...
Ref = LR(0).

eleveldb:status(Ref, <<"leveldb.total-bytes">>).

Result (size in bytes / use list_to_integer to convert to number):

{ok, <<"3373592">>}
@jonmeredith jonmeredith reopened this
@jrwest

works as advertised.

@jtuple we discussed last week adding in the memtable stats to "total-bytes" in order to preserve the upper-bound on transfer progress. were you going to add that as part of this PR?

@jrwest jrwest referenced this pull request in basho/riak_kv
Merged

Expose Backend Size to Handoff for Progress Tracking #526

db/db_impl.cc
@@ -1677,6 +1677,18 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
} else if (in == "sstables") {
*value = versions_->current()->DebugString();
return true;
+ } else if (in == "total-bytes") {
+ char buf[50];
+ double total = 0;

why a double and not a uint64_t?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
db/db_impl.cc
@@ -1677,6 +1677,18 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
} else if (in == "sstables") {
*value = versions_->current()->DebugString();
return true;
+ } else if (in == "total-bytes") {
+ char buf[50];
+ double total = 0;
+ for (int level = 0; level < config::kNumLevels; level++) {
+ int files = versions_->NumLevelFiles(level);
+ if (stats_[level].micros > 0 || files > 0) {

Why this test of micros? In fact why even test files? If files is zero the NumLevelBytes is going to return really fast anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
db/db_impl.cc
@@ -1677,6 +1677,18 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
} else if (in == "sstables") {
*value = versions_->current()->DebugString();
return true;
+ } else if (in == "total-bytes") {
+ char buf[50];
+ double total = 0;
+ for (int level = 0; level < config::kNumLevels; level++) {
+ int files = versions_->NumLevelFiles(level);
+ if (stats_[level].micros > 0 || files > 0) {
+ total += versions_->NumLevelBytes(level);
+ }
+ }
+ snprintf(buf, sizeof(buf), "%.0f", total);

so if total becomes uint64_t, this would need to change also

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@matthewvon

+1 ... updated code from using double to uint64_t. Executed manual test script from above. All is good.

@matthewvon matthewvon merged commit 75b76a2 into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 19, 2013
  1. @jtuple

    Add status query for total bytes used by a LevelDB instance

    jtuple authored
    This commit adds a new LevelDB property "leveldb.total-bytes" that
    returns the total bytes used by LevelDB for SSTs. This value is
    equivalent to the sum of the reported per-level sizes shown in the
    "leveldb.status" output.
Commits on May 29, 2013
  1. change total from float to uint64_t

    matthewvon authored
This page is out of date. Refresh to see the latest.
Showing with 9 additions and 0 deletions.
  1. +9 −0 db/db_impl.cc
View
9 db/db_impl.cc
@@ -1677,6 +1677,15 @@ bool DBImpl::GetProperty(const Slice& property, std::string* value) {
} else if (in == "sstables") {
*value = versions_->current()->DebugString();
return true;
+ } else if (in == "total-bytes") {
+ char buf[50];
+ uint64_t total = 0;
+ for (int level = 0; level < config::kNumLevels; level++) {
+ total += versions_->NumLevelBytes(level);
+ }
+ snprintf(buf, sizeof(buf), "%" PRIu64, total);
+ value->append(buf);
+ return true;
} else if (-1!=gPerfCounters->LookupCounter(in.ToString().c_str())) {
char buf[66];
Something went wrong with that request. Please try again.