New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv/rocksdb: supports SliceParts interface #15058

Merged
merged 2 commits into from May 30, 2017

Conversation

Projects
None yet
4 participants
@yuyuyu101
Member

yuyuyu101 commented May 12, 2017

No description provided.

rocksdb::Slice key_slice(key);
rocksdb::Slice value_slices[to_set_bl.buffers().size()];
bat.Put(nullptr, rocksdb::SliceParts(&key_slice, 1),
rocksdb::SlicePartsi(&value_slices, to_set_bl.buffers().size()));

This comment has been minimized.

@liewegas

liewegas May 12, 2017

Member

I don't see the part where value_slices is populated with the bufferptr data pointers...

@yuyuyu101

This comment has been minimized.

Member

yuyuyu101 commented May 12, 2017

oh, yes... lost git operation

@yuyuyu101

This comment has been minimized.

Member

yuyuyu101 commented May 12, 2017

@liewegas @markhpc the third commit add prefix_filter supports in rocksdb(https://github.com/facebook/rocksdb/wiki/Prefix-Seek-API-Changes). it will help a lot for million keys case. But it will lose iterate backward capacity. I scan the iterate use case, if accepted this, it won't be problem for current omap iterator users.

@liewegas

This comment has been minimized.

Member

liewegas commented May 12, 2017

Slice parts thing looks great (once it compiles :).

I can't think of any case where we use a reverse iterator. I'm not clear where we want to use the prefix, though.. are you thinking for the bluestore omap key prefix for each object? Or for the bluestore one-character type prefixes (it seems unlikely to help there?).

@yuyuyu101

This comment has been minimized.

Member

yuyuyu101 commented May 12, 2017

@liewegas yes, this commit just is a example.

we could define "PREFIX"(1 byte) + "INDEX_PREFIX"(8 bytes) as the rocksdb prefix
I expect we could see benefit from bluestore onode shards, rgw bucket index lookup/scan.

yuyuyu101 added some commits Apr 26, 2017

kv/RocksDBStore: make non-contiguous bl calls with SliceParts to avoi…
…d copy

Signed-off-by: Haomai Wang <haomai@xsky.com>
os/BlueRocksEnv: remove unsupported rocksdb::Env methods
upstream:
facebook/rocksdb@cdad04b

Signed-off-by: Haomai Wang <haomai@xsky.com>

@liewegas liewegas merged commit d79c772 into ceph:master May 30, 2017

3 checks passed

Signed-off-by all commits in this PR are signed
Details
Unmodifed Submodules submodules for project are unmodified
Details
default Build finished.
Details
@wjwithagen

This comment has been minimized.

Contributor

wjwithagen commented Jul 26, 2017

@yuyuyu101 @liewegas
Not sure why it only surfaces now, but Clang does not like the slices, or the way they are used.
This is on a freshly installed system, so could also be missing something.

/usr/srcs/Ceph/work.v20170726/ceph/src/kv/RocksDBStore.cc:611:32: error: variable length array of non-POD element type
      'rocksdb::Slice'
    rocksdb::Slice value_slices[to_set_bl.buffers().size()];
                               ^
/usr/srcs/Ceph/work.v20170726/ceph/src/kv/RocksDBStore.cc:632:32: error: variable length array of non-POD element type
      'rocksdb::Slice'
    rocksdb::Slice value_slices[to_set_bl.buffers().size()];
                               ^
/usr/srcs/Ceph/work.v20170726/ceph/src/kv/RocksDBStore.cc:710:32: error: variable length array of non-POD element type
      'rocksdb::Slice'
    rocksdb::Slice value_slices[to_set_bl.buffers().size()];
                               ^

@yuyuyu101 yuyuyu101 deleted the yuyuyu101:wip-rocksdb-opt branch Jul 26, 2017

@tchaikov

This comment has been minimized.

Contributor

tchaikov commented Jul 27, 2017

@wjwithagen could use a vector<rocksdb::Slice> to workaround this.

@tchaikov

This comment has been minimized.

Contributor

tchaikov commented Jul 27, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment