Add API for writing wide-column entities #10242

ltamasi · 2022-06-23T03:31:41Z

Summary:
The patch builds on #9915 and adds
a new API called PutEntity that can be used to write a wide-column entity
to the database. The new API is added to both DB and WriteBatch. Note
that currently there is no way to retrieve these entities; more precisely, all
read APIs (Get, MultiGet, and iterator) return NotSupported when they
encounter a wide-column entity that is required to answer a query. Read-side
support (as well as other missing functionality like Merge, compaction filter,
and timestamp support) will be added in later PRs.

Test Plan:
make check

…y when reading from write batch; add KV protection logic

…or now)

… a check

…tContext

facebook-github-bot · 2022-06-23T03:32:03Z

@ltamasi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-06-23T05:17:22Z

@ltamasi has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-23T05:17:41Z

@ltamasi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

riversand963 · 2022-06-23T21:41:18Z

db/compaction/compaction_iterator.cc

@@ -519,7 +520,8 @@ void CompactionIterator::NextFromInput() {
      // In the previous iteration we encountered a single delete that we could
      // not compact out.  We will keep this Put, but can drop it's data.
      // (See Optimization 3, below.)
-      if (ikey_.type != kTypeValue && ikey_.type != kTypeBlobIndex) {
+      if (ikey_.type != kTypeValue && ikey_.type != kTypeBlobIndex &&
+          ikey_.type != kTypeWideColumnEntity) {


This means, if a logical entity spans across n column families, then the entity can go through the compaction filter n times.

I think this line is part of the SingleDelete processing logic (not compaction filter), right?

Otherwise, yes, when an entity is split across column families, the compaction filter would get called for each part separately. (It might actually be a different compaction filter for each, since compaction filter is a CF option.)

riversand963 · 2022-06-23T21:45:01Z

db/compaction/compaction_iterator.cc

@@ -533,7 +535,7 @@ void CompactionIterator::NextFromInput() {
        assert(false);
      }

-      if (ikey_.type == kTypeBlobIndex) {
+      if (ikey_.type == kTypeBlobIndex || ikey_.type == kTypeWideColumnEntity) {


When should compaction convert a wide column entity into a normal PUT?

This is also related to SingleDelete, namely "optimization 3" where we preserve a SingleDelete + Put combination for the purposes of txn conflict checking but clear out the Put's value to save space.

ah I misread the code before. Thanks

riversand963 · 2022-06-23T21:54:40Z

db/write_batch.cc

+}
+
+Status WriteBatch::PutEntity(ColumnFamilyHandle* column_family,
+                             const Slice& key, const WideColumns& columns) {


Since this is new API, I wonder if we should enforce column_family != nullptr from the beginning. Not sure.

Yeah, good point, we can definitely do that. Thanks!

riversand963 · 2022-06-23T21:55:19Z

include/rocksdb/utilities/write_batch_with_index.h

@@ -113,6 +113,13 @@ class WriteBatchWithIndex : public WriteBatchBase {
  Status Put(ColumnFamilyHandle* column_family, const Slice& key,
             const Slice& ts, const Slice& value) override;

+  Status PutEntity(ColumnFamilyHandle* /* column_family */,
+                   const Slice& /* key */,
+                   const WideColumns& /* columns */) override {


I think we can at least enforce column_family != nullptr.

facebook-github-bot · 2022-06-24T21:52:48Z

@ltamasi has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-24T21:53:04Z

@ltamasi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

riversand963

Thanks @ltamasi for the PR! Took another pass and left some comments.

riversand963 · 2022-06-24T21:47:20Z

db/db_kv_checksum_test.cc

@@ -62,6 +63,9 @@ std::pair<WriteBatch, Status> GetWriteBatch(ColumnFamilyHandle* cf_handle,
    case WriteBatchOpType::kMerge:
      s = wb.Merge(cf_handle, "key", "val");
      break;
+    case WriteBatchOpType::kPutEntity:
+      s = wb.PutEntity(cf_handle, "key", {{"foo", "bar"}, {"hello", "world"}});


Nit: maybe consider

s = wb.PutEntity(cf_handle, "key", {{"attr_name1", "bar"}, {"attr_name2", "world"}});

riversand963 · 2022-06-24T21:59:59Z

db/wide/wide_column_serialization.cc

    const Slice& name = column.name();
    if (name.size() >
        static_cast<size_t>(std::numeric_limits<uint32_t>::max())) {
      return Status::InvalidArgument("Wide column name too long");
    }
+    if (i > 0 && columns[i - 1].name().compare(name) >= 0) {
+      return Status::Corruption("Wide columns out of order");


If we sort the column names in WriteBatchInternal, is it possible that the column names here are out-of-order?

This will be called from other places too but yeah, strictly out of order is not possible during PutEntity. Duplicates are possible though.

riversand963 · 2022-06-24T22:03:49Z

table/get_context.h

@@ -74,6 +74,7 @@ class GetContext {
    kCorrupt,
    kMerge,  // saver contains the current merge result (the operands)
    kUnexpectedBlobIndex,
+    kUnexpectedWideColumnEntity,


Will this be removed in the future if wide column is generally supported in RocksDB? If so, maybe put a TODO.

riversand963 · 2022-06-24T22:18:44Z

db/compaction/compaction_iterator.cc

@@ -533,7 +535,7 @@ void CompactionIterator::NextFromInput() {
        assert(false);
      }

-      if (ikey_.type == kTypeBlobIndex) {
+      if (ikey_.type == kTypeBlobIndex || ikey_.type == kTypeWideColumnEntity) {


ah I misread the code before. Thanks

riversand963 · 2022-06-24T22:20:26Z

include/rocksdb/write_batch.h

+    virtual Status PutEntityCF(uint32_t /* column_family_id */,
+                               const Slice& /* key */,
+                               const Slice& /* entity */) {
+      return Status::InvalidArgument("PutEntityCF not implemented");


NotSupported or InvalidArgument?

Will change it to NotSupported

riversand963 · 2022-06-24T22:40:34Z

db/wide/db_wide_basic_test.cc

+
+class DBWideBasicTest : public DBTestBase {
+ protected:
+  DBWideBasicTest()


Nit: explicit

Don't think it makes much of a difference in this context but sure, we can make it explicit

ltamasi · 2022-06-25T20:27:52Z

Thanks so much for the review @riversand963 !

facebook-github-bot · 2022-06-25T20:27:55Z

@ltamasi has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-25T20:28:33Z

@ltamasi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-06-25T20:36:16Z

@ltamasi has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-25T20:36:38Z

@ltamasi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

ltamasi added 30 commits June 22, 2022 14:30

Add PutEntity API

41b4171

Disallow PutEntity in compacted/read-only/secondary mode

3325f7d

Partial support in WriteBatch (WIP)

3c4da5e

Use length-prefixed encoding and do not deserialize wide-column entit…

2239161

…y when reading from write batch; add KV protection logic

Revert formatting change

3b4b4fe

Implement DB::PutEntity

f8b61bf

Implement MemTableInserter::PutEntityCF

4d0117e

Add kTypeColumnFamilyWideColumnEntity

a2983ab

Follow up on some recent changes after rebase

aecc846

More of the same

9197cbb

Add PutEntityCF to ProtectionInfoUpdater

2a208f9

Error out when encountering wide-column entities in MergeHelper

061b7a6

Initial memtable support

9772b96

Initial support in CompactionIterator (no compaction filter support f…

131f10e

…or now)

Add stub logic to DBIter

e433109

Rework wide-column support in SaveValue a bit

6f88cff

Small follow-up for some recent protection info related changes

41925c6

Stub logic in GetContext

9c8f4da

Update DataBlockIter::SeekForGetImpl

fc52e82

Pass protection_bytes_per_key when constructing WriteBatch

394fe1e

Disallow Merge for now in DBIter

0ec41ac

Small cleanup around disallowing wide-column entities in GetContext

ef41e5d

Rename method to SetWideColumnValueIfNeeded

89e947e

Disallow Merge in even more places

e0fc497

Sort columns in WriteBatchInternal::PutEntity

cae54d3

Fix typo in sort order

32f81b9

Turn assertion about columns being properly ordered in Serialize into…

79499d4

… a check

Use NotSupported instead of Corruption in DBIter

9f17518

Return NotSupported from memtable

edc3940

Use NotSupported instead of Corruption for wide-column entities in Ge…

3553c24

…tContext

ltamasi added 4 commits June 22, 2022 16:16

Update db_kv_checksum_test

1cf8b77

Add some TODOs

8365d5f

Add unit test coverage for Merge

e7c5012

Small cleanup in SaveValue

982ce69

ltamasi requested review from siying and riversand963 June 23, 2022 03:31

facebook-github-bot added the CLA Signed label Jun 23, 2022

ltamasi changed the title ~~Add API for writing a wide-column entities~~ Add API for writing wide-column entities Jun 23, 2022

Attempt to fix Windows build

53e3d21

riversand963 reviewed Jun 23, 2022

View reviewed changes

ltamasi added 2 commits June 24, 2022 14:33

Make column_family mandatory

c9f2ccb

Fix up db_kv_checksum_test

cc39473

riversand963 approved these changes Jun 24, 2022

View reviewed changes

ltamasi added 4 commits June 25, 2022 13:17

Update some column names in unit tests

b8a64ff

Add a TODO to remove kUnexpectedWideColumnEntity

fb56a9c

Return NotSupported from WriteBatch::Handler::PutEntityCF

987c2c8

Make ctor explicit

5091ad9

Add 'under construction' comment for public APIs

8d8a0c6

facebook-github-bot closed this in c73d2a9 Jun 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add API for writing wide-column entities #10242

Add API for writing wide-column entities #10242

ltamasi commented Jun 23, 2022

facebook-github-bot commented Jun 23, 2022

facebook-github-bot commented Jun 23, 2022

facebook-github-bot commented Jun 23, 2022

riversand963 Jun 23, 2022

ltamasi Jun 24, 2022

riversand963 Jun 23, 2022

ltamasi Jun 24, 2022

riversand963 Jun 24, 2022

riversand963 Jun 23, 2022

ltamasi Jun 24, 2022

riversand963 Jun 23, 2022

facebook-github-bot commented Jun 24, 2022

facebook-github-bot commented Jun 24, 2022

riversand963 left a comment

riversand963 Jun 24, 2022

riversand963 Jun 24, 2022

ltamasi Jun 25, 2022

riversand963 Jun 24, 2022

riversand963 Jun 24, 2022

riversand963 Jun 24, 2022

ltamasi Jun 25, 2022

riversand963 Jun 24, 2022

ltamasi Jun 25, 2022

ltamasi commented Jun 25, 2022

facebook-github-bot commented Jun 25, 2022

facebook-github-bot commented Jun 25, 2022

facebook-github-bot commented Jun 25, 2022

facebook-github-bot commented Jun 25, 2022

Add API for writing wide-column entities #10242

Add API for writing wide-column entities #10242

Conversation

ltamasi commented Jun 23, 2022

facebook-github-bot commented Jun 23, 2022

facebook-github-bot commented Jun 23, 2022

facebook-github-bot commented Jun 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Jun 24, 2022

facebook-github-bot commented Jun 24, 2022

riversand963 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ltamasi commented Jun 25, 2022

facebook-github-bot commented Jun 25, 2022

facebook-github-bot commented Jun 25, 2022

facebook-github-bot commented Jun 25, 2022

facebook-github-bot commented Jun 25, 2022