Index of edges and vertices in storage layer #1360

bright-starry-sky · 2019-11-30T18:20:24Z

Support for edges and vertices insert、delete and update

nebula-community-bot · 2019-12-06T11:14:23Z

Unit testing failed.

bright-starry-sky · 2019-12-06T13:54:31Z

Jenkins go

nebula-community-bot · 2019-12-06T15:02:25Z

Unit testing passed.

src/common/base/NebulaKeyUtils.cpp

src/interface/storage.thrift

src/storage/AddEdgesProcessor.cpp

bright-starry-sky · 2019-12-15T02:30:22Z

Addressed all @dangleptr comments since 2019-12-15 10:15:00:
1: Using kIndex instead of kData.
2: Removed 'kIndexPrefix' from index key.
3: Removed 'IndexItem' from all thrift requests.
4: Avoid the use of asyncAtomicOp if no index exists.
5: Avoid the double copy for NebulaKeyUtils::vertexIndexKey and NebulaKeyUtils::edgeIndexKey.

nebula-community-bot · 2019-12-15T02:50:27Z

Unit testing passed.

nebula-community-bot · 2019-12-15T03:06:09Z

Unit testing passed.

src/common/base/NebulaKeyUtils.cpp

src/storage/AddEdgesProcessor.cpp

src/storage/test/IndexTest.cpp

src/storage/test/DeleteVertexTest.cpp

src/storage/AddVerticesProcessor.cpp

bright-starry-sky · 2019-12-17T03:19:34Z

Resolve the conflict

nebula-community-bot · 2019-12-17T03:41:52Z

Unit testing passed.

critical27 · 2019-12-18T07:34:26Z

src/storage/AddEdgesProcessor.cpp

+        std::for_each(req.parts.begin(), req.parts.end(), [&](auto& partEdges){
+            auto partId = partEdges.first;
+            const auto &edges = partEdges.second;
+            auto atomic = [&]() -> std::string {


Maybe you should capture by value here?

Sorry, I don't quite follow you. your means [&] ?

wadeliuyi · 2019-12-19T15:46:07Z

src/common/base/NebulaKeyUtils.cpp

@@ -88,6 +88,59 @@ std::string NebulaKeyUtils::kvKey(PartitionID partId, const folly::StringPiece&
    return key;
 }

+// static
+void NebulaKeyUtils::indexRaw(const IndexValues &values, std::string& raw) {
+    std::vector<int32_t> colsLen;


colsLen init with size first better

wadeliuyi · 2019-12-19T15:52:42Z

src/storage/AddEdgesProcessor.cpp

-                                               edge.key.ranking, edge.key.dst, version);
-            data.emplace_back(std::move(key), std::move(edge.get_props()));
+    if (indexes_.empty()) {
+        std::for_each(req.parts.begin(), req.parts.end(), [&](auto& partEdges){


no need capture all local variables

your means [&] ?

nebula-community-bot · 2019-12-20T12:13:06Z

Unit testing passed.

nebula-community-bot · 2020-01-02T01:18:43Z

Unit testing passed.

critical27 · 2020-01-02T10:51:05Z

src/storage/mutate/AddEdgesProcessor.cpp

+     *     kv(part1_src1_edgeType1_rank1_dst1 , v4)
+     *
+     * Ultimately, kv(part1_src1_edgeType1_rank1_dst1 , v4) . It's just what I need.
+     */


critical27 · 2020-01-02T11:13:55Z

src/common/base/NebulaKeyUtils.h

+     */
+
+    static std::string encodeInt64(int64_t v) {
+        v ^= folly::to<int64_t>(1) << 63;


How about define a constexpr?

A good idea, Do you mean for method encodeInt64 or constant folly::to<int64_t>(1) << 63?

nebula-community-bot · 2020-01-02T16:24:42Z

Unit testing passed.

src/storage/mutate/DeleteEdgesProcessor.cpp

src/storage/mutate/AddEdgesProcessor.cpp

bright-starry-sky · 2020-01-03T14:55:10Z

Performance optimization for index write .
The performance report please refer to #1580

nebula-community-bot · 2020-01-03T15:23:03Z

Unit testing passed.

src/storage/mutate/AddEdgesProcessor.cpp

dangleptr · 2020-01-06T03:28:40Z

src/storage/StorageFlags.cpp

@@ -16,3 +16,5 @@ DEFINE_int32(waiting_catch_up_interval_in_secs, 30,
 DEFINE_int32(waiting_new_leader_retry_times, 30, "retry times when waiting for catching up data");
 DEFINE_int32(waiting_new_leader_interval_in_secs, 5,
             "interval between two requests for catching up state");
+DEFINE_bool(ignore_index_check_pre_insert, false,


src/interface/common.thrift

darionyaphet · 2020-01-06T03:34:47Z

src/common/base/NebulaKeyUtils.cpp

+        }
+        raw.append(col.second.data(), col.second.size());
+    }
+    for (auto len : colsLen) {


why append column size at the end of encoded-value?

Used for variable-length column type ， for example string .
if an index row likes string('abc') + string('abc') , when the where expression is col1 == 'abca', the current row should be skip.

dangleptr · 2020-01-06T05:55:11Z

src/storage/mutate/UpdateVertexProcessor.cpp

+                                                              spaceId_,
+                                                              u.first);
+                    const auto &cols = index.get_cols();
+                    auto values = collectIndexValues(std::move(reader).get(), cols);


why std::move(reader) ?

why std::move(reader) ?

Good point! It got me thinking about performance optimization. I've solved the performance problem for delete|update vertices or edges.

dangleptr · 2020-01-06T05:55:43Z

src/storage/mutate/UpdateVertexProcessor.cpp

+                                                                   spaceId_,
+                                                                   u.first);
+                        const auto &oCols = index.get_cols();
+                        auto oValues = collectIndexValues(std::move(oReader).get(), oCols);


bright-starry-sky · 2020-01-06T16:08:53Z

1,Performance optimization. 2,Resolve the conflict. 3,Improve test cases

nebula-community-bot · 2020-01-06T16:10:13Z

Unit testing passed.

dangleptr

Well done. The PR looks good to me now.

nebula-community-bot · 2020-01-07T07:33:29Z

Unit testing passed.

jude-zhu · 2020-02-13T07:25:04Z

close #458 & #467

@dangleptr

* online index * Address all comments since 2019-12-15 10:15:00 * Resolve the conflict * Addressed all @dangleptr comments since 2019-12-26 23:00:00 * Addressed dangleptr's comment since 2019-12-27 21:00:00 * Addressed dangleptr's comment since 2019-12-30 17:45:00 * Improved processing logic forAddEdges and AddVertices * Performance optimization for index write * 1,Performance optimization. 2,Resolve the conflict. 3,Improve test cases 1: Using kIndex instead of kData. 2: Removed 'kIndexPrefix' from index key. 3: Removed 'IndexItem' from all thrift requests. 4: Avoid the use of asyncAtomicOp if no index exists. 5: Avoid the double copy for NebulaKeyUtils::vertexIndexKey and NebulaKeyUtils::edgeIndexKey. Co-authored-by: yaphet <darion.wang@vesoft.com>

bright-starry-sky requested review from darionyaphet, sherman-the-tank, zhangguoqing, zlcook, critical27, dangleptr and whitewum November 30, 2019 18:20

bright-starry-sky added the ready-for-testing PR: ready for the CI test label Dec 6, 2019

dangleptr reviewed Dec 12, 2019

View reviewed changes

dangleptr reviewed Dec 13, 2019

View reviewed changes

src/storage/AddEdgesProcessor.cpp Outdated Show resolved Hide resolved

critical27 reviewed Dec 16, 2019

View reviewed changes

src/common/base/NebulaKeyUtils.cpp Show resolved Hide resolved

critical27 reviewed Dec 16, 2019

View reviewed changes

src/storage/AddEdgesProcessor.cpp Outdated Show resolved Hide resolved

critical27 reviewed Dec 16, 2019

View reviewed changes

src/storage/test/IndexTest.cpp Show resolved Hide resolved

darionyaphet reviewed Dec 17, 2019

View reviewed changes

src/storage/test/DeleteVertexTest.cpp Show resolved Hide resolved

darionyaphet reviewed Dec 17, 2019

View reviewed changes

src/storage/AddVerticesProcessor.cpp Outdated Show resolved Hide resolved

jude-zhu requested review from darionyaphet, critical27 and dangleptr December 17, 2019 05:17

critical27 reviewed Dec 18, 2019

View reviewed changes

wadeliuyi reviewed Dec 19, 2019

View reviewed changes

critical27 reviewed Jan 2, 2020

View reviewed changes

dangleptr reviewed Jan 3, 2020

View reviewed changes

src/storage/mutate/DeleteEdgesProcessor.cpp Outdated Show resolved Hide resolved

src/storage/mutate/AddEdgesProcessor.cpp Outdated Show resolved Hide resolved

dangleptr reviewed Jan 6, 2020

View reviewed changes

src/interface/common.thrift Show resolved Hide resolved

darionyaphet reviewed Jan 6, 2020

View reviewed changes

dangleptr reviewed Jan 6, 2020

View reviewed changes

bright-starry-sky added 9 commits January 6, 2020 20:58

online index

89c95fc

Address all comments since 2019-12-15 10:15:00

17f476a

Resolve the conflict

1716b48

Addressed all @dangleptr comments since 2019-12-26 23:00:00

592fd9d

Addressed dangleptr's comment since 2019-12-27 21:00:00

90910cd

Addressed dangleptr's comment since 2019-12-30 17:45:00

3075bc3

Improved processing logic forAddEdges and AddVertices

74bf446

Performance optimization for index write

fa009ed

1,Performance optimization. 2,Resolve the conflict. 3,Improve test cases

d559d30

dangleptr approved these changes Jan 7, 2020

View reviewed changes

jude-zhu requested a review from darionyaphet January 7, 2020 02:55

darionyaphet approved these changes Jan 7, 2020

View reviewed changes

Merge branch 'master' into online_index

32f6632

dutor merged commit fc1760e into vesoft-inc:master Jan 7, 2020

bright-starry-sky deleted the online_index branch February 27, 2020 06:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index of edges and vertices in storage layer #1360

Index of edges and vertices in storage layer #1360

bright-starry-sky commented Nov 30, 2019

nebula-community-bot commented Dec 6, 2019

bright-starry-sky commented Dec 6, 2019

nebula-community-bot commented Dec 6, 2019

bright-starry-sky commented Dec 15, 2019

nebula-community-bot commented Dec 15, 2019

nebula-community-bot commented Dec 15, 2019

bright-starry-sky commented Dec 17, 2019

nebula-community-bot commented Dec 17, 2019

critical27 Dec 18, 2019 •

edited

Loading

bright-starry-sky Dec 20, 2019

wadeliuyi Dec 19, 2019

wadeliuyi Dec 19, 2019

bright-starry-sky Dec 20, 2019

nebula-community-bot commented Dec 20, 2019

nebula-community-bot commented Jan 2, 2020

critical27 Jan 2, 2020

critical27 Jan 2, 2020

bright-starry-sky Jan 2, 2020

nebula-community-bot commented Jan 2, 2020

bright-starry-sky commented Jan 3, 2020

nebula-community-bot commented Jan 3, 2020

dangleptr Jan 6, 2020

darionyaphet Jan 6, 2020

bright-starry-sky Jan 6, 2020

dangleptr Jan 6, 2020

bright-starry-sky Jan 6, 2020

dangleptr Jan 6, 2020

bright-starry-sky commented Jan 6, 2020

nebula-community-bot commented Jan 6, 2020

dangleptr left a comment

nebula-community-bot commented Jan 7, 2020

jude-zhu commented Feb 13, 2020 •

edited

Loading

Index of edges and vertices in storage layer #1360

Index of edges and vertices in storage layer #1360

Conversation

bright-starry-sky commented Nov 30, 2019

nebula-community-bot commented Dec 6, 2019

bright-starry-sky commented Dec 6, 2019

nebula-community-bot commented Dec 6, 2019

bright-starry-sky commented Dec 15, 2019

nebula-community-bot commented Dec 15, 2019

nebula-community-bot commented Dec 15, 2019

bright-starry-sky commented Dec 17, 2019

nebula-community-bot commented Dec 17, 2019

critical27 Dec 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nebula-community-bot commented Dec 20, 2019

nebula-community-bot commented Jan 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nebula-community-bot commented Jan 2, 2020

bright-starry-sky commented Jan 3, 2020

nebula-community-bot commented Jan 3, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bright-starry-sky commented Jan 6, 2020

nebula-community-bot commented Jan 6, 2020

dangleptr left a comment

Choose a reason for hiding this comment

nebula-community-bot commented Jan 7, 2020

jude-zhu commented Feb 13, 2020 • edited Loading

critical27 Dec 18, 2019 •

edited

Loading

jude-zhu commented Feb 13, 2020 •

edited

Loading