support olap writeback for cassandra and rocksdb #1506

zhoney · 2021-06-17T13:14:42Z

Change-Id: I947c3a96eb7aa67ee9eff5f7fb248e9f4539e91b

hugegraph-api/src/main/java/com/baidu/hugegraph/api/schema/PropertyKeyAPI.java

hugegraph-api/src/main/java/com/baidu/hugegraph/serializer/JsonSerializer.java

...h-cassandra/src/main/java/com/baidu/hugegraph/backend/store/cassandra/CassandraFeatures.java

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/SchemaTransaction.java

javeme · 2021-06-17T14:50:06Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/query/Query.java

+    }
+
+    public boolean olap() {
+        return this.olap;


can return olapPks != null && olapPks not empty

olapPks is for query Vertex, olap is for index query

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/query/Query.java

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/GraphIndexTransaction.java

codecov · 2021-06-25T03:58:52Z

Codecov Report

Merging #1506 (295355a) into master (379859a) will increase coverage by 0.88%.
The diff coverage is 81.05%.

@@             Coverage Diff              @@
##             master    #1506      +/-   ##
============================================
+ Coverage     59.47%   60.36%   +0.88%     
- Complexity     6212     6394     +182     
============================================
  Files           412      417       +5     
  Lines         33657    34293     +636     
  Branches       4638     4735      +97     
============================================
+ Hits          20019    20701     +682     
+ Misses        11545    11458      -87     
- Partials       2093     2134      +41

Impacted Files	Coverage Δ
...api/src/main/java/com/baidu/hugegraph/api/API.java	`69.04% <ø> (ø)`
...ava/com/baidu/hugegraph/api/profile/GraphsAPI.java	`10.71% <0.00%> (ø)`
...ava/com/baidu/hugegraph/serializer/Serializer.java	`0.00% <ø> (ø)`
...e/src/main/java/com/baidu/hugegraph/HugeGraph.java	`65.30% <ø> (ø)`
...hugegraph/backend/serializer/TextBackendEntry.java	`83.43% <0.00%> (-3.08%)`	⬇️
...om/baidu/hugegraph/backend/store/BackendEntry.java	`69.44% <0.00%> (-1.99%)`	⬇️
.../hugegraph/backend/store/BackendStoreProvider.java	`0.00% <0.00%> (ø)`
...hugegraph/backend/store/mysql/MysqlSerializer.java	`0.00% <0.00%> (ø)`
...egraph/backend/store/mysql/MysqlStoreProvider.java	`0.00% <ø> (ø)`
...idu/hugegraph/backend/store/mysql/MysqlTables.java	`0.00% <0.00%> (ø)`
... and 82 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 379859a...295355a. Read the comment docs.

javeme · 2021-06-25T13:26:34Z

Error:  Tests run: 371, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 628.431 s <<< FAILURE! - in com.baidu.hugegraph.unit.UnitTestSuite
Error:  testToString(com.baidu.hugegraph.unit.core.QueryTest)  Time elapsed: 0.013 s  <<< FAILURE!
org.junit.ComparisonFailure: expected:<...e id in [1, 3] and [[LABEL == 3, PROPERTIES < 18]]`> but was:<...e id in [1, 3] and [[PROPERTIES < 18, LABEL == 3]]`>
	at com.baidu.hugegraph.unit.core.QueryTest.testToString(QueryTest.java:120)

javeme · 2021-06-25T13:30:29Z

hugegraph-core/src/main/java/com/baidu/hugegraph/StandardHugeGraph.java

    @Override
    public void addIndexLabel(SchemaLabel schemaLabel, IndexLabel indexLabel) {
-        assert this.name.equals(schemaLabel.graph().name());
+        assert ALL_VL.equals(schemaLabel) ||


ALL_VL means what

ALL_VL means all of vertex labels

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/GraphIndexTransaction.java

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/SchemaTransaction.java

javeme · 2021-06-25T13:41:13Z

hugegraph-core/src/main/java/com/baidu/hugegraph/StandardHugeGraph.java


+    private void clearVertexCache() {
+        Future<?> future = this.graphEventHub.notify(Events.CACHE, "clear",
+                                                     HugeType.VERTEX);


need to clear edges cache?

olap properties only affect vertex query results, seems no need to clear edge cache

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

javeme · 2021-06-25T14:10:22Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

+        }
+
+        @Override
+        public void checkAndRegisterOlapTable(Id pkId) {


not covered by tests

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBTables.java

javeme · 2021-06-25T14:12:37Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBTables.java

+
+    public static class OlapRangeIntIndex extends RangeIntIndex {
+
+        public static final String TABLE = HugeType.OLAP.string();


is it the same table name as OlapSecondaryIndex

no, it's like g_ap_ii

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/SchemaTransaction.java

javeme · 2021-07-07T09:02:01Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/BackendStore.java

-        return this.store() + "_" + HugeType.OLAP.string() + "_" + type.string();
+        return this.store() + "_" +
+               HugeType.OLAP.string().toLowerCase() + "_" +
+               type.string().toLowerCase();


each query will call this method?
use StringBuilder with init-capacity

javeme · 2021-07-07T09:02:07Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/BackendStore.java

    public default String olapTableName(Id id) {
-        return this.store() + "_" + HugeType.OLAP.string() + "_" + id.asLong();
+        return this.store() + "_" +
+               HugeType.OLAP.string().toLowerCase() + "_" + id.asLong();


javeme · 2021-07-07T09:08:51Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

+                if (entry.type() == HugeType.OLAP) {
+                    table = this.table(this.olapTableName(entry.subId()));
+                    session = this.session(HugeType.OLAP);
+                } else {
+                    table = this.table(entry.type());
+                }
+                table.insert(session, entry);


move this block to line 446

javeme · 2021-07-07T09:14:03Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

        }
    }

+    protected RocksDBSessions db(HugeType tableType) {


move to the line of db(String disk) method

javeme · 2021-07-07T09:15:36Z

...ph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStoreProvider.java

         * [1.10] #1333: support read frequency for property key
-         * [1.11] #1506: support olap properties
+         * [1.11] #1506: support olap properties and rename read frequency to
+         *               write type


remove "rename read frequency..."?

javeme · 2021-07-20T09:51:50Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/BackendStore.java

+        StringBuilder sb = new StringBuilder(7);
+        sb.append(this.store())
+          .append("_")
+          .append(HugeType.OLAP.string().toLowerCase())


HugeType itself is LowerCase?

no. HugeType.string() is UpperCase

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/BackendStore.java

javeme · 2021-07-20T09:54:56Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStore.java

        BackendEntry entry = item.entry();

        RocksDBTable table;
+        if (entry.type() != HugeType.OLAP && !entry.olap()) {


not the same between entry.olap() and entry.type() == HugeType.OLAP?

not same. entry.type() == HugeType.OLAP means olap vertex, but entry.olap() means olap vertex index

Change-Id: I947c3a96eb7aa67ee9eff5f7fb248e9f4539e91b

Change-Id: Iaacdbaca95630dd37fedd134c8f9ff04516f4d17

Change-Id: I068266c57aa7b0f5b1ed8979868cf5ebd6ee33cd

Change-Id: Id1ffefc10cba8306743acee20e20dee926865da6

Change-Id: Ic48c2cba1d2f4d427cc9eeea2dd6268c1b9fa334

Change-Id: Ifa5c0fba1488fec0636c58892992ffa49d75265f

Change-Id: Ice6dce962b54d54e59303338cfe42921221e5c8f

Change-Id: I1173061e9aed6e1f1d171f5c72f0bea487aa864a

Change-Id: I72007fea93f7a32d3a25794b184640656d28e7f3

Change-Id: Ie92f81aec2cd1d10a815c02f946ef01f9d657823

Change-Id: I97d2cea75ec344aa6ccd5587e24351f3e7ed05a9

Change-Id: I54e3f4399bcebaa587d263962b01303b2e387676

Change-Id: If1ccf204837b671cd53904834a80f72020ed9d90

Change-Id: I273d215355ec5d49ebb9ba3cf8159b1af013f937

javeme reviewed Jun 17, 2021

View reviewed changes

zhoney force-pushed the olap-writeback branch 2 times, most recently from a651bbf to 87b0d62 Compare June 25, 2021 03:36

javeme reviewed Jun 25, 2021

View reviewed changes

zhoney force-pushed the olap-writeback branch from bb2ef78 to 4e59c46 Compare June 28, 2021 07:48

javeme reviewed Jun 28, 2021

View reviewed changes

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/SchemaTransaction.java Outdated Show resolved Hide resolved

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/SchemaTransaction.java Outdated Show resolved Hide resolved

javeme reviewed Jul 7, 2021

View reviewed changes

zhoney force-pushed the olap-writeback branch from ae731b3 to 469981a Compare July 7, 2021 11:09

javeme reviewed Jul 20, 2021

View reviewed changes

zhoney added 13 commits July 20, 2021 18:02

support olap writeback for cassandra and rocksdb

c824080

Change-Id: I947c3a96eb7aa67ee9eff5f7fb248e9f4539e91b

add olap property key tests

4d82a1e

Change-Id: Iaacdbaca95630dd37fedd134c8f9ff04516f4d17

rebase master and adapt unit test

38e79bb

Change-Id: I068266c57aa7b0f5b1ed8979868cf5ebd6ee33cd

improve

5932650

Change-Id: Id1ffefc10cba8306743acee20e20dee926865da6

clear vertex cache when olap pk changed

08bfe7c

Change-Id: Ic48c2cba1d2f4d427cc9eeea2dd6268c1b9fa334

improve

fdff4af

Change-Id: Ifa5c0fba1488fec0636c58892992ffa49d75265f

improve

be067e6

Change-Id: Ice6dce962b54d54e59303338cfe42921221e5c8f

support config data disk for olap pk when using rocksdb backend

98384d4

Change-Id: I1173061e9aed6e1f1d171f5c72f0bea487aa864a

rename read frequency to write type

fe2429f

Change-Id: I72007fea93f7a32d3a25794b184640656d28e7f3

improve

34686c8

Change-Id: Ie92f81aec2cd1d10a815c02f946ef01f9d657823

improve

57586f0

Change-Id: I97d2cea75ec344aa6ccd5587e24351f3e7ed05a9

rebase and improve

bed0239

Change-Id: I54e3f4399bcebaa587d263962b01303b2e387676

improve

4e10dd1

Change-Id: If1ccf204837b671cd53904834a80f72020ed9d90

zhoney force-pushed the olap-writeback branch from 469981a to 4e10dd1 Compare July 20, 2021 10:11

improve

295355a

Change-Id: I273d215355ec5d49ebb9ba3cf8159b1af013f937

javeme approved these changes Jul 22, 2021

View reviewed changes

Linary approved these changes Jul 22, 2021

View reviewed changes

zhoney merged commit ef37b3a into master Jul 22, 2021

zhoney deleted the olap-writeback branch July 22, 2021 15:53


		public static class OlapRangeIntIndex extends RangeIntIndex {

		public static final String TABLE = HugeType.OLAP.string();

support olap writeback for cassandra and rocksdb #1506

support olap writeback for cassandra and rocksdb #1506

Uh oh!

Conversation

zhoney commented Jun 17, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jun 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

javeme commented Jun 25, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Jun 25, 2021 •

edited

Loading