Hbase shard improve and implement mysql/memory shard #974

zhoney · 2020-04-27T13:41:36Z

Hbase shard range参考：https://blog.csdn.net/shenxiaoming77/article/details/22675919

codecov · 2020-04-28T15:04:14Z

Codecov Report

Merging #974 into master will decrease coverage by 0.29%.
The diff coverage is 53.82%.

@@             Coverage Diff              @@
##             master     #974      +/-   ##
============================================
- Coverage     70.02%   69.73%   -0.30%     
- Complexity     5255     5276      +21     
============================================
  Files           324      324              
  Lines         25433    25730     +297     
  Branches       3559     3619      +60     
============================================
+ Hits          17810    17943     +133     
- Misses         5920     6067     +147     
- Partials       1703     1720      +17

Impacted Files	Coverage Δ	Complexity Δ
...gegraph/backend/store/rocksdb/RocksDBSessions.java	`100.00% <ø> (ø)`	`1.00 <0.00> (ø)`
...h/backend/store/rocksdbsst/RocksDBSstSessions.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...om/baidu/hugegraph/backend/store/BackendTable.java	`25.82% <14.28%> (-66.18%)`	`11.00 <0.00> (-1.00)`
...ugegraph/backend/store/memory/InMemoryDBTable.java	`67.14% <28.12%> (-10.84%)`	`42.00 <10.00> (+1.00)`	⬇️
...u/hugegraph/backend/store/mysql/MysqlFeatures.java	`80.00% <50.00%> (-5.00%)`	`16.00 <1.00> (-1.00)`
...raph/backend/store/rocksdb/RocksDBStdSessions.java	`70.02% <63.63%> (+0.29%)`	`30.00 <0.00> (ø)`
.../hugegraph/backend/store/rocksdb/RocksDBTable.java	`72.30% <70.37%> (-0.08%)`	`37.00 <0.00> (+2.00)`	⬇️
...aidu/hugegraph/backend/store/hbase/HbaseTable.java	`78.75% <83.67%> (+1.07%)`	`37.00 <0.00> (ø)`
...aidu/hugegraph/backend/store/mysql/MysqlTable.java	`79.77% <85.24%> (+0.69%)`	`92.00 <11.00> (+15.00)`
...egraph/backend/store/mysql/MysqlEntryIterator.java	`79.48% <91.66%> (+0.38%)`	`19.00 <5.00> (+2.00)`
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cadc15a...4405da1. Read the comment docs.

javeme · 2020-04-30T08:35:27Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/memory/InMemoryDBTable.java

+
+        @Override
+        protected long estimateDataSize(BackendSession session) {
+            return 0;


javeme · 2020-04-30T09:36:39Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/BackendEntryIterator.java

@@ -80,9 +80,17 @@ public BackendEntry next() {

        this.current = null;
        this.count += this.sizeOf(current);
+        if (this.query.paging() && !this.query.nolimit() &&
+            this.count > this.query.limit()) {
+            this.reduce2limit(current);


move to fetch()

javeme · 2020-04-30T09:59:56Z

hugegraph-test/src/main/java/com/baidu/hugegraph/core/EdgeCoreTest.java

+        } else {
+            query.scan(String.valueOf(Long.MIN_VALUE),
+                       String.valueOf(Long.MAX_VALUE));
+        }


TODO: unify format
at least unify start and end

set START=END=""
and just set cassandra as long value

javeme · 2020-04-30T10:03:50Z

hugegraph-mysql/src/main/java/com/baidu/hugegraph/backend/store/mysql/MysqlTable.java

+        }
+
+        return select;
+    }


improve code

javeme · 2020-04-30T10:04:08Z

hugegraph-mysql/src/main/java/com/baidu/hugegraph/backend/store/mysql/MysqlEntryIterator.java

+        int size = e.subRows().size();
+        // Remove last paging row
+        e.subRows().remove(size - 1);
+    }


javeme · 2020-05-06T13:35:32Z

hugegraph-mysql/src/main/java/com/baidu/hugegraph/backend/store/mysql/MysqlTable.java

@@ -402,7 +402,8 @@ public Number queryNumber(Session session, Query query) {
                this.wrapOrderBy(selection, query);
            }
            if (query.paging()) {
-                this.wrapPage(selection, query);
+                this.wrapPage(selection, query, false);
+                wrapLimit(selection, query);


this.wrapLimit

javeme · 2020-05-06T13:35:54Z

hugegraph-mysql/src/main/java/com/baidu/hugegraph/backend/store/mysql/MysqlTable.java


        if (page != null && !page.isEmpty()) {
            // >= page
-            wrapPage(select, query, prefixAnd);
+            wrapPage(select, query, true);


this.wrapPage

javeme · 2020-05-06T13:38:42Z

hugegraph-test/src/main/java/com/baidu/hugegraph/core/EdgeCoreTest.java

+        } else {
+            query.scan(String.valueOf(Long.MIN_VALUE),
+                       String.valueOf(Long.MAX_VALUE));
+        }


set START=END=""
and just set cassandra as long value

javeme · 2020-05-06T13:38:56Z

hugegraph-test/src/main/java/com/baidu/hugegraph/core/VertexCoreTest.java

+            query.scan("00", BackendTable.ShardSpliter.END);
+        } else if (backend.equals("mysql") || backend.equals("postgresql")) {
+            query.scan(BackendTable.ShardSpliter.END,
+                       BackendTable.ShardSpliter.END);


javeme · 2020-05-13T16:46:01Z

hugegraph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBSessions.java

@@ -63,6 +64,7 @@ public abstract RocksDBSessions copy(HugeConfig config,
        public static final int SCAN_LTE_END = 0x30;

        public abstract String property(String table, String property);
+        public abstract Pair<byte[], byte[]> getKeyRange(String table);


javeme · 2020-05-13T16:47:39Z

...raph-rocksdb/src/main/java/com/baidu/hugegraph/backend/store/rocksdb/RocksDBStdSessions.java

+                startKey = iter.key();
+                iter.seekToLast();
+                if (!iter.isValid()) {
+                    return null;


return Pair.of(startKey, null);

javeme · 2020-05-13T16:49:20Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/store/BackendTable.java

+        private static final byte[] EMPTY = new byte[0];
+        public static final byte[] START_BYTES = new byte[]{0x0};
+        public static final byte[] END_BYTES = new byte[]{
+                -1, -1, -1, -1, -1, -1, -1, -1,


Change-Id: I5a660a08239f63b7da4cc382d6ddb449d1532e5f

Change-Id: Ibb840388667a0ed3e57b965ff607427ff429dcf9

Change-Id: I4dbfd4b726c715cc2f00816d574ae438d1300b2c

Change-Id: Ie146c5abe98b441ecd67de8e5d04900e46e72471

Change-Id: I9d478119d343dc9aab5bd38680090f9b6175dc9e

Change-Id: I0e58d0b1c7947ac3ad10a59d052e87066dce0b67

Change-Id: Id3bf5d13cc3a6cd98e2710e1fb0abefc4afa349f

Change-Id: I6c98ba38bd8e2439dcab21d4cdca3eba04358e92

javeme reviewed Apr 30, 2020

View reviewed changes

javeme reviewed May 6, 2020

View reviewed changes

zhoney force-pushed the hbase-shard-improve branch 2 times, most recently from 27496d5 to 750d037 Compare May 13, 2020 03:10

javeme reviewed May 13, 2020

View reviewed changes

zhoney added 8 commits May 14, 2020 11:09

improve hbase shard with region start/end key

22224e9

Change-Id: I5a660a08239f63b7da4cc382d6ddb449d1532e5f

support shard for mysql and memory backend

e478009

Change-Id: Ibb840388667a0ed3e57b965ff607427ff429dcf9

improve hbase and rocksdb shards end

40840e6

Change-Id: I4dbfd4b726c715cc2f00816d574ae438d1300b2c

improve

b7d9fb2

Change-Id: Ie146c5abe98b441ecd67de8e5d04900e46e72471

fix mysql/postgresql mistakenly return paging edge row

58e3ec3

Change-Id: I9d478119d343dc9aab5bd38680090f9b6175dc9e

improve

8181b96

Change-Id: I0e58d0b1c7947ac3ad10a59d052e87066dce0b67

improve rocksdb shard by get first and last row key

dd7726c

Change-Id: Id3bf5d13cc3a6cd98e2710e1fb0abefc4afa349f

improve

4405da1

Change-Id: I6c98ba38bd8e2439dcab21d4cdca3eba04358e92

zhoney force-pushed the hbase-shard-improve branch from 750d037 to 4405da1 Compare May 14, 2020 03:26

javeme approved these changes May 19, 2020

View reviewed changes

zhoney merged commit 071b3b3 into master May 19, 2020

zhoney deleted the hbase-shard-improve branch May 19, 2020 08:36

zhoney mentioned this pull request Jun 29, 2020

后端存储使用Memory，使用backup/Restore，出现Not support metadata 'splits' #370

Closed

This was referenced Jul 25, 2020

hugegraph-tools backup出现问题 #777

Closed

希望能支持mysql后端 apache/incubator-hugegraph-tools#47

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hbase shard improve and implement mysql/memory shard #974

Hbase shard improve and implement mysql/memory shard #974

zhoney commented Apr 27, 2020 •

edited

codecov bot commented Apr 28, 2020 •

edited

javeme Apr 30, 2020

javeme Apr 30, 2020

javeme Apr 30, 2020

javeme May 6, 2020

javeme Apr 30, 2020

javeme Apr 30, 2020

javeme May 6, 2020

javeme May 6, 2020

javeme May 6, 2020

javeme May 6, 2020

javeme May 13, 2020

javeme May 13, 2020

javeme May 13, 2020

Hbase shard improve and implement mysql/memory shard #974

Hbase shard improve and implement mysql/memory shard #974

Conversation

zhoney commented Apr 27, 2020 • edited

codecov bot commented Apr 28, 2020 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhoney commented Apr 27, 2020 •

edited

codecov bot commented Apr 28, 2020 •

edited