Fix rocksdb oom with g.V().hasLabel().count() #823

javeme · 2020-01-09T11:48:26Z

fix #793
rebase for #798

codecov · 2020-01-09T11:52:35Z

Codecov Report

Merging #823 into master will increase coverage by 0.09%.
The diff coverage is 74.27%.

@@             Coverage Diff              @@
##             master     #823      +/-   ##
============================================
+ Coverage     72.13%   72.23%   +0.09%     
- Complexity     4456     4486      +30     
============================================
  Files           287      287              
  Lines         21227    21407     +180     
  Branches       2992     3017      +25     
============================================
+ Hits          15313    15463     +150     
- Misses         4401     4433      +32     
+ Partials       1513     1511       -2

Impacted Files	Coverage Δ	Complexity Δ
...api/src/main/java/com/baidu/hugegraph/api/API.java	`66.29% <0.00%> (-1.53%)`	`0.00 <0.00> (ø)`
...ugegraph/api/traversers/FusiformSimilarityAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
.../baidu/hugegraph/backend/query/ConditionQuery.java	`85.97% <ø> (+0.70%)`	`97.00 <0.00> (+1.00)`
...ava/com/baidu/hugegraph/backend/query/IdQuery.java	`97.22% <ø> (-0.15%)`	`13.00 <0.00> (-1.00)`
...du/hugegraph/traversal/optimize/TraversalUtil.java	`67.59% <0.00%> (+0.69%)`	`82.00 <0.00> (ø)`
...a/com/baidu/hugegraph/variables/HugeVariables.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...egraph/backend/store/mysql/MysqlEntryIterator.java	`79.10% <0.00%> (ø)`	`17.00 <0.00> (ø)`
...du/hugegraph/backend/cache/CachedBackendStore.java	`9.67% <33.33%> (ø)`	`0.00 <0.00> (ø)`
...n/java/com/baidu/hugegraph/structure/HugeEdge.java	`74.40% <50.00%> (-0.16%)`	`73.00 <0.00> (ø)`
...java/com/baidu/hugegraph/structure/HugeVertex.java	`78.69% <50.00%> (-0.10%)`	`76.00 <0.00> (ø)`
... and 41 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f1f1d0e...0808022. Read the comment docs.

javeme · 2020-01-09T12:04:19Z

TODO: let tx.queryXX() return CIter

javeme · 2020-01-15T10:29:43Z

TODO: add gremlin job test with g.V(), g.V().hasLabel(xx), g.E(), g.E().hasLabel(xx)

javeme · 2020-02-27T17:28:26Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/serializer/BinaryEntryIterator.java

@@ -76,6 +76,9 @@ protected final boolean fetch() {
            } else if (merged == this.current) {
                // The next entry belongs to the current entry
                assert this.current != null;
+                if (this.sizeOf(this.current) >= INLINE_BATCH_SIZE) {


same as cassandra and hbase

Linary · 2020-02-28T05:22:46Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/page/IdHolder.java

+
+        /**
+         * For paging situation
+         * @param query         original query


'original query' align

Change-Id: I99c9471d75d320f3980d38f291d75c0b7237adfd

Change-Id: Iae0716e4624e8389b9c3bb74c8f784c54d29d0ba

Change-Id: I3f8da8b39d64dfc8a3aa9e888c6c07feace0878f

Change-Id: I4a14339548b3906f6be024293a8c3bb0e76d65e7

Change-Id: I98047052ceb791eba5e3d915cc2bfeacc2186a01

Change-Id: I75cd3d4bb66bfbbc584033b8f279c6781bf125cc

Change-Id: I625ff7d826f3f3f85ebeff72a94d70e416aee242

Change-Id: I7f4c03d17ad584b97142a07fa10d04868f5be53a

Change-Id: Ib030a2758e73b4c9a9fccf1e310875d58610e4de

Change-Id: I0d178d44ab3b28f1add736eac00fecca06fb8e47

Change-Id: I6851863901143b6b607a7ad8ebc611c1807a17ae

Change-Id: I3a96ade9563a56b8cdd840aaccb7664922ac6950

Change-Id: If8561419d87642986a2f942c1998d78070617163

Change-Id: Id1dd6973d0209598c710069128690e1f58e21011

Change-Id: I0e5a590aafe34f53489901b9c595964c26cde977

Change-Id: If196b2450addbe62cf92fa433b661e1bc28c16cc

Change-Id: I5e5d1e05800ce9763831a49962d84b9a4944d7bb

Change-Id: Ia578154ef5c58dd3fb3a11d779180a68df9f28fe

Change-Id: Id309f7101e81a643017de3e94c01eceddf52fa8d

Change-Id: I118d49b45467eaefea4b0017b84bbc158692f17b

Change-Id: I4c772747c06890257743283ff5559ac84efbe8d9

Change-Id: I632b9f847c55249c96d6aabfa6e037d317124fcb

Change-Id: I773df0769bb25e139e3242b5c8436fc5fd1eec8d

Linary · 2020-03-09T14:34:15Z

hugegraph-test/src/main/java/com/baidu/hugegraph/unit/core/QueryTest.java

+    @Test
+    public void testToStringOfIdRangeQuery() {
+        IdRangeQuery query = new IdRangeQuery(HugeType.EDGE,
+                                               IdGenerator.of(1),


Linary · 2020-03-09T15:33:01Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/GraphIndexTransaction.java

@@ -1211,6 +1275,10 @@ private static boolean hasNullableProp(HugeElement element, Id key) {
        return indexLabels;
    }

+    private static void increaseLimit(Query query) {
+        query.limit(query.limit() * 2L + 8L);


What's the basis?

Linary · 2020-03-09T15:34:56Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/tx/GraphIndexTransaction.java

+            return null;
+        }
+
+        public Query asQuery() {


asJointQuery maybe clearer

Change-Id: Ib41ae85b65f95707386e3c5df5ed4286580009c1

zhoney · 2020-03-11T06:44:08Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/cache/CachedGraphTransaction.java

-                vertices.add(vertex);
+            // Generally there are not too much data with id query
+            ListIterator<HugeVertex> listIterator = QueryResults.toList(rs);
+            for (HugeVertex vertex :listIterator.list()) {


add empty after ":"

zhoney · 2020-03-11T10:29:13Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/optimize/TraversalUtil.java

            if (dir == Directions.OUT && vertex.equals(edge.outVertex()) ||
-                dir == Directions.IN && vertex.equals(edge.inVertex())) {
-                list.add(edge);
+                dir == Directions.IN  && vertex.equals(edge.inVertex())) {


delete empty

zhoney · 2020-03-11T11:14:24Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/query/Query.java

@@ -48,6 +51,7 @@
    private HugeType resultType;


capacityContext

zhoney · 2020-03-11T11:31:57Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/page/IdHolder.java

+            if (this.exhausted) {
+                return false;
+            }
+            boolean hasNext= this.entries.hasNext();


zhoney · 2020-03-11T11:33:22Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/page/IdHolder.java

+        public PageIds fetchNext(String page, long batchSize) {
+            E.checkArgument(page == null,
+                            "Not support page parameter by BatchIdHolder");
+            E.checkArgument(batchSize >= 0L,


batchSize might be 0?

yes, it's 0 when hit the last page, and then set exhausted true

zhoney · 2020-03-11T11:57:33Z

hugegraph-core/src/main/java/com/baidu/hugegraph/backend/query/QueryResults.java

+                T result = iterator.next();
+                if (iterator.hasNext()) {
+                    throw new HugeException(
+                              "Expect just one result but got two: [%s, %s]",


Expect just one result but got at least two: [%s, %s, ...]

Change-Id: I63c2adec8365e481682bd511dea6c6849a3d5114

Change-Id: I13daaa513b2f5ddc69bcae4d548f263ad1f5b08d

javeme mentioned this pull request Jan 9, 2020

Fix rocksdb oom with g.V().hasLabel().count() #798

Closed

javeme commented Feb 27, 2020

View reviewed changes

Linary requested changes Feb 28, 2020

View reviewed changes

javeme added 20 commits March 4, 2020 23:54

debug

27f4a3a

Change-Id: I99c9471d75d320f3980d38f291d75c0b7237adfd

fix iter gloabal

8a052b0

Change-Id: Iae0716e4624e8389b9c3bb74c8f784c54d29d0ba

Manually close the iterators

84d13ba

Change-Id: I3f8da8b39d64dfc8a3aa9e888c6c07feace0878f

tiny improve

947b131

Change-Id: I4a14339548b3906f6be024293a8c3bb0e76d65e7

catch locks in every batch

e3d1b16

Change-Id: I98047052ceb791eba5e3d915cc2bfeacc2186a01

limit iterator to list

e337c9a

Change-Id: I75cd3d4bb66bfbbc584033b8f279c6781bf125cc

improve cache iter

f7f0c01

Change-Id: I625ff7d826f3f3f85ebeff72a94d70e416aee242

fix system index-label graph() return null

6faf4f6

Change-Id: I7f4c03d17ad584b97142a07fa10d04868f5be53a

add QueryResults.one() method

4eef1dc

Change-Id: Ib030a2758e73b4c9a9fccf1e310875d58610e4de

fix BatchIdHolder dead loop with batchSize=0

66afee3

Change-Id: I0d178d44ab3b28f1add736eac00fecca06fb8e47

fix test iterators don't be closed

8b927f3

Change-Id: I6851863901143b6b607a7ad8ebc611c1807a17ae

add option query.batch_size

4a003c3

Change-Id: I3a96ade9563a56b8cdd840aaccb7664922ac6950

remove debug info

47da22c

Change-Id: If8561419d87642986a2f942c1998d78070617163

force limit BatchIdHolder.all()

a1f1bed

Change-Id: Id1dd6973d0209598c710069128690e1f58e21011

adapt ListIterator.list() return Collection<T>

a151db7

Change-Id: I0e5a590aafe34f53489901b9c595964c26cde977

improve log

afd6147

Change-Id: If196b2450addbe62cf92fa433b661e1bc28c16cc

tiny fix

0d0f07d

Change-Id: I5e5d1e05800ce9763831a49962d84b9a4944d7bb

set default value of check_customized_id_exist option to false

e0a36a9

Change-Id: Ia578154ef5c58dd3fb3a11d779180a68df9f28fe

fix cassandra oom

0120d85

Change-Id: Id309f7101e81a643017de3e94c01eceddf52fa8d

improve comment

632553c

Change-Id: I118d49b45467eaefea4b0017b84bbc158692f17b

javeme force-pushed the fix-rocksdb-oom-rebase branch from c8d519e to dc5f365 Compare March 5, 2020 16:41

fix offset bug

a48603e

Change-Id: I4c772747c06890257743283ff5559ac84efbe8d9

javeme force-pushed the fix-rocksdb-oom-rebase branch from dc5f365 to a48603e Compare March 5, 2020 17:25

Increase limit for intersection

3fbe60b

Change-Id: I632b9f847c55249c96d6aabfa6e037d317124fcb

fix check subRows().size() <= INLINE_BATCH_SIZE

05622ae

Change-Id: I773df0769bb25e139e3242b5c8436fc5fd1eec8d

Linary requested changes Mar 9, 2020

View reviewed changes

tiny improve

4a82d5f

Change-Id: Ib41ae85b65f95707386e3c5df5ed4286580009c1

Linary previously approved these changes Mar 10, 2020

View reviewed changes

zhoney requested changes Mar 11, 2020

View reviewed changes

tiny fix

f4e8853

Change-Id: I63c2adec8365e481682bd511dea6c6849a3d5114

javeme dismissed Linary’s stale review via f4e8853 March 12, 2020 02:47

Linary previously approved these changes Mar 12, 2020

View reviewed changes

fix typo capacityContex -> capacityContext

0808022

Change-Id: I13daaa513b2f5ddc69bcae4d548f263ad1f5b08d

javeme dismissed Linary’s stale review via 0808022 March 12, 2020 14:21

javeme force-pushed the fix-rocksdb-oom-rebase branch from 1a406a6 to 0808022 Compare March 12, 2020 14:21

Linary approved these changes Mar 13, 2020

View reviewed changes

zhoney approved these changes Mar 13, 2020

View reviewed changes

javeme merged commit 9166c26 into master Mar 13, 2020

javeme deleted the fix-rocksdb-oom-rebase branch March 13, 2020 05:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix rocksdb oom with g.V().hasLabel().count() #823

Fix rocksdb oom with g.V().hasLabel().count() #823

javeme commented Jan 9, 2020

codecov bot commented Jan 9, 2020 •

edited

javeme commented Jan 9, 2020

javeme commented Jan 15, 2020 •

edited

javeme Feb 27, 2020

Linary Feb 28, 2020

Linary Mar 9, 2020

Linary Mar 9, 2020

Linary Mar 9, 2020

zhoney Mar 11, 2020

zhoney Mar 11, 2020

zhoney Mar 11, 2020

zhoney Mar 11, 2020

zhoney Mar 11, 2020

javeme Mar 12, 2020 •

edited

zhoney Mar 11, 2020

Fix rocksdb oom with g.V().hasLabel().count() #823

Fix rocksdb oom with g.V().hasLabel().count() #823

Conversation

javeme commented Jan 9, 2020

codecov bot commented Jan 9, 2020 • edited

Codecov Report

javeme commented Jan 9, 2020

javeme commented Jan 15, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme Mar 12, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 9, 2020 •

edited

javeme commented Jan 15, 2020 •

edited

javeme Mar 12, 2020 •

edited