Custom kout/kneighbor, multi-node-shortest-path, jaccard-similar and template paths #1174

zhoney · 2020-09-15T03:28:55Z

No description provided.

codecov · 2020-09-15T04:17:02Z

Codecov Report

Merging #1174 into master will increase coverage by 0.14%.
The diff coverage is 2.72%.

@@             Coverage Diff              @@
##             master    #1174      +/-   ##
============================================
+ Coverage     64.98%   65.12%   +0.14%     
- Complexity     5652     5748      +96     
============================================
  Files           361      366       +5     
  Lines         29429    29875     +446     
  Branches       4158     4222      +64     
============================================
+ Hits          19123    19456     +333     
- Misses         8352     8443      +91     
- Partials       1954     1976      +22

Impacted Files	Coverage Δ	Complexity Δ
.../hugegraph/api/traversers/AllShortestPathsAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...graph/api/traversers/CustomizedCrosspointsAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...u/hugegraph/api/traversers/CustomizedPathsAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ugegraph/api/traversers/FusiformSimilarityAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...hugegraph/api/traversers/JaccardSimilarityAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...m/baidu/hugegraph/api/traversers/KneighborAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...va/com/baidu/hugegraph/api/traversers/KoutAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...graph/api/traversers/MultiNodeShortestPathAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...a/com/baidu/hugegraph/api/traversers/PathsAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...idu/hugegraph/api/traversers/SameNeighborsAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
... and 67 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e35f9fb...20c1ff5. Read the comment docs.

javeme · 2020-09-18T12:33:11Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/algorithm/Consumers.java

+import com.baidu.hugegraph.util.ExecutorUtil;
+import com.baidu.hugegraph.util.Log;
+
+public class Consumers<V> {


move Consumers to com/baidu/hugegraph/util

rebase https://github.com/hugegraph/hugegraph/pull/1183

…h and jaccard similar * fix OLTP algorithm not check if source/target vertex exist (#1156) * support customized kout and kneighbor * support multi node shortest path API * support template paths api * support jaccard similars find oltp api * use multi-thread to accelerate customized kout/kneighbor, multi-node-shortest-path, jaccard-similar and template-paths * big depth and both direction use multi threads, otherwise single threead * fix shortest path api NLP Change-Id: I7eb4ceaccdc4a6b2a4a7b944edc83dff64a98f5d

* template path use concurrent hashset to save paths in concurrent mode * support property filter for paths api Change-Id: I2e589f58cdfc48b4b8d16b7780b78cc17ab107d6

also paths and template paths use single thread Change-Id: I4375c455f377d9e4d43c0284ec30da1725fdf6bc

paths supports nearest args oltp multiple threads reuse add multiple thread depth config Change-Id: I8c2f921ffb56302fb8d300c09a5a13857dac187a

Change-Id: I6d972894f3eec339181e5626bf45d01f36dbd4c0

implements: #1173 Change-Id: I85aa1d4274554d65f85a0deb7ac596e65dbb503b

Change-Id: I73d877fbd2099d9e63ef29ee0a10df30354595f9

Change-Id: Ib6fa985ff48fab203611218cc955079ad8cc90c3

Change-Id: Ia6ff5efd7e2a195f85427992b7bff875d30dec48

Change-Id: Ib43614c252912427c6ddaeb52c974a59b06934e6

Change-Id: I5ef4b33e8c5b4ade0b372d3810e596d335f37f29

Change-Id: Icbc65aa24e069b67bd5653b79473da48e2f973c6

Change-Id: I606772946a2a55f7472274242fb18f4ca91c189a

Change-Id: I4ab46bc68644d35fab1fa966e27a0d7ab87b433b

javeme · 2020-09-24T11:29:08Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

+
+            // Traversal vertices of previous level
+            RepeatEdgeStep finalCurrentStep = currentStep;
+            traverseIds(this.sources.keySet().iterator(), vid -> {


add common method

Change-Id: If6f0c1370a2376ef176ce2db39e002ae11590e8d

javeme · 2020-09-29T12:52:53Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/HugeTraverser.java

@@ -551,6 +551,26 @@ public static void checkSkipDegree(long skipDegree, long degree,
        return new MultivaluedHashMap<>();
    }

+    protected static List<Id> joinPath(Node pre, Node back, boolean ring) {


rename pre to prev

javeme · 2020-09-29T12:57:02Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

-                new ConcurrentMultiValuedMap<>();
-        private ConcurrentMultiValuedMap<Id, Node> targetsAll =
-                new ConcurrentMultiValuedMap<>();
+        private void traverseOne(Id v, RepeatEdgeStep step, boolean forward) {


rename v to source

javeme · 2020-09-29T13:00:22Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

-            // Re-init sources
-            if (currentStep.remainTimes() > 0) {
-                this.sources = newVertices;
+        private void reInitCurrentIfNeeded(RepeatEdgeStep step,


reInitCurrentStepIfNeeded

javeme · 2020-09-29T13:01:08Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

+        private void reInitAllIfNeeded(boolean forward) {
+            if (forward) {
+                // Re-init source all if last forward finished one super step
+                // and not last super step


not finished last super step? or current step is not the last super step?
and use /*

javeme · 2020-09-29T13:01:17Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

+                }
+            } else {
+                // Re-init target all if last forward finished one super step
+                // and not last super step


javeme · 2020-09-29T13:06:29Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

+        @Override
+        public void traverseOneLayer(
+                    Map<Id, List<Node>> vertices, RepeatEdgeStep step,
+                    BiConsumer<Id, RepeatEdgeStep> biConsumer) {


javeme · 2020-09-29T13:08:14Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

        }

-        protected boolean lastStep() {
-            return this.stepCount == this.totalSteps - 1;
+        public RepeatEdgeStep step(boolean forward) {


prefer nextStep

javeme · 2020-09-29T13:12:38Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

-                    this.sourceIndex = i;
-                    break;
+        private void processOneForForward(Id source, Id target) {
+            for (Node n : this.sources.get(source)) {


rename source to sourceV, and rename n to source, same as target

javeme · 2020-09-29T13:17:24Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

-                if (this.reachLimit()) {
-                    return;
+        private void processOneForBackward(Id source, Id target) {
+            for (Node n : this.targets.get(source)) {


javeme · 2020-09-29T13:18:15Z

...graph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TemplatePathsTraverser.java

+            if (forward) {
+                processOneForForward(source, target);
+            } else {
+                processOneForBackward(source, target);


can we merge the two methods with one, and pass sourcesAll or targetsAll to that method

javeme · 2020-09-29T13:24:29Z

hugegraph-api/src/main/java/com/baidu/hugegraph/core/GraphManager.java

@@ -165,7 +165,7 @@ private HugeAuthenticator authenticator() {

    @SuppressWarnings("unused")
    private void installLicense(HugeConfig config, String md5) {
-        LicenseVerifier.instance().install(config, this, md5);
+//        LicenseVerifier.instance().install(config, this, md5);


keep origin

javeme · 2020-09-29T13:24:51Z

hugegraph-api/src/main/java/com/baidu/hugegraph/serializer/JsonSerializer.java

+                                     Iterator<Vertex> iterator,
+                                     boolean countOnly) {
+        List<Map<String, Object>> pathList;
+        pathList = new ArrayList<>();


keep in one line

javeme · 2020-09-29T13:25:22Z

hugegraph-core/src/main/java/com/baidu/hugegraph/config/CoreOptions.java

+                    rangeInt(0, 65535),
+                    10
+            );
+


remove empty line

javeme · 2020-09-29T13:26:07Z

...aph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/CollectionPathsTraverser.java

+    private Traverser concurrentTraverser(List<Id> sources, List<Id> targets,
+                                          EdgeStep step, boolean nearest,
+                                          long capacity, long limit) {
+        return new ConcurrentTraverser(sources, targets, step, capacity, limit);


also refactor this class, and can we share code with templatepath?

javeme · 2020-09-29T13:29:42Z

...core/src/main/java/com/baidu/hugegraph/traversal/algorithm/CustomizedKneighborTraverser.java

+    }
+
+    public Set<Node> customizedKneighborSingle(Id source, EdgeStep step,
+                                               int maxDepth, long limit) {


refactor with one method and call newSet() of Single and Concurrent

javeme · 2020-09-29T13:30:53Z

...-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/FusiformSimilarityTraverser.java

+        if (groupProperty == null) {
+            E.checkArgument(minGroups == 0,
+                            "Can not set min group count when " +
+                                    "group property not set");


javeme · 2020-09-29T13:31:40Z

...-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/FusiformSimilarityTraverser.java

-        if (groupProperty != null) {
+        if (groupProperty == null) {
+            E.checkArgument(minGroups == 0,
+                            "Can not set min group count when " +


javeme · 2020-09-29T13:35:09Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/HugeTraverser.java

@@ -259,6 +262,35 @@ public double jaccardSimilarity(Id vertex, Id other, Directions dir,
        });
    }

+    protected Set<Node> adjacentVertices(Set<Node> vertices, EdgeStep step,


move Iterator adjacentVertices() to line 293

javeme · 2020-09-29T13:36:50Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/HugeTraverser.java

+        return jaccardSimilarity(sourceNeighbors, targetNeighbors);
+    }
+
+    public double jaccardSimilarity(Set<Id> set1, Set<Id> set2) {


prefer move kout/jaccardSimilarity out of this class

javeme · 2020-09-29T13:37:31Z

...raph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/JaccardSimilarTraverser.java

+        return results;
+    }
+
+    public Map<Id, Double> jaccardSimilarsSingle(Id source, EdgeStep step,


Change-Id: I7d6912f33836ac9a36fd0dda82b446780675a390

Change-Id: I5b7441f562106fa3810d6d844c897add3010ef30

javeme · 2020-10-13T14:04:05Z

...aph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/CollectionPathsTraverser.java

+        } else {
+            strategy = new SingleTraverseStrategy(this.graph());
+            if (nearest) {
+                traverser = new SingleNearestTraverser(sourceList, targetList,


no nearest when Concurrent?

javeme · 2020-10-13T14:05:01Z

...aph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/CollectionPathsTraverser.java

-            if (--depth < 0 || traverser.reachLimit()) {
-                break;
+        Traverser traverser;
+


unused empty line

javeme · 2020-10-13T14:07:15Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/HugeTraverser.java

+        return newSet(true);
+    }
+
+    protected static <V> Set<V> newSet(boolean single) {


prefer rename to concurrent

javeme · 2020-10-13T14:08:35Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TpTraverser.java

+
+    }
+
+    public abstract class PathTraverser {


add a new java file

Change-Id: I760194b0f02a41873c2157a5ddf59dd4d1c3ceff

javeme · 2020-10-20T11:52:29Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/PathTraverser.java

+        this.beforeTraverse(true);
+
+        // Traversal vertices of previous level
+        traverseOneLayer(this.sources, currentStep, this::forward);


this.traverseOneLayer

javeme · 2020-10-20T11:54:08Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/PathTraverser.java

+
+        currentStep.swithDirection();
+        // Traversal vertices of previous level
+        traverseOneLayer(this.targets, currentStep, this::backward);


javeme · 2020-10-20T11:54:30Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/PathTraverser.java

+
+    private void processOne(Id source, Id target, boolean forward) {
+        if (forward) {
+            processOneForForward(source, target);


javeme · 2020-10-20T11:54:35Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/PathTraverser.java

+        if (forward) {
+            processOneForForward(source, target);
+        } else {
+            processOneForBackward(source, target);


javeme · 2020-10-20T11:55:25Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/PathTraverser.java

+        return this.paths.size();
+    }
+
+    protected boolean finish() {


javeme · 2020-10-20T12:25:02Z

...c/main/java/com/baidu/hugegraph/traversal/algorithm/strategy/ConcurrentTraverseStrategy.java

+    @Override
+    public Iterator<Edge> edgesOfVertex(Id source, EdgeStep edgeStep) {
+        return super.edgesOfVertex(source, edgeStep);
+    }


javeme · 2020-10-20T12:25:15Z

...ph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/strategy/TraverseStrategy.java

@@ -44,4 +47,6 @@ public abstract void addNode(Map<Id, List<HugeTraverser.Node>> vertices,
    public abstract void addNewVerticesToAll(
                         Map<Id, List<HugeTraverser.Node>> newVertices,
                         Map<Id, List<HugeTraverser.Node>> targets);
+
+    public Iterator<Edge> edgesOfVertex(Id source, EdgeStep edgeStep);


Change-Id: I68226c55ca387ecfdd103e4af8b5e45faa9a934e

javeme · 2020-10-21T08:12:12Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TpTraverser.java

 import com.baidu.hugegraph.util.Consumers;

 import jersey.repackaged.com.google.common.base.Objects;

 public abstract class TpTraverser extends HugeTraverser
                                  implements AutoCloseable {

-    protected static ExecutorService executor;
+    private static final String EXECUTOR_NAME = "oltp";


also rename TpTraverser to OltpTraverser

javeme · 2020-10-21T08:12:40Z

...ph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/strategy/TraverseStrategy.java

+    public static TraverseStrategy create(boolean concurrent, HugeGraph graph) {
+        return concurrent ? new ConcurrentTraverseStrategy(graph) :
+                            new SingleTraverseStrategy(graph);
+


javeme · 2020-10-21T08:15:36Z

hugegraph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/TpTraverser.java

        if (executor == null) {
            int workers = this.config().get(CoreOptions.OLTP_CONCURRENT_THREADS);
            if (workers > 0) {
-                executor = Consumers.newThreadPool(name, workers);
+                executor = Consumers.newThreadPool(EXECUTOR_NAME, workers);


lock for null judgment and assignment

javeme · 2020-10-21T08:16:14Z

...aph-core/src/main/java/com/baidu/hugegraph/traversal/algorithm/CollectionPathsTraverser.java

-            if (--depth < 0 || traverser.reachLimit()) {
-                break;
+        Traverser traverser;
+


unused empty line

Change-Id: I882e472d001c40c0f1ffdc89c1c3c28bd6d34a17

zhoney force-pushed the custom-kout-knei-template-path branch from b2cab74 to 9eb68d8 Compare September 15, 2020 03:32

javeme reviewed Sep 18, 2020

View reviewed changes

zhoney mentioned this pull request Sep 21, 2020

5 tp algo #1157

Closed

zhoney and others added 13 commits September 21, 2020 17:48

Szzq paths improve (#51)

5660ca9

* template path use concurrent hashset to save paths in concurrent mode * support property filter for paths api Change-Id: I2e589f58cdfc48b4b8d16b7780b78cc17ab107d6

fix configuration not supported with auth (#55)

851760b

also paths and template paths use single thread Change-Id: I4375c455f377d9e4d43c0284ec30da1725fdf6bc

template path supports repeat times args (#57)

64088e5

paths supports nearest args oltp multiple threads reuse add multiple thread depth config Change-Id: I8c2f921ffb56302fb8d300c09a5a13857dac187a

fix group perperty can't be empty

b120508

Change-Id: I6d972894f3eec339181e5626bf45d01f36dbd4c0

upgrade api version to 58

e178508

implements: #1173 Change-Id: I85aa1d4274554d65f85a0deb7ac596e65dbb503b

add Consumers class

e98555e

Change-Id: I73d877fbd2099d9e63ef29ee0a10df30354595f9

move customized kout/kneighbor to kout/kneighbor

a0f44c0

Change-Id: Ib6fa985ff48fab203611218cc955079ad8cc90c3

move jaccard similar into jaccard similarity

924c5e1

Change-Id: Ia6ff5efd7e2a195f85427992b7bff875d30dec48

improve

f31cf3a

Change-Id: Ib43614c252912427c6ddaeb52c974a59b06934e6

improve

b9706e1

Change-Id: I5ef4b33e8c5b4ade0b372d3810e596d335f37f29

fix kout/kneighbor withVertex not work

ebe3157

Change-Id: Icbc65aa24e069b67bd5653b79473da48e2f973c6

rebase ramtable

25cb47a

Change-Id: I606772946a2a55f7472274242fb18f4ca91c189a

zhoney force-pushed the custom-kout-knei-template-path branch from 5635e57 to 25cb47a Compare September 21, 2020 10:04

improve

bff5a6a

Change-Id: I4ab46bc68644d35fab1fa966e27a0d7ab87b433b

houzhizhen previously approved these changes Sep 22, 2020

View reviewed changes

javeme reviewed Sep 25, 2020

View reviewed changes

extract common traverser from template paths traversers

8523889

Change-Id: If6f0c1370a2376ef176ce2db39e002ae11590e8d

zhoney dismissed houzhizhen’s stale review via 8523889 September 29, 2020 07:11

javeme requested changes Sep 29, 2020

View reviewed changes

javeme reviewed Sep 29, 2020

View reviewed changes

zhoney added 2 commits October 12, 2020 21:10

improve

21ea44c

Change-Id: I7d6912f33836ac9a36fd0dda82b446780675a390

extract path traverser for template paths and collection paths

876b3e1

Change-Id: I5b7441f562106fa3810d6d844c897add3010ef30

javeme reviewed Oct 13, 2020

View reviewed changes

improve

9e8372a

Change-Id: I760194b0f02a41873c2157a5ddf59dd4d1c3ceff

javeme reviewed Oct 20, 2020

View reviewed changes

normalize traverser

4fd93f9

Change-Id: I68226c55ca387ecfdd103e4af8b5e45faa9a934e

javeme reviewed Oct 21, 2020

View reviewed changes

improve

20c1ff5

Change-Id: I882e472d001c40c0f1ffdc89c1c3c28bd6d34a17

javeme approved these changes Oct 27, 2020

View reviewed changes

houzhizhen approved these changes Oct 28, 2020

View reviewed changes

Linary approved these changes Oct 28, 2020

View reviewed changes

zhoney merged commit d5cc79b into master Oct 28, 2020

zhoney deleted the custom-kout-knei-template-path branch October 28, 2020 09:58

Custom kout/kneighbor, multi-node-shortest-path, jaccard-similar and template paths #1174

Custom kout/kneighbor, multi-node-shortest-path, jaccard-similar and template paths #1174

Conversation

zhoney commented Sep 15, 2020

codecov bot commented Sep 15, 2020 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 15, 2020 •

edited