add entrance for pagerank and wcc #1114

zhoney · 2020-07-27T13:02:11Z

Change-Id: I9d0a35ac8ea2cd7f1c9b6cf86a62a06e68d6498d

Change-Id: I5dcd536180e34d25a72b84b0160b198f2429f70b

codecov · 2020-07-27T13:22:22Z

Codecov Report

Merging #1114 into master will decrease coverage by 3.25%.
The diff coverage is 1.93%.

@@             Coverage Diff              @@
##             master    #1114      +/-   ##
============================================
- Coverage     69.38%   66.13%   -3.26%     
- Complexity     5625     5652      +27     
============================================
  Files           331      354      +23     
  Lines         27126    28686    +1560     
  Branches       3898     4044     +146     
============================================
+ Hits          18821    18971     +150     
- Misses         6454     7847    +1393     
- Partials       1851     1868      +17

Impacted Files	Coverage Δ	Complexity Δ
.../java/com/baidu/hugegraph/api/job/ComputerAPI.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...main/java/com/baidu/hugegraph/job/ComputerJob.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...baidu/hugegraph/job/computer/AbstractComputer.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...com/baidu/hugegraph/job/computer/ComputerPool.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...baidu/hugegraph/job/computer/PageRankComputer.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...h/job/computer/WeakConnectedComponentComputer.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
.../java/com/baidu/hugegraph/task/HugeServerInfo.java	`67.39% <0.00%> (-1.00%)`	`18.00 <1.00> (ø)`
...om/baidu/hugegraph/task/StandardTaskScheduler.java	`79.14% <0.00%> (-0.26%)`	`90.00 <0.00> (ø)`
...n/java/com/baidu/hugegraph/util/ParameterUtil.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...n/java/com/baidu/hugegraph/config/CoreOptions.java	`99.25% <100.00%> (+0.20%)`	`2.00 <0.00> (ø)`
... and 49 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3cac836...fabce45. Read the comment docs.

javeme · 2020-07-27T13:45:13Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/compute/WeakConnectedComponentCompute.java

+        this.userDefinedParameters.put(PRECISION, threshold(parameters));
+    }
+
+    private static double threshold (Map<String, Object> parameters) {


remove space in "threshold ("

javeme · 2020-07-27T13:54:34Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/compute/AbstractCompute.java

+        return value;
+    }
+
+    public static String parameterString(Map<String, Object> parameters,


add ParameterUtil class and move these methods to it

javeme · 2020-07-27T13:56:46Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/compute/AbstractCompute.java

+    }
+
+    private Map<String, Object> readCommonConfig() {
+        return readSubConfig(COMMON);


this.readSubConfig

javeme · 2020-07-27T13:58:29Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/ComputeJob.java

+        ComputePool pool = ComputePool.instance();
+        Compute compute = pool.find(name);
+        E.checkArgument(compute != null,
+                        "There is no compute named '%s'", name);


There is no compute method named

Change-Id: I7ddfa6ee3959739ed81529bcc7ae078a537fd699

javeme · 2020-07-28T06:34:59Z

hugegraph-api/src/main/java/com/baidu/hugegraph/api/job/ComputeAPI.java

+        builder.name("compute:" + compute)
+               .input(JsonUtil.toJson(input))
+               .job(new ComputeJob());
+        return ImmutableMap.of("task_id", builder.schedule().id());


define var for builder.schedule()

javeme · 2020-07-28T06:36:13Z

hugegraph-core/src/main/java/com/baidu/hugegraph/config/CoreOptions.java

+
+    public static final ConfigOption<String> COMPUTER_CONFIG =
+            new ConfigOption<>(
+                    "computer.config",


compute/computer unify into one

javeme · 2020-07-28T06:36:30Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/ComputeJob.java

+    public static final String COMPUTE = "compute";
+
+    public static boolean check(String name, Map<String, Object> parameters) {
+        Compute algorithm = ComputePool.instance().find(name);


javeme · 2020-07-28T06:41:31Z

hugegraph-core/src/main/java/com/baidu/hugegraph/task/HugeServerInfo.java

@@ -212,6 +212,10 @@ public static HugeServerInfo fromVertex(Vertex vertex) {
    }

    public <V> boolean suitableFor(HugeTask<V> task, long now) {
+        if (task.compute() && !this.role.computer() ||
+            !task.compute() && this.role.computer()) {


use task.compute() != this.role.computer()

javeme · 2020-07-28T06:42:57Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/compute/WeakConnectedComponentCompute.java

+        this.userDefinedParameters.put(PRECISION, threshold(parameters));
+    }
+
+    private static double threshold(Map<String, Object> parameters) {


rename to precision

javeme · 2020-07-28T06:47:26Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/compute/AbstractCompute.java

+    @SuppressWarnings("unchecked")
+    private void setComputeSpecifiedParameters() {
+        ((Map<String, Object>) this.commonConfig.get(EXTRA_ARGS))
+                               .putAll(this.userDefinedParameters);


align with "."

javeme · 2020-07-28T06:50:16Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/compute/PageRankCompute.java

+    @Override
+    public void checkParameters(Map<String, Object> parameters) {
+        this.userDefinedParameters.put(MAX_STEPS, maxSteps(parameters));
+        this.userDefinedParameters.put(ALPHA, alpha(parameters));


add precision

javeme · 2020-07-28T06:56:41Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/ComputeJob.java

+        Compute compute = pool.find(name);
+        E.checkArgument(compute != null,
+                        "There is no compute method named '%s'", name);
+        compute.checkParameters(parameters);


move into call and rename checkParameters to checkAndCollectParameters

javeme · 2020-07-28T06:57:02Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/compute/AbstractCompute.java

+
+    private YamlConfiguration config;
+    private Map<String, Object> commonConfig = new HashMap<>();
+    protected Map<String, Object> userDefinedParameters = new HashMap<>();


don't define as field

Change-Id: I6038bd61f5ca9a664acabb4f2036efbda912e62a

javeme · 2020-07-28T09:29:30Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/computer/AbstractComputer.java

-        // Set current compute job's specified parameters
-        this.setComputeSpecifiedParameters();
+        // Set current computer job's specified parameters
+        this.commonConfig.putAll(this.checkAndCollectParameters(parameters));


don't update commonConfig

javeme · 2020-07-28T09:29:52Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/computer/AbstractComputer.java


-        // Construct shell command for compute job
+        // Construct shell command for computer job


this.constructShellCommands

javeme · 2020-07-28T09:32:40Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/ComputerJob.java

 import com.baidu.hugegraph.util.E;
 import com.baidu.hugegraph.util.JsonUtil;

-public class ComputeJob extends SysJob<Object> {
+public class ComputerJob extends SysJob<Object> {


extends UserJob

need get computer conf from SysJob's params

Change-Id: I869d0d895917839269293a97b26ace4b7269b7ac

javeme · 2020-07-28T12:34:50Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/ComputerJob.java

+        return true;
+    }
+
+    public HugeConfig config() {


do not expose HugeConfig to the outside

javeme · 2020-07-28T12:35:53Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/computer/AbstractComputer.java

+    private static final String ARG_SYMBOL = "C";
+    private static final String MINUS = "-";
+    private static final String EQUAL = "=";
+    private static final String EMPTY = " ";


javeme · 2020-07-28T12:36:21Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/computer/AbstractComputer.java

+        try {
+            Process process = Runtime.getRuntime().exec(command);
+            exitCode = process.waitFor();
+        } catch (Exception e) {


cache throwable

javeme · 2020-07-28T12:37:26Z

hugegraph-core/src/main/java/com/baidu/hugegraph/job/computer/AbstractComputer.java

+            return DEFAULT_PRECISION;
+        }
+        double precision = ParameterUtil.parameterDouble(parameters, PRECISION);
+        E.checkArgument(precision > 0 && precision < 1,


change to 0.0 1.0

javeme · 2020-07-28T12:39:58Z

hugegraph-dist/src/assembly/static/conf/computer.yaml

+  worker_heap: 10000,
+  vertex_input_dir: /data/data_wdc_600/arc,
+  workers: 51,
+  zookeeper_list: "10.157.11.170:2181",


Change-Id: I731fc6374992b834f22403be0734ca9288b21b34

zhoney added 2 commits July 27, 2020 21:01

add entrance for pagerank and wcc

77a100b

Change-Id: I9d0a35ac8ea2cd7f1c9b6cf86a62a06e68d6498d

improve

54ef4bf

Change-Id: I5dcd536180e34d25a72b84b0160b198f2429f70b

javeme reviewed Jul 27, 2020

View reviewed changes

improve

592ec66

Change-Id: I7ddfa6ee3959739ed81529bcc7ae078a537fd699

javeme requested changes Jul 28, 2020

View reviewed changes

format all args to "-C key=value"

80df0a6

Change-Id: I6038bd61f5ca9a664acabb4f2036efbda912e62a

javeme reviewed Jul 28, 2020

View reviewed changes

improve

2a4ed24

Change-Id: I869d0d895917839269293a97b26ace4b7269b7ac

javeme reviewed Jul 28, 2020

View reviewed changes

improve

fabce45

Change-Id: I731fc6374992b834f22403be0734ca9288b21b34

Linary approved these changes Jul 28, 2020

View reviewed changes

javeme approved these changes Jul 28, 2020

View reviewed changes

javeme merged commit 5f60a29 into master Jul 28, 2020

javeme deleted the compute branch July 28, 2020 14:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add entrance for pagerank and wcc #1114

add entrance for pagerank and wcc #1114

zhoney commented Jul 27, 2020

codecov bot commented Jul 27, 2020 •

edited

javeme Jul 27, 2020

javeme Jul 27, 2020

javeme Jul 27, 2020

javeme Jul 27, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

zhoney Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020

javeme Jul 28, 2020


		// Construct shell command for compute job
		// Construct shell command for computer job

add entrance for pagerank and wcc #1114

add entrance for pagerank and wcc #1114

Conversation

zhoney commented Jul 27, 2020

codecov bot commented Jul 27, 2020 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 27, 2020 •

edited