[GAIA Compiler] Build algebra layer structures for filter and project in `GraphBuilder` #2482

shirly121 · 2023-03-03T12:40:00Z

What do these changes do?

Related issue number

Fixes

…ructure

…phBuilder

…ayer

codecov-commenter · 2023-03-06T05:44:30Z

Codecov Report

Merging #2482 (85819bc) into main (a012a92) will decrease coverage by 32.76%.
The diff coverage is n/a.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2482       +/-   ##
===========================================
- Coverage   72.62%   39.87%   -32.76%     
===========================================
  Files          88       88               
  Lines        9816     9816               
===========================================
- Hits         7129     3914     -3215     
- Misses       2687     5902     +3215

see 47 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a012a92...85819bc. Read the comment docs.

longbinlai · 2023-03-07T15:36:09Z

interactive_engine/compiler/src/test/java/com/alibaba/graphscope/common/ir/FilterTest.java

+    }
+
+    /**
+     * 10 > 20 -> always returns false, create {@link LogicalValues} which carries all data types of the node before the filter


Why the behavior?

longbinlai · 2023-03-07T15:38:14Z

interactive_engine/compiler/src/test/java/com/alibaba/graphscope/common/ir/ProjectTest.java

+                        .variable("a", "age");
+        Project project = (Project) builder.project(ImmutableList.of(variable)).build();
+        Assert.assertEquals("[a.age]", project.getProjects().toString());
+        Assert.assertEquals("RecordType(INTEGER age)", project.getRowType().toString());


Where are we getting the DataType of age?

project.getRowType()返回所有expressions的类型信息，是一个RelRecord(List<RelDataTypeField>)类型，每一个expression都一一对应一个RelDataTypeField，RelDataTypeField.getType()就能拿到对应expression的类型；

longbinlai · 2023-03-07T15:38:42Z

interactive_engine/compiler/src/test/java/com/alibaba/graphscope/common/ir/ProjectTest.java

+                project.getRowType().toString());
+    }
+
+    // project("a.age") -> expr: "a.age", alias: "age"


Why alias: age ?

project/aggregate都使用了calcite的推断方式：

如果expression是variable类型，则尽量使用variable中的tag/property来命名，i.e. a->a, a.age -> age ;

对于其他的expression，则通过$f前缀加上id来命名，其中id是从0开始直到推断出一个不重复的命名为止 ;

这里是推断的主要逻辑：https://github.com/shirly121/GraphScope/blob/013f8d69492ae18ac524173aa3c37fb9d31b5619/interactive_engine/compiler/src/main/java/com/alibaba/graphscope/common/ir/tools/AliasInference.java#L77

longbinlai · 2023-03-07T15:39:18Z

interactive_engine/compiler/src/test/java/com/alibaba/graphscope/common/ir/ProjectTest.java

+        Assert.assertEquals("RecordType(INTEGER age)", project.getRowType().toString());
+    }
+
+    // project("a.age+1") -> expr: "a.age+1", alias: "$f0"


What is $f0 standing for?

longbinlai · 2023-03-07T15:42:48Z

...e/compiler/src/main/java/com/alibaba/graphscope/common/ir/tools/config/PathExpandConfig.java

@@ -93,6 +97,27 @@ public RelNode getGetV() {
        return getV;
    }

+    @Override
+    public String toString() {
+        return "PathExpandConfig{"


Always use StringBuilder for such long string concatenation

longbinlai · 2023-03-07T15:44:53Z

...ne/compiler/src/main/java/com/alibaba/graphscope/common/ir/rel/graph/GraphLogicalExpand.java

    }

    public static GraphLogicalExpand create(
            GraphOptCluster cluster, List<RelHint> hints, RelNode input, TableConfig tableConfig) {
        return new GraphLogicalExpand(cluster, hints, input, tableConfig);
    }

-    private GraphOpt.Expand directionOpt() {
+    public GraphOpt.Expand getOpt() {
        ObjectUtils.requireNonEmpty(hints);


What is the behavior of requireNonEmpty? Throwing error, or quick the program. How does it affect the working thread?

只是抛出异常，当前线程返回一个异常信息，不会影响整个服务

longbinlai · 2023-03-07T15:46:36Z

...gine/compiler/src/main/java/com/alibaba/graphscope/common/ir/rel/graph/GraphLogicalGetV.java

    }

    public static GraphLogicalGetV create(
            GraphOptCluster cluster, List<RelHint> hints, RelNode input, TableConfig tableConfig) {
        return new GraphLogicalGetV(cluster, hints, input, tableConfig);
    }

-    private GraphOpt.GetV getVOpt() {
+    public GraphOpt.GetV getOpt() {
        ObjectUtils.requireNonEmpty(hints);
        RelHint optHint = hints.get(0);


What is hints?

List是用来额外传算子配置的config，类似于kv键值对；点/边/path_expand的opt都是通过kv给到对应算子，再通过getOpt函数取出来；

这里是GraphBuilder创建图算子时设置的地方：https://github.com/shirly121/GraphScope/blob/ir_algebra_tmp/interactive_engine/compiler/src/main/java/com/alibaba/graphscope/common/ir/tools/GraphBuilder.java#L239

…ators

longbinlai

LGTM

longbinlai · 2023-03-10T02:04:35Z

...engine/compiler/src/main/java/com/alibaba/graphscope/common/ir/type/GraphPxdElementType.java

+/**
+ * each element type of path expand
+ */
+public class GraphPxdElementType extends AbstractSqlType {


Use GraphPathExpandElementType. The short name is not very clear.

longbinlai · 2023-03-10T02:05:39Z

...ctive_engine/compiler/src/main/java/com/alibaba/graphscope/common/ir/type/GraphProperty.java

+    }
+
+    public enum Opt {
+        ID,


Please comment these options.

shirly121 and others added 30 commits November 4, 2022 16:45

[IR Compiler] introduce interfaces for a new compiler intermediate st…

f3a125c

…ructure

[IR Compiler] update interfaces

fe9531f

Merge branch 'main' into ir_cypher

b47cba5

refine codes

bba09ae

add relational expression for LogicalMatch

9026d66

refine interfaces

f32140d

update GraphBuilder to build algebra structures

bf5ff13

implement graph operator source and expressions

5e95e15

Merge branch 'main' into ir_cypher

3111d2a

minor fix

6467e57

fix pom

c66a3fe

remove useless tests

e8d4a3f

[GAIA Compiler] minor fix

558326d

[GAIA Compiler] add filter implementation in GraphBuilder

9f1b765

[GAIA Compiler] add match/aggregate/sort/limit implementations in Gra…

815dfe6

…phBuilder

[GAIA Compiler] fix format

a07712d

[GAIA Compiler] minor fix

3ac4df0

[GAIA Compiler] minor fix

32d11b0

[GAIA Compiler] add expression test, [a.age='X']

2dd7d9a

[GAIA Compiler] add support to [POWER, MOD, UNARY_MINUS] in algebra l…

391f5b8

…ayer

[GAIA Compiler] refact package name

fac477c

Merge branch 'main' into ir_algebra

dc5279f

[GAIA Compiler] refact opt types

1b3ab39

[GAIA Compiler] refine interfaces of GraphBuilder

d44bee5

[GAIA Compiler] pre-compute complex expressions in sort or group

ccc99c9

[GAIA Compiler] make property name to id configurable

30b9535

[GAIA Compiler] make property name to id configurable

04835bd

[GAIA Compiler] minor fix

e191460

[GAIA Compiler] convert alias name to id

e13bbb9

[GAIA Compiler] remove exception resources

76fc04c

shirly121 added 8 commits February 28, 2023 16:03

[GAIA Compiler] convert RelNode to ir logical plan

f6802d7

update

3e84256

Merge branch 'main' into ir_algebra

eb5fa5c

[GAIA Compiler] add codes

d74c033

add codes

0d38aea

[GAIA Compiler] convert RelNode to ir core pb

52061de

[GAIA Compiler] build filter and project in GraphBuilder

1c1ac82

[GAIA Compiler] minor fix

c7c256a

shirly121 requested a review from longbinlai March 3, 2023 12:40

[GAIA Compiler] fix path expand builder

013f8d6

longbinlai reviewed Mar 7, 2023

View reviewed changes

[GAIA Compiler] refine ir algebra tests and codes according to reviews

3fc0d4c

shirly121 requested a review from longbinlai March 9, 2023 12:04

[GAIA Compiler] remove hints from constructor arguments of graph oper…

43ba45f

…ators

longbinlai approved these changes Mar 10, 2023

View reviewed changes

Merge branch 'main' into ir_algebra_tmp

85819bc

longbinlai merged commit cf5d75e into alibaba:main Mar 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GAIA Compiler] Build algebra layer structures for filter and project in `GraphBuilder` #2482

[GAIA Compiler] Build algebra layer structures for filter and project in `GraphBuilder` #2482

shirly121 commented Mar 3, 2023

codecov-commenter commented Mar 6, 2023 •

edited

longbinlai Mar 7, 2023

longbinlai Mar 7, 2023

shirly121 Mar 9, 2023 •

edited

longbinlai Mar 7, 2023

shirly121 Mar 9, 2023 •

edited

longbinlai Mar 7, 2023

shirly121 Mar 9, 2023

longbinlai Mar 7, 2023

shirly121 Mar 9, 2023

longbinlai Mar 7, 2023

shirly121 Mar 9, 2023 •

edited

longbinlai Mar 7, 2023

shirly121 Mar 9, 2023 •

edited

longbinlai left a comment

longbinlai Mar 10, 2023

longbinlai Mar 10, 2023

[GAIA Compiler] Build algebra layer structures for filter and project in GraphBuilder #2482

[GAIA Compiler] Build algebra layer structures for filter and project in GraphBuilder #2482

Conversation

shirly121 commented Mar 3, 2023

What do these changes do?

Related issue number

codecov-commenter commented Mar 6, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shirly121 Mar 9, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shirly121 Mar 9, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shirly121 Mar 9, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shirly121 Mar 9, 2023 • edited

Choose a reason for hiding this comment

longbinlai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[GAIA Compiler] Build algebra layer structures for filter and project in `GraphBuilder` #2482

[GAIA Compiler] Build algebra layer structures for filter and project in `GraphBuilder` #2482

codecov-commenter commented Mar 6, 2023 •

edited

shirly121 Mar 9, 2023 •

edited

shirly121 Mar 9, 2023 •

edited

shirly121 Mar 9, 2023 •

edited

shirly121 Mar 9, 2023 •

edited