Introducing IR query service (#1407)

* [Gaia/IR] Fix compiling error due to updating Sink pb * [Gaia/IR] Fix compiling error due to updating Sink pb * [Runtime UT] update path_expand ut according to new sink pb * [IR Runtime] fix apply ut due to updating Sink pb * support by empty * refine interface IrCustomizedTraversalSource * [IR Runtime] set Head as None after project multiple columns * [IR Runtime] Do not sink column if corresponding entry is None * [IR Runtime] add more path_expand uts * minor refine * [IR Core] update union pb * [IR Core] upadte union in ffi * add union * [Runtime UT] add shuffle op in physical plans in integration tests * add union ci test * [Runtime UT] add worker_num = 2 for some test cases in integration * [Gaia/IR] Update Makefile. * [IR Runtime] update EntryAccumulator, and impl Encode/Decode for it * [IR Runtime] rename get_entry() and get_arc_entry() in TagKey * [Gaia/IR] Update ldbc schema file for exp store. * update union * minor: rename * update union * fix union sink * add result parser of path expand * fix minor * [IR Runtime] use Count<()> in RecordAccumulator * add support to range(0, 1) in gremlin * restrict global id with integer type only in antlr * fix antlr parser of integerLiteralList && add ci tests * [Gaia/IR] Fix union one branch * [Gaia/IR] Fix compiling issue. * [Gaia/IR] Fix an error of missing tag for group-by alias. * [IR Runtime] Support Max, Min, ToSet in Group * [Gaia/IR] Remove useless print. * [IR Runtime] Support Count_Distinct in Group * preserve order of group entries when parsing results && add fusion strategy of scan * [Gaia/IR] Add Property::Length Key * [Gaia/IR] Add partition test for subtask * fix variable of path expand length * [Gaia/IR] Fix missing shuffle issue in subtask. * fix optimization of scan fusion * [Gaia/IR] Add support of multiple current nodes for union. * [Gaia/IR] Add support of multiple current nodes for union. * [Gaia/IR] Add support of multiple current nodes for union. * [Gaia/IR] Add test cases for union operator. * [Gaia/IR] Refine codes. * [Runtime Ut] add more uts for accumulator * [IR Runtime] add extra_params in QueryParam * [Graph Adapter] adapt to graphscope storage * [Graph Adapter] remove ID_MASK * [IR EXECUTOR] try to start ir executor on vineyard * [Graph Adapter] throw query_store_error in graph adapter * [Gaia/IR] Move `len()` from `GraphElement` to `Element`. * [Gaia/IR] Supporting getting all properties via a special "~all" key. * [Gaia/IR] All test cases for evaluating getting all properties. * [IR Proto] add is_all_columns flag in QueryParams for getting all properties * [IR Runtime] support get all props in QueryParams * [IR Runtime] refine get all props in QueryParams * [Gaia/IR] Fix compiling issues while adding is_all_columns in pb::QueryParams. * [IR Runtime] renmae properties as columns in QueryParams * [Runtime UT] add ut for valueMap() * minor: code refine * [Gaia/IR] Refine Match operator * minor: code refine * fix a bug in accumulator ut * [Gaia/IR] Add QueryParams for GetV. * [Gaia/IR] Minor refine. * [Gaia/IR] Minor refine. * [Gaia/IR] Fix some typos. * [IR Proto] update ExtraParams in proto, runtime and graph_adapter * [Gaia/IR] Redefine QueryParams to use a map instead of a vector for the extra parameters. * [Gaia/IR] Simplify the definition of QueryParams in test cases. * [Gaia/IR] Refine the error returning message. * minor refine * minor refine * [Gaia/IR] Add Patmat proposal * [Gaia/IR] Refine codes to fix building errors. * minor refine * [Gaia/IR] Fix id-mappings while transforming LogicalPlan and pb::LogicalPlan * [Graph Adapter] remove schema accessing in Runtime * update op tests with the changes of ir core * add newline at the end of file * support more patterns of subtask * fix valueMap().as(a) * add ldbc test * initialize configs from system property * return empty if id not exist in test properties * add comments * add comments * update ldbc 6 * update test * [IR Runtime] do not throw error when # of subtask results > 1 * [IR Runtime] process None obj in TagKey to make it identical to eval() * fix minor * refine makefile * [Gaia/Engine]: Add -l limit_one configuration for amsk and spmsk benchmarking tools. * [Bug Fix] sink CommonObject::None as a None object * [Gaia/IR] Implementing building logical plan for a `Pattern` operator. * [Gaia/IR] Refine codes and add test cases for sentence composition/join. * [Gaia/IR] Add test cases for building logical plan for pattern matching * set graph.schema from conf file if conmmand line not exist * add alias manager and update tests * add select(keys/values) && update ldbc test * [Gaia/IR] Add test cases for building logical plan for pattern matching * [Gaia/IR] Prevent pb::Patten to reference existing tags. * [Gaia/IR] Refine codes. * refine codes according to code reviews * refine interface of judging whether a sub_traversal can be converted to expression * add select(keys/values) && select(keys/values).values/valueMap(..) as expression * refine codes according to code reviews * [Gaia/IR] Make pattern matching runnable at runtime * rename ExprRes as ExprResult * [Gaia/IR] Add Ffi functions for the compiler to call constructing a `Pattern` operator. * [GAIA/engine] fix bug in any operator caused by notify_end * degrade tinkerpop version to 3.4.12 && update gremlin tests * refine interface of ir compiler to be compatible with groot & vineyard * adapt ir compiler to groot store * adapt ir compiler to vineyard store * [Graph Adapter] try to start ir executor on v2 * [Gaia/IR] Merge new gaia-x * [Gaia/IR] Fix a bug of throwing error in case of referring to a non-exist tag. * [Benchmark] init benchmark * [IR Benchmark] add ir_queries in benchmark * [IR Benchmark] ignore subquery for test * [IR Benchmark] add LDBC_3 & LDBC_4 in benchmark * [IR Benchmark] update query plans with select(keys), select(values) etc. * [IR Benchmark] move benchmark to query_service/benchmark * [IR Benchmark] move ldbc queries for ir or gaiax under benchmark/queries * support where().by(subtask) * revert changes to ir.compiler.properties * [Gaia/IR] Change `is_anti` to `JoinKind` in pb::patttern::Sentence. * [Gaia/IR] Refine protobuf data field. * add match step && related ci tests * update match according to changes with ir_core * [Gaia/IR] Fix a bug of throwing TagNotExist while referring to a tag that is neither a vertex/edge. * tmp commit: replace gaia executor with ir executor, for e2e test * [Graph Adapter] rename ffi for ir executor * [Graph Adapter] set si as default value if get si failed * update tests according to changes with ir_core * [Graph Adapter] rename ffi for ir executor * [Gaia/IR] Prepare parsing tag from string to id. * add expr step * add copyrights * [Gaia/IR] Set tag id for pb::Sink * [Gaia/IR] Set tag id for pb::Dedup * update according to changes with return code of ir core * update * update * Ir return code (#131) * [Gaia/IR] Return FfiResult that includes both ResultCode and error message in Ffi functions. * [Gaia/IR] Return FfiData instead of FfiJobBuffer for build_physical_plan to include error message from building logical/physical plan * [Gaia/IR] Fix bug of returning error message in Ffi functions. * [Gaia/IR] Fix bug of returning error message in Ffi functions. * [Gaia/IR] Fix bug of returning error message in Ffi functions. * [Gaia/IR] Fix bug of returning error message in Ffi functions. * [Gaia/IR] Fix bug of returning error message in Ffi functions. * [Gaia/IR] Add `destory_ffi_error()` * [Gaia/IR] Add [no_mangle] for `destory_ffi_error()` Co-authored-by: longbin.lailb <longbin.lailb@alibaba-inc.com> * add enV() step * [Gaia/IR] Fix a bug of converting a c string to Rust string without checking it to be `NULL`. * [Gaia/IR] Fix bugs of pattern matching. * [Gaia/IR] Fix bugs of pattern matching. * [Gaia/IR] Fix bugs of pattern matching. * [Gaia/IR] Refine codes. * refine interface of ir compiler to be compatible with gs * fix bugs of ir-groot adaptor * fix bugs of ir-vineyard adaptor * add ir_core lib to distribution target * [Gaia/IR] Try to reverse sentence if possible. * remove servers in job conf from compiler * [Gaia/IR] Try to reverse sentence if possible. * set servers in job conf * [GraphAdapter] Start IR Service when given GAIA option to ensure e2e test * fix bugs of gremlin parser when column key is NameOrId::ID * add ir core lib in docker build phase * revert BASE_VERSION in graphscope-store.Dockerfile * revert changes of degrading tinkerpop * [GraphAdapter] remove dependency of gaia gremlin-runtime in graphscope for groot and vineyard * add copyrights && refine codes * [Pegasus Server/Client] add server-v0 and client back * add executor copy in k8s and Makefile * merge ir compiler to main * update versions of grpc & protobuf in pegasus java client * set is_column_id as false for exp store * add back executor of ir on graphscope store * add pegasus-client module to interactive_engine * [Gaia/IR] Unify setting query parameters for GetV/EdgeExpand/PathExpand/Scan * update compiler codes * [Gaia/IR] Fix a bug. * [Gaia/IR] Refine codes * [BugFix] set is_column_id as false in ldbc schema for exp store * format java codes of ir compiler * minor: add some copyright * minor: code refactor * minor: code refactor * fix errors of code refactor * rm useless file * format java codes * minor: remove useless file * add ci tests for ir on experimental & groot store * fix errors of code refactor * add shell of ci tests * update gaia ci env to vineyard:v0.3.21 * update ci flags of gaia * update gaia ci process Co-authored-by: longbin.lailb <longbin.lailb@alibaba-inc.com> Co-authored-by: shirly121 <zxlmillie@163.com> Co-authored-by: Longbin Lai <longbin.lai@gmail.com> Co-authored-by: Neng Li <lnfjpt@gmail.com>
alibaba · Mar 28, 2022 · 7d2f92c · 7d2f92c
1 parent 88e7670
commit 7d2f92c
Show file tree

Hide file tree

Showing 431 changed files with 53,668 additions and 295 deletions.
diff --git a/.github/workflows/gaia.yml b/.github/workflows/gaia.yml
@@ -6,21 +6,22 @@ on:
     branches:
       - main
     paths:
-      - 'research/gaia/**'
+      - 'research/**'
       - '.github/workflows/gaia.yml'
-      - '!research/gaia/**.md'
+      - '!research/**.md'
   pull_request:
     branches:
       - main
     paths:
-      - 'research/gaia/**'
+      - 'research/**'
       - '.github/workflows/gaia.yml'
-      - '!research/gaia/**.md'
+      - '!research/**.md'
 
 jobs:
   gaia-test:
-    runs-on: ubuntu-20.04
-    if: false
+    runs-on: self-hosted
+    container:
+      image: registry.cn-hongkong.aliyuncs.com/graphscope/graphscope-vineyard:v0.3.21
     defaults:
       run:
         shell: bash --noprofile --norc -eo pipefail {0}
@@ -31,40 +32,50 @@ jobs:
 
     - name: Install Dependencies
       run: |
-        # Due to an observation of changing hostname in github runners,
-        # append 127.0.0.1 to etc/hosts to avoid DNS lookup.
-        r=`cat /etc/hosts | grep $(hostname) || true`
-        if [ -z "${r}" ];then export hn=$(hostname); sudo -E bash -c 'echo "127.0.0.1 ${hn}" >> /etc/hosts'; fi
-        cat /etc/hosts
+        # install cargo cause github change $HOME to /github/home, so we can't use
+        # `source ~/.bashrc` or `source ${HOME}/.bashrc` to find cargo command.
+        sudo chown -R $(id -u):$(id -g) ${HOME}
+        sudo chmod -R a+wrx ${HOME} ${GITHUB_WORKSPACE}
+        wget --no-verbose https://golang.org/dl/go1.15.5.linux-amd64.tar.gz
+        sudo tar -C /usr/local -xzf go1.15.5.linux-amd64.tar.gz
+        curl -sf -L https://static.rust-lang.org/rustup.sh | sh -s -- -y --profile minimal --default-toolchain 1.54.0
+        echo "source ~/.cargo/env" >> ~/.bashrc
+        source ${HOME}/.bashrc
+        rustup component add rustfmt
+        # install rdkafka
+        sudo yum install -y librdkafka-devel
+        # install cppkafka
+        git clone -b 0.4.0 --single-branch --depth=1 \
+            https://github.com/mfontanini/cppkafka.git /tmp/cppkafka && \
+          cd /tmp/cppkafka && git submodule update --init && \
+          mkdir -p build && cd build && \
+          cmake .. && make -j && sudo make install && \
+          rm -fr /tmp/cppkafka
 
-        # install dependencies
-        ${GITHUB_WORKSPACE}/scripts/install_deps.sh --dev --verbose
-
-    - name: Build GraphScope
+    - name: Build Ir on Experimental Store
       run: |
-        source ${HOME}/.graphscope_env
-        make BUILD_TYPE=debug
+        source ${HOME}/.bashrc
+        cd ${GITHUB_WORKSPACE}/research/query_service/ir/compiler
+        make build
 
-    - name: Build MaxGraph Store
+    - name: Build Ir on Groot Store
       run: |
-        source ${HOME}/.graphscope_env
+        source ${HOME}/.bashrc
         cd ${GITHUB_WORKSPACE}/interactive_engine
         mvn clean install -DskipTests -Pv2
 
-    - name: Gaia test on MaxGraph Store
+    - name: Ir Unit Test
       run: |
-        source ${HOME}/.graphscope_env
-        cd ${GITHUB_WORKSPACE}/interactive_engine/gaia-adaptor && ./gremlin_test.sh
+        source ${HOME}/.bashrc
+        cd ${GITHUB_WORKSPACE}/research/query_service/ir/compiler && make test
 
-    - name: Run Backup Test on Maxgraph Store
+    - name: Ir Integration Test on Experimental Store
       run: |
-        source ${HOME}/.graphscope_env
-        cd ${GITHUB_WORKSPACE}/interactive_engine/sdk && ./backup_test.sh
+        source ${HOME}/.bashrc
+        cd ${GITHUB_WORKSPACE}/research/query_service/ir/compiler && ./ir_exprimental_ci.sh
 
-    - name: Gaia runtime test
+    - name: Ir Integration Test on Groot Store
       run: |
-        source ${HOME}/.graphscope_env
+        source ${HOME}/.bashrc
+        cd ${GITHUB_WORKSPACE}/interactive_engine/ir-adaptor && ./ir_groot_ci.sh
 
-        cd ${GITHUB_WORKSPACE}/research/query_service/gremlin/gremlin_core/tests
-        sh prepare_all_test_plan.sh
-        cargo test
diff --git a/.github/workflows/pegasus.yml b/.github/workflows/pegasus.yml
@@ -0,0 +1,84 @@
+name: GraphScope Pegasus CI
+
+on:
+  push:
+    branches:
+      - gaia-x
+    paths:
+      - 'research/engine/pegasus/**'
+      - '.github/workflows/pegasus.yml'
+
+  pull_request:
+    branches:
+      - gaia-x
+    paths:
+      - 'research/engine/pegasus/**'
+      - '.github/workflows/pegasus.yml'
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  peagsus-build:
+    runs-on: ubuntu-20.04
+    defaults:
+      run:
+        shell: bash --noprofile --norc -eo pipefail {0}
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Build & Test
+      run: |
+        echo $(pwd)
+        cd research/engine/pegasus
+        cargo build --verbose
+        cargo test --verbose
+
+    - name: Run example
+      run: |
+        cd research/engine/pegasus
+
+        cargo build --release --examples
+
+        # Run word_count_toy
+        target/release/examples/word_count_toy
+
+        target/release/examples/word_count_toy -w 2
+
+        target/release/examples/word_count_toy -w 4
+
+        # Run logistic regression
+        target/release/examples/logistic_regression --data pegasus/examples/data/binary.csv
+
+  k8s-test:
+    runs-on: self-hosted
+    defaults:
+      run:
+        shell: bash --noprofile --norc -eo pipefail {0}
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name : Prepare Image
+      run: |
+        sudo docker pull registry.cn-hongkong.aliyuncs.com/graphscope/pegasus-base:latest
+        sudo docker build -t registry.cn-hongkong.aliyuncs.com/graphscope/pegasus:${{ github.sha }} \
+                          --network=host \
+                          -f research/engine/pegasus/test/k8s/pegasus.Dockerfile .
+        echo "        - name: CONFIG_MAP_NAME" >> research/engine/pegasus/test/k8s/pegasus-set.yaml
+        echo "          value: \"${{ github.sha }}\"" >> research/engine/pegasus/test/k8s/pegasus-set.yaml
+        echo "        image: registry.cn-hongkong.aliyuncs.com/graphscope/pegasus:${{ github.sha }}" >> research/engine/pegasus/test/k8s/pegasus-set.yaml
+
+    - name: Start k8s cluster
+      run: |
+        kubectl get ns pegasus-ci || kubectl create ns pegasus-ci
+        kubectl --namespace pegasus-ci create -f research/engine/pegasus/test/k8s/role_binding.yaml
+        kubectl --namespace pegasus-ci create -f research/engine/pegasus/test/k8s/pegasus-set.yaml
+        bash research/engine/pegasus/test/k8s/read_result.sh ${{ github.sha }}
+
+    - name: Clean
+      if: always()
+      run: |
+        kubectl delete ns pegasus-ci || true
+        docker rmi -f registry.cn-hongkong.aliyuncs.com/graphscope/pegasus:${{ github.sha }} --wait=false || true
diff --git a/.gitignore b/.gitignore
@@ -65,6 +65,7 @@ interactive_engine/executor/store/src/db/proto/*
 interactive_engine/data_load_tools/dependency-reduced-pom.xml
 interactive_engine/gaia-adaptor/dependency-reduced-pom.xml
 interactive_engine/executor/Cargo.lock
+research/engine/pegasus/benchmark/src/graph/storage/clickhouse/pb_gen/*
 
 # java sdk
 java/**/dependency-reduced-pom.xml

diff --git a/Makefile b/Makefile
@@ -115,6 +115,7 @@ gie:
 	mkdir -p $(WORKING_DIR)/.install_prefix && \
 	tar -xf $(WORKING_DIR)/interactive_engine/assembly/target/maxgraph-assembly-0.0.1-SNAPSHOT.tar.gz --strip-components 1 -C $(WORKING_DIR)/.install_prefix && \
 	cp $(WORKING_DIR)/interactive_engine/executor/target/$(BUILD_TYPE)/executor $(WORKING_DIR)/.install_prefix/bin/executor && \
+	cp $(WORKING_DIR)/interactive_engine/executor/target/$(BUILD_TYPE)/gaia_executor $(WORKING_DIR)/.install_prefix/bin/gaia_executor && \
 	sudo cp -r $(WORKING_DIR)/.install_prefix/* $(INSTALL_PREFIX) && \
 	rm -fr $(WORKING_DIR)/.install_prefix
 

diff --git a/interactive_engine/distribution/src/assembly/assembly.xml b/interactive_engine/distribution/src/assembly/assembly.xml
@@ -13,6 +13,13 @@
       </includes>
       <outputDirectory>native</outputDirectory>
     </fileSet>
+    <fileSet>
+      <directory>${project.parent.basedir}/../research/query_service/ir/target/release</directory>
+      <includes>
+        <include>libir_core*</include>
+      </includes>
+      <outputDirectory>native</outputDirectory>
+    </fileSet>
     <fileSet>
       <directory>${project.parent.basedir}/lgraph/build/install</directory>
       <excludes>

diff --git a/interactive_engine/executor/Cargo.toml b/interactive_engine/executor/Cargo.toml
@@ -5,7 +5,8 @@ members = [
     "store",
     "server",
     "Pegasus",
-    "ffi"
+    "ffi",
+    "gaia_runtime"
 ]
 
 [profile.release]

diff --git a/interactive_engine/executor/ffi/Cargo.toml b/interactive_engine/executor/ffi/Cargo.toml
@@ -13,6 +13,10 @@ maxgraph-store = { path = "../store" }
 maxgraph-common = { path = "../../rust-common" }
 maxgraph-server = { path = "../server" }
 pegasus = { path = "../Pegasus", package = "pegasus" }
+gaia_pegasus = { path = "../../../research/engine/pegasus/pegasus", package = "pegasus" }
+pegasus_network = { path = "../../../research/engine/pegasus/network" }
+pegasus_server = { path = "../../../research/engine/pegasus/server-v0" }
+graph_proxy =  { path = "../../../research/query_service/ir/graph_proxy" }
 itertools = "0.7.8"
 log = "0.3"
 log4rs = "0.8.0"

diff --git a/interactive_engine/executor/ffi/src/executor/gaia/engine_ports_response.rs b/interactive_engine/executor/ffi/src/executor/gaia/engine_ports_response.rs
@@ -0,0 +1,64 @@
+//
+//! Copyright 2022 Alibaba Group Holding Limited.
+//!
+//! Licensed under the Apache License, Version 2.0 (the "License");
+//! you may not use this file except in compliance with the License.
+//! You may obtain a copy of the License at
+//!
+//! http://www.apache.org/licenses/LICENSE-2.0
+//!
+//! Unless required by applicable law or agreed to in writing, software
+//! distributed under the License is distributed on an "AS IS" BASIS,
+//! WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+//! See the License for the specific language governing permissions and
+//! limitations under the License.
+//!
+
+use std::os::raw::c_char;
+use std::ffi::CString;
+
+#[repr(C)]
+#[allow(non_snake_case)]
+pub struct EnginePortsResponse {
+    success: bool,
+    errMsg: *const c_char,
+    enginePort: i32,
+    rpcPort: i32,
+}
+
+impl EnginePortsResponse {
+    pub fn new(engine_port: i32, rpc_port: i32) -> Box<EnginePortsResponse> {
+        Box::new(EnginePortsResponse {
+            success: true,
+            errMsg: std::ptr::null(),
+            enginePort: engine_port,
+            rpcPort: rpc_port,
+        })
+    }
+
+    pub fn new_with_error(err_msg: &str) -> Box<EnginePortsResponse> {
+        let msg = CString::new(err_msg).unwrap();
+        let response = EnginePortsResponse {
+            success: false,
+            errMsg: msg.as_ptr(),
+            enginePort: 0,
+            rpcPort: 0,
+        };
+        ::std::mem::forget(msg);
+        Box::new(response)
+    }
+}
+
+impl Drop for EnginePortsResponse {
+    fn drop(&mut self) {
+        unsafe {
+            if !self.errMsg.is_null() {
+                CString::from_raw(self.errMsg as *mut c_char);
+            }
+        }
+    }
+}
+
+#[no_mangle]
+#[allow(non_snake_case)]
+pub extern fn dropEnginePortsResponse(_: Box<EnginePortsResponse>) {}