Enhance setup script (#26)

* enhance setup script

* modify doc

* fix github action

* fix action

* use wget

.github/workflows/import.yaml

@@ -46,6 +46,10 @@ jobs:
         with:
           maven-version: 3.5.4
 
+      - uses: actions/setup-go@v1
+        with:
+          go-version: 1.16.4
+
       - name: Cache Hadoop
         uses: actions/cache@v2
         with:
@@ -64,6 +68,15 @@ jobs:
           popd
           popd
 
+      - name: setup go tools
+        env:
+          CGO_ENABLED: 0
+        run: |
+          export GOPATH=$HOME/go
+          export GOBIN=$GOPATH/bin
+          export PATH=$PATH:$GOBIN
+          /bin/bash scripts/setup.sh
+
       - name: generate the data
         run: |
           python3 run.py data -s 0.1

.gitignore

@@ -26,4 +26,6 @@ mysql/data
 nebula-bench.db
 .vscode
 output
-third/*/data
+third/*/data
+scripts/k6
+scripts/nebula-importer

README.md

@@ -29,7 +29,6 @@ sudo yum install -y git \
                     java-1.8.0-openjdk \
                     maven 
 
-
 ```
 
 ```bash
@@ -39,6 +38,17 @@ pip3 install --user -r requirements.txt
 python3 run.py --help
 ```
 
+prepare nebula tools.
+
+* [nebula-importer](https://github.com/vesoft-inc/nebula-importer)
+* [xk6-nebula](https://github.com/HarrisChu/xk6-nebula)
+
+```bash
+/bin/bash scripts/setup.sh
+```
+
+After compilation, it would put binaries in `scripts` folder.
+
 ### generate ldbc data
 
 ```bash
@@ -75,9 +85,6 @@ python3 run.py nebula importer
 
 Render the import config file according to the header files, and then run nebula-importer.
 
-Be careful, the default `nebula-import` in scripts folder is built in Linux, if you want to
-run the tool in Mac OS, please build the nebula-import by yourself.
-
 ```bash
 # after prepare the data, you could import the data to any nebula graph as you want.
 # space is mytest, graph address is 127.0.0.1:9669
@@ -96,8 +103,6 @@ python3 run.py nebula importer --dry-run
 ### nebula benchmark
 
 Use [k6](https://github.com/k6io/k6) with [xk6-nebula](https://github.com/HarrisChu/xk6-nebula) extension.
-Be careful, the default `k6` in scripts folder is built in Linux, if you want to
-run the tool in Mac OS, please download by youself. [xk6-nebula](https://github.com/HarrisChu/xk6-nebula/tags)
 
 Scenarios are in `nebula_bench/scenarios/`.
 
@@ -132,6 +137,30 @@ jq .metrics.latency output/result_Go1Step.json
 awk -F ',' 'NR>1{print $NF}' output/output_Go1Step.csv |sort|uniq -c
 ```
 
+or, just review the sumary result in stdout. e.g.
+
+```bash
+     ✓ IsSucceed
+
+     █ setup
+
+     █ teardown
+
+     checks...............: 100.00% ✓ 113778      ✗ 0
+     data_received........: 0 B     0 B/s
+     data_sent............: 0 B     0 B/s
+     iteration_duration...: min=747.84µs avg=52.76ms      med=40.77ms max=1.17s   p(90)=98.68ms p(95)=147.15ms  p(99)=263.03ms
+     iterations...........: 113778  1861.550127/s
+     latency..............: min=462      avg=49182.770298 med=37245   max=1160358 p(90)=93377   p(95)=142304.15 p(99)=258465.89
+     responseTime.........: min=662      avg=52636.793537 med=40659   max=1177651 p(90)=98556.5 p(95)=147036.15 p(99)=262869.63
+     vus..................: 100     min=0         max=100
+     vus_max..............: 100     min=100       max=100
+```
+
+As one iteration has one check, it means run `113778` queries.
+
+The unit of latency is `us`.
+
 ## and more
 
 * The file with `aaa_xxYY_bbb` format, like `comment_hasTag_tag`, should be an edge, and the edge name shoud be `XX_YY`. Keep the same format with [ldbc_snb_interactive](https://github.com/ldbc/ldbc_snb_interactive/blob/main/cypher/queries/interactive-complex-1.cypher)

README_cn.md

@@ -36,6 +36,25 @@ pip3 install --user -r requirements.txt
 python3 run.py --help
 ```
 
+准备 nebula graph 的工具。
+
+安装 golang，然后编译相关工具。
+
+* [nebula-importer](https://github.com/vesoft-inc/nebula-importer)
+* [xk6-nebula](https://github.com/HarrisChu/xk6-nebula)
+
+```bash
+/bin/bash scripts/setup.sh
+```
+
+如果 go get 的包下载不下来，需要设置 golang 代理。
+
+```bash
+export GOPROXY=https://goproxy.cn
+```
+
+编译后，二进制包在 `scripts` 文件夹中。
+
 ### 生成 LDBC 数据
 
 ```bash
@@ -69,7 +88,6 @@ python3 run.py nebula importer
 ```
 
 会根据 header 文件，自动生成 importer 的配置文件，然后运行 importer 导入。
-需要注意，默认的 `nebula-importer` 是 linux 下编译的，如果需要在 Mac OS 上使用，请自行编译一个新的 `nebula-importer`。
 
 ```bash
 # after prepare the data, you could import the data to any nebula graph as you want.
@@ -89,7 +107,6 @@ python3 run.py nebula importer --dry-run
 ### nebula benchmark
 
 使用带有 [xk6-nebula](https://github.com/HarrisChu/xk6-nebula) 插件的 [K6](https://github.com/k6io/k6) 来进行压测。
-需要注意，默认的 `k6` 是 linux 下编译的，如果需要在 Mac OS 上使用，请自行下载对应的二进制文件。[xk6-nebula](https://github.com/HarrisChu/xk6-nebula/tags)
 
 自动化的场景，在 `nebula_bench/scenarios/` 中。
 
@@ -124,6 +141,30 @@ jq .metrics.latency output/result_Go1Step.json
 awk -F ',' 'NR>1{print $NF}' output/output_Go1Step.csv |sort|uniq -c
 ```
 
+或者从标准输出看测试的结果。
+
+```bash
+     ✓ IsSucceed
+
+     █ setup
+
+     █ teardown
+
+     checks...............: 100.00% ✓ 113778      ✗ 0
+     data_received........: 0 B     0 B/s
+     data_sent............: 0 B     0 B/s
+     iteration_duration...: min=747.84µs avg=52.76ms      med=40.77ms max=1.17s   p(90)=98.68ms p(95)=147.15ms  p(99)=263.03ms
+     iterations...........: 113778  1861.550127/s
+     latency..............: min=462      avg=49182.770298 med=37245   max=1160358 p(90)=93377   p(95)=142304.15 p(99)=258465.89
+     responseTime.........: min=662      avg=52636.793537 med=40659   max=1177651 p(90)=98556.5 p(95)=147036.15 p(99)=262869.63
+     vus..................: 100     min=0         max=100
+     vus_max..............: 100     min=100       max=100
+```
+
+因为一个查询有一个检查点，所以上面代表执行了 113778 个 query，所有都成功了。
+
+latency 的单位是 `us`。
+
 如果使用 Jmeter，暂时没有自动化操作，可以通过手动调整 Jmeter 来测试，具体参考 [jmx](ldbc/jmx/go_step.jmx) 和 [java](util/LdbcGoStep/src/main/java/vesoft/LdbcGoStep.java)。
 
 ## 更多

scripts/env.sh

@@ -3,4 +3,9 @@
 LDBC_SNB_DATAGEN_VERSION=v0.3.3
 HADOOP_VERSION=3.2.1
 
-scaleFactor=${scaleFactor:-1}
+scaleFactor=${scaleFactor:-1}
+
+NEBULA_IMPORTER_VERSION=${NEBULA_IMPORTER_VERSION:-master}
+NEBULA_XK6_VERSION=${NEBULA_XK6_VERSION:-master}
+
+GOLANG_VERSION=${GOLANG_VERSION:-1.16.6}

scripts/generate-data.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+set -e
 
 # Directory of this script
 SCRIPT_DIR=$(cd "$(dirname "$0")"; pwd)
@@ -17,7 +18,7 @@ if [ -d ${HADOOP_HOME} ];then
   echo "Hadoop is existed"
 else
   cd ${DATA_DIR} && \
-  wget -c http://archive.apache.org/dist/hadoop/core/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz
+  wget -c   http://archive.apache.org/dist/hadoop/core/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz 
   echo "extract hadoop files"
   tar zxvf hadoop-${HADOOP_VERSION}.tar.gz -C ${DATA_DIR} >  /dev/null
   echo "extract hadoop files done"

scripts/setup.sh

@@ -0,0 +1,73 @@
+#!/bin/bash
+# setup dependency tools, including nebula-importer, k6.
+
+set -e
+
+# Directory of this script
+SCRIPT_DIR=$(cd "$(dirname "$0")"; pwd)
+# Directory of this project
+PROJECT_DIR=$(dirname ${SCRIPT_DIR})
+# target data
+TEMP_DIR=${PROJECT_DIR}/temp
+source ${SCRIPT_DIR}/env.sh
+
+
+
+NEBULA_IMPORTER_VERSION=${NEBULA_IMPORTER_VERSION}
+NEBULA_XK6_VERSION=${NEBULA_XK6_VERSION}
+GOLANG_VERSION=${GOLANG_VERSION}
+
+
+function setup_nebula_importer(){
+  git clone  --branch ${NEBULA_IMPORTER_VERSION} https://github.com/vesoft-inc/nebula-importer ${TEMP_DIR}/nebula-importer
+  cd ${TEMP_DIR}/nebula-importer
+  make build
+  mv nebula-importer ${PROJECT_DIR}/scripts/.
+}
+
+
+function setup_nebula_k6(){
+  git clone  --branch ${NEBULA_XK6_VERSION} https://github.com/HarrisChu/xk6-nebula ${TEMP_DIR}/xk6-nebula
+  cd ${TEMP_DIR}/xk6-nebula
+  make build
+  mv k6 ${PROJECT_DIR}/scripts/.
+}
+
+function setup_go(){
+  # setup go 
+  if [ `command -v go` ];then
+    echo 'already install golang environment.'
+    return 
+  fi
+  echo "begin install golang environment"
+
+  os=`uname | tr "[A-Z]" "[a-z]"`
+  case $(uname -m) in
+    x86_64)  arch=amd64;;
+    aarch64) arch=arm64;;
+  esac
+  wget -c https://golang.org/dl/go${GOLANG_VERSION}.${os}-${arch}.tar.gz 
+
+  if [ $? != 0 ] ; then
+    wget -c https://golang.google.cn/dl/go${GOLANG_VERSION}.${os}-${arch}.tar.gz
+    if [ $? !=0 ] ; then
+      echo "cannot download golang installation package, please install manually."
+      exit 1
+    fi
+  fi
+  tar zxvf go${GOLANG_VERSION}.${os}-${arch}.tar.gz -C ${PROJECT_DIR}  > /dev/null
+  export GOPATH=${PROJECT_DIR}/gopath
+  export PATH=$PATH:${PROJECT_DIR}/go/bin:${GOPATH}/bin
+}
+
+function main(){
+  rm -rf ${TEMP_DIR}
+  setup_go
+  setup_nebula_importer
+  setup_nebula_k6
+  rm -rf ${TEMP_DIR}
+  echo "Finish"
+
+}
+
+main

scripts/split-data.sh

@@ -1,4 +1,5 @@
 #!/bin/bash
+set -e
 
 # Directory of this script
 SCRIPT_DIR=$(cd "$(dirname "$0")"; pwd)

templates/k6_config.js.j2

@@ -31,7 +31,7 @@ export function setup() {
 export default function (data) {
 
   // get csv data from csv file
-  let d = pool.getData()
+  let d = session.getData()
   // d[0] means the first column data in the csv file
   let ngql = "{{ nGQL }}".format({{ param }})
   let response = session.execute(ngql)

templates/nebula-import-vid-int.yaml.j2

@@ -12,7 +12,6 @@ clientSettings:
     address: {{ address }}
   postStart:
     commands: |
-      UPDATE CONFIGS heartbeat_interval_secs=1;
       CREATE SPACE IF NOT EXISTS {{ space }}(PARTITION_NUM = 24, REPLICA_FACTOR = 3, vid_type = int64);
       USE {{ space }};
       {% for vertex in vertex_set -%}

templates/nebula-import-vid-string.yaml.j2

@@ -12,7 +12,6 @@ clientSettings:
     address: {{ address }}
   postStart:
     commands: |
-      UPDATE CONFIGS heartbeat_interval_secs=1;
       CREATE SPACE IF NOT EXISTS {{ space }}(PARTITION_NUM = 24, REPLICA_FACTOR = 3, vid_type = fixed_string(20));
       USE {{ space }};
       {% for vertex in vertex_set -%}