Dogfooding by taoqian #2656

TaoLbr1993 · 2023-04-25T09:50:28Z

TaoLbr1993
Apr 25, 2023
Collaborator

系统环境：

uname -a
Linux iZj6cehjts12r3b8nzhtqvZ 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

python3 --version
Python 3.10.6

pip3 --version
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)

java -version
openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment (build 17.0.7+7-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 17.0.7+7-Ubuntu-0ubuntu120.04, mixed mode, sharing)

Task 1: 在本地试用 GraphScope

安装 GraphScope
载入 ogbn_mag 内置图
Project ogbn_mag到一个简单图，选择点标签为 paper, 边标签为 cites, 属性都为空。
在简单图上运行一个 Kcore 算法, 并打印结果为 pandas.DataFrame
在 ogbn_mag图上启动交互引擎（Interactive）
通过 gremlin 计算图有多少点，边
对于 ogbn_mag图，使用 learning 定义并执行一个 GCN 训练过程，可参考 CI 或文档中定义的过程
退出 GraphScope。

Task 2: 在本地开发调试 GraphScope

Task 3: 运行本地测试

找到 CI 定义的位置和文件
找到单元测试的位置，参考在 CI 中是如何运行这些单元测试的步骤，并在本地运行CI上的本地测试（如果没有，可开 Issue 加入运行）
GAE 的单元测试
GIE 的单元测试
GLE 的单元测试
运行一组 Python 中的单元测试 test_min.py 。（Python 中的单元测试更像是集成测试，因为往往由多个引擎参与）
在 python/graphscope/tests/minitest下新增一个test 文件，在里面添加一个涉及多个引擎的测试，其中要包括载入一张 ogbn_mag 图，打印 Kcore 算法结果，通过 Gremlin 查询图的点边数量，执行 GCN 训练。

Task 4: 在 Kubernetes 上使用 GraphScope （Python SDK）

通过客户端在 K8S 环境下安装 Graphscope，建立一个 Session，通过 kubectl 查看拉起来的 Pod 和 Service （可通过 minikube/kind 在单机搭建 K8S 环境）
通过载图 API 载一个图 ogbn_mag
在图上运行一个 LPA 算法，并打印结果为 pandas.DataFrame
在图上启动交互引擎（Interactive）
通过 gremlin 计算图有多少点，边
使用 learning 定义并执行一个 GCN 训练过程
关闭 Session

Task 5: 在 Kubernetes 上使用 GraphScope （Helm）

通过 Helm 安装 GraphScope，并载图，运行算法，Gremlin 和 Learning

Task 6: 独立部署

独立部署 GAE，并载图 modern_graph，运行 LPA 算法
独立部署 GIE，发送 gremlin 查询点边数量

Task 7: 打包镜像与了解 Kubernetes 上运行的基本概念

构建 GAE 的镜像
构建 GIE 的镜像（包括 Frontend 和 Executor）
构建 GLE 的镜像
构建 Coordinator 的镜像
为 GAE 添加一条日志 ( analytical_engine/core/server/analytical_server.cc) 并重新构建镜像，在不使用 docker push的情况下，使用新的镜像启动并在日志中可以找到新添加的日志（使用 registry 和 tag 来选择使用哪个镜像）
为 GIE 的 Frontend (interactive_engine/frontend/src/main/java/com/alibaba/graphscope/frontend/Frontend.java) 和 executor (interactive_engine/executor/assembly/v6d/src/bin/gaia_executor.rs) 各添加一条日志并重新构建 Frontend 和 Executor 镜像，在不使用 docker push的情况下，使用新的镜像启动并可以找到新添加的日志
为 Coordinator 添加一个异常 (coordinator/gscoordinator/coordinator.py)，并重新构建 Coordinator 镜像。启动并找到异常的报错。（日志在控制台中或者在 Coordinator Pod 的日志中）
清除所有 GraphScope 的资源，包括镜像，容器。

Task 8: 运行 K8S 上的测试

找到在 Kubernetes 上运行的测试，找到在 CI 中是如何运行这些单元测试的
参考 CI 的步骤，运行 test_demo_scripts.py 中的 test_demo_distribute 测试
尝试添加一个在 Kubernetes 上运行的测试，并运行。其中要包括载入一张 ogbn_mag 图，打印 Kcore 算法结果，通过 Gremlin 查询图的点边数量，执行 GCN 训练。

TaoLbr1993 · 2023-04-26T08:52:20Z

TaoLbr1993
Apr 26, 2023
Collaborator Author

Got error when creating interactive engine

>>> interactive = graphscope.gremlin(g)
2023-04-26 16:49:30,895 [ERROR][local_launcher:182]: Cannot get version of java
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gscoordinator/local_launcher.py", line 179, in create_interactive_instance
    java_version = get_java_version()
  File "/usr/local/lib/python3.10/dist-packages/graphscope/framework/utils.py", line 201, in get_java_version
    java_exec = find_java()
  File "/usr/local/lib/python3.10/dist-packages/graphscope/framework/utils.py", line 196, in find_java
    raise RuntimeError("java command not found.")
RuntimeError: java command not found.

1 reply

TaoLbr1993 Apr 26, 2023
Collaborator Author

Fixed after installing Java JDK.

apt install openjdk-17-jre-headless

TaoLbr1993 · 2023-04-26T11:25:10Z

TaoLbr1993
Apr 26, 2023
Collaborator Author

Cannot import graphscope after installing tensorflow.

package version:
Tensorflow: 2.12.0; protobuf: 4.22.3

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/graphscope/__init__.py", line 19, in <module>
    from graphscope.analytical.app import *
  File "/usr/local/lib/python3.10/dist-packages/graphscope/analytical/app/__init__.py", line 19, in <module>
    from graphscope.analytical.app import flash
  File "/usr/local/lib/python3.10/dist-packages/graphscope/analytical/app/flash/__init__.py", line 19, in <module>
    from graphscope.analytical.app.flash.centrality import betweenness_centrality
  File "/usr/local/lib/python3.10/dist-packages/graphscope/analytical/app/flash/centrality.py", line 20, in <module>
    from graphscope.framework.app import AppAssets
  File "/usr/local/lib/python3.10/dist-packages/graphscope/framework/app.py", line 30, in <module>
    from graphscope.framework.context import create_context_node
  File "/usr/local/lib/python3.10/dist-packages/graphscope/framework/context.py", line 28, in <module>
    from graphscope.framework import dag_utils
  File "/usr/local/lib/python3.10/dist-packages/graphscope/framework/dag_utils.py", line 22, in <module>
    from graphscope.framework import utils
  File "/usr/local/lib/python3.10/dist-packages/graphscope/framework/utils.py", line 39, in <module>
    from graphscope.framework.errors import check_argument
  File "/usr/local/lib/python3.10/dist-packages/graphscope/framework/errors.py", line 21, in <module>
    from graphscope.proto import error_codes_pb2
  File "/usr/local/lib/python3.10/dist-packages/graphscope/proto/error_codes_pb2.py", line 33, in <module>
    _descriptor.EnumValueDescriptor(
  File "/usr/local/lib/python3.10/dist-packages/google/protobuf/descriptor.py", line 796, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

3 replies

vegetableysm Apr 27, 2023
Collaborator

Trying a lower version of tensorflow (e.g. 2.8.0)might solve this problem. The latest version of tensorflow has some dependency packages that conflict with graphscope.

TaoLbr1993 Apr 27, 2023
Collaborator Author

Downgrading tensorflow==2.11 solves the conflict.

sighingnow Apr 28, 2023
Maintainer

Fixed by #2650

TaoLbr1993 · 2023-05-19T09:35:04Z

TaoLbr1993
May 19, 2023
Collaborator Author

This issue happens when following local deployment in the documents (https://graphscope.io/docs/deployment.html#deployment-on-local). So I guess this is because the document has not been updated as the newest one.

Argument --dev can not be recognized when deploy Graphscope locally.
--dev is the argument for local deployment. However, run command ./scripts/install_deps.sh --dev and obtain unrecognized option '--dev'.
It seems the script does not implement the code for --dev.

while test $# -ne 0; do
  arg=$1; shift
  case ${arg} in
    -h|--help)        usage; exit ;;
    --verbose)        VERBOSE=true; readonly VERBOSE; ;;
    --cn)             CN_MIRROR=true; readonly CN_MIRROR; ;;
    --k8s) install_deps_k8s "$@"; exit;;
    *)
      echo "unrecognized option '${arg}'"
      usage; exit 1;;
  esac
done

0 replies

TaoLbr1993 · 2023-05-22T07:05:52Z

TaoLbr1993
May 22, 2023
Collaborator Author

Cannot compile from source code even MPI is installed.

Could NOT find MPI_C (missing: MPI_open-rte_LIBRARY MPI_open-pal_LIBRARY MPI_hwloc_LIBRARY MPI_event_LIBRARY MPI_event_pthreads_LIBRARY MPI_C_WORKS)
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find MPI (missing: MPI_C_FOUND) (found version "3.1")

1 reply

TaoLbr1993 May 24, 2023
Collaborator Author

Solved. Not deleting build files after installing necessary packages causes the problem.

TaoLbr1993 · 2023-05-25T02:03:12Z

TaoLbr1993
May 25, 2023
Collaborator Author

Another issue due to using the mirror of apt from Tsinghua: Cannot find the package libre2-9.
Problem is solved after adding the source or using the mirror from aliyun.

deb http://cz.archive.ubuntu.com/ubuntu jammy main

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dogfooding by taoqian #2656

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 5 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Dogfooding by taoqian #2656

TaoLbr1993 Apr 25, 2023 Collaborator

Task 1: 在本地试用 GraphScope

Task 2: 在本地开发调试 GraphScope

Task 3: 运行本地测试

Task 4: 在 Kubernetes 上使用 GraphScope （Python SDK）

Task 5: 在 Kubernetes 上使用 GraphScope （Helm）

Task 6: 独立部署

Task 7: 打包镜像与了解 Kubernetes 上运行的基本概念

Task 8: 运行 K8S 上的测试

Replies: 5 comments · 5 replies

TaoLbr1993 Apr 26, 2023 Collaborator Author

TaoLbr1993 Apr 26, 2023 Collaborator Author

TaoLbr1993 Apr 26, 2023 Collaborator Author

vegetableysm Apr 27, 2023 Collaborator

TaoLbr1993 Apr 27, 2023 Collaborator Author

sighingnow Apr 28, 2023 Maintainer

TaoLbr1993 May 19, 2023 Collaborator Author

TaoLbr1993 May 22, 2023 Collaborator Author

TaoLbr1993 May 24, 2023 Collaborator Author

TaoLbr1993 May 25, 2023 Collaborator Author

TaoLbr1993
Apr 25, 2023
Collaborator

Replies: 5 comments 5 replies

TaoLbr1993
Apr 26, 2023
Collaborator Author

TaoLbr1993 Apr 26, 2023
Collaborator Author

TaoLbr1993
Apr 26, 2023
Collaborator Author

vegetableysm Apr 27, 2023
Collaborator

TaoLbr1993 Apr 27, 2023
Collaborator Author

sighingnow Apr 28, 2023
Maintainer

TaoLbr1993
May 19, 2023
Collaborator Author

TaoLbr1993
May 22, 2023
Collaborator Author

TaoLbr1993 May 24, 2023
Collaborator Author

TaoLbr1993
May 25, 2023
Collaborator Author