Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reorganized coordinator and redesigned the logic to create engines. #2152

Merged
merged 19 commits into from Oct 25, 2022

Conversation

siyuan0322
Copy link
Collaborator

@siyuan0322 siyuan0322 commented Oct 20, 2022

Coordinator:

  1. Split coordinator to launcher and op_executor
  2. Revised the logic for connecting session
  3. Create and close engines now use their own RPC call instead of Operation in DAG
  4. Simplify and refine error handling
  5. Organized local launcher and kubernetes launcher
  6. Move the create gl handle parts to client, in order to keep coordinator simple

Interactive engine:

  1. Separate groot and v6d rust package compilation as they are non-related
  2. Thanks to the separation of compilation and the dependency reduction, the image size shrinks from 7GB to 0.9GB
  3. Clean more target in make clean

Client:

  1. Refactored session and RPC client according to changes in coordinator
  2. Removed some unused parameters in functions
  3. Creating GIE and GLE now are not parts of DAG, hence their DAGNode are removed
  4. Remove some unused parameters

Dev infra:

  1. Use NUMPROC to control compilation parallelism instead of hard coded value
  2. Eliminate usage of sudo inside Makefile
  3. Make now only do make in local, and make install will copy it to destination directory
  4. Make GAE, GIE, and GLE now will use cached artifact instead of build from beginning
  5. Add a bunch of dockerfiles for single modules (GAE, GIE, GLE, Coordinator).

Miscellaneous:

  1. Fix several typos

Fixes #2020
Fixes #2019
Fixes #2155
Fixes #2156

@codecov-commenter
Copy link

codecov-commenter commented Oct 20, 2022

Codecov Report

Merging #2152 (8f057cf) into main (ea4ca1d) will increase coverage by 32.84%.
The diff coverage is 68.12%.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##             main    #2152       +/-   ##
===========================================
+ Coverage   40.08%   72.92%   +32.84%     
===========================================
  Files          87       89        +2     
  Lines        9912     9789      -123     
===========================================
+ Hits         3973     7139     +3166     
+ Misses       5939     2650     -3289     
Impacted Files Coverage Δ
...n/graphscope/deploy/kubernetes/resource_builder.py 56.84% <ø> (+1.62%) ⬆️
python/graphscope/framework/graph.py 86.58% <0.00%> (+16.84%) ⬆️
python/graphscope/tests/conftest.py 80.95% <ø> (+32.13%) ⬆️
python/graphscope/tests/minitest/test_min.py 0.00% <ø> (ø)
python/graphscope/tests/unittest/test_graph.py 85.46% <ø> (+85.46%) ⬆️
...thon/graphscope/tests/unittest/test_scalability.py 96.00% <ø> (+96.00%) ⬆️
python/graphscope/tests/unittest/test_session.py 63.82% <ø> (+63.82%) ⬆️
python/graphscope/deploy/kubernetes/cluster.py 74.71% <14.28%> (+1.11%) ⬆️
python/graphscope/deploy/kubernetes/utils.py 70.47% <25.00%> (+0.95%) ⬆️
python/graphscope/client/utils.py 85.45% <50.00%> (+3.63%) ⬆️
... and 58 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea4ca1d...8f057cf. Read the comment docs.

Makefile Outdated Show resolved Hide resolved
@siyuan0322 siyuan0322 merged commit 695cb9c into alibaba:main Oct 25, 2022
@siyuan0322 siyuan0322 deleted the zsy/layer branch October 25, 2022 05:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants