Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Launch engines with each components inside their own container. (#2247)
Changes on interface:
1. Images identifier are controlled by registry and tag, the image name is fixed by convention.
2. Users now could control a subset of components to be launched. by
`with_analytical`, `with_analytical_java`, `with_interactive` and `with_learning`.
Additionally, user could use `with_dataset` to launched a dataset container.
```python
sess = graphscope.session(
k8s_image_registry="registry.cn-hongkong.aliyuncs.com", # the default value. Could be "" to use local images
k8s_image_tag="0.19.0", # default __version__
num_workers=2,
with_analytical=True, # default True
with_analytical_java=False, # default False
with_interactive=True, # default True
with_learning=True, # default True,
k8s_vineyard_image="ghcr.io/v6d-io/v6d/vineyardd:v0.11.1", # the default value.
)
# the session will have a coordinator pod, 2 engine pod, which have 4 containers.
coordinator is a deployment with 1 replicas
engine is a statefulset with have 2 replicas.
etcd is started by vineyard container, which is a small and handy image.
```
Major changes are:
1. Refactored CI pipelines
1. Dedicate a k8s CI workflow by building images
2. Dedicate a local CI workflow by installing built graphscope wheel.
3. Amend dummy ci workflows
4. Do not upload GIE log when succeeded.
2. Increased coordinator launch speed by reduce the initial delay seconds, the delay is not necessary as coordinator started very fast.
3. Rename `mount_dataset` to `with_dataset` and mount to a fixed path to align with other `with_X` jargons.
4. Updated GraphScope Helm charts accordingly
5. GraphScope cluster use statefulsets instead of deployments
6. Use standard vineyard image
7. Get rid of etcd pods and rely on vineyard image to start etcd
8. Frontend now has its own pod.
9. Vineyardd is launched in the container args.
10. Disabled Mars
11. Use classes defined in Kubernetes client instead of self-maintained JSON to build resources.
12. Move compiling process to engine pod
13. Refactored issue commands snippets, gives error logs when failed.
14. Optimize many inefficient logging statements, and use f-string over str.format
15. Add `.git` to .dockerignore and commented out the `*.pb.*` pattern.
16. Handle exceptions when delete dangling coordinator
17. Remove cmake from requirements.txt of coordinator, as installed cmake is broken.
18. Upload a dummy dev image `graphscope-dev:ci` for CI process
19. Install rapidjson and msgpack to analytical container
20. Install Hadoop and get rid of its huge docs directory to vineyard-dev.
21. Enable launch without analyticlal engine.- Loading branch information
1 parent
12a3654
commit c8eaf7a
Showing
65 changed files
with
3,426 additions
and
3,432 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| name: Build GraphScope Images on Linux | ||
|
|
||
| # on: [push, pull_request] | ||
| on: | ||
| workflow_dispatch: | ||
| schedule: | ||
| # The notifications for scheduled workflows are sent to the user who | ||
| # last modified the cron syntax in the workflow file. | ||
| # Trigger the workflow at 03:00(CST) every day. | ||
| - cron: '00 19 * * *' | ||
| push: | ||
| tags: | ||
| - "v*" | ||
|
|
||
| env: | ||
| REGISTRY: registry.cn-hongkong.aliyuncs.com | ||
|
|
||
| jobs: | ||
| build-image: | ||
| if: (github.ref == 'refs/heads/main' && github.repository == 'alibaba/GraphScope') || (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && github.repository == 'alibaba/GraphScope') | ||
| runs-on: ubuntu-20.04 | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| with: | ||
| submodules: true | ||
|
|
||
| - name: Add envs to GITHUB_ENV | ||
| run: | | ||
| short_sha=$(git rev-parse --short HEAD) | ||
| echo "SHORT_SHA=${short_sha}" >> $GITHUB_ENV | ||
| - name: Build GraphScope Image | ||
| run: | | ||
| cd ${GITHUB_WORKSPACE}/k8s | ||
| make coordinator CI=false | ||
| make analytical CI=false | ||
| make analytical-java CI=false | ||
| make interactive-frontend CI=false | ||
| make interactive-executor CI=false | ||
| make learning CI=false | ||
| # make jupyter-image | ||
| # cd ${GITHUB_WORKSPACE} | ||
| # docker build --build-arg CI=${CI} -t graphscope/jupyter:${SHORT_SHA} -f ./k8s/internal/jupyter.Dockerfile . | ||
| # dataset image doesn't changed, we can just use the latest one | ||
| # make dataset-image | ||
| - name: Release Nightly Image | ||
| if: ${{ github.ref == 'refs/heads/main' && github.repository == 'alibaba/GraphScope' }} | ||
| env: | ||
| docker_password: ${{ secrets.DOCKER_PASSWORD }} | ||
| docker_username: ${{ secrets.DOCKER_USER }} | ||
| run: | | ||
| echo "${docker_password}" | sudo docker login --username="${docker_username}" ${{ env.REGISTRY }} --password-stdin | ||
| # docker tag: 0.15.0 -> 0.15.0a20220808 | ||
| time=$(date "+%Y%m%d") | ||
| version=$(cat ${GITHUB_WORKSPACE}/VERSION) | ||
| tag="${version}a${time}" | ||
| # graphscope image | ||
| sudo docker tag graphscope/coordinator:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/coordinator:${tag} | ||
| sudo docker tag graphscope/analytical:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/analytical:${tag} | ||
| sudo docker tag graphscope/analytical-java:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/analytical-java:${tag} | ||
| sudo docker tag graphscope/interactive-frontend:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/interactive-frontend:${tag} | ||
| sudo docker tag graphscope/interactive-executor:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/interactive-executor:${tag} | ||
| sudo docker tag graphscope/learning:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/learning:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/coordinator:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/analytical:${tag} | ||
| # sudo docker push ${{ env.REGISTRY }}/graphscope/analytical-java:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/interactive-frontend:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/interactive-executor:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/learning:${tag} | ||
| # jupyter image | ||
| # sudo docker tag graphscope/jupyter:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/jupyter:${tag} | ||
| # sudo docker push ${{ env.REGISTRY }}/graphscope/jupyter:${tag} | ||
| # dataset image | ||
| # sudo docker tag graphscope/dataset:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/dataset:${tag} | ||
| # sudo docker push ${{ env.REGISTRY }}/graphscope/dataset:${tag} | ||
| - name: Extract Tag Name | ||
| if: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && github.repository == 'alibaba/GraphScope' }} | ||
| id: tag | ||
| run: echo "TAG=${GITHUB_REF#refs/tags/v}" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: Release Image | ||
| if: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && github.repository == 'alibaba/GraphScope' }} | ||
| env: | ||
| docker_password: ${{ secrets.DOCKER_PASSWORD }} | ||
| docker_username: ${{ secrets.DOCKER_USER }} | ||
| run: | | ||
| echo "${docker_password}" | sudo docker login --username="${docker_username}" ${{ env.REGISTRY }} --password-stdin | ||
| # Release version tag | ||
| tag=${{ steps.tag.outputs.TAG }} | ||
| # graphscope image | ||
| sudo docker tag graphscope/coordinator:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/coordinator:${tag} | ||
| sudo docker tag graphscope/analytical:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/analytical:${tag} | ||
| sudo docker tag graphscope/analytical-java:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/analytical-java:${tag} | ||
| sudo docker tag graphscope/interactive-frontend:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/interactive-frontend:${tag} | ||
| sudo docker tag graphscope/interactive-executor:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/interactive-executor:${tag} | ||
| sudo docker tag graphscope/learning:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/learning:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/coordinator:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/analytical:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/analytical-java:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/interactive-frontend:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/interactive-executor:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/learning:${tag} | ||
| # jupyter image | ||
| # sudo docker tag graphscope/jupyter:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/jupyter:${tag} | ||
| # sudo docker tag graphscope/jupyter:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/jupyter:latest | ||
| # sudo docker push ${{ env.REGISTRY }}/graphscope/jupyter:${tag} | ||
| # dataset image | ||
| # Note! dataset image are built mannually just use the latest one. | ||
| sudo docker pull ${{ env.REGISTRY }}/graphscope/dataset:latest | ||
| sudo docker tag ${{ env.REGISTRY }}/graphscope/dataset:latest ${{ env.REGISTRY }}/graphscope/dataset:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/dataset:${tag} | ||
| # Release the latest tag | ||
| tag=latest | ||
| # graphscope image | ||
| sudo docker tag graphscope/coordinator:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/coordinator:${tag} | ||
| sudo docker tag graphscope/analytical:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/analytical:${tag} | ||
| sudo docker tag graphscope/analytical-java:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/analytical-java:${tag} | ||
| sudo docker tag graphscope/interactive-frontend:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/interactive-frontend:${tag} | ||
| sudo docker tag graphscope/interactive-executor:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/interactive-executor:${tag} | ||
| sudo docker tag graphscope/learning:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/learning:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/coordinator:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/analytical:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/analytical-java:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/interactive-frontend:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/interactive-executor:${tag} | ||
| sudo docker push ${{ env.REGISTRY }}/graphscope/learning:${tag} | ||
| # jupyter image | ||
| # sudo docker tag graphscope/jupyter:${SHORT_SHA} ${{ env.REGISTRY }}/graphscope/jupyter:${tag} | ||
| # sudo docker push ${{ env.REGISTRY }}/graphscope/jupyter:${tag} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| name: GraphScope CI on Kubernetes (Dummy) | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - '**' | ||
| - '!.github/workflows/ci.yml' | ||
| - '!Makefile' | ||
| - '!analytical_engine/**' | ||
| - '!charts/**' | ||
| - '!coordinator/**' | ||
| - '!interactive_engine/**' | ||
| - '!k8s/**' | ||
| - '!learning_engine/**' | ||
| - '!proto/**' | ||
| - '!python/**' | ||
| - '**.md' | ||
| - '**.rst' | ||
|
|
||
| concurrency: | ||
| group: ${{ github.repository }}-${{ github.event.number || github.head_ref || github.sha }}-${{ github.workflow }} | ||
| cancel-in-progress: true | ||
|
|
||
| env: | ||
| GS_IMAGE: registry.cn-hongkong.aliyuncs.com/graphscope/graphscope | ||
|
|
||
| jobs: | ||
| # JOB to run change detection | ||
| changes: | ||
| runs-on: ubuntu-20.04 | ||
| # Set job outputs to values from filter step | ||
| outputs: | ||
| gae-python: ${{ steps.filter.outputs.gae-python }} | ||
| networkx: ${{ steps.filter.outputs.networkx }} | ||
| gie-function-test: ${{ steps.filter.outputs.gie-function-test }} | ||
| steps: | ||
| # For push it's necessary to checkout the code | ||
| - uses: actions/checkout@v3 | ||
| # For pull requests it's not necessary to checkout the code | ||
| - uses: dorny/paths-filter@v2 | ||
| id: filter | ||
| with: | ||
| base: main # Change detection against merge-base with main before push | ||
| filters: | | ||
| gae-python: | ||
| - 'proto/**' | ||
| - 'analytical_engine/**' | ||
| - 'python/graphscope/analytical/**' | ||
| - 'python/graphscope/client/**' | ||
| - 'python/graphscope/dataset/**' | ||
| - 'python/graphscope/deploy/**' | ||
| - 'python/graphscope/framework/**' | ||
| - 'python/graphscope/tests/unittest/**' | ||
| - 'coordinator/gscoordinator/**' | ||
| - '.github/workflows/ci.yml' | ||
| networkx: | ||
| - 'analytical_engine/apps/**' | ||
| - 'analytical_engine/frame/**' | ||
| - 'analytical_engine/core/**' | ||
| - 'python/graphscope/nx/**' | ||
| gie-function-test: | ||
| - 'interactive_engine/**' | ||
| - 'python/graphscope/interactive/**' | ||
| - '.github/workflows/ci.yml' | ||
| build-analytical: | ||
| runs-on: ubuntu-20.04 | ||
| needs: [build-wheels, changes] | ||
| if: ${{ github.repository == 'alibaba/GraphScope' }} | ||
| steps: | ||
| - run: 'echo "No action required" ' | ||
|
|
||
| build-analytical-java: | ||
| runs-on: ubuntu-20.04 | ||
| needs: [build-wheels, changes] | ||
| if: ${{ github.repository == 'alibaba/GraphScope' }} | ||
| steps: | ||
| - run: 'echo "No action required" ' | ||
|
|
||
| build-interactive: | ||
| runs-on: ubuntu-20.04 | ||
| needs: [build-wheels, changes] | ||
| if: ${{ github.repository == 'alibaba/GraphScope' }} | ||
| steps: | ||
| - run: 'echo "No action required" ' | ||
|
|
||
| build-learning: | ||
| runs-on: ubuntu-20.04 | ||
| needs: [build-wheels, changes] | ||
| if: ${{ github.repository == 'alibaba/GraphScope' }} | ||
| steps: | ||
| - run: 'echo "No action required" ' | ||
|
|
||
| # build-coordinator: | ||
| # runs-on: ubuntu-20.04 | ||
| # needs: [build-wheels, changes] | ||
| # if: ${{ github.repository == 'alibaba/GraphScope' }} | ||
| # steps: | ||
| # - run: 'echo "No action required" ' | ||
|
|
||
| k8s-test: | ||
| runs-on: ubuntu-20.04 | ||
| if: ${{ github.repository == 'alibaba/GraphScope' }} | ||
| needs: [build-analytical, build-analytical-java, build-interactive, build-learning] | ||
| steps: | ||
| - run: 'echo "No action required" ' | ||
|
|
||
| gie-test: | ||
| runs-on: ubuntu-20.04 | ||
| needs: [build-analytical, build-analytical-java, build-interactive, build-learning] | ||
| if: ${{ (needs.changes.outputs.gie-function-test == 'false' || github.ref == 'refs/heads/main') && github.repository == 'alibaba/GraphScope' }} | ||
| steps: | ||
| - run: 'echo "No action required" ' |
Oops, something went wrong.