Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Add estimate memory usage command for gsctl #3816

Merged
merged 4 commits into from
May 20, 2024

Conversation

siyuan0322
Copy link
Collaborator

@siyuan0322 siyuan0322 commented May 20, 2024

Example usage:

❯ PYTHONPATH=`pwd` python3 graphscope/gsctl/gsctl.py estimate -v 222111111 -e 1610000000 -vf 2 -ef 28 -g gae -p 4

Disclaimer: The `estimate` command serves as a estimator for various kind of GraphScope resources.
    - GAE memory usage estimator:
        It would do a rough estimation of the total memory usage of the GAE.
        The actual memory usage may vary vastly due to the complexity of the graph algorithm and the data distribution.

        Here's some assumption when estimating the memory usage:
            1. Assuming the graph is a simple graph, e.g. only has 1 label and at most 1 property for each label.
            2. Assuming the graph algorithms has a fixed amount of memory consumption, e.g. PageRank, SSSP, CC;
               It should not generate a huge amount of intermediate result, like K-Hop, Louvain.
            3. Assuming the vertex map is global vertex map

        User should take this estimation as a reference and a start point for tuning, and adjust the memory allocation according to the actual situation.

[SUCCESS] The estimated memory usage is 30.61 GB per pod for 4 pods.

python/graphscope/gsctl/commands/common.py Outdated Show resolved Hide resolved
python/graphscope/gsctl/commands/common.py Outdated Show resolved Hide resolved
python/graphscope/gsctl/commands/common.py Outdated Show resolved Hide resolved
python/graphscope/gsctl/commands/common.py Outdated Show resolved Hide resolved
python/graphscope/gsctl/commands/common.py Outdated Show resolved Hide resolved
@siyuan0322 siyuan0322 requested a review from yecol May 20, 2024 09:46
yecol
yecol previously approved these changes May 20, 2024
@siyuan0322 siyuan0322 merged commit bf4bd71 into alibaba:main May 20, 2024
40 of 42 checks passed
@siyuan0322 siyuan0322 deleted the zsy/gsctl branch May 20, 2024 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants