Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Progress report for computation-heavy steps #756

Closed
sighingnow opened this issue Sep 3, 2021 · 1 comment
Closed

Progress report for computation-heavy steps #756

sighingnow opened this issue Sep 3, 2021 · 1 comment

Comments

@sighingnow
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

It would be great if we could have a progress report for computation-heavy steps, e.g., loading super large graphs, run analytical apps, etc. We could integration things like tqdm, if possible.

It could deliver better user experiences. e.g., loading a graph with GBs files may taking miniutes to run, we could let users know what we are doing and how far to finish.

It could be not very precise, and requires efforts from both vineyard and graphscope.

Additional context
Add any other context or screenshots about the feature request here.

@yecol
Copy link
Collaborator

yecol commented Sep 14, 2021

the most time-consuming phases are

  • loading graph;
  • launching pods;

Let's add some stubs on these 2 phases by print one-line logging in the format of:

progress-[TAG]-[SUB_PHASE]-[COUNT]

# example
progress-LG-readingVertexLines-100000
progress-LP-launchingCdntorSeconds-20

then extract/process this info in Python with tqmd

To estimate the rest time, we can empirically assume some key-points,
e.g., for loading graphs:

  • achieves 30% when reads the vertex file
  • achieves 80% when reads the edge file
  • ...(please give a proposal @siyuan0322 )

for lauching pods, (give a proposal @lidongze0629 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants