Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug(analytical): context.output raise error when result is huge #3578

Closed
acezen opened this issue Feb 27, 2024 · 4 comments
Closed

bug(analytical): context.output raise error when result is huge #3578

acezen opened this issue Feb 27, 2024 · 4 comments
Assignees
Labels

Comments

@acezen
Copy link
Collaborator

acezen commented Feb 27, 2024

Describe the bug
User run flash.harmonic_centrality algorithm on a large string id dataset (1.2 billion vertices, 3.2 billion edges), and dumps the result with contest.output. Got error:
915736CC-9809-4A26-93C6-B0F7051D5740

To Reproduce

#!/usr/bin/python3

import os
import graphscope
from graphscope.client.session import get_default_session
from graphscope.framework.loader import Loader

graphscope.set_option(show_log=True)


def load_graph(sess, path):
    graph = sess.g(oid_type="string", directed=True, generate_eid=False, retain_oid=False)
    edge = Loader(path, header_row=False, delimiter=",", directed=True)
    graph = graph.add_edges(edge, label='link', src_label="domain", dst_label="domain") 
    return graph

def hamonic_centrality(graph):
    context = graphscope.flash.harmonic_centrality(graph)
    return context

if __name__ == "__main__":
    sess = graphscope.session(cluster_type="hosts", num_workers=1)
    g = load_graph(sess, "/xxx//domain_graph.txt")
    ctx = hamonic_centrality(g)
    ctx.output("/xxxx/harmonic_result.txt", {"id": "v.id", "centrality": "r"})

Environment (please complete the following information):

  • GraphScope version: v0.26.0 through pip install graphscope
  • OS: Linux
  • Version :Ubuntu 20.04
@sighingnow sighingnow self-assigned this Feb 28, 2024
@sighingnow
Copy link
Collaborator

Fixed in upstream: apache/arrow#40271

@yecol
Copy link
Collaborator

yecol commented Feb 29, 2024

@acezen Please verify the fix and close this issue if possible.

@github-actions github-actions bot added the stale label Mar 12, 2024
Copy link
Contributor

/cc @sighingnow, this issus/pr has had no activity for for a long time, could you folks help to review the status ?
To suppress further notifications,

  • for issues,
    • if it is waiting for further response from the reporter/author, please help to add the label requires-further-info,
    • if you have already started working on it, please add the label work-in-progress to the issue,
    • if this issue requires further designing discussion and not in current plan, or won't be fixed, please add the label requires-further-discussion or wontfix to the issue,
  • for pull requests,
    • if you are still working on it and it is not ready for reviewing, please convert this pull request as draft PR,
    • if you have decided to hold this development on, please add the requires-further-discussion label to the pull request.
      Thanks!

@sighingnow
Copy link
Collaborator

Closing as fixed in upstream: apache/arrow#40271

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants