Skip to content

Conversation

@Fengzdadi
Copy link
Contributor

What type of PR is this?

  • bug
  • feature
  • enhancement

What problem(s) does this PR solve?

Issue(s) number:

N/A

Description:

  • Added two new asynchronous scan APIs for Python clients: scan_vertex_async() and scan_edge_async()
  • These methods enable multi‑partition concurrency and streaming batch retrieval of vertices and edges in an asyncio‑friendly way, filling the gap where only blocking, single‑threaded scans existed.

How do you solve it?

  • Use concurrent.futures.ThreadPoolExecutor to submit one scan_*_with_part job per partition (via meta_cache.get_part_leaders), dispatching them in parallel.
  • As each future completes, iterate its ScanResult in a while scan_result.has_next(): yield part, batch loop to stream (partition_id, batch) pairs.

Special notes for your reviewer, ex. impact of this fix, design document, etc:

  • Verified on NebulaGraph 3.8 with manual testing.

Copy link

@Malone-AI Malone-AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the scan_vertex_async function,

scan_result = future.result()  

may throw an exception but is not handled.

In the scan_edge_async function,

while scan_result and scan_result.has_next():  
    batch = scan_result.next()  

If batch = scan_result.next() throws an exception, the scan_result may not be closed, leading to improper resource cleanup.

The comment may be incorrect:
"""
:type prop_names: if given empty, return all property # Should be :param
"""

Perhaps it should be:
"""
:param prop_names: if given empty, return all property
"""

@wey-gu wey-gu requested a review from Copilot July 29, 2025 15:07
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds asynchronous scanning capabilities to the GraphStorageClient by introducing two new methods: scan_vertex_async() and scan_edge_async(). These methods enable concurrent, multi-partition scanning of vertices and edges using thread pools, addressing the limitation of only having blocking, single-threaded scan operations.

  • Added concurrent.futures import for thread pool execution
  • Implemented scan_vertex_async() method for parallel vertex scanning across partitions
  • Implemented scan_edge_async() method for parallel edge scanning across partitions


for future in concurrent.futures.as_completed(future_to_part):
part = future_to_part[future]
scan_result = future.result() # ScanResult
Copy link

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The future.result() call can raise exceptions from the background thread, but there's no exception handling. Consider wrapping this in a try-except block to handle potential errors gracefully, especially since the TODO comment mentions error handling as a known issue.

Copilot uses AI. Check for mistakes.

for future in concurrent.futures.as_completed(future_to_part):
part = future_to_part[future]
scan_result = future.result()
Copy link

Copilot AI Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as in scan_vertex_async: the future.result() call can raise exceptions from the background thread, but there's no exception handling. Consider wrapping this in a try-except block to handle potential errors gracefully.

Copilot uses AI. Check for mistakes.
@Fengzdadi
Copy link
Contributor Author

Tks to @Malone-AI @wey-gu, I will fix these errors in the next few days.

Fengzdadi and others added 3 commits July 30, 2025 12:30
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@CLAassistant
Copy link

CLAassistant commented Oct 16, 2025

CLA assistant check
All committers have signed the CLA.

@BeautyyuYanli BeautyyuYanli merged commit 1f71404 into vesoft-inc:master Oct 16, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants