Skip to content

make thread = 32 to speedup #112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hipudding merged 1 commit into from
Mar 10, 2026
+1 −1
Merged

make thread = 32 to speedup #112

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion scripts/fetch_ascend_data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,7 @@ def main():
})

# 创建工作线程（限制最大线程数为32，避免API限流）
num_threads = min(os.cpu_count() or 4, 32)
num_threads = 32
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment on line 318 still says "限制最大线程数为32，避免API限流" (limit max threads to 32, avoid API rate limiting), but the original intent of the code was to use min(os.cpu_count() or 4, 32) — meaning 32 was already the upper bound to avoid API rate limiting. By hardcoding num_threads = 32, the thread count is now fixed at 32 regardless of the number of available CPUs or tasks, which can cause unnecessary resource consumption on machines with fewer cores (e.g. CI runners with 2–4 CPUs). More critically, the original comment's rationale about avoiding API rate limiting is now inverted: the original code treated 32 as the maximum to stay safe, but the new code always spawns the maximum, which increases the risk of hitting rate limits on the external hiascend.com API. The comment should be updated to reflect this change, or better yet, the original min(os.cpu_count() or 4, 32) logic should be preserved to adaptively scale threads based on the machine.

Copilot uses AI. Check for mistakes.
threads = []
for i in range(num_threads):
t = threading.Thread(target=worker, args=(task_queue, config, config_lock), name=f"Worker-{i}")
Expand Down