Skip to content

fix: wrap blocking MirrorClient calls with asyncio.to_thread to prevent event loop starvation - #945

Merged
anderdc merged 5 commits into
entrius:testfrom
Tet-9:fix/876-wrap-mirror-client-blocking-calls-asyncio-to-thread
May 7, 2026
Merged

fix: wrap blocking MirrorClient calls with asyncio.to_thread to prevent event loop starvation#945
anderdc merged 5 commits into
entrius:testfrom
Tet-9:fix/876-wrap-mirror-client-blocking-calls-asyncio-to-thread

Conversation

@Tet-9

@Tet-9 Tet-9 commented May 3, 2026

Copy link
Copy Markdown
Contributor

Closes #876

Problem

MirrorClient uses the synchronous requests library for all HTTP calls. The validator's forward path (forward.py) is async def, and all mirror scoring and issue discovery functions are reached from it. Every blocking requests.get(...) call ties up the event loop for the full round-trip time — often hundreds of milliseconds per call, seconds under Cloudflare throttle or cold cache conditions.

This starves:

  • Bittensor metagraph heartbeats
  • bt.logging flush callbacks
  • Any concurrent dendrite calls

The validator's apparent latency to other subnet participants spikes during every mirror scoring round.

Root Cause

Three blocking call sites were running synchronous I/O directly in the async context:

  1. mirror_scan.pyclient.get_miner_issues() in run_mirror_issue_discovery (async)
  2. mirror_scan.pyclient.get_pr_files() in _resolve_solving_pr_score (called from async chain)
  3. scoring.pyclient.get_pr_files() in score_mirror_pr (called from async chain)
  4. reward.pyload_mirror_miner_prs + score_mirror_miner_prs called synchronously inside evaluate_miners_pull_requests (async)

Fix

gittensor/validator/issue_discovery/mirror_scan.py

  • Made _resolve_solving_pr_score and _score_miner_mirror_issues async def
  • Wrapped client.get_pr_files() with await asyncio.to_thread(...) inside _resolve_solving_pr_score
  • Wrapped client.get_miner_issues() with await asyncio.to_thread(...) in run_mirror_issue_discovery
  • Added await to all call sites of the newly async functions
  • Added import asyncio

gittensor/validator/oss_contributions/mirror/scoring.py

  • Made score_mirror_pr and score_mirror_miner_prs async def
  • Wrapped client.get_pr_files() with await asyncio.to_thread(...) inside score_mirror_pr
  • Added await to score_mirror_pr call site in score_mirror_miner_prs
  • Added import asyncio

gittensor/validator/oss_contributions/reward.py

  • load_mirror_miner_prs remains synchronous — wrapped with await asyncio.to_thread(...)
  • score_mirror_miner_prs is now async — called directly with await
  • Added import asyncio

Result

All blocking mirror HTTP calls now run on the thread pool via asyncio.to_thread, freeing the event loop during mirror I/O. No changes to scoring logic, multipliers, or data flow.

Files Changed

  • gittensor/validator/issue_discovery/mirror_scan.py
  • gittensor/validator/oss_contributions/mirror/scoring.py
  • gittensor/validator/oss_contributions/reward.py

@xiao-xiao-mao xiao-xiao-mao Bot added the enhancement New feature or request label May 3, 2026
@Tet-9
Tet-9 force-pushed the fix/876-wrap-mirror-client-blocking-calls-asyncio-to-thread branch from e7fd415 to 49f18c0 Compare May 3, 2026 14:21

@anderdc anderdc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move import asyncio from inside evaluate_miners_pull_requests to the top of gittensor/validator/oss_contributions/reward.py with the other module-level imports.

@Tet-9
Tet-9 requested a review from anderdc May 6, 2026 22:37
@Tet-9

Tet-9 commented May 6, 2026

Copy link
Copy Markdown
Contributor Author

Move import asyncio from inside evaluate_miners_pull_requests to the top of gittensor/validator/oss_contributions/reward.py with the other module-level imports.

Done 🫡

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] MirrorClient is synchronous but called from async validator forward path — blocks the event loop

2 participants