fix: drain native stderr pipe concurrently to unblock EP compilation - #1223
Merged
Conversation
capture_native_stderr redirected fd 2 into an OS pipe but only drained it after the wrapped block returned. A native EP that wrote more than the pipe buffer holds (~64 KB on Windows) blocked in write() and never returned, so the block never completed and the drain never ran. VitisAI model compilation emits several hundred lines of compiler output on native stderr and reliably filled the buffer, stalling session creation indefinitely: winml perf -m facebook/convnext-tiny-224 --device npu --ep vitisai --precision fp16 ran for 17m52s without completing, with no compile artifacts written for the last 14m45s, while the same model and EP compiled in 185s through a session created outside the context manager. Start a daemon reader thread before the yield so the pipe is drained while the wrapped block runs and the buffer never fills. Restoring fd 2 drops the last write-end reference, which signals EOF to the reader; it is then joined and the captured lines are re-emitted through logging as before. Also clear the VitisAI EP compilation cache in the eval harness. It lives outside the user profile (C:\temp\<user>\vaip\.cache), so clearing only the HuggingFace and WinML caches left compiled NPU artifacts behind. AMD documents that these directories must not be reused across VitisAI EP or NPU driver versions, and a stale entry also lets a run load a previously compiled model instead of exercising the compile path under test. After the fix the same perf run completes in 3m23s, and the eval harness reports [PASS] 172.4s (1/1) for the vitisai_npu fp16 job that previously timed out.
DingmaomaoBJTU
approved these changes
Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
capture_native_stderrredirected fd 2 into an OS pipe but only drained itafter the wrapped block returned. A native EP that wrote more than the pipe
buffer holds (~64 KB on Windows) blocked in
write()and never returned, so thewrapped block never completed and the drain never ran.
VitisAI model compilation emits several hundred lines of compiler output on
native stderr and reliably filled the buffer, stalling session creation
indefinitely.
Measured on an AMD Ryzen AI 9 HX 370 (VitisAI EP 1.8.72.0, EP cache emptied
before each run):
winml perf ... --ep vitisai --precision fp16capture_native_stderrThe last variant isolates the context manager as the cause: same model, same EP,
same session options, same empty cache.
What changed
capture_native_stderrdrains concurrently. A daemon reader thread startsbefore the
yield, so the pipe is emptied while the wrapped block runs and thebuffer never fills. Restoring fd 2 drops the last write-end reference, which
signals EOF to the reader; it is then joined and the captured lines are
re-emitted through logging exactly as before. No change to the public
behaviour (capture, ANSI stripping,
[ORT]prefix).The eval harness now clears the VitisAI EP compilation cache. It lives
outside the user profile (
C:\temp\<user>\vaip\.cache), so clearing only theHuggingFace and WinML caches left compiled NPU artifacts behind. AMD documents
that these directories must not be reused across VitisAI EP or NPU driver
versions, and a stale entry also lets a run load a previously compiled model
instead of exercising the compile path under test.
Regression coverage.
test_no_deadlock_on_large_outputwrites ~720 KBinside the context manager under a 30s timeout, so a reintroduced deadlock
fails instead of hanging.
TestClearDiskCachescovers the new cache directoryand the
Nonecase on non-Windows.Validation
pytest tests/unit/utils/test_native_stderr.py— 8 passed, 1 skippedpytest tests/unit/eval/test_run_eval_script.py— 111 passedruff checkclean on all touched fileswinml perf -m facebook/convnext-tiny-224 --device npu --ep vitisai --precision fp16completes in 3m23s (was: no completion after 17m52s)
run_eval.py --hf-model facebook/convnext-tiny-224 --device npu --ep vitisaireports
[PASS] 172.4s, perf pass 1/1 (was:[FAIL (TIMEOUT)] 601.9s)