Stop the backend suites logging captured output - #22581
Merged
Merged
Conversation
Since #22246 moved these suites to OSDC, arm_ethos_u55 and arm_ethos_u85 models have died partway through reporting their failures. The runner agent throws System.OutOfMemoryException on the step, so it is the agent's heap rather than anything in the pod: AMD r7a vs Intel r7i at the same size, 8 vs 15 vs 32 workers, 116Gi vs 256Gi and disk 300 vs 400 all failed identically, while every other cell of the same suite passed on the same label. The two that failed are the two that run Corstone FVP on large models, where each failure captures a few hundred thousand lines of Vela operator listings. --show-capture=no drops those sections from the report. Nothing else changes: same runner, same worker count. On mt-l-x86iavx512-16-128 with 15 workers both cells now complete, 9 failed 13 passed 20 skipped, and the log goes from 458k lines to 12.6k. Each failure keeps its traceback and its exception message, so "Corstone simulation failed" and the arena sizes are still in the log. The failures themselves are pre-existing and unrelated, and the job was green with them before because test_backend.sh exits with the summary generator's status rather than pytest's. Authored with Claude Code.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22581
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 47dd799 with merge base bf88c64 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
This was referenced Sep 5, 2026
shoumikhin
approved these changes
Sep 5, 2026
Contributor
Author
|
Testing pass now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #22246 moved these suites to OSDC,
arm_ethos_u55 / modelsandarm_ethos_u85 / modelshave died partway through reporting their failures. The cause is the runner agent's heap, not the pod: the log ends with##[error]Exception of type 'System.OutOfMemoryException' was thrown.Each Ethos-U failure captures a few hundred thousand lines of Vela operator listings, and the agent cannot process a step that size.--show-capture=nodrops those sections from the terminal report. Verified on the default label and worker count: https://github.com/pytorch/executorch/actions/runs/33935989761Nothing is lost.
--show-captureonly affects what the terminal prints, so the captured streams still ship in the JSON report artifact (19MB, against 17MB before), and each failure keeps its traceback and exception message —RuntimeError: Corstone simulation failedand the arena sizes are still in the log.This applies to every backend suite, Linux and macOS, since
_test_backend.ymlsources the script in both jobs. Only the Ethos-U cells produce output at this scale today, but any of them could.The failures themselves are pre-existing and unrelated; the job was green with them before because
test_backend.shexits with the summary generator's status rather than pytest's.Authored with Claude Code.
cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani