Hold the Ethos-U backend suites at 8 pytest workers - #22575
Closed
huydhn wants to merge 1 commit into
Closed
Conversation
test-arm's arm_ethos_u85 models cell has killed its pod on every main run since the reusable workflows moved to linux_job_v3 (#22246): "Executing the custom container implementation failed", no pytest summary, always while reporting test_alexnet. arm_ethos_u55 passes on the same label with a larger log, and the suite completed on linux.4xlarge.memory, so this is memory rather than the runner or log volume. linux.4xlarge.memory is r5.4xlarge, 16 vCPU and 128GiB, and r5 is hyperthreaded, so `-n auto` counted 8 physical cores and each worker had 16GiB. OSDC pods run with SMT off, so the pod's cpuset is all physical cores and the worker cap follows it: 15 workers with 7.7GiB each. Every Ethos-U failure emits a few hundred thousand lines of Vela output, and 15 of those in flight is more than the pod has. No label helps, the whole fleet is about 8GiB per vCPU, so the worker count is the only lever. pytest-workers is an input rather than a change to pytest-parallelism.sh: a memory-derived cap would also cut the arm no-driver job on the 8-64 label from 8 workers to 4, and that one is fine as it is. The other eight callers of _test_backend.yml keep the default. Authored with Claude Code.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22575
Note: Links to docs will display an error until the docs builds have been completed. ⏳ No Failures, 1 PendingAs of commit f719f26 with merge base bf88c64 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
huydhn
marked this pull request as draft
September 4, 2026 23:56
rascani
approved these changes
Sep 4, 2026
Contributor
Author
|
Wrong axis: the OOM is the runner agent (System.OutOfMemoryException), not the pod, so worker count is irrelevant. 8 workers failed and also regressed arm_ethos_u55. Fixed properly in #22581. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test-arm'sarm_ethos_u85 / modelscell has killed its pod on every main run since #22246 landed:Executing the custom container implementation failed, no pytest summary, always while reportingtest_alexnet. Examples: f624c34665, bf88c643ce.It's memory, not the runner or the log volume:
arm_ethos_u55passes on the same label with a larger log (768k lines vs 457k), and the suite completed onlinux.4xlarge.memory.linux.4xlarge.memoryis r5.4xlarge, 16 vCPU / 128GiB, and r5 is hyperthreaded — so-n autocounted 8 physical cores and each worker got 16GiB. OSDC pods run with SMT off, so the cpuset is all physical cores and the worker cap follows it: 15 workers at 7.7GiB. Every Ethos-U failure emits a few hundred thousand lines of Vela output, and 15 of those in flight exceeds the pod. No label fixes this — the fleet is ~8GiB/vCPU throughout — so worker count is the only lever.An input rather than a change to
pytest-parallelism.sh: a memory-derived cap would also cut the arm no-driver job on the8-64label from 8 workers to 4, and that one is fine. The other eight callers of_test_backend.ymlkeep the default.Test plan.
arm_ethos_u85only runs on push, so dispatched on this branch (dispatch takes the non-pull_requestbranch of theflowsexpression and runs all four flows): https://github.com/pytorch/executorch/actions/runs/33931026107Authored with Claude Code.
cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani