Skip dtensor ops on CPU-only runner due to flaky timeout #98868
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/98868
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 FailuresAs of commit 7ef4187: BROKEN TRUNK - The following jobs failed but were present on the merge base def50d2:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
huydhn
requested review from
H-Huang,
awgu,
fegin,
kwen2501,
mrshenli,
rohan-varma,
wanchaol and
zhaojuanmao
as code owners
| run_tests() | ||
| # NB: CPU dtensor ops test frequently timeout https://github.com/pytorch/pytorch/issues/98816 | ||
| # so running it only on CUDA | ||
| if torch.cuda.is_available(): |
There was a problem hiding this comment.
hmmm I don't think test_dtensor_ops runs in CUDA, so this essentially means it would only run in CUDA available machines with cpu tests https://github.com/pytorch/pytorch/blob/master/test/distributed/_tensor/test_dtensor_ops.py#L559
There was a problem hiding this comment.
which I think it might be fine for short term while we are investigating the timeouts. I just want to make sure there's at least something running in the CI so that we can capture errors when submitting PRs :)
There was a problem hiding this comment.
Yeah, the test would still be run on machine with CUDA, which curiously is never getting timed out. I will update the PR title and description accordingly
huydhn
changed the title
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 3 jobs have failed, first few of them are: trunk / macos-12-py3-arm64 / test (default, 1, 3, macos-m1-12), trunk / macos-12-py3-arm64 / test (default, 2, 3, macos-m1-12), trunk / macos-12-py3-arm64 / test (default, 3, 3, macos-m1-12) Details for Dev Infra teamRaised by workflow job |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
4 participants
distributed/_tensor/test_dtensor_opsis still flaky in trunk with a curious timeout issue, for example https://hud.pytorch.org/pytorch/pytorch/commit/ce4df4cc596aa10534ac6d54912f960238264dfd. It seems that the test just hang without any failure. The root cause is unclear. On the other hang, #98816 might offer a solution for this. Anyway, I'm disable the test on CPU for now while the investigation is being done.The test is still being run on CUDA-available runner because it's not flaky there.