reset dynamo cache before each test #126586

shunting314 · 2024-05-17T23:30:07Z

Stack from ghstack (oldest at bottom):

-> reset dynamo cache before each test #126586

In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests.

This PR clear dynamo cache before each unit test so we get more deterministic result for unit test

[ghstack-poisoned]

pytorch-bot · 2024-05-17T23:30:10Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126586

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (5 Unrelated Failures)

As of commit 9472bcf with merge base 5905207 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no log or failing log classifier)
inductor / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no log or failing log classifier)
inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no log or failing log classifier)
inductor / rocm6.1-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2) (gh) (detected as infra flaky with no log or failing log classifier)

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu, unstable) (gh) ()
functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_linalg_lu_factor_cuda_float32

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 94cbf126e5c128577d388ecf24d362d8594c1e00 Pull Request resolved: #126586

huydhn · 2024-05-18T01:13:34Z

torch/testing/_internal/common_utils.py

+        # reset dynamo cache to avoid issues like
+        # https://github.com/pytorch/pytorch/issues/125967#issuecomment-2118483919
+        # which depends on test order.
+        torch._dynamo.reset()


I vaguely remember that this causes a perf hit on CI testing, so it couldn't be rolled out before. @zou3519 would know more

We do need to do this (reset dynamo cache). I rolled out this change to all tests in the dynamo shard, but didn't do the tests in the inductor shard. I'm not sure about the performance impact on inductor tests, but it is hopefully not bad because we do not run that many inductor tests.

@shunting314 can you get the tests to pass in this PR so that we can see the before/after test timing numbers?

yea. 2 of the failures are due to previously fail tests now pass. I removed the file marking the test failure.

For the other failures, I'm not able to repro. E.g., for https://github.com/pytorch/pytorch/actions/runs/9135411752/job/25122927855 , I run

PYTORCH_TEST_WITH_INDUCTOR=1 python test/functorch/test_ops.py -k test_extremal_numerics_l1_loss_cpu

And it passes on my devserver..

Any suggestions for this?

it might have to do with python version. Try to skip these for now. Also, the keep-going label will make all tests run (after a re-push) -- it's possible you have more failures than just these two

Since the PR is clean now (the previous failed tests shows up as flaky and non-blocking in CI), I'll merge it.

Thanks for letting me know the 'keep-going' label.

Since the PR is clean now (the previous failed tests shows up as flaky and non-blocking in CI), I'll merge it.

Yup go for it

In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]

shunting314 · 2024-05-21T20:22:47Z

@pytorchbot merge

pytorchmergebot · 2024-05-21T20:24:53Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-05-21T20:25:05Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

pull / linux-focal-py3.11-clang10 / test (dynamo, 2, 3, linux.2xlarge)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

shunting314 · 2024-05-21T20:32:07Z

hmm, the test failures blocks the merge even though CI is green before I try to merge.

I'll skip those tests for now.

In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]

Fix #125967 . The test actually fail for empty 4D or 5D tensors when checking for memory format. I'm not exactly sure what recent inductor change cause the failure, but it may be not that important to maintain strides for an empty tensor. (?) I just skip the check for empty tensor. Pull Request resolved: #126593 Approved by: https://github.com/ezyang ghstack dependencies: #126586

…#126594) This reverts commit 0a9c6e9. enable the test since it's fixed. Pull Request resolved: #126594 Approved by: https://github.com/huydhn ghstack dependencies: #126586, #126593

clee2000 · 2024-05-23T04:52:11Z

@pytorchbot revert -m "broke tests on inductor? test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_CTCLoss_cuda_float64 https://hud.pytorch.org/pytorch/pytorch/commit/43f2f43eb3b6d8cbe8eb7f45acb50376092f1a16 https://github.com/pytorch/pytorch/actions/runs/9200644034/job/25308511495" -c nosignal

pytorchmergebot · 2024-05-23T04:54:41Z

@shunting314 your PR has been successfully reverted.

shunting314 · 2024-05-23T18:13:13Z

@clee2000 is this due to land race? The test is broken even without my PR stacks. If yes, anything else you want me to do before I reland?

In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]

ghstack-source-id: c4c01ae7c17caafd1514e3ea659dd1af00bd84b4 Pull Request resolved: #126586

In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]

ghstack-source-id: 969d13a2e533bc9511ff23624fe0af9ce200b485 Pull Request resolved: #126586

In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]

ghstack-source-id: ecbcf9f6925d3f87129cef5d640588add6a77acc Pull Request resolved: #126586

shunting314 · 2024-05-24T22:33:06Z

@pytorchbot merge

pytorchmergebot · 2024-05-24T22:35:21Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-05-24T23:17:51Z

Merge failed

Reason: 11 mandatory check(s) failed. The first few are:

pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 1, 5, linux.4xlarge.nvidia.gpu, unstable)
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 2, 5, linux.4xlarge.nvidia.gpu, unstable)
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 3, 5, linux.4xlarge.nvidia.gpu, unstable)
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 4, 5, linux.4xlarge.nvidia.gpu, unstable)
pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu, unstable)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

huydhn · 2024-05-25T01:27:44Z

@pytorchbot drci

huydhn · 2024-05-25T01:30:54Z

@pytorchbot merge

pytorchmergebot · 2024-05-25T01:32:39Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

malfet · 2024-05-25T17:15:17Z

@pytorchbot revert -m "Broke tons of tests, see https://hud.pytorch.org/pytorch/pytorch/commit/bd24991f461476036d6ba20fed92651c7e46ef7c " -c ignoredsignal

pytorchmergebot · 2024-05-25T17:17:11Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

This reverts commit bd24991. Reverted #126586 on behalf of https://github.com/malfet due to Broke tons of tests, see https://hud.pytorch.org/pytorch/pytorch/commit/bd24991f461476036d6ba20fed92651c7e46ef7c ([comment](#126586 (comment)))

pytorchmergebot · 2024-05-25T17:17:22Z

@shunting314 your PR has been successfully reverted.

In pytorch#125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test Pull Request resolved: pytorch#126586 Approved by: https://github.com/jansel

This reverts commit bd24991. Reverted pytorch#126586 on behalf of https://github.com/malfet due to Broke tons of tests, see https://hud.pytorch.org/pytorch/pytorch/commit/bd24991f461476036d6ba20fed92651c7e46ef7c ([comment](pytorch#126586 (comment)))

reset dynamo cache before each test

ed27879

[ghstack-poisoned]

shunting314 requested a review from a team as a code owner May 17, 2024 23:30

shunting314 added a commit that referenced this pull request May 17, 2024

reset dynamo cache before each test

b211a8b

ghstack-source-id: 94cbf126e5c128577d388ecf24d362d8594c1e00 Pull Request resolved: #126586

shunting314 requested review from ezyang, jansel and anijain2305 May 17, 2024 23:32

This was referenced May 18, 2024

don't check memory format for empty tensors #126593

Closed

Revert "Skip test_memory_format_nn_BatchNorm2d in inductor (#125970)" #126594

Closed

jansel approved these changes May 18, 2024

View reviewed changes

huydhn reviewed May 18, 2024

View reviewed changes

zou3519 added the keep-going Don't stop on first failure, keep running tests until the end label May 21, 2024

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 21, 2024

shunting314 added the topic: not user facing topic category label May 21, 2024

pytorchmergebot added the merging label May 21, 2024

pytorchmergebot removed the merging label May 21, 2024

shunting314 requested review from Chillee and kshitij12345 as code owners May 21, 2024 20:34

pytorchmergebot closed this in 43f2f43 May 22, 2024

pytorchmergebot added the Merged label May 22, 2024

pytorchmergebot added the Reverted label May 23, 2024

pytorchmergebot reopened this May 23, 2024

shunting314 added 2 commits May 24, 2024 13:39

shunting314 added a commit that referenced this pull request May 24, 2024

reset dynamo cache before each test

202120d

ghstack-source-id: c4c01ae7c17caafd1514e3ea659dd1af00bd84b4 Pull Request resolved: #126586

shunting314 added a commit that referenced this pull request May 24, 2024

reset dynamo cache before each test

61fb995

ghstack-source-id: 969d13a2e533bc9511ff23624fe0af9ce200b485 Pull Request resolved: #126586

shunting314 added a commit that referenced this pull request May 24, 2024

reset dynamo cache before each test

e098094

ghstack-source-id: ecbcf9f6925d3f87129cef5d640588add6a77acc Pull Request resolved: #126586

pytorchmergebot added the merging label May 24, 2024

pytorchmergebot removed the merging label May 24, 2024

pytorchmergebot added the merging label May 25, 2024

pytorchmergebot closed this in bd24991 May 25, 2024

pytorchmergebot removed the merging label May 25, 2024

pytorchmergebot reopened this May 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

reset dynamo cache before each test #126586

reset dynamo cache before each test #126586

shunting314 commented May 17, 2024 •

edited

pytorch-bot bot commented May 17, 2024 •

edited

huydhn May 18, 2024 •

edited

zou3519 May 20, 2024 •

edited

shunting314 May 20, 2024

zou3519 May 21, 2024 •

edited

shunting314 May 21, 2024 •

edited

zou3519 May 21, 2024

shunting314 commented May 21, 2024

pytorchmergebot commented May 21, 2024

pytorchmergebot commented May 21, 2024

shunting314 commented May 21, 2024

clee2000 commented May 23, 2024

pytorchmergebot commented May 23, 2024

shunting314 commented May 23, 2024

shunting314 commented May 24, 2024

pytorchmergebot commented May 24, 2024

pytorchmergebot commented May 24, 2024

huydhn commented May 25, 2024

huydhn commented May 25, 2024

pytorchmergebot commented May 25, 2024

malfet commented May 25, 2024

pytorchmergebot commented May 25, 2024

pytorchmergebot commented May 25, 2024

reset dynamo cache before each test #126586

Are you sure you want to change the base?

reset dynamo cache before each test #126586

Conversation

shunting314 commented May 17, 2024 • edited

pytorch-bot bot commented May 17, 2024 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126586

✅ You can merge normally! (5 Unrelated Failures)

huydhn May 18, 2024 • edited

Choose a reason for hiding this comment

zou3519 May 20, 2024 • edited

Choose a reason for hiding this comment

shunting314 May 20, 2024

Choose a reason for hiding this comment

zou3519 May 21, 2024 • edited

Choose a reason for hiding this comment

shunting314 May 21, 2024 • edited

Choose a reason for hiding this comment

zou3519 May 21, 2024

Choose a reason for hiding this comment

shunting314 commented May 21, 2024

pytorchmergebot commented May 21, 2024

Merge started

pytorchmergebot commented May 21, 2024

Merge failed

shunting314 commented May 21, 2024

clee2000 commented May 23, 2024

pytorchmergebot commented May 23, 2024

shunting314 commented May 23, 2024

shunting314 commented May 24, 2024

pytorchmergebot commented May 24, 2024

Merge started

pytorchmergebot commented May 24, 2024

Merge failed

huydhn commented May 25, 2024

huydhn commented May 25, 2024

pytorchmergebot commented May 25, 2024

Merge started

malfet commented May 25, 2024

pytorchmergebot commented May 25, 2024

pytorchmergebot commented May 25, 2024

shunting314 commented May 17, 2024 •

edited

pytorch-bot bot commented May 17, 2024 •

edited

huydhn May 18, 2024 •

edited

zou3519 May 20, 2024 •

edited

zou3519 May 21, 2024 •

edited

shunting314 May 21, 2024 •

edited