reset dynamo cache before each test #126586

shunting314 · 2024-05-17T23:30:07Z

Stack from ghstack (oldest at bottom):

-> reset dynamo cache before each test #126586

In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests.

This PR clear dynamo cache before each unit test so we get more deterministic result for unit test

[ghstack-poisoned]

pytorch-bot · 2024-05-17T23:30:10Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126586

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (5 Unrelated Failures)

As of commit 9472bcf with merge base 5905207 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no log or failing log classifier)
inductor / cuda12.1-py3.12-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no log or failing log classifier)
inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor, 1, 1, linux.g5.4xlarge.nvidia.gpu) (gh) (detected as infra flaky with no log or failing log classifier)
inductor / rocm6.1-py3.8-inductor / test (inductor, 1, 1, linux.rocm.gpu.2) (gh) (detected as infra flaky with no log or failing log classifier)

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu, unstable) (gh) ()
functorch/test_ops.py::TestOperatorsCUDA::test_vmapjvpall_linalg_lu_factor_cuda_float32

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 94cbf12 Pull Request resolved: #126586

huydhn · 2024-05-18T01:13:34Z

torch/testing/_internal/common_utils.py

+        # reset dynamo cache to avoid issues like
+        # https://github.com/pytorch/pytorch/issues/125967#issuecomment-2118483919
+        # which depends on test order.
+        torch._dynamo.reset()


I vaguely remember that this causes a perf hit on CI testing, so it couldn't be rolled out before. @zou3519 would know more

We do need to do this (reset dynamo cache). I rolled out this change to all tests in the dynamo shard, but didn't do the tests in the inductor shard. I'm not sure about the performance impact on inductor tests, but it is hopefully not bad because we do not run that many inductor tests.

@shunting314 can you get the tests to pass in this PR so that we can see the before/after test timing numbers?

yea. 2 of the failures are due to previously fail tests now pass. I removed the file marking the test failure.

For the other failures, I'm not able to repro. E.g., for https://github.com/pytorch/pytorch/actions/runs/9135411752/job/25122927855 , I run

PYTORCH_TEST_WITH_INDUCTOR=1 python test/functorch/test_ops.py -k test_extremal_numerics_l1_loss_cpu

And it passes on my devserver..

Any suggestions for this?

it might have to do with python version. Try to skip these for now. Also, the keep-going label will make all tests run (after a re-push) -- it's possible you have more failures than just these two

Since the PR is clean now (the previous failed tests shows up as flaky and non-blocking in CI), I'll merge it.

Thanks for letting me know the 'keep-going' label.

Since the PR is clean now (the previous failed tests shows up as flaky and non-blocking in CI), I'll merge it.

Yup go for it

In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]

shunting314 · 2024-05-21T20:22:47Z

@pytorchbot merge

pytorchmergebot · 2024-05-21T20:24:53Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-05-21T20:25:05Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

pull / linux-focal-py3.11-clang10 / test (dynamo, 2, 3, linux.2xlarge)

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

shunting314 · 2024-05-21T20:32:07Z

hmm, the test failures blocks the merge even though CI is green before I try to merge.

I'll skip those tests for now.

In #125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test [ghstack-poisoned]

Fix #125967 . The test actually fail for empty 4D or 5D tensors when checking for memory format. I'm not exactly sure what recent inductor change cause the failure, but it may be not that important to maintain strides for an empty tensor. (?) I just skip the check for empty tensor. Pull Request resolved: #126593 Approved by: https://github.com/ezyang ghstack dependencies: #126586

…#126594) This reverts commit 0a9c6e9. enable the test since it's fixed. Pull Request resolved: #126594 Approved by: https://github.com/huydhn ghstack dependencies: #126586, #126593

clee2000 · 2024-05-23T04:52:11Z

@pytorchbot revert -m "broke tests on inductor? test_modules.py::TestModuleCUDA::test_cpu_gpu_parity_nn_CTCLoss_cuda_float64 https://hud.pytorch.org/pytorch/pytorch/commit/43f2f43eb3b6d8cbe8eb7f45acb50376092f1a16 https://github.com/pytorch/pytorch/actions/runs/9200644034/job/25308511495" -c nosignal

pytorchmergebot · 2024-05-25T17:17:11Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

This reverts commit bd24991. Reverted #126586 on behalf of https://github.com/malfet due to Broke tons of tests, see https://hud.pytorch.org/pytorch/pytorch/commit/bd24991f461476036d6ba20fed92651c7e46ef7c ([comment](#126586 (comment)))

pytorchmergebot · 2024-05-25T17:17:22Z

@shunting314 your PR has been successfully reverted.

In pytorch#125967, we found test results depend on test order. The root cause is due to earlier tests populate dynamo cache and affect the later tests. This PR clear dynamo cache before each unit test so we get more deterministic result for unit test Pull Request resolved: pytorch#126586 Approved by: https://github.com/jansel

This reverts commit bd24991. Reverted pytorch#126586 on behalf of https://github.com/malfet due to Broke tons of tests, see https://hud.pytorch.org/pytorch/pytorch/commit/bd24991f461476036d6ba20fed92651c7e46ef7c ([comment](pytorch#126586 (comment)))

…torch.py" #126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_torch.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_torch.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

…torch.py" #126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_torch.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_torch.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

…ops_gradients.py" #126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_ops_gradients.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

…module.py" #126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_module.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_ops_gradients.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_module.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

…module.py" #126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_module.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_module.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

…ops_gradients.py" #126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_ops_gradients.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_ops_gradients.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

…torch.py" #126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_torch.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_torch.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. [ghstack-poisoned]

github-actions · 2024-07-24T23:33:45Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_torch.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. Pull Request resolved: #131388 Approved by: https://github.com/zou3519 ghstack dependencies: #131551

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_module.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. Pull Request resolved: #131372 Approved by: https://github.com/zou3519 ghstack dependencies: #131551, #131388

#126586 tried to reset dynamo before each unit test. That PR get reverted a couple of times because we see post-land test failures that we don't see before merge. This PR only reset dynamo before each tests in `test_ops_gradients.py` to make it easier to land. Eventually after we reset dynamo in each individual test files, we can move the change to the base class (TestCase) and remove the change in individual test files. Pull Request resolved: #131397 Approved by: https://github.com/zou3519 ghstack dependencies: #131551, #131388, #131372

reset dynamo cache before each test

ed27879

[ghstack-poisoned]

shunting314 requested a review from a team as a code owner May 17, 2024 23:30

shunting314 added a commit that referenced this pull request May 17, 2024

reset dynamo cache before each test

b211a8b

ghstack-source-id: 94cbf12 Pull Request resolved: #126586

shunting314 requested review from ezyang, jansel and anijain2305 May 17, 2024 23:32

This was referenced May 18, 2024

don't check memory format for empty tensors #126593

Closed

Revert "Skip test_memory_format_nn_BatchNorm2d in inductor (#125970)" #126594

Closed

jansel approved these changes May 18, 2024

View reviewed changes

huydhn reviewed May 18, 2024

View reviewed changes

zou3519 added the keep-going Don't stop on first failure, keep running tests until the end label May 21, 2024

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 21, 2024

shunting314 added the topic: not user facing topic category label May 21, 2024

pytorchmergebot added the merging label May 21, 2024

pytorchmergebot removed the merging label May 21, 2024

shunting314 requested review from Chillee and kshitij12345 as code owners May 21, 2024 20:34

pytorchmergebot closed this in 43f2f43 May 22, 2024

pytorchmergebot added the Merged label May 22, 2024

pytorchmergebot reopened this May 25, 2024

This was referenced Jul 22, 2024

BE: reset dynamo before each test in test_module.py #131372

Closed

BE: reset dynamo before each test in test_torch.py #131388

Closed

BE: reset dynamo before each test in test_ops_gradients.py #131397

Closed

github-actions bot added the Stale label Jul 24, 2024

github-actions bot closed this Aug 23, 2024

github-actions bot deleted the gh/shunting314/147/head branch September 23, 2024 02:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

reset dynamo cache before each test #126586

reset dynamo cache before each test #126586

Uh oh!

shunting314 commented May 17, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented May 17, 2024 •

edited

Loading

Uh oh!

huydhn May 18, 2024 •

edited

Loading

Uh oh!

zou3519 May 20, 2024 •

edited

Loading

Uh oh!

shunting314 May 20, 2024

Uh oh!

zou3519 May 21, 2024 •

edited

Loading

Uh oh!

shunting314 May 21, 2024 •

edited

Loading

Uh oh!

zou3519 May 21, 2024

Uh oh!

shunting314 commented May 21, 2024

Uh oh!

pytorchmergebot commented May 21, 2024

Uh oh!

pytorchmergebot commented May 21, 2024

Uh oh!

shunting314 commented May 21, 2024

Uh oh!

clee2000 commented May 23, 2024

Uh oh!

pytorchmergebot commented May 25, 2024

Uh oh!

pytorchmergebot commented May 25, 2024

Uh oh!

github-actions bot commented Jul 24, 2024

Uh oh!

Uh oh!

reset dynamo cache before each test #126586

reset dynamo cache before each test #126586

Uh oh!

Conversation

shunting314 commented May 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented May 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126586

✅ You can merge normally! (5 Unrelated Failures)

Uh oh!

huydhn May 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zou3519 May 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shunting314 May 20, 2024

Choose a reason for hiding this comment

Uh oh!

zou3519 May 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shunting314 May 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zou3519 May 21, 2024

Choose a reason for hiding this comment

Uh oh!

shunting314 commented May 21, 2024

Uh oh!

pytorchmergebot commented May 21, 2024

Merge started

Uh oh!

pytorchmergebot commented May 21, 2024

Merge failed

Uh oh!

shunting314 commented May 21, 2024

Uh oh!

clee2000 commented May 23, 2024

Uh oh!

pytorchmergebot commented May 25, 2024

Uh oh!

pytorchmergebot commented May 25, 2024

Uh oh!

github-actions bot commented Jul 24, 2024

Uh oh!

Uh oh!

shunting314 commented May 17, 2024 •

edited

Loading

pytorch-bot bot commented May 17, 2024 •

edited

Loading

huydhn May 18, 2024 •

edited

Loading

zou3519 May 20, 2024 •

edited

Loading

zou3519 May 21, 2024 •

edited

Loading

shunting314 May 21, 2024 •

edited

Loading