Tests modify global state cause later tests to fail

[clee2000]

Context

I've been working on getting the Dynamo tests to reset the Dynamo state before and after each unittest, otherwise, the state from one test affects subsequent tests, leading to unexpected behavior.

However, Dynamo is not the only place where we have global state. We've ran into situations in the past where changing e.g. the global default dtype in one test had a downstream effect on other tests.

Can we make unittests better by better isolating them somehow?

Pitch

Some ideas:

• One naive way to do this is to run each test in its own process, but this is probably not feasible due to the amount of overhead it adds to each test.
• Another idea is to have some centralized place where "all pytorch state" is kept and reset this state before and after each test runs.
• Linters for abuse of APIs that modify global state? For example, instead of using torch.set_default_dtype, we should use a context-manager version (if we have one) so it's clear the global state gets cleaned up. This would still require us to identify all of the APIs that do modify global state and put them somewhere.

cc @ZainRizvi @kit1980 @huydhn @clee2000 @janeyx99, @jbschlosser, @pytorch/pytorch-dev-infra for ideas

This is more of a tracker for tests that modify global state somehow, and then tests that run after them change outcome depending on whether the first test was run or not. I want to keep track of this to see how often it happens and why it happens.

The tests afterwards generally get marked as flaky because they usually succeed on file level retry, which starts at the failing test and doesn't run the first test.

Examples of this include:

• Test 1 adds to dictionary that is imported or declared at the module level. Test 2 reads that dictionary and changes outcome after seeing the new addition
• Test 1 imports extra modules. Test 2 crawls through imports to check functions in each module. The new module causes the outcome to change.
• Test 1 sets an attribute. Test 2 sees the new attribute. (Clean up test_external_module_register #110254)
• APIs - set_default_dtype
• dynamo state, dynamo cache

Fixes can include cleaning up after the function or moving tests to run in different files (then CI will run in different processes).

cc @seemethere @malfet @pytorch/pytorch-dev-infra @ZainRizvi @kit1980 @huydhn