Create a mechanism to trigger tests that only warn once

[mruberry]

Today in PyTorch we have TORCH_WARN and TORCH_WARN_ONCE. TORCH_WARN_ONCE, as its name suggests, only throws a warning one time. This makes it tricky to test for. Our test suite, for example, has maybeWarnsRegex, but this just verifies that no other warnings are thrown.

This feature request is for a mechanism to consistently trigger warnings that use TORCH_WARN_ONCE. One mechanism for this would be to add a new context that, if set, causes these warnings to always warn. This would let us reliably test that these warnings are thrown correctly.

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @mruberry @VitalyFedyunin @walterddr

[mattip]

If I understand the code correctly, it seems TORCH_WARN_ONCE and TORCH_WARN do not currently always use python. In a call like setDeterministic, the default warning_handler is used, which goes to LOG_AT_FILE_LINE which uses MessgeLogger to print to std::err. Most python wrappers use the HANDLE_TH_ERRORS macro to set a warning_handler by calling PyWarningHandler. For some reason the wrapper for setDeterministic does not use HANDLE_TH_ERRORS.

But that is only one part of the story. Overriding the behaviour of TORCH_WARN_ONCE to make it warn on each call is more complicated. As defined in c10/util/Exception.h, it instantiates a static class instance, making it a singleton, and uses that instance to call c10::Warning::warn only once ever. Refactoring that to call each time would make this equivalent to TORCH_WARN.

[mruberry]

I was hoping we could do something like set a global context, "setAlwaysWarn(true/false)", that TORCH_WARN_ONCE would query and, if the global flag was true, it would then warn like TORCH_WARN does.

[mattip]

Is this the direction you were thinking?

#define TORCH_WARN_MAYBE(...) \
  if (g_WARN_ONLY_ONCE) { \
    TORCH_WARN_ONCE(...) \
  } else { \
    TORCH_WARN(...) \
  }

Then

• Some/All of the uses of TORCH_WARN_ONCE would be refactored to TORCH_WARN_MAYBE.
• The new boolean global value g_WARN_ONLY_ONCE would be set false by the python warning handler, passing responsibility for filtering warnings to python.
• Tests would need a warning filter, and could check that the warning is emitted.

Is there a warning context manager in the testing framework, or should we use the numpy.testing.suppress_warnings context manager to work around python issues with warning filters

[mruberry]

That's close to what I was thinking of.

I was thinking we'd keep TORCH_WARN_ONCE, and there would be a flag, like g_WARN_ONLY_ONCE, that had the behavior you're describing. It's true that this would make TORCH_WARN_ONCE a bit of a misnomer, but that's OK.

I was thinking the flag would be controlled by a PyTorch global context (like the deterministic flag).

We currently use a the Python warning filter in our test suite and that seems to be OK, I don't think we'd need to use numpy.testing.suppress_warnings.

Basically we want this flag to be just for debugging. We'll set it to true during the test suite so we can verify that these warnings are thrown.

Does that sound good?

[mattip]

Now that the assertWarnsOnceRegex is inplace, the next step is to replace maybeWarnsRegex with it.

[mruberry]

I think this issue is closed. @mattip, would you file a separate issue to audit uses of maybeWarnsRegex in the test suite?

[mattip]

See issue #52025