Command to reproduce error is incorrect

[kshitij12345]

🐛 Describe the bug

To execute this test, run the following from the base repo dir: TEST_WITH_TORCHINDUCTOR=1 python test/test_torch.py -k test_nondeterministic_alert_NLLLoss_cuda

But I had to run it with

PYTORCH_TEST_WITH_INDUCTOR=1 python test/test_torch.py -k test_nondeterministic_alert_NLLLoss_cuda

Ref CI failure: https://github.com/pytorch/pytorch/actions/runs/5678841216/job/15391208767?pr=106118
Ref PR: #106118

Versions

master

cc @mruberry @ZainRizvi

[ezyang]

cc @jbschlosser

This shouldn't be too hard to fix

[NavneetSinghArora]

Hi,
I am new to open source contributions and would love to contribute to PyTorch starting with this ticket.
It would be great if you can assign me this ticket and give an insight on how to proceed.

Thank You

[gordbegli]

Hello. This is my first time touching open source and PyTorch. I took a look at this error.

---

I was unable to reproduce the bug.

I added this failing test to /test/test_torch.py

    def test_that_fails(self):
        self.assertEqual(True, False)

Then ran

PYTORCH_TEST_WITH_INDUCTOR=1 python test/test_torch.py -k test_that_fails

Which correctly output

AssertionError: Booleans mismatch: True is not False

To execute this test, run the following from the base repo dir:
    PYTORCH_TEST_WITH_INDUCTOR=1 python test/test_torch.py -k test_that_fails_cpu

---

I also investigated the logic in /torch/testing/_internal/common_utils.py. It seems to check out.

The message uses {env_var_prefix} to display “PYTORCH_TEST_WITH_INDUCTOR” : Line 2591
{env_var_prefix} is instantiated with TestEnvironment.repro_env_var_prefix() : Line 2572
TestEnvironment.repro_env_var_prefix() converts dictionary repro_env_vars into a string: Line 159
Values are added to repro_env_vars via def_flag()’s argument env_var : Line 148

I created this script to verify the logic in def_flag() and it also correctly output "PYTORCH_TEST_WITH_INDUCTOR=1"

import os
class TestEnvironment:
    repro_env_vars: dict = {}
    @staticmethod
    def def_flag(
        name,
        env_var=None,
        default=False,
        include_in_repro=True,
        enabled_fn=lambda env_var_val, default: (
            (env_var_val != "0") if default else (env_var_val == "1")),
        implied_by_fn=lambda: False,
    ):
        enabled = default
        if env_var is not None:
            env_var_val = os.getenv(env_var)
            enabled = enabled_fn(env_var_val, default)
        implied = implied_by_fn()
        enabled = enabled or implied
        if include_in_repro and (env_var is not None) and (enabled != default) and not implied:
            TestEnvironment.repro_env_vars[env_var] = env_var_val
        assert name not in globals(), f"duplicate definition of flag '{name}'"
        globals()[name] = enabled
    @staticmethod
    def repro_env_var_prefix() -> str:
        return " ".join([f"{env_var}={value}"
                         for env_var, value in TestEnvironment.repro_env_vars.items()])
os.environ["PYTORCH_TEST_WITH_INDUCTOR"] = "1"
TestEnvironment.def_flag("TEST_WITH_TORCHINDUCTOR", env_var="PYTORCH_TEST_WITH_INDUCTOR")
env_var_prefix = TestEnvironment.repro_env_var_prefix()
print("Environment Variable Prefix:", env_var_prefix)

---

Hopefully this is helpful to anyone who decides to pursue this issue.

[ezyang]

To be clear, the problem is not that the test fails, it's that if the test fails (e.g., you artificially induce a failure) it gives the wrong repro command.

[jbschlosser]

Just saw this issue when crawling through the testing issue backlog. I'm also unable to reproduce the stated buggy behavior - I see a repro command with the correct env var name:

PYTORCH_TEST_WITH_INDUCTOR=1 python test/test_torch.py -k TestTorchDeviceTypeCUDA.test_nondeterministic_alert_NLLLoss_cuda

@kshitij12345 have you happened to run into this again or can I close this one?

[joldov]

I will be picking up this issue and working on it - hope it is possible to do so! Please feel free to let me know otherwise :)