Various functions exhibit nondeterministic behavior

[ttanpcs]

🐛 Describe the bug

When testing the nondeterminism_seeded and bitwise tags on operators, the following operators fail:
Nondeterministic_seeded:

• aten.empty.memory_format
• aten.empty_like.default
• aten.new_empty.default
• aten._ctc_loss.default
• aten.resize_.default
• prims.uniform.default
• aten.empty_strided.default
• aten.empty.SymInt

Nondeterministic_bitwise:
-aten.linalg_lstsq.default

(I've bolded the ops that are interesting). The 'empty' ops fail due to their use of uninitialized data.

The following lines of code were added to test_ops.py and test_tags was run.

def test_nondeterministic_seeded(func, args, kwargs):
    if func is None or torch.Tag.inplace_view in func.tags:
        return
    results = []
    for i in range(2):
        results.append(func(*args, **kwargs))
    try:
        TestCase.assertEqual(TestCase(), results[0], results[1])
    except AssertionError:
        assert torch.Tag.nondeterministic_seeded in func.tags, f'{func} should be nondeterministic_seeded'
    try:
        TestCase.assertEqual(TestCase(), results[0], results[1], atol=0, rtol=0)
    except AssertionError:
        has_nondeterminism_tag = torch.Tag.nondeterministic_bitwise in func.tags or torch.Tag.nondeterministic_seeded in func.tags
        assert has_nondeterminism_tag, f'{func} should be nondeterministic_bitwise'

...

class TestTagsMode(TorchDispatchMode):
    def __torch_dispatch__(self, func, types, args=(), kwargs=None):
        test_nondeterministic_seeded(func, args, kwargs)

...

Versions

N/A

cc @ezyang @gchanan @zou3519 @mruberry @kurtamohler

[ezyang]

good call. When deterministic mode is on we should fill empty with deterministic garbage to make it deterministic

[lezcano]

Mandatory meme:

[lezcano]

On a more serious note, lstsq is known to have a non-deterministic LAPACK implementation. There should be a few open issues where we have discussed this. There's not much that we can do on that front other than throwing an error.

[albanD]

From offline discussion, we should fill empty* functions values with NaN in deterministic mode to make sure we don't "hide" any bug by filling it with ok-looking values.

[kurtamohler]

I can make that change

[kurtamohler]

And I assume for integer types we should just fill with zeros?

[lezcano]

more like MAX_INT, no?

[albanD]

I think some not-ok looking value would be best yes.

[kurtamohler]

I'm having trouble finding where at::empty_symint is implemented. git grep -w empty_symint only shows call sites. I see that it shows up in build/aten/src/ATen/ops/empty.h , but I'm not sure what build/aten/src/ATen/ops is or what generates it. Where can I find it's implementation so I can add the deterministic fill to it?

EDIT: Nevermind, I used gdb to find out that at::empty_symint just ends up calling at::native::empty_cpu

[albanD]

at::empty_symint is indeed codegened like all other aten native functions. You can find them at build/aten/src/ATen/ops/{op_name}.h for your local build (you can see there the magic about the regular and regular_symint version of these functions!). The impl for the ::call is in the Operators_{i}.cpp files next to this. You can see it calls the op via the dispatcher.
So in your case, I guess you passed CPU Tensors and so called into the native cpu impl.

[kurtamohler]

After #101849, the remaining functions from the issue description still need to be addressed:

• aten._ctc_loss.default
• aten.resize_.default
• prims.uniform.default
• aten.linalg_lstsq.default

For resize_, I imagine we'd want to do almost the same thing that empty does, but we'd only want to fill new elements with NaN/MAX_INT, keeping old elements untouched. I will submit a PR for this. (EDIT: fixed in #104300)

For linalg.lstsq, discussion is tracked in #71222.

For ctc_loss, discussion is tracked in #17798.

If I understand correctly, prims.uniform just calls into Tensor.uniform_. I would like to see a reproducer of the nondeterministic behavior for this one. It seems that it should be deterministic if the random seed is always initialized to the same value, but perhaps that's not true for some reason

[kurtamohler]

The only thing left for this issue is the nondeterminism of Tensor.uniform_, which I haven't been able to reproduce. I tried running the modification to test_ops.py that was mentioned in the issue summary, and test_tags_uniform_cpu_float32 passes, which is the only test_tags test that seems to be related to uniform. So I think it's probably safe to close this issue now. If Tensor.uniform_ shows any nondeterminism in the future either this issue can be reopened or a new issue can be created