[vision_maskrcnn] Accuracy error

[desertfire]

python benchmarks/torchbench.py -d cuda --float16 --inductor --no-skip -k vision_maskrcnn

[Chillee]

Can I get a command for this too :)

[Chillee]

I'm getting a

torchinductor.exc.LoweringException: AssertionError: (1 - s2, -4999)
  target: aten.index.Tensor
  args[0]: TensorBox(StorageBox(
    InputBuffer(name='arg0_1_1', layout=FixedLayout('cuda', torch.float32, size=[s0, s1, s0], stride=[s0*s1, s0, 1]))
  ))
  args[1]: [TensorBox(StorageBox(
    InputBuffer(name='unsqueeze_default_1', layout=FixedLayout('cuda', torch.int64, size=[s0, 1], stride=[1, 1]))
  )), TensorBox(StorageBox(
    InputBuffer(name='cat_default_1_1', layout=FixedLayout('cuda', torch.int64, size=[s0, s2], stride=[s2, 1]))
  ))]

for this.

Minimal repro here:

from torchinductor.compile_fx import compile_fx_inner
from torch.fx.experimental.proxy_tensor import make_fx

inps = [(torch.Size([4, 359613, 4]), torch.float32), (torch.Size([4, 5000]), torch.int64), (torch.Size([4, 1]), torch.int64), (torch.Size([4, 5000]), torch.float32)]
inps = [torch.zeros(())] + [torch.ones(shape, dtype=dtype, device='cuda') for (shape, dtype) in inps]



def forward(self, arg0_1, cat_default_1, unsqueeze_default, index_tensor):
    index_tensor_2 = torch.ops.aten.index.Tensor(arg0_1, [unsqueeze_default, cat_default_1]);  arg0_1 = unsqueeze_default = cat_default_1 = None
    return []
forward = make_fx(forward)(*inps)
compile_fx_inner(forward, inps)

[Chillee]

New error

  File "<eval_with_key>.734", line 7, in forward
    to_dtype = ops.to_dtype(load, torch.float16);  load = None
  File "/scratch/chilli/fresh/torchdynamo/torchinductor/codegen/common.py", line 579, in inner
    self.compute, getattr(parent_handler, name)(*args, **kwargs)
  File "/scratch/chilli/fresh/torchdynamo/torchinductor/codegen/cpp.py", line 157, in to_dtype
    assert dtype in DTYPE_TO_CPP, f"{dtype} missing from {__name__}.DTYPE_TO_CPP"
AssertionError: torch.float16 missing from torchinductor.codegen.cpp.DTYPE_TO_CPP

Running minifier now...

[Chillee]

2 minimal repros:

import torch
from torch import tensor, device
import torch.fx as fx
from torchdynamo.testing import rand_strided
from math import inf
from torchinductor.compile_fx import compile_fx_inner
from torch.fx.experimental.proxy_tensor import make_fx

inps = [(torch.Size([]), torch.int64, 'cuda'), (torch.Size([37]), torch.int32, 'cuda'), (torch.Size([1406, 1]), torch.int32, 'cuda'), (torch.Size([1406, 1]), torch.int32, 'cuda'), (torch.Size([1406, 1]), torch.int32, 'cuda')]
inps = [torch.zeros(())] + [torch.ones(shape, dtype=dtype, device='cuda') for (shape, dtype, device) in inps]



def forward(self, fill__scalar_6, arange_7, unsqueeze_default_12, unsqueeze_default_13, unsqueeze_default_14):
    mul_tensor_7 = torch.ops.aten.mul.Tensor(arange_7, fill__scalar_6);  arange_7 = fill__scalar_6 = None
    view_default_15 = torch.ops.aten.view.default(mul_tensor_7, [-1, 1]);  mul_tensor_7 = None
    expand_default_6 = torch.ops.aten.expand.default(view_default_15, [37, 38]);  view_default_15 = None
    clone_default_7 = torch.ops.aten.clone.default(expand_default_6, memory_format = torch.contiguous_format);  expand_default_6 = None
    _unsafe_view_default_7 = torch.ops.aten._unsafe_view.default(clone_default_7, [1406]);  clone_default_7 = None
    unsqueeze_default_15 = torch.ops.aten.unsqueeze.default(_unsafe_view_default_7, 1);  _unsafe_view_default_7 = None
    cat_default_3 = torch.ops.aten.cat.default([unsqueeze_default_12, unsqueeze_default_13, unsqueeze_default_14, unsqueeze_default_15], 1);  unsqueeze_default_12 = unsqueeze_default_13 = unsqueeze_default_14 = unsqueeze_default_15 = None
    return (cat_default_3,)


compile_fx_inner(make_fx(forward)(*inps), inps)

This is not the one it's actually originally erroring with but was another one the minimizer found along the way.

  File "/scratch/chilli/fresh/torchdynamo/torchinductor/lowering.py", line 613, in cat
    return TensorBox(ir.ConcatKernel.create(inputs, dim))
  File "/scratch/chilli/fresh/torchdynamo/torchinductor/ir.py", line 1933, in create
    assert inputs[i].get_dtype() == dtype
AssertionError

[Chillee]

Ok the other failure is no mystery. Somehow, one of the inputs to the graph is a CPU tensor. Something like this

from torchinductor.compile_fx import compile_fx_inner
from torch.fx.experimental.proxy_tensor import make_fx

inps = [(torch.Size([3, 4]), torch.float32, 'cpu'), (torch.Size([4, 359613, 4]), torch.float16, 'cuda')]
inps = [torch.zeros(())] + [torch.ones(shape, dtype=dtype, device=device) for (shape, dtype, device) in inps]



def forward(self, arg15_1, view_default_40):
    _to_copy_default_4 = torch.ops.aten._to_copy.default(arg15_1, dtype = torch.float16, device = device(type='cuda', index=0));  arg15_1 = None
    return [view_default_40]

compile_fx_inner(make_fx(forward)(*inps), inps)

[ngimel]

cat is type promoting, whereas lowering asserts that all the inputs to cat are the same type. It's interesting though that maskrcnn would actually use type promoting cat, it's a pretty old model from the time when cat didn't type promote. Are we sure something else is not going wrong, and the inputs were actually all supposed to be of the same type?

[desertfire]

The latest error:

Traceback (most recent call last):
  File "/fsx/users/binbao/torchdynamo-tip/benchmarks/torchbench.py", line 355, in <module>
    main(TorchBenchmarkRunner(), original_dir)
  File "/fsx/users/binbao/torchdynamo-tip/benchmarks/common.py", line 2001, in main
    runner.run_one_model(
  File "/fsx/users/binbao/torchdynamo-tip/benchmarks/common.py", line 1388, in run_one_model
    status = self.check_accuracy(
  File "/fsx/users/binbao/torchdynamo-tip/benchmarks/common.py", line 1245, in check_accuracy
    if not same(
  File "/fsx/users/binbao/torchdynamo-tip/torchdynamo/utils.py", line 720, in same
    return len(ref) == len(res) and all(
  File "/fsx/users/binbao/torchdynamo-tip/torchdynamo/utils.py", line 721, in <genexpr>
    same(ai, bi, fp64_refi, cos_similarity, tol, equal_nan, exact_dtype)
  File "/fsx/users/binbao/torchdynamo-tip/torchdynamo/utils.py", line 731, in same
    same(
  File "/fsx/users/binbao/torchdynamo-tip/torchdynamo/utils.py", line 768, in same
    if torch.allclose(ref, res, atol=tol, rtol=tol, equal_nan=equal_nan):
RuntimeError: The size of tensor a (36) must match the size of tensor b (34) at non-singleton dimension 0

Repro:

benchmarks/torchbench.py -d cuda --float32 --inductor --no-skip --accuracy --only vision_maskrcnn

[mlazos]

Small update - this is actually an accuracy issue, basically the number of valid boxes being returned is incorrect because the box scores are wrong. Continuing to look into it