aot_autograd error: `argument 'pad' must be tuple of ints, but found element of type Tensor`

[jansel]

I see this same error in many detectron2 benchmarks.

Works with --backend=eager

$ ./benchmarks/torchbench.py --backend=eager --only detectron2_fcos_r_50_fpn                                                                                                    
cuda eval  detectron2_fcos_r_50_fpn           0.914x p=0.00

Error with --backend=aot_nop

$ ./benchmarks/torchbench.py --backend=aot_nop --only detectron2_fcos_r_50_fpn
cuda eval  detectron2_fcos_r_50_fpn           ERROR:root:unhandled error
Traceback (most recent call last):
  File "/home/jansel/torchdynamo/benchmarks/common.py", line 1137, in run_one_model
    new_result = optimized_model_iter_fn(model, example_inputs)
  File "/home/jansel/torchdynamo/torchdynamo/eval_frame.py", line 153, in _fn
    return fn(*args, **kwargs)
  File "./benchmarks/torchbench.py", line 370, in forward_pass
    def forward_pass(self, mod, inputs, collect_outputs=True):
  File "/home/jansel/pytorch/torch/nn/modules/module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jansel/conda/envs/torchdynamo/lib/python3.8/site-packages/detectron2/modeling/meta_arch/dense_detector.py", line 72, in forward
    def forward(self, batched_inputs: List[Dict[str, Tensor]]):
  File "/home/jansel/torchdynamo/torchdynamo/eval_frame.py", line 153, in _fn
    return fn(*args, **kwargs)
  File "/home/jansel/pytorch/functorch/functorch/_src/aot_autograd.py", line 880, in forward
    return compiled_f(
  File "/home/jansel/pytorch/functorch/functorch/_src/aot_autograd.py", line 866, in new_func
    compiled_fn = create_aot_dispatcher_function(
  File "/home/jansel/pytorch/functorch/functorch/_src/aot_autograd.py", line 444, in create_aot_dispatcher_function
    return aot_dispatch_base(flat_fn, fake_flat_tensor_args, aot_config)
  File "/home/jansel/pytorch/functorch/functorch/_src/aot_autograd.py", line 268, in aot_dispatch_base
    fw_module = make_fx(flat_fn, aot_config.decompositions)(*flat_args)
  File "/home/jansel/pytorch/torch/fx/experimental/proxy_tensor.py", line 618, in wrapped
    t = dispatch_trace(wrap_key(func, args, fx_tracer), tracer=fx_tracer, concrete_args=tuple(phs))
  File "/home/jansel/torchdynamo/torchdynamo/eval_frame.py", line 153, in _fn
    return fn(*args, **kwargs)
  File "/home/jansel/pytorch/torch/fx/experimental/proxy_tensor.py", line 318, in dispatch_trace
    graph = tracer.trace(root, concrete_args)
  File "/home/jansel/torchdynamo/torchdynamo/eval_frame.py", line 153, in _fn
    return fn(*args, **kwargs)
  File "/home/jansel/pytorch/torch/fx/_symbolic_trace.py", line 739, in trace
    (self.create_arg(fn(*args)),),
  File "/home/jansel/pytorch/torch/fx/experimental/proxy_tensor.py", line 332, in wrapped
    out = f(*tensors)
  File "<string>", line 1, in <lambda>
  File "/home/jansel/pytorch/functorch/functorch/_src/aot_autograd.py", line 829, in functional_call
    out = Interpreter(mod).run(*args[params_len:], **kwargs)
  File "/home/jansel/pytorch/torch/fx/interpreter.py", line 129, in run
    self.env[node] = self.run_node(node)
  File "/home/jansel/pytorch/torch/fx/interpreter.py", line 162, in run_node
    return getattr(self, n.op)(n.target, args, kwargs)
  File "/home/jansel/pytorch/torch/fx/interpreter.py", line 234, in call_function
    return target(*args, **kwargs)
TypeError: pad(): argument 'pad' must be tuple of ints, but found element of type Tensor at pos 2
ERROR
pad(): argument 'pad' must be tuple of ints, but found element of type Tensor at pos 2

[tanaymeh]

Hi! I would like to work on this but not sure where to start since this is my first contribution to PyTorch. Any guidance would be highly appreciated!

[jansel]

@anijain2305 @Chillee any pointers on how to get started on this one?

[anijain2305]

Give me a few hours. I am going to see if minifier can give a starting point.

[anijain2305]

Here is a repro
repro.tar.gz

However, it fails with a different error

  File "/scratch/anijain/work/pytorch/torch/fx/interpreter.py", line 256, in call_method
    return getattr(self_obj, target)(*args_tail, **kwargs)
TypeError: new_full(): argument 'size' must be tuple of ints, but found element of type Tensor at pos 3

cc @Chillee This could be something related to factory functions?

[anijain2305]

The error has changed from the time the issue was created

Traceback (most recent call last):
  File "/scratch/anijain/work/torchdynamo/torchdynamo/variables/tensor.py", line 77, in run_proxy
    return getattr(args[0], proxy.node.target)(*args[1:], **kwargs)
TypeError: new_full(): argument 'size' must be tuple of ints, but found element of type FakeTensor at pos 3

Repro is

import torch
import torchdynamo


max_size = torch.tensor([800, 1216], dtype=torch.int64)

def fn(tensors):
    batch_shape = [len(tensors)] + list(tensors[0].shape[:-2]) + list(max_size)
    batched_imgs = tensors[0].new_full(batch_shape, 0.0)
    return batched_imgs


a = torch.randn(3, 800, 1199)
b = torch.randn(3, 800, 800)
inputs = [a, b]

ref = fn(inputs)

opt_fn = torchdynamo.optimize("eager")(fn)
res = opt_fn(inputs)

NOTE that this fake tensor propagation in TorchDynamo issue (and not fake tensor issue in Aot Autograd)

cc @eellison @ezyang