Multiple issues training a conv model with torchinductor

[anj-s]

I ran a simple E2E example with a conv model + MNIST data and these are the errors I ran into:

1. TensorBoard profiling does not work
   --> works with nvfuser and without the torchdynamo annotation
2. TORCHINDUCTOR_TRACE = 1 raises an error
   --> has trouble writing the pre fusion IR graph for some reason. i did turn on graph writing manually in the config file.
3. max_pool2d_with_indices_backward and max_pool2d_with_indices lowering throw an assertion error in len(strides) == 2.
   --> we expect strides to be None or a value but it can be an empty list which we don't check for.
   See paste for full repro, stack trace and other debugging info. If you don't have access to the repo, here is the code

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym

[voznesenskym]

Thank you, I can take a look at these later this week.

CUDA error: an illegal memory access was encountered

Thats probably unrelated to tensorboard. I can look.

cannot determine truth value of Relational

Can you get latest dynamo? A whole bunch of these were recently fixed.

max_pool2d_with_indices_backward
assert len(stride) == 2

Maybe Fixed in pytorch/torchdynamo#1211 ?

Code:https://github.com/fairinternal/fairscale_examples/blob/main/ptd/transformer_inductor.py

I don't have access (404)

[ngimel]

1. That's unfortunately a long standing bug, kineto profiling doesn't work with cudagraphs https://fb.workplace.com/groups/1144215345733672/permalink/2303849063103622, can you try disabling cudagraphs and see if it's fixed?

[anj-s]

Thank you, I can take a look at these later this week.
Sounds good! Its not blocking me but figured I'd report them here anyway.

CUDA error: an illegal memory access was encountered

Thats probably unrelated to tensorboard. I can look.
Yeah, works with nvfuser backend.

cannot determine truth value of Relational

Can you get latest dynamo? A whole bunch of these were recently fixed.

max_pool2d_with_indices_backward
assert len(stride) == 2

Maybe Fixed in pytorch/torchdynamo#1211 ?
The paste has a context about this but essentially we don't expect strides to be an empty list.

Code:https://github.com/fairinternal/fairscale_examples/blob/main/ptd/transformer_inductor.py

I don't have access (404)
https://www.internalfb.com/phabricator/paste/view/P529090378