BatchNorm training=True in some Timm classes

[yuanyao-nv]

Previously we resolved the issue (#1262) where instance norm in pytorch was decomposing into batch norm with training mode set to True.

I've also encountered some issues with BN set to True in certain timm classes. If you look in this file https://github.com/huggingface/pytorch-image-models/blob/main/timm/layers/std_conv.py, all calls to F.batch_norm() have training=True. Any thoughts on what would be the best solution here? I'm wondering if there's a way to always set training to False during tracing.

[justinchuby]

I wonder if it would make sense to replace when to false after we obtain the model?

[justinchuby]

@thiagocrepaldi

[yuanyao-nv]

I wonder if it would make sense to replace when to false after we obtain the model?

Yea I think someone brought up this point before: unless we anticipate use cases for training ops, onnx-rewriter can potentially just set all the training attributes it sees to False? But then that's assuming the user will always run the onnx-rewriter a posteriori.

[justinchuby]

I wonder why training=True in the first place? Was pytorch trying to create some kind of training behavior even when eval() was called on the model?

[BowenBao]

But then that's assuming the user will always run the onnx-rewriter a posteriori.

This can be integrated into exporter since it operates on standard onnx domains.

In onnx, training only affects whether bn emits extra outputs. cc @gramalingam I think that's our conclusion last time. So if these outputs are unused, it is safe to remove the outputs and flip the flag in the node.