Usage report of the efficient_conv_bn_eval feature

[youkaichao]

Please report here how enabling efficient_conv_bn_eval feature helps your training. We expect the feature to seamlessly reduce memory footprint without affecting the performance. Currently it is an optional feature, but if the feedback is good enough, we can turn this feature into default :)

[JolyonWu]

I used the maskrcnn model as a baseline to test the difference of turning on this function. When the model first started training, it could indeed save 13% of the memory. But as the training continued, I used nvidia-smi to check the memory. I found that the memory continued to increase from 7GB. It soared until it occupied 24GB, but the memory usage in the training log record remained as low as it was at the beginning

[youkaichao]

Hi, numbers shown in nvidia-smi is known to be unreliable. Please refer to the documentation of pytorch https://pytorch.org/docs/stable/notes/cuda.html#memory-management for details. In your case, I suppose there are some memory cache showing up in nvidia-smi. The log record might be a more faithful number for memory footprint.

[JolyonWu]

Thanks for your prompt reply, I'll check again!

[youkaichao]

@buxuewushu1314 for example, please check the memory field at https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r101_fpn_2x_coco/faster_rcnn_r101_fpn_2x_coco_20200504_210455.log.json .

[buxuewushu1314]

Train resnet18 on mmpretrain-v1.0.2 using '-- cfg options effective'_ Conv_ Bn_ After eval="[backbone]" ', there is almost no savings memory. Do I need to specify a version for mmretrain, or do I need any other settings?

[youkaichao]

Can you show your full training config? e.g. which task you are training on, and the config file you are using.

In general, pre-training uses BN in train mode, where efficient_conv_bn_eval cannot help. As the name suggests, it works when you train with BN in eval mode, which often occurs in mmdetection training.

[buxuewushu1314]

Thanks for your reply，yeah，it should be train mode，so does mmpretrain provide an interface for setting BN in eval mode, how should to do?

[youkaichao]

Generally we don't pretrain in eval mode. Pretraining requires train mode.

[buxuewushu1314]

Thank you again. classification model was still effective during fine-tuning by setting 'norm_eval', testing shows that it can save 33.8% memory,good job!

[youkaichao]

Glad it helps :)

[Jano-rm-rf]

再次感谢你。通过设置norm_eval，分类模型在微调时仍然有效，测试表明可以节省33.8%的内存，干得好！

Thank you again. classification model was still effective during fine-tuning by setting 'norm_eval', testing shows that it can save 33.8% memory,good job!

Hello, I have set "norm_val", but there is an error：torch.fx.proxy.TraceError: symbolically traced variables cannot be used as inputs to control flow,how should to do?

[Jano-rm-rf]

https://github.com/Jano-rm-rf/Test/blob/main/DINO-Test6-Config.py
command: python tools/train.py configs/dino/DINO-Test6.py --cfg-options efficient_conv_bn_eval="[backbone]"

[Jano-rm-rf]

code running environment：torch2.0.1, mmdet3.2.0,mmengine0.9.0
Is version incompatibility cause the error？

[hvlgo]

This is not a problem caused by version incompatibility. This is due to the current implementation of efficient_conv_ bn_eval feature is based on the symbolic trace in torch.fx. Due to the limited ability of symbolic trace, it is not possible to handle expressions that are only known at runtime or depend on parameters such as values or shapes of input. In the future, we will use torch.compile to implement this feature, and there will be no such issue at that time. If you want to learn more about symbolic trace, you can check this link https://pytorch.org/docs/stable/fx.html#limitations-of-symbolic-tracing or https://zhuanlan.zhihu.com/p/644590863.

[hvlgo]

The specific reason for this issue is that your config uses eficientnetv2, and its initialization specifies the conv type as Conv2dAdaptivePadding (https://github.com/open-mmlab/mmpretrain/blob/main/mmpretrain/models/backbones/efficientnet_v2.py#L186). The forward function of Conv2dAdaptivePadding contains condition judgment of pad_h calculated based on x.size() (https://github.com/open-mmlab/mmcv/blob/main/mmcv/cnn/bricks/conv2d_adaptive_padding.py#L58).

[Jano-rm-rf]

Thanks for your reply, I will modify conv type and test it

[Ztreasure]

Can this method still work if the batch_size is set to 1 during training?

[youkaichao]

As long as your BN is in eval mode during training, this method should work. Nevertheless, smaller batchsize may lead to less benefit in memory reduction.

[Ztreasure]

I understand, thanks for your reply!

[makecent]

Tested on the SlowOnly model (torch.hub.load("facebookresearch/pytorchvideo", model='slow_r50', pretrained=True). It reduced the memory from 4576Mb to 2623Mb:

[youkaichao]

Glad it helps :)