finetune训练时，内存会随着训练步数增多而增大，直到占满内存报错 Cannot allocate memory 

Notice: In order to resolve issues more efficiently, please raise issue following the template.
（注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充细节）

## 🐛 Bug

[rank2]: OSError: [Errno 12] Cannot allocate memory 

### To Reproduce

examples/industrial_data_pretraining/paraformer/finetune.sh

### Additional context

finetune训练时，内存会随着训练步数增多而增大，又可能是哪里的内存占用后没有释放吗
