Memory leak during CPU inference

[debasish-mihup]

I have trained efficient conformer transducer and during inference on a CPU in a flask based web app, I see there is a memory leak at

EfficientConformer/models/encoders.py

Line 128 in 2f59ed2

x, attention, hidden = block(x, mask)

x, attention, hidden = block(x, mask)

The memory used at the above line while running an inference is never released, thus causing OOM is some time. Increase in memory in each iteration gets lowered but not totally removed if I use jemalloc.

[debasish-mihup]

Using last release version of jemalloc 5.2.1 and using torch.no_grad() fixed the memory leak.

[burchim]

Hi,

Thanks for pointing this out!
The gradient of attention weights and hidden states aren't detached in attention layers.