Card 0 OOM during block inference

[xin3he]

With this PR, #981
got OOM when quantizing block 11/61 for DeepSeek on CUDA 3x 80GB cards
got OOM when quantizing block 14/80 for llama 70b on Intel GPU 3x 24GB cards

[xin3he]

@wenhuach21 With clear_memory in each iteration, we had fixed the issue.
Could we close it?