OutOfMemoryError: CUDA out of memory

[Sicmatr1x]

  File "C:\Users\sicma/.cache\huggingface\modules\transformers_modules\local\modeling_chatglm.py", line 277, in attention_fn
    attention_scores = attention_scores * query_key_layer_scaling_coeff
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 116.00 MiB (GPU 0; 6.00 GiB total capacity; 5.07 GiB already allocated; 0 bytes free; 5.31 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

My card is NVIDIA GeForce RTX2060, memory size is 6144MB

[josStorer]

Attempt to close some gpu memory-consuming programs. For chatglm, 6GB of memory is just enough to run

[Terrency]

您好， 进行多轮对话的时候，会出现显存不足的情况，请问main.py上有没有好的办法限制呢？
当多轮对话的内容超过一定的长度，程序就会崩溃掉，需要重启

[josStorer]

@Terrency 我正在开发RWKV-Runner 以获得最佳的体验, 避免爆显存, 并适配2GB-20GB各规模显存, 这个模型可商用, 灵活性强, 模型结构方面也比较有潜力, 大概一周内会出一个初版

[Terrency]

@josStorer 你咋这么牛逼呢，期待最新作品， 我就为了搭建以后团队内部能查查资料。

[josStorer]

@Terrency https://github.com/josStorer/RWKV-Runner
@Sicmatr1x