Generation process very slow!

[walkingwithGod2017]

Hello, thank you for your excellent research work.
The results of this repo sound very impressive. The only problem is that it consumes a large amount of GPU memory—generating text with 200 characters already takes up 11GB of VRAM, which makes the generation process very slow. Is there any way to solve this problem?
Looking forward to your reply. ^^ ^^

[weedge]

ai-bot-pro/achatbot#140

[jiang-ziyue]

@walkingwithGod2017
You may consider installing FlashAttention to reduce GPU memory usage and speed up inference, or integrate vLLM with the G2P model for better performance. Our project prioritizes maximal simplicity, and there remains significant room for engineering optimizations.

[walkingwithGod2017]

@walkingwithGod2017 You may consider installing FlashAttention to reduce GPU memory usage and speed up inference, or integrate vLLM with the G2P model for better performance. Our project prioritizes maximal simplicity, and there remains significant room for engineering optimizations.

OK, I'll try it, thank you for your kind reply. ^^ ^^