Slow performance on NVIDIA MX450 GPU - Possible to optimize or downsize the model for ordinary users?

[brynne8]

I am using an NVIDIA MX450 GPU on my laptop and was excited to try your selfhostedAI project. After installing GCC and CUDA 11, I managed to get it running. However, the performance is quite slow.

I am wondering if the model might be too large for my relatively modest laptop hardware. Is there any way to optimize the model or provide a smaller version that would be more suitable for ordinary users like myself?

[josStorer]

The recommended minimum video memory is 6GB, and for the MX450, CPU will be used, which may result in slower speed.

[josStorer]

Try RWKV-Runner https://github.com/josStorer/RWKV-Runner

[josStorer]

Minimum VRAM requirement is 2GB