GPU Memory Requirements

[gunpal5]

I want to know how much GPU memory would it need to run Yi-6b-200k and Yi-34b-200k for production environment?

I tried to run Yi-6b and Yi-6b-200k on 12GB GPU locally, but it is giving me out of memory exception.

[findmyway]

Several parameters might affect the required GPU memory. So you should try it out yourself with your typical use cases.