Is your feature request related to a problem? Please describe.
For local inference, partial offload is required for many models of reasonable quality. ik_llama.cpp is a fork of llama.cpp that has additional support for more modern and efficient quants and significantly better performance on modern hardware. I have personally been able to run significantly larger quants (2x size) at significantly better speeds (1.5x inference) using ik_llama, which completely redefines the capabilities of my local AI inference. LocalAI would become much more usable for local inference in my setup with the addition of an ik_llama cpp backend.
Describe the solution you'd like
Addition of ik_llama.cpp as a selectable backend.
Describe alternatives you've considered
Considered using OobaBooga text generation webui but it's clearly not built for the same use case, can't run multiple models, and doesn't support auto load/unload. LocalAI is a true workhorse for multi-purpose local AI inference.
Additional context
N/A
Is your feature request related to a problem? Please describe.
For local inference, partial offload is required for many models of reasonable quality. ik_llama.cpp is a fork of llama.cpp that has additional support for more modern and efficient quants and significantly better performance on modern hardware. I have personally been able to run significantly larger quants (2x size) at significantly better speeds (1.5x inference) using ik_llama, which completely redefines the capabilities of my local AI inference. LocalAI would become much more usable for local inference in my setup with the addition of an ik_llama cpp backend.
Describe the solution you'd like
Addition of ik_llama.cpp as a selectable backend.
Describe alternatives you've considered
Considered using OobaBooga text generation webui but it's clearly not built for the same use case, can't run multiple models, and doesn't support auto load/unload. LocalAI is a true workhorse for multi-purpose local AI inference.
Additional context
N/A