You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I believe adding the metal cpp files from "LLaMACpp" repository to src and some small changes on "extconf.rb" should be sufficient enough to build the native extension with metal support in case anyone runs sidekiq queues on mac devices to generate the responses 10x faster.
I couldn't manage to build the gem even without changing the code. I always get "failed build native extensions" with .o files are not found error.
I will try to create a pull request if I can figure out why I get ".o" files are not found error.
Just wanted to bring it up if it's a 5-minute thing to add from your end since you created this gem. I can do my best to help with testing or wherever you need help with.
The text was updated successfully, but these errors were encountered:
Hi yoshuku, it works great. Thank you, it's not 10x but it's definitely 3x faster. It went down from 200-240 seconds of processing time for my long prompts to 40-75 seconds.
I believe adding the metal cpp files from "LLaMACpp" repository to src and some small changes on "extconf.rb" should be sufficient enough to build the native extension with metal support in case anyone runs sidekiq queues on mac devices to generate the responses 10x faster.
https://github.com/ggerganov/llama.cpp
I couldn't manage to build the gem even without changing the code. I always get "failed build native extensions" with .o files are not found error.
I will try to create a pull request if I can figure out why I get ".o" files are not found error.
Just wanted to bring it up if it's a 5-minute thing to add from your end since you created this gem. I can do my best to help with testing or wherever you need help with.
The text was updated successfully, but these errors were encountered: