Using mlx for apple silicon

[sandangel]

Is your feature request related to a problem? Please describe.

Currently we are using llama.cpp for apple silicon. I wonder if we can use mlx developed by Apple instead so it will get more performance and better ultilizing apple hardware?

Describe the solution you'd like

Use mlx to run inference on apple silicon

Describe alternatives you've considered

Additional context

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

I understand that you would like to use mlx instead of llama.cpp for running inference on apple silicon. This is a great idea and can potentially improve performance and utilization of Apple hardware.
To implement this, you will need to modify the code in your project to use mlx instead of llama.cpp. You may also need to update any dependencies or libraries that are used in your project to ensure they are compatible with mlx.
As for alternatives, you could consider using other frameworks or libraries such as TensorFlow or PyTorch, which also support running inference on apple silicon. However, these may have different APIs and require additional setup and configuration.
Overall, I recommend exploring the documentation and resources available for mlx to determine the best approach for integrating it into your project. If you have any specific questions or need further assistance, feel free to ask!

Sources:

• https://github.com/go-skynet/LocalAI/tree/master/pkg/grpc/proto/backend.pb.go
• https://github.com/go-skynet/LocalAI/tree/master/backend/cpp/llama/json.hpp

[JTZ18]

just curious if there's a online community of mlx enthusiasts looking to integrate mlx into different AI applications like Automatic1111, comfyUI, Foooocus, ollama etc. is there a discord / reddit community for mlx?

[qdrddr]

Would that be of any help the LM Studio has implemented MLX. And here is Anemll ANE library to work with MLX it is MIT Licensed.