MPS Support

[fakerybakery]

Hi,
Congrats on the launch!
Is MPS (Apple Silicon) or MLX support planned?
Thank you!

[sidroopdaska]

Hey @fakerybakery, can you share your use-case in more detail?
MPS / MLX support is not planned in the short-term.
But, we'd love to collaborate if you would like to implement this for our model?

[daankortenbach]

Great work on metavoice, sounds really good! Wish I could use it om my Mac.

Use case: all Apple Silicon (Mac, iPhone, iPad, Vision).
There is a huge lack of good tts for Apple and there is demand.
Take note of @fakerybakery, he's everywhere and doing good work.

[cocktailpeanut]

Is it difficult to implement MPS support? The use case is basically the entire Apple ecosystem users. I was going to write a cross platform installer for the app (I work on https://pinokio.computer) and ended up here, but now the fact that there's no plan for supporting MPS i'm kind of confused. I believe the true hockey stick growth for your model will come not from a small number of entrepreneurs who run a server and charge for service, but every single user with a laptop.

[fakerybakery]

I think a major issue is that this uses flash-attn, which is currently not supported on either MPS or ROCm.

[vatsalaggarwal]

yep @fakerybakery is right. But, a chunk of what's required to fix this is already implemented, it’s just not been hooked up properly. Will try to do it, but happy for someone to beat me to it!

Here’s the ref : https://github.com/metavoiceio/metavoice-src/blob/main/fam/llm/layers/attn.py

It contains code to mix and match the following options:

• flash decoding (which does kv-caching and attn calculations within a single optimised kernel, requires newer NVIDIA GPUs),
• fa2 + vanilla kv caching (which uses fa2 - an optimised attn kernel - for attn calculation but does kv caching using standard PyTorch ops. But due to fa2 it still requires newer NVIDIA GPUs),
• torch attn + vanilla kv caching (which does both attn calculation and kv caching using torch standard ops, so should work across the board)

So one needs to change some plumbing in the code iirc to use the third option and I believe it should work!

[cocktailpeanut]

That sounds awesome, and really glad to see that you guys care. Looking forward to the progress!

[sidroopdaska]

Thanks for participating. Can't wait to review a PR here :)

[groovybits]

+1 for this :) need this so bad on my Mac Ultra M2! Mimic3 TTS is just not great and OpenAI costs a bit much for 24/7 use.

I may try to see if I can figure it out with the "torch attn + vanilla kv caching" but not too hopeful of my luck with that :D Any details on what needs to be done may help me at least try, I just haven't done MPS / GPU support very much (hacked at getting bark.cpp working but was a kludge / not a full MPS dev experience).

Curious about where resources are to learn about that more, since I am really coding a lot and have this Mac I want to work with everything so might as well put some effort in on the MPS support like this if I can get the hang of it.

[pyetras]

Hey folks, I pushed a change that should fix the problems mentioned here, could you give it another go?

[ahmetkca]

Hey folks, I pushed a change that should fix the problems mentioned here, could you give it another go?

I tried to install requirements.txt but now getting ModuleNotFoundError: No module named 'torch'. I am on Mac apple silicon. Do I need to install torch seperatly?

[sidroopdaska]

Looking into this now :)

[ahmetkca]

Looking into this now :)

idk if this could help, facebookresearch/xformers#740 (comment)