LFX Mentorship (Mar-May, 2024): Integrate whisper.cpp as a new WASI-NN backend

[hydai]

Summary

WasmEdge supports PyTorch, TensorFlow Lite, llama.cpp, and more NN backend. Dealing with the voice-to-text is a big thing that we want to achieve. To make it possible, we would like to integrate whisper.cpp, a port of OpenAI's Whisper model in C/C++ as a new WASI-NN backend.

Details

• Pre-test: To apply for this mentorship, you MUST finish a pre-test. Pre-test link: LFX Mentorship 2024-01 Pre-test #3182
• Expected Outcome: A new plugin provides a whisper.cpp WASI-NN backend, a test suite for validating the plugin, documents, and examples for explaining how to use the plugin.
• Recommended Skills: C++, Wasm
• Mentor(s):
  • Hung-Ying Tai (@hydai)
  • dm4 (@dm4)

Appendix

1. whisper.cpp: https://github.com/ggerganov/whisper.cpp
2. WASI-NN: https://github.com/second-state/wasmedge-wasi-nn
3. WASI-NN llama.cpp backend(check ggml.h and ggml.cpp): https://github.com/WasmEdge/WasmEdge/tree/master/plugins/wasi_nn

[Selomn]

Hello, sir. I have two questions. Could the project be done in Rust? And when is the estimated time for the pre-test?

[hydai]

Could the project be done in Rust?

No. Unless you can prove that there is a Rust version that provides a faster and better performance solution than the whisper.cpp project.

And when is the estimated time for the pre-test?

We will announce the pre-test when the application opens.

[Selomn]

Thank you sir.

[octonawish-akcodes]

I am interested in this issue for the LFX mentorship. Looking forward to apply.

[hydai]

Because the previous mentee didn't finish this, there was no response to the second stage of the LFX mentorship. We decide to pick it up and implement it by our maintainer, @q82419

[q82419]

Replaced by the roadmap issue #3538 .