Adding LLM(Large Language Model) support using Ollama

[JamesClarke7283]

Also, we could add Large language models to the application. Starting with smaller models. And adding bigger ones over time.
This could be really helpful, as LLM's that are open source that replace ChatGPT, are getting more powerful every day.
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

What are your thoughts on this?
Is this feasible?

[mkiol]

Thank you for interesting ideas.

Actually I was just thinking about adding "text processing" features like punctuation correction, smart spell-checker and more...

Expanding backlog for:

• tool for summarizing the text
• (maybe) text completion with LLMs

[JamesClarke7283]

I was thinking we could do something like what Chatbox does, where we can have conversations with a LLM, and do it using Ollama, that way its easy to pull the models, you just get them from the ollama registry. The majour benefit of doing this, is you could embed TTS/STT/MT functionality for the LLM, which would be pretty cool. ive never seen TTS/STT/MT(Machine Translation) functionality at the same time for a Open Source software that uses LLMs in ollama.

We could call the tab, "Chat".

For more on Ollama, check this out:
https://ollama.com/

[mkiol]

I was thinking we could do something like what Chatbox does

It is quite advance and impressive. This app works offline?

We could call the tab, "Chat".

LLMs and new tab are great ideas but I'm not super convinced to add "Chat" functionality. I understand that "chat" interface is natural and useful but there so many other apps that already going in this direction. I'm not sure that Speech Note can bring any new value in this space. On the other hand, LLMs have much bigger potential that just "chat". I've already made an early prototype of text conversions like grammar error correction or text summation with Llama2 and Mistral models. It worked really well and the speed was decent.

Maybe this new tab could be just for Text-to-Text processing. This may be a major feature in version 5.0 🤔

functionality at the same time for a Open Source software that uses LLMs in ollama

Ollama is great indeed. It uses llama.cpp library that is highly optimized and can run LLMs efficiently on both CPU and GPU. Actually, Ollama and llama.cpp are the reason I am optimistic about LLMs in general. We can do something useful with LLMs without losing our privacy 😃

[JamesClarke7283]

I was thinking we could do something like what Chatbox does

It is quite advance and impressive. This app works offline?

We could call the tab, "Chat".

LLMs and new tab are great ideas but I'm not super convinced to add "Chat" functionality. I understand that "chat" interface is natural and useful but there so many other apps that already going in this direction. I'm not sure that Speech Note can bring any new value in this space. On the other hand, LLMs have much bigger potential that just "chat". I've already made an early prototype of text conversions like grammar error correction or text summation with Llama2 and Mistral models. It worked really well and the speed was decent.

Maybe this new tab could be just for Text-to-Text processing. This may be a major feature in version 5.0 🤔

functionality at the same time for a Open Source software that uses LLMs in ollama

Ollama is great indeed. It uses llama.cpp library that is highly optimized and can run LLMs efficiently on both CPU and GPU. Actually, Ollama and llama.cpp are the reason I am optimistic about LLMs in general. We can do something useful with LLMs without losing our privacy 😃

Yea its great!
I think chat functionality would be great, because you already have speech to text and text to speech, so you can make LLMs more accessible than ever if we integrated this.

I think "Chat" functionality in Speech Note with Ollama, would be one of the most transformative things you could add.
reason being:

1. We could have seemless conversations with a LLM via speech, potentially (Only the ChatGPT app seems to do this and its proprietary).
2. Make a voice for the AI, with text to speech you can make it easy to read out the text of the conversation, i also dont see any open source tech, going in that direction.
3. Swappability of Speech/Voice models, one thing ive never seen, is being able to swap out the Speech/Voice models (already a capability in Speech Note) but with use of a Open Source LLM, this has not been done before in a way that a GUI exists for it.

Those are my reasons, i would really like to see Speech Note also be a way to use LLMs. Currently i use Chatbox which does not enable speech/voice model use inside it, and when it does implement it, i doubt it will be as flexible as in speech note.