❔ Semantic router on top of ollama 🦥

[adriens]

❔ About

I'd like to run semantic router fully locally with custom local functions.

Actually, this is possible (Local Dynamic Routes).

But currently, there is no dedicated Notebook that show how to do this (for example on mistral, openhermes or dolphin), except maybe using LangChain throught Intro LangChain Agents with Semantic Router.

🎯 Questions

• Would you recommand use LangChain to pass the llm built on top on ollama
• Would you share a dedicated Notebook that would put best practices available to community ?

🔖 Resources

• Llama.cpp for FULL LOCAL Semantic Router (James Briggs)
• 🦙 ollama running local models w/LlamaIndex (CPU)
• I'd like to port this agent demo (🦜🔗 Track Package Delivery w. Langchain Agent Custom Tools , based on Lanchain tools / gpt-3.5)
• "Semantic Router superfast decision layer for LLMs and AI agents" on Geeky Gadgets

[jamescalam]

Hey @adriens it is possible thanks to a recent PR from @bruvduroiu — you can follow the /docs/05-local-execution.ipynb notebook or this video walkthrough of the same.

Main 'best practice' advise I can give for getting started is to implement some routes and then begin chatting with your agent. When you see routes bit being activated when they should, add that exact phrase to the Route.utterances — and build you your routes with that type of iterative process

[adriens]

Thanks a lot for the quick feedback @jamescalam .
In the code you mention, we have to reference the modelfile itself.

What I would like to do from my code is just :

• assume ollama is running (ollama serve)
• the proper model has already be downloaded by ollama
• Build the llm on top of local ollama API

Is my approach correct ? From my point of view it would make local models even easier to intergrate (I mean as easy as openai's ones) 😅

[adriens]

• Fully local Semantic Router with llama.cpp and HuggingFace Encoder

[jamescalam]

hey @adriens I think this is now complete, you can find the example notebook for it here

[adriens]

I m going to give it à try and Will let you know

[mintisan]

Great!

[adriens]

Hey, sorry for very late delay on my feedback.

I'm currently testing it with success on ollama, running a gemma2:9b.

It's working and wonder a little thing :

I need to define question in french :

politics = Route(
    name="politics",
    utterances=[
        "isn't politics the best thing ever",
        "why don't you tell me about your political opinions",
        "don't you just love the president" "don't you just hate the president",
        "they're going to destroy this country!",
        "they will save the country!",
        "élection législatives de ce week-end ?"
    ],
)

if I don't do that (stay with current) :

politics = Route(
    name="politics",
    utterances=[
        "isn't politics the best thing ever",
        "why don't you tell me about your political opinions",
        "don't you just love the president" "don't you just hate the president",
        "they're going to destroy this country!",
        "they will save the country!"
    ],
)

It seems like it's not able to decide a route.

Any idea ?

I'll work on producing a concrete example within the next weeks, only on open source llms, and of course I will let you know. the outputs.

[adriens]

🎁 👇

• feat: 🎁 🔀 Semantic Router ollama/gemma2 hotline #346

[valonebot]

Nous venons de dev.to. Félicitations pour votre projet. Tous nos encouragements. Nous avons enregistré votre article.

[adriens]

Nous venons de dev.to. Félicitations pour votre projet. Tous nos encouragements. Nous avons enregistré votre article.

Merci pour le petit message bien sympa en français. C bien cool.🤩