-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature request]: LocalLLM(ollama) support #178
Comments
It has that support. Ollama and Claude. —model. For a list of models run —listmodels. If you have ollama running on your machine it will list the ollama models you have running. |
i have ollama but not on this machine(its remote). i am looking to configure it as custom endpoint. |
I don’t have a custom endpoint coded in yet. I was looking through the ollama python library and I couldn’t find a way to do it. If anyone knows how to do this I’ll add it |
check this out side quest. integration as tools for crewai to use fabric |
I believe ollama support needs some refining:
Some enhancements:
I used ollama v0.1.28, latest commit of fabric. Ollama has multiple models pulled:
Fabric reports these models:
Looking forward for the next drop, |
fixed. try it now without an api key and let me know if it works |
Still does not work, additionally fabric does not distinguish between the variants of models (mistral for example). Please see below's output
To fix the output of the model regarding ollama line 298 needs to be modified on utils to say
|
Ok I think I got it working with ollama models. Please see the below code from my working utils.py
|
Thanks! good looking out. I fixed it. Make sure it works for you and let me know |
it can use different Ollama location (not just the default localhost and port) if we change ... i try to integrate Fabric with Slack as a Bot, using local Ollama with local LLM in different docker containers |
Well currently I am using the ollama python library, and I can't find a way to change the default ollama url in that library in the documentation. If you can figure that out I'll incorporate it |
https://github.com/ollama/ollama-python/blob/main/ollama/_client.py
"host" is an optional paramteter of AsyncClient |
OK I added it as --remoteOllamaServer. Please test it and make sure it works and I didn't break anything :) |
Seems it works. My steps are as below:
|
the update was very fast. thank you a few improvement notes
and so on.... and a System wide env variable for ~/fabric.conf file location, perhaps user needs a different path for it, can setup like this
fabric automatically look for this file and read it as ollama endpointsin terminal calling fabric like this
if s1 or --s1 is not called it should default to local ollama or GPT4 etc... and one last thing i think this is the best way to approach this, what do you guys think |
thank you for the fast update |
Something not work, what am I missed? But i cannot see my local models: Local Models: Claude Models: I guess instead of this: |
to be openai compatible endpoint on ollama it still needs key try this in your curl request question for the dev. |
the openai package |
I run Fabric on windows WSL, and run Ollama on native windows. "> fabric --listmodel" don't seem to list any models from ollama. How do I setup Fabric to access LLM loaded via Ollama in this case? 127.0.0.1 doesn't seem to work, as wsl and navtive windows seem to act as two different system/network. thanks, |
run ollama on WSL much more stable, windows version is buggy, |
I am sure running both ollama and fabric on WSL would work out. I am also running Open/Ollama WebUI on native windows via docker desktop. I am not sure the dockered-webui would work with WSL-ollama. I will just stay with OAI for fabric now, until the fabric supports native windows. |
Since Ollama is OAI compatible and there are tons of OAI-compatible API-services out there, like groq, together, etc. as @rony432 has suggested:
one could predefine models by usecases, i.e. |
Hello, Firstly, congratulation for this project that seems to be very interessting. Unfortunatly I can't make it work with Ollama. I have a local Ollama service with Gemma that I can curl with no issue :
But when I try to configure fabric :
It ask me for OpenAI, Claude and YT API but did nothing more :
If I then ask for model list :
Even with this command :
In fact every request send me back to the setup :
There may certainly be something I didn't do right, but I have no clue for the moment. |
Just run --setup first without the remote server flag. Fabric --setup". If you don't want to use OpenAI or Claude just click enter. Once you do that that message will go away |
I should have mention that I did that first.
|
I have exactly the same problem as @Papoulos. After
Skipping keys for OpenAI, Claude and YB, I tried the following:
Also:
A curl request from another console works fine:
Incidentally, it would be nice to have an option to skip updating patterns when we use |
I try the same command as you did but still got the message for configuration :
|
I'm also running both fabric (latest from git commit c3df1e7 ) - and ollama 0.1.29 on Windows 11:
And it's working correctly:
And here is the local model respnding:
However, fabric still does not see the local models:
|
I'm really wondering if you guys just aren't missing the .config/fabric/.env file & fabric isn't setup to access Window's user file path as its Mac/Linux based. From what I understant, the .config/fabric dir gets created when you run that setup stuff but as Windows doesn't recognize This is all just a theory though! I did see the author of fabric say Mac/Linux/WSL are the only officially supported distributions atm. I hope I'm wrong though. ;) |
@Gerkinfeltser if you see in the The
|
Ah, kk. Well glad that works then. Sorry for the misdirection. Edit: I just attempted to run fabric from Windows & I do seem to be able to access both the wsl ollama install & the windows one. I wonder if there's a way to get more debug-ish info concerning ollama & fabric on your systems. |
@Gerkinfeltser What version of Windows? What version of Python? Here's what happens in WSL for me:
And in Windows native, basically the same thing, except I don't specify the alternate Ollama server, since it's on localhost:11434 |
I'm windows 10 & python 3.11.5. I'm able to access:
I'm unable to access:
FWIW I've never had to specify Let me know if you want me to check anything else. |
@Gerkinfeltser let ollama bind to all interfaces by setting its server address to 0.0.0.0 and you'll be able to access Windows ollama from WSL by specifying the gateway address, like in my WSL example above. See this discussion: ollama/ollama#703 I'm starting to think this might be a Windows firewall issue. I'll look into it more later this afternoon. |
Just chiming in to note that I am also seeing the same issue. Ollama locally works great. Remote Ollama doesn't appear to function. During setup it never queries the remote Ollama server. During a pattern run it will ask GPT to use the specified model. |
@systerchristian I have some fixes. Stay tuned. The crux of the issue appears to be that as mentioned in Ollama docs, I set This causes the
And then:
|
Doens't Ollama provide an OpenAI compatible API? Wouldn't is just be needed to change the export OPENAI_BASE_URL=http://127.0.0.1:5000/v1/ to something like this? I'm using textgen webui using this method and it works great. |
I'm getting this error
Does anyone know what I'm doing wrong? I have these under
|
Try with "OPENAI_BASE_URL=https://127.0.0.1:11434/v1/" |
@JaySurplus it worked! 🎉 🥳 |
What do you need?
this is awesome project, but it needs ollama support.
the OpenAI api is the ease way out.
please add support for local LLMs too.
thank you
The text was updated successfully, but these errors were encountered: