Replies: 5 comments 10 replies
|
The context window will be too small when you want to run reasonable local LLM models. You need at least 64-128GB of VRAM or unified memory 🤷♂️ |
|
I've loaded it up in Windows, I have Ollama running against my poor little 10GB 3080 graphics card and hooked up Odysseus to it. I downloaded a few models that would reasonably fit in my VRAM and tried asking a few questions and it works!! it can answer basic stuff, but it can't seem to do any research or stuff like that? It just sort of stops with half of a sentence... So clearly not smart enough for the agentic stuff, but will it work at all with this dumb LLM hooked up? Maybe there is a real practical minimum amount of VRAM you need to be able to make this environment really functional in any way, like maybe a 40B param model is the minimum and I just can't run that... or maybe I am doing it wrong IDK... It seems to me that some of the agentic "Preload" could be very simple for some basic tasks, so maybe the tool itself can be tuned to work on lower spec hardware, just in a more limited capacity. |
|
Well, people here asking dum questions and testing and receiving semi good answers never pushed the app to its limits, or used it extensively. You need at least a 200k context for a resonable task, then you need to manage memory, skills, MCPs, that eat up context really fast. Then you need semi fast inference. These consumer GPUs with 10-12, 16GB of VRAM are tragically bad for these kind of workflows. Again, to use these 30b+ models in a normal way, you need at least 128GB of unified fast memory, or just use APIs, whatever. But local models need more memory and more bandwidth. Today, you need a good harness that can use both, local and frontier models in cooperation. This is the modus operandi for the next 1-2 years. Im planning to get a home Mac Studio with 128GB of unified ram (M5MAX) and tunnel requests to its local models from my laptop. |
|
I think at this early stage a really helpful resource for new users will be a guide for what models to configure on low-end hardware and what features will and won't work, depending on what context size you have and what model you're using. Then we can either guide users toward disabling features that don't currently work on their hardware, so that all enabled tools work effectively with what they currently have, or suggest the best value-for-money upgrades to add in a second GPU or more RAM to an existing machine to increase the size of models they are able to run effectively. The "Cookbook" feature already goes most of the way toward this. I think that is the area that needs the most work in terms of getting everything seamless and automated. What it really needs is a good set of baselines from lots of different real-world hardware configurations to have a decent chance of making useful suggestions and loading decent models as defaults, the first time a user runs the service. |
|
oh and I think "need" is relative. Most of the tasks we are asking of agents in Odysseus don't need frontier models. Obviously the larger the model the smarter it will be, a 128GB Mac Studio should be able to come close to older ChatGPT or Codex models in its capability, but it obviously won't be anywhere near as smart as Claude or even Grok. I think for more complex tasks like advanced coding or cloud infrastructure tasks then yeah you will either "need" a much larger model to assist you now and then, or a very good understanding already of what you are doing and the ability to break down a large task into smaller chunks and decide on strategy yourself. For tasks users are already familiar with and where we aren't really breaking new ground or inventing anything new, like editing documents / emails, managing our calendars, filtering out spam or low value emails, giving us advice about how to dress based on the weather forecast, and so on, I have already validated personally that this harness / framework performs at a very competent level on a single 3080 with 10gb, with limited coaching and prompting from me. I am using simple, default builds of Qwen, Llama and Deepseek variants that fit into my hardware. So I am certain that the app could eventually auto-detect the hardware it is running on, start the process of downloading, self-hosting and running some models tuned and optimised for that hardware, assign sensible default models to all functionality, disable functionality that won't work on that hardware, and tune the frontload prompts to all of its internal tools, specific to the models and context sizes that were auto-selected, and be fully "ready to use" on first boot. |
|
my setup has rx 9070xt + rtx 2070 super total vram 23~ GB and 32 ram I'm running https://huggingface.co/DavidAU/Qwen3.6-27B-Heretic-Uncensored-FINETUNE-NEO-CODE-Di-IMatrix-MAX-GGUF at Q4KM doing 131k context and it's glorious. |
|
Can I ask you how it copes with unequal amount of VRAM on each GPU? Ollama is pretty bad at this so if I have an 8GB and a 16GB in the same box it will load about the same amount of weights to both cards and basically leave half of the 16GB card VRAM empty. Interested to see how Odysseus handles asymmetric model sharding. |
|
You can easily run a super powerful model like Bartowski's Q6 quant of Gemma 4 31b with that kind of VRAM at your disposal (or Q5_K_M if you don't mind it being a bit dumber and less consistent in exchange for a bit more VRAM headroom). Use a temperature of 1, top_p of 0.95 and top_k of 64 and start with like 32k context, then scale as needed or drop down a tier to Q5 if the maximum you can fit is still inadequate. |
|
Oh but word of advice, limit the number of context checkpoints it can make if you find it slows to a crawl or crashes after a bit, known issue |
|
It won't be an issue at all. All depends on the model's parameter count. I have an RTX 5060 which has 8 GBs of RAM and I can run qwen2.5-coder:7b pretty smoothly. |
|
You can do a lot better than Qwen 2.5 coder, try a Q8 quant of Gemma 4 26BA4B with --n-cpu-moe for offloading starting at around 20ish and increasing in increments until the model fits, you can probably still get 20-30 tokens per second that way. If you end up unable to fit Gemma you could also give GPT-OSS 20B a shot with a similar strategy since that's considerably smaller being an MXFP4 native model. |
|
Thank you. Will give it a try definitely. |
|
Lol you people. 16gb vram 64gb ram here. you can totally run any sort of model. just tinker offload if your model is too big. you can download lmstudio for example where you can change each models parameters to suit you. load on gpu as much as possible and put context as much as needed. havin MoE helps alot. then rest you can dial there untill you LLM is stable. gongrats you are now running 32b llm on 16gbVram! |
|
Right, we can run any size model but the real question is what's going to be useful and effective in this type of setup. Clearly anyone can load up any model and start chatting to it, you can even get some basic coding done with a simple Qwen or Deepseek coder model on a single 10GB GPU... What I'm saying is that the more advanced stuff this platform does like making its own memories or learning from your chat sessions, I'm not sure any of that really works with a single GPU, and if you offloaded most of a much larger model to RAM, it would be so slow that it would not really be usable. Maybe if this is a server that you just leave up and running 24/7, having it run a big model on CPU overnight and "learn" slowly is a good thing to do, then during the day when you're interacting with it, it loads a much smaller model into VRAM-only and you interact with that. It is certainly going to be interesting to see how people use it and what we can actually get out of this platform on lower specs. |
|
"Maybe if this is a server that you just leave up and running 24/7, having it run a big model on CPU overnight and "learn" slowly is a good thing to do, then during the day when you're interacting with it, it loads a much smaller model into VRAM-only and you interact with that." Offcourse you could make and massive datasheet with all bunch of stuff and let it parse into database to act as a memory so it does it all night long :DD but my point to the original question was just that dude has 32gb of vram. he can cook almost anything with it. |
Uh oh!
There was an error while loading. Please reload this page.
Like can it run on 32gb vram and 64gb ram?
All reactions