Work to do #978
Replies: 2 comments
|
a little hard to follow - there is a lot going on here. One thing you can try is to have your favourite agent try to build from source and see if it can sort things out. There are often things guessed wrong when ti downloads a runtime. hard to action much of this, but can update docs. Might want ot break things down a bit, and grind away at just one? (memory reporting was off with an older version) |
|
@ivan-kiselev Some fixes since you tried (as you got the web/docs just before the new release): mesh-llm setup missing but obviusly a lot more to tackle! thanks! on it |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hey folks!
I come with peace ☮️
I got really excited about the project as I have plenty of wasted compute at home in shape of old PCs/laptops and phones, and all together, compute-wise, they are plenty to run a decent model.
I understand nothing in internals of LLMs and ML in general, but being in profession of dealing with computers for 12 or so years, I generally don't struggle wrangling with computer-related problems, and so I was hoping that mesh-llm will cover this gap for me, allowing me to run a local LLM whilst still being blissfully unaware of how and what and why, ya know, good old one button click success experience.
Unfortunately, it's not the case at the moment, and I am writing up this discussion just to share general feedback.
First of all, documentation on the website is outdated. Gives bitter taste right away. Config structure is wrong,
mesh-llm setupdoesn't exist. Documentation is important. Even employing external Claude I couldn't figure many things out within acceptable period of time.Second, when installing with a standard install script on my old NixOS box (it has old
NVIDIA GeForce GTX 970) - mesh-llm binary that was installed was one with CUDA support and was crushing in absence of the CUDA library installed. After some time, I figured it out that my old GPU isn't supported by CUDA13 at all, and that I need the vulkan backend if I want to try anything at all.Third, achieving mesh it troublesome. Iroh should be implementation detail and not have its complexity exposed to the user. I had a simple setup of my laptop and old NixOS box on the same LAN, and I spent first hour on just trying to make them friends. For that I had to enable Avahi service on NixOS and switch to mDNS discovery, super not-obvious things to figure out for Iroh-unaware people.
Fourth, I had to figure out what model I have to use. At this moment, I'd like to remind you that I know nothing about ML or LLMs internals and it's all dark forest for me (and I generally don't want to specialize in LLMs), I only get by with general computer knowledge and vast sys-admining past amplified by LLMs themselves. I had to understand that
Qwen3.6-27B-UD-Q4_K_XL-layersis what I want because it's split to layers by some randos on internetz, and the vanillaQwen3.6-27Bdoesn't fit to my NixOS box memory-wise, because it's like.. monolithic? If I try to run vanilla on the small worker box - it complains about not having someggufmetadata, necessary to split the thing. And the split one is not in catalogue on the web, but it is in catalogue on the CLI.Fifth, after jumping through all of the hoops above - my whole hype around the project was the idea that I had in mind is that I will have LLM running on all available hardware, on the same LAN as my laptop, partially on my laptop itself, but outsourcing as much as possible to the "workers". No matter what I tried - no load went to my NixOS worker, all the work remained on my laptop.
Aside of this, "automatic routing" of the mesh in the chat web ui always give 5xx error, and I had to choose model manually.
Memory reporting seems to be off for the mesh, the vulkan-backend box shows RAM and not GPU VRAM (maybe it's correct? you tell me).
Anyhow.
I am excited for the future of the project, and therefore taking time to write out these ergonomics issues, I bet these things will shed whole lot of potential users for you.
I had plenty of free time and patience on the hands yesterday, and the result was still a failure. Ultimate goal for a project like this to be competitive is success after 5 minutes.
Looking forward to see the future of mesh-llm!
Ivan
All reactions