Repository navigation
Replies: 1 comment 8 replies
|
Hey, did you end up discovering a decent workflow with local models? |
8 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
There seems to be 2 options depending on what kind of device you are running inference on. Some devices have very high vram but very poor compute (eg:Macbooks), and other devices have very high compute but very low vram (eg:desktops with nvidia gpus). For option one you are best served using a MoE model with a very high parameter model but with very few active at any given time (eg: qwen 80b3a) for option 2 you would want a dense non-MoE model with the entire model active at once (eg:gemma4 31b).
Anyways has anyone put any real time and effort into using any of these models? Have you found any local models to actually be useful for this plugin? Historically they haven't been anywhere near the ones coming out of the proprietary labs. Just thought it could be interesting to see if anyone has had any experience or has any thoughts about this. I plan to try out the new qwen 35b3a on my macbook and hope it's able to work well.
All reactions