Repository navigation
Request for the next Gemma generation: a 150–250B MoE with ~8–12B active parameters #785
iamtooohandsome-cpu
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Gemma has excellent small and mid-sized open models, but there is still a gap for users who want frontier-class open weights on workstation-class hardware.
A large sparse MoE—roughly 150–250B total parameters with only 8–12B active per token—would fill this gap extremely well.
This configuration could offer:
much higher knowledge/model capacity than a 20–30B dense model;
substantially lower decode cost than a similarly sized dense model;
realistic local deployment on 128–256GB RAM workstations with 24–48GB GPUs;
a compelling open-weight alternative for coding, agents, multilingual use, and long-context workloads.
Models such as recent sparse MoEs demonstrate that very large total capacity does not require very large active compute. A Gemma model designed around this deployment class would be much more interesting to workstation users than simply scaling another dense model.
An approximate target such as 200B total / 10B active would be especially attractive, but the important request is the general deployment tier rather than the exact parameter count.
Please keep it a general-purpose Gemma model, not only a specialized derivative.
All reactions