Summary
Add a command-line flag --expert-override or --moe-topk to allow users to change the number of active experts at runtime for Mixture of Experts (MoE) models.
Motivation
Currently, the expert_used_count (top-k) is hardcoded in the GGUF file metadata (LLM_KV_EXPERT_USED_COUNT). Users cannot experiment with different expert counts without re-conver
ting models.
Real-world use case: The Nemotron-3-Nano-30B-A3B model defaults to 6 experts, but testing shows that 8-10 experts significantly improves summarization quality for long-context ta
sks while 4-6 is sufficient for chat.
Proposed API
# Override to use 8 experts instead of GGUF default
./llama-cli -m model.gguf --expert-override 8 -p "Hello"
# Or --moe-topk alias
./llama-cli -m model.gguf --moe-topk 8 -p "Hello"
Implementation
The change would be minimal:
• Add expert_override to common_params
• Modify llm_graph_params initialization in llama-graph.cpp
• Add CLI argument in common/arg.cpp
Benefits
1. Experimentation: Users can tune expert counts for specific tasks
2. Flexibility: Same model file works for different scenarios
3. No re-quantization: Saves disk space and time
Affected Models
• Nemotron-H series
• Mixtral 8x7B/8x22B
• DeepSeek-V2/V3
• Qwen2.5-MoE
Summary
Add a command-line flag
--expert-overrideor--moe-topkto allow users to change the number of active experts at runtime for Mixture of Experts (MoE) models.Motivation
Currently, the
expert_used_count(top-k) is hardcoded in the GGUF file metadata (LLM_KV_EXPERT_USED_COUNT). Users cannot experiment with different expert counts without re-converting models.
Real-world use case: The Nemotron-3-Nano-30B-A3B model defaults to 6 experts, but testing shows that 8-10 experts significantly improves summarization quality for long-context ta
sks while 4-6 is sufficient for chat.
Proposed API