Changing the experts number can lead to better results for MoE?

[magikRUKKOLA]

How do we increasing the experts number for MoE? I suggest we change the number of active experts and measure the PPL results to figure out if that can be used to lower the PPL.

REFERENCE:

Unlock GLM 4.7 & MiniMax M2.1 - Active Expert Control for SPEED or IQ
https://www.youtube.com/watch?v=h4lMWOk_iZk

[saood06]

Yes this testing can be interesting.

See an example here.

[saood06]

@saood06

With --override-kv "deepseek2.expert_used_count=int:6"

I also tried --override-kv "deepseek2.expert_used_count=int:10" \ to try to increase the number of experts per token. That did not work as well (for GLM-4.7 at least).

Maybe 16 will be better? I only say that because for my quant of GLM-4.7-Flash 6 was not better than 4, but 8 was. And similarly for Deepseek R1 I remember reports that 16 experts was better than 10 or 12 (but 16 was still slightly worse than 8).

Edit: Oh I misread work as well to mean it did not perform as good as 8 experts, not to mean that it didn't work because you were using the wrong model architecture.

[ikawrakow]

PPL for the 3.068 bpw GLM-4.7
7 active experts: 4.4491
9 active experts: 4.5471

[magikRUKKOLA]

7 active experts: 4.4491

Uh oh. 7 experts is better than the default 8? Interesting.

[magikRUKKOLA]

@ikawrakow

Uh oh. The --split-mode-f16 was causing nan in the PPL.

[EDIT]: *for the increased experts count.

[ikawrakow]

That's why there is -grt bf16 instead. I know you like the long form, so it is --graph-reduce-type bf16.

[hunterx2591]

Hello - tried changing expert override all the way up to 32 experts on GLM 4.7 Q8_0 and GLM 4.7 Q8_K_XL. first off the override commands is required as well as the Ser 16,1 or 32,1 whatever commands. TG drops by respective amount. Quality (worse for 32 and 24) - model router has large amount of experts that get averaged so potentially better but here is the trick... it has to be significantly balanced because entropy is unstable. Please dont use default topp,k etc samplers you need to find the "golden zone" for entropy where the experts dont kill each other which varies by quant. for 8 expert i only got this stable. example = Mirostat 2 Ent 3 Lr 0.05 for Q8_K_XL works, i tested 16 experts the results are just higher entropy still hard to balance the logits. here is 8 expert at least. I tried it on "Draw an accurate outline of a Sony DualShock 4 (PS4) controller in SVG"

I got some okay results for 16 experts but nothing better than below the zones for the logit learning rate zones are like 0.005 increments. I did use both the fixed seed and confirmed without fixed seed. custom template just makes it do instruct instead of thinking. Model overthinks a lot and degrades performance if the model isnt setup just right

./build/bin/llama-server
--model "/mnt/ExtraStorage/Models/GLM-4.7-UD-Q8_K_XL-00001-of-00009.gguf"
--alias "GLM-4.7-Q8_K_XL"
-c 32765 -ctk q8_0 -ctv q8_0
-b 4096
-ub 2048
-fa on
-ngl 99
-mea 256
--n-cpu-moe 99
-ts 24,24
--parallel 1
--threads 54
--host 0.0.0.0
--port 8080
--special
--jinja
-sm layer
-gr
-smgs
-ger
--numa distribute
--merge-qkv
--split-mode graph
--mirostat 2 --mirostat-ent 3 --mirostat-lr 0.05
--chat-template-file ./models/templates/GLM-4.7.jinja

Here is example of Ent 4 Lr 0.031 which is one of the better outputs in the space of entropy zones.

[magikRUKKOLA]

GLM-4.7/THIREUS-3.3560bpw

command

for ((i=9;i<=16;i++)); do

/opt/ik_llama.cpp/ik_llama.cpp/build/bin/llama-perplexity \
    -f /opt/ik_llama.cpp/wiki.test.raw \
    --model /opt/THIREUS/GLM-4.7/THIREUS-3.3560bpw/GLM-4.7-THIREUS-BF16-SPECIAL_TENSOR-00001-of-01762.gguf \
    --alias THIREUS/GLM-4.7-3.3560bpw \
    --ctx-size $((512)) \
    -b $((2 * 1024)) -ub $((512)) \
    --mlock \
    --temp 0.5 --top-k 0 --top-p 1.0 --min-p 0.1 --repeat-penalty 1.1 \
    -ctk f16 \
    -ctv f16 \
    -ger \
    -smgs \
    -sas \
    -mea 256 \
    --merge-qkv \
    --split-mode-f32 \
    -amb 256 \
    --split-mode graph \
    --tensor-split 23,24,24,24,24,24,24 \
    --main-gpu 0 \
    --max-gpu 0 \
    --n-gpu-layers 99 \
    --override-kv "glm4moe.expert_used_count=int:${i}" \
    --threads $(grep ^cpu\\scores /proc/cpuinfo | uniq | awk '{print $4}' | xargs -I{} echo "{}-0" | bc) \
    --host 0.0.0.0 \
    --port 8080 \
    --log-enable \
    --logdir /var/log/ \
    --jinja \
    --special \
    --verbosity 1 \
    --verbose-prompt \
    --reasoning-format auto \
    --prompt-cache "$HOME/.cache/ik_llama.cpp/prompt-cache.bin" --prompt-cache-all \
    --slot-save-path "$HOME/.cache/ik_llama.cpp/slot.bin" \
    --lookup-cache-dynamic "$HOME/.cache/ik_llama.cpp/slot.bin" \
    --keep -1 \
    --slot-prompt-similarity 0.35 \
    --metrics \
    -cuda fusion=1 &> ./ppl-expert_used_count-${i}.log

done

summary:

 grep -rn -Ea --colour 'PPL over|prompt eval time' ppl-expert_used_count-{9..16}*log
ppl-expert_used_count-9.log:655:llama_print_timings: prompt eval time =  465734.49 ms / 289280 tokens (    1.61 ms per token,   621.13 tokens per second)
ppl-expert_used_count-9.log:659:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.3507 +/- 0.02778
ppl-expert_used_count-10.log:655:llama_print_timings: prompt eval time =  468591.07 ms / 289280 tokens (    1.62 ms per token,   617.34 tokens per second)
ppl-expert_used_count-10.log:659:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.4351 +/- 0.02867
ppl-expert_used_count-11.log:655:llama_print_timings: prompt eval time =  471550.35 ms / 289280 tokens (    1.63 ms per token,   613.47 tokens per second)
ppl-expert_used_count-11.log:659:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.5030 +/- 0.02936
ppl-expert_used_count-12.log:655:llama_print_timings: prompt eval time =  474510.24 ms / 289280 tokens (    1.64 ms per token,   609.64 tokens per second)
ppl-expert_used_count-13.log:655:llama_print_timings: prompt eval time =  477032.81 ms / 289280 tokens (    1.65 ms per token,   606.42 tokens per second)
ppl-expert_used_count-13.log:659:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.6561 +/- 0.03088
ppl-expert_used_count-15.log:655:llama_print_timings: prompt eval time =  482728.30 ms / 289280 tokens (    1.67 ms per token,   599.26 tokens per second)
ppl-expert_used_count-15.log:659:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.8285 +/- 0.03255
ppl-expert_used_count-16.log:655:llama_print_timings: prompt eval time =  470915.01 ms / 289280 tokens (    1.63 ms per token,   614.29 tokens per second)
ppl-expert_used_count-16.log:659:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.9154 +/- 0.03338

12 experts fail:

[1]3.2404,[2]4.0713,[3]3.0764,[4]2.8607,[5]2.8555,[6]3.0974,[7]3.2631,[8]3.4657,[9]3.6542,[10]3.5624,[11]3.4602,[12]3.6422,[13]3.7012,[14]3.6666,[15]3.8448,[16]3.8958,[17]4.0685,[18]4.3303,[19]4.2670,[20]4.2719,[21]4.3164,[22]4.2576,[23]4.1416,[24]4.0179,[25]3.9219,[26]3.8531,[27]3.8090,[28]3.8195,[29]3.8616,[30]3.9471,[31]4.0099,[32]4.0789,[33]4.1489,[34]4.1914,[35]4.2672,[36]4.3125,[37]4.3165,[38]4.3915,[39]4.4246,[40]4.4703,[41]4.5576,[42]4.5914,[43]4.5958,[44]4.6301,[45]4.7457,[46]4.8079,[47]4.7436,[48]4.6558,[49]4.5793,[50]4.5412,[51]4.5540,[52]4.5684,[53]4.6044,[54]4.6006,[55]4.6155,[56]4.6391,[57]4.6085,[58]4.6054,[59]4.5940,[60]4.6436,[61]4.6810,[62]4.7377,[63]4.7695,[64]4.7812,[65]4.7850,[66]4.7691,[67]4.7255,[68]4.6871,[69]4.7094,[70]4.7085,[71]4.7031,[72]4.7002,[73]4.7082,[74]4.7465,[75]4.7608,[76]4.7016,[77]4.6627,[78]4.6312,[79]4.5828,[80]4.5292,[81]4.5368,[82]4.5239,[83]4.5208,[84]4.5307,[85]4.4986,[86]4.4966,[87]4.4853,[88]4.4982,[89]4.4758,[90]4.4390,[91]4.4017,[92]4.4051,[93]4.4163,[94]4.4021,[95]4.3812,[96]4.4239,[97]4.4855,[98]4.4994,[99]4.4768,[100]4.4555,[101]4.4737,[102]4.4800,[103]4.5061,[104]4.4995,[105]4.5178,[106]4.5552,[107]4.6441,[108]4.6484,[109]4.6514,[110]4.6996,[111]4.7212,[112]4.6903,[113]4.6690,[114]4.6479,[115]4.6196,[116]4.6178,[117]4.6023,[118]4.6002,[119]4.5918,[120]4.5774,[121]4.5738,[122]4.5559,[123]4.5205,[124]4.4954,[125]4.4797,[126]4.4568,[127]4.4493,[128]4.4381,[129]4.4399,[130]4.4309,[131]4.4242,[132]4.4153,[133]4.4085,[134]4.4155,[135]4.4357,[136]4.4273,[137]4.4329,[138]4.4217,[139]4.4097,[140]4.4260,[141]4.4187,[142]4.4178,[143]4.4038,[144]4.3980,[145]4.3892,[146]4.3905,[147]4.3914,[148]4.3936,[149]4.3927,[150]4.3915,[151]4.3750,[152]4.3636,[153]4.3633,[154]4.3533,[155]4.3498,[156]4.3477,[157]4.3444,[158]4.3436,[159]4.3611,[160]4.3722,[161]4.3817,[162]nan,[163]nan,[164]nan,[165]nan,[166]nan,[167]nan,[168]nan,[169]nan,[170]nan,[171]nan,[172]nan,[173]nan,[174]nan,[175]nan,[176]nan,[177]nan,[178]nan,[179]nan,[180]nan,[181]nan,[182]nan,[183]nan,[184]nan,[185]nan,[186]nan,[187]nan,[188]nan,[189]nan,[190]nan,[191]nan,[192]nan,[193]nan,[194]nan,[195]nan,[196]nan,[197]nan,[198]nan,[199]nan,[200]nan,[201]nan,[202]nan,[203]nan,[204]nan,[205]nan,[206]nan,[207]nan,[208]nan,[209]nan,[210]nan,[211]nan,[212]nan,[213]nan,[214]nan,[215]nan,[216]nan,[217]nan,[218]nan,[219]nan,[220]nan,[221]nan,[222]nan,[223]nan,[224]nan,[225]nan,[226]nan,[227]nan,[228]nan,[229]nan,[230]nan,[231]nan,[232]nan,[233]nan,[234]nan,[235]nan,[236]nan,[237]nan,[238]nan,[239]nan,[240]nan,[241]nan,[242]nan,[243]nan,[244]nan,[245]nan,[246]nan,[247]nan,[248]nan,[249]nan,[250]nan,[251]nan,[252]nan,[253]nan,[254]nan,[255]nan,[256]nan,[257]nan,[258]nan,[259]nan,[260]nan,[261]nan,[262]nan,[263]nan,[264]nan,[265]nan,[266]nan,[267]nan,[268]nan,[269]nan,[270]nan,[271]nan,[272]nan,[273]nan,[274]nan,[275]nan,[276]nan,[277]nan,[278]nan,[279]nan,[280]nan,[281]nan,[282]nan,[283]nan,[284]nan,[285]nan,[286]nan,[287]nan,[288]nan,[289]nan,[290]nan,[291]nan,[292]nan,[293]nan,[294]nan,[295]nan,[296]nan,[297]nan,[298]nan,[299]nan,[300]nan,[301]nan,[302]nan,[303]nan,[304]nan,[305]nan,[306]nan,[307]nan,[308]nan,[309]nan,[310]nan,[311]nan,[312]nan,[313]nan,[314]nan,[315]nan,[316]nan,[317]nan,[318]nan,[319]nan,[320]nan,[321]nan,[322]nan,[323]nan,[324]nan,[325]nan,[326]nan,[327]nan,[328]nan,[329]nan,[330]nan,[331]nan,[332]nan,[333]nan,[334]nan,[335]nan,[336]nan,[337]nan,[338]nan,[339]nan,[340]nan,[341]nan,[342]nan,[343]nan,[344]nan,[345]nan,[346]nan,[347]nan,[348]nan,[349]nan,[350]nan,[351]nan,[352]nan,[353]nan,[354]nan,[355]nan,[356]nan,[357]nan,[358]nan,[359]nan,[360]nan,[361]nan,[362]nan,[363]nan,[364]nan,[365]nan,[366]nan,[367]nan,[368]nan,[369]nan,[370]nan,[371]nan,[372]nan,[373]nan,[374]nan,[375]nan,[376]nan,[377]nan,[378]nan,[379]nan,[380]nan,[381]nan,[382]nan,[383]nan,[384]nan,[385]nan,[386]nan,[387]nan,[388]nan,[389]nan,[390]nan,[391]nan,[392]nan,[393]nan,[394]nan,[395]nan,[396]nan,[397]nan,[398]nan,[399]nan,[400]nan,[401]nan,[402]nan,[403]nan,[404]nan,[405]nan,[406]nan,[407]nan,[408]nan,[409]nan,[410]nan,[411]nan,[412]nan,[413]nan,[414]nan,[415]nan,[416]nan,[417]nan,[418]nan,[419]nan,[420]nan,[421]nan,[422]nan,[423]nan,[424]nan,[425]nan,[426]nan,[427]nan,[428]nan,[429]nan,[430]nan,[431]nan,[432]nan,[433]nan,[434]nan,[435]nan,[436]nan,[437]nan,[438]nan,[439]nan,[440]nan,[441]nan,[442]nan,[443]nan,[444]nan,[445]nan,[446]nan,[447]nan,[448]nan,[449]nan,[450]nan,[451]nan,[452]nan,[453]nan,[454]nan,[455]nan,[456]nan,[457]nan,[458]nan,[459]nan,[460]nan,[461]nan,[462]nan,[463]nan,[464]nan,[465]nan,[466]nan,[467]nan,[468]nan,[469]nan,[470]nan,[471]nan,[472]nan,[473]nan,[474]nan,[475]nan,[476]nan,[477]nan,[478]nan,[479]nan,[480]nan,[481]nan,[482]nan,[483]nan,[484]nan,[485]nan,[486]nan,[487]nan,[488]nan,[489]nan,[490]nan,[491]nan,[492]nan,[493]nan,[494]nan,[495]nan,[496]nan,[497]nan,[498]nan,[499]nan,[500]nan,[501]nan,[502]nan,[503]nan,[504]nan,[505]nan,[506]nan,[507]nan,[508]nan,[509]nan,[510]nan,[511]nan,[512]nan,[513]nan,[514]nan,[515]nan,[516]nan,[517]nan,[518]nan,[519]nan,[520]nan,[521]nan,[522]nan,[523]nan,[524]nan,[525]nan,[526]nan,[527]nan,[528]nan,[529]nan,[530]nan,[531]nan,[532]nan,[533]nan,[534]nan,[535]nan,[536]nan,[537]nan,[538]nan,[539]nan,[540]nan,[541]nan,[542]nan,[543]nan,[544]nan,[545]nan,[546]nan,[547]nan,[548]nan,[549]nan,[550]nan,[551]nan,[552]nan,[553]nan,[554]nan,[555]nan,[556]nan,[557]nan,[558]nan,[559]nan,[560]nan,[561]nan,[562]nan,[563]nan,[564]nan,[565]nan,
llama_print_timings:        load time =   37113.18 ms
llama_print_timings:      sample time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings: prompt eval time =  474510.24 ms / 289280 tokens (    1.64 ms per token,   609.64 tokens per second)
llama_print_timings:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =  480510.10 ms / 289281 tokens

[hunterx2591]

Just a note for PS4 controller we are looking for shape. This model really struggle with rendering the body shape as it's not part of training data. Looks like 7 is closer it's missing the cut out to the controller and more of a shield shape. The shape the model is averaging output so if it's safe it won't render the straight rectangle line code and if it's too risky it might have an exploded shape or separate parts to the shape that are not connected.
At that point you could play around with k sampling but that's a different benchmark I guess. This test I was doing is part of a mirostat paper 2025 summer of precomputed logits and having a surprise value (jitter) that would enable fast compute of these shapes so it could output straight lines and curves (can be creative with besier curves but also no output values outside the designated 'shape'. Most of this test is on how accurately can the model reason spatially and then put in the values it 'thinks' about without drift) since ppl is computed before mirostat or sampling it won't reflect that portion of it I believe based on what I know. If you out in mirostat 2 ent 3 lr 0.05 it should give you the same shape or similar to mine above if the logits of bf16 are similar to q8kxl

[magikRUKKOLA]

if the logits of bf16 are similar to q8kxl

bf16? But its for the intermediate results between the GPUs.

I had been using the THIREUS-3.3560bpw (to fit everything into 7 GPUs).

[hunterx2591]

i dont know why I though the model was at bf16... first pass : xeon@xeon-System-Product-Name:~/ik_llama.cpp$ grep -E 'PPL over' ./ppl-sweep-q8/*.log
./ppl-sweep-q8/ppl-q8-exp4.log:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.3513 +/- 0.02640
Its not the model size right now itll matter where the perplexity is lowest and compare to smaller ones until im done... 1 hour thirty minute per pass so itll take 20 hours

[hunterx2591]

Hi - got 4-9 done its still running so gotta use claude Opus to extract images, i have verified the numbers with terminal. We can see that lowest PPL is at 6 or 7 experts even at larger model sizes so its not a function of model quantization I think. Model is more sure of the answer in that range of 6-7 experts (based on distribution of answers it has) edit: for the wiki text specifically that we are testing ppl on. I havent tested complex geometry output or harder questions or anything but straight up writing out tokens seems 6 might even be better. I will continue to test until 18 experts and then probably do a pass at a very low quant to see what the ideals are there. I have GLM 4.7 IQ2 KL from ubergarm or if anyone else gets to it before me. 4-18 experts range test.

This is the prompt : #!/bin/bash

MODEL="/mnt/ExtraStorage/Models/GLM-4.7-UD-Q8_K_XL-00001-of-00009.gguf"
WIKI="/home/xeon/ik_llama.cpp/wikitext-2-raw/wiki.test.raw"
OUTDIR="./ppl-sweep-q8"
LLAMA_PPL="/home/xeon/ik_llama.cpp/build/bin/llama-perplexity"

mkdir -p "$OUTDIR"

for ((i=5;i<=18;i++)); do
echo "=== Testing experts=$i ==="

$LLAMA_PPL
-f "$WIKI"
-m "$MODEL"
--alias "GLM-4.7-Q8_K_XL"
--ctx-size 512
-b 2048
-ub 512
-ctk f16
-ctv f16
-fa on
-ger
-smgs
-mea 256
--merge-qkv
--split-mode graph
-ts 24,24
--main-gpu 0
-ngl 99
--n-cpu-moe 99
--override-kv "glm4moe.expert_used_count=int:${i}"
--threads 54
--numa distribute
2>&1 | tee "$OUTDIR/ppl-q8-exp${i}.log"

done

echo ""
echo "=== SUMMARY ==="
grep -E 'PPL over|prompt eval time' "$OUTDIR"/ppl-q8-exp*.log | sort -V

edit: i know above says 5-18 test i did 4 and then broke the test to verify it saved

[magikRUKKOLA]

lol deepseek v32-speciale iq3_k:

(after 30k tokens of deliberation?)

[hunterx2591]

No dips in PPL for q8kxl at higher Expert counts. Best around 6-8 experts same as above findings for lower bpw quants. Ill do Low quants now though not expecting any different result than below. If 6 experts is good or better than 8 experts then we can correlate PPL to smartness if 9 or 10 is better then smartness is not a result of PPL and adjusting for noise floor is more important. usually with more experts the images just get more "AVERAGE" like you see the ps4 controller form speciale. Both the increased thinking and the increased experts would just increase to averaging out a shape instead of actually designing it as per my example at 8 experts Entropy 3 Lr 0.05 no thinking (instruct template).

=== SUMMARY ===
./ppl-sweep-q8/ppl-q8-exp4.log:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.3513 +/- 0.02640
./ppl-sweep-q8/ppl-q8-exp4.log:llama_print_timings: prompt eval time = 5563646.11 ms / 289280 tokens ( 19.23 ms per token, 51.99 tokens per second)
./ppl-sweep-q8/ppl-q8-exp5.log:Final estimate: PPL over 565 chunks for n_ctx=512 = 3.9803 +/- 0.02396
./ppl-sweep-q8/ppl-q8-exp5.log:llama_print_timings: prompt eval time = 5148507.55 ms / 289280 tokens ( 17.80 ms per token, 56.19 tokens per second)
./ppl-sweep-q8/ppl-q8-exp6.log:Final estimate: PPL over 565 chunks for n_ctx=512 = 3.8698 +/- 0.02334
./ppl-sweep-q8/ppl-q8-exp6.log:llama_print_timings: prompt eval time = 5762430.57 ms / 289280 tokens ( 19.92 ms per token, 50.20 tokens per second)
./ppl-sweep-q8/ppl-q8-exp7.log:Final estimate: PPL over 565 chunks for n_ctx=512 = 3.8705 +/- 0.02358
./ppl-sweep-q8/ppl-q8-exp7.log:llama_print_timings: prompt eval time = 6450444.83 ms / 289280 tokens ( 22.30 ms per token, 44.85 tokens per second)
./ppl-sweep-q8/ppl-q8-exp8.log:Final estimate: PPL over 565 chunks for n_ctx=512 = 3.9198 +/- 0.02419
./ppl-sweep-q8/ppl-q8-exp8.log:llama_print_timings: prompt eval time = 6986628.48 ms / 289280 tokens ( 24.15 ms per token, 41.40 tokens per second)
./ppl-sweep-q8/ppl-q8-exp9.log:Final estimate: PPL over 565 chunks for n_ctx=512 = 3.9943 +/- 0.02496
./ppl-sweep-q8/ppl-q8-exp9.log:llama_print_timings: prompt eval time = 7182664.03 ms / 289280 tokens ( 24.83 ms per token, 40.27 tokens per second)
./ppl-sweep-q8/ppl-q8-exp10.log:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.0800 +/- 0.02581
./ppl-sweep-q8/ppl-q8-exp10.log:llama_print_timings: prompt eval time = 21712186.18 ms / 289280 tokens ( 75.06 ms per token, 13.32 tokens per second)
./ppl-sweep-q8/ppl-q8-exp11.log:Final estimate: PPL over 565 chunks for n_ctx=512 = 4.1617 +/- 0.02659
./ppl-sweep-q8/ppl-q8-exp11.log:llama_print_timings: prompt eval time = 21423350.37 ms / 289280 tokens ( 74.06 ms per token, 13.50 tokens per second)
./ppl-sweep-q8/ppl-q8-exp12.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.4738 +/- 0.02971
./ppl-sweep-q8/ppl-q8-exp12.log:llama_print_timings: prompt eval time = 3671268.96 ms / 288768 tokens ( 12.71 ms per token, 78.66 tokens per second)
./ppl-sweep-q8/ppl-q8-exp13.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.5394 +/- 0.03034
./ppl-sweep-q8/ppl-q8-exp13.log:llama_print_timings: prompt eval time = 3222612.72 ms / 288768 tokens ( 11.16 ms per token, 89.61 tokens per second)
./ppl-sweep-q8/ppl-q8-exp14.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.5978 +/- 0.03091
./ppl-sweep-q8/ppl-q8-exp14.log:llama_print_timings: prompt eval time = 3580828.81 ms / 288768 tokens ( 12.40 ms per token, 80.64 tokens per second)
./ppl-sweep-q8/ppl-q8-exp15.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.6665 +/- 0.03159
./ppl-sweep-q8/ppl-q8-exp15.log:llama_print_timings: prompt eval time = 3464123.02 ms / 288768 tokens ( 12.00 ms per token, 83.36 tokens per second)
./ppl-sweep-q8/ppl-q8-exp16.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.7391 +/- 0.03230
./ppl-sweep-q8/ppl-q8-exp16.log:llama_print_timings: prompt eval time = 3520107.98 ms / 288768 tokens ( 12.19 ms per token, 82.03 tokens per second)
./ppl-sweep-q8/ppl-q8-exp17.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.8051 +/- 0.03291
./ppl-sweep-q8/ppl-q8-exp17.log:llama_print_timings: prompt eval time = 3454010.86 ms / 288768 tokens ( 11.96 ms per token, 83.60 tokens per second)
./ppl-sweep-q8/ppl-q8-exp18.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.8825 +/- 0.03363
./ppl-sweep-q8/ppl-q8-exp18.log:llama_print_timings: prompt eval time = 3558301.46 ms / 288768 tokens ( 12.32 ms per token, 81.15 tokens per second)

[hunterx2591]

OK got data for iq2_k_L The PPL is more toward the 7 experts. But i tried 6 expert on the q8 version and its a no go. Ill try to tune the Q2 version to see if I can get the same or similar reasoning ability at least for SVG stuff. Based on PPL the optimal is within that 7 Expert runs but unsure if that actually translates to anything.

=== SUMMARY === ./ppl-sweep-q2/ppl-q8-exp4.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.3079 +/- 0.03432 ./ppl-sweep-q2/ppl-q8-exp4.log:llama_print_timings: prompt eval time = 1271603.13 ms / 288768 tokens ( 4.40 ms per token, 227.09 tokens per second) ./ppl-sweep-q2/ppl-q8-exp5.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.9963 +/- 0.03243 ./ppl-sweep-q2/ppl-q8-exp5.log:llama_print_timings: prompt eval time = 1372764.15 ms / 288768 tokens ( 4.75 ms per token, 210.36 tokens per second) ./ppl-sweep-q2/ppl-q8-exp6.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.8656 +/- 0.03177 ./ppl-sweep-q2/ppl-q8-exp6.log:llama_print_timings: prompt eval time = 1452607.45 ms / 288768 tokens ( 5.03 ms per token, 198.79 tokens per second) ./ppl-sweep-q2/ppl-q8-exp7.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.8258 +/- 0.03173 ./ppl-sweep-q2/ppl-q8-exp7.log:llama_print_timings: prompt eval time = 1350470.34 ms / 288768 tokens ( 4.68 ms per token, 213.83 tokens per second) ./ppl-sweep-q2/ppl-q8-exp8.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 4.8720 +/- 0.03247 ./ppl-sweep-q2/ppl-q8-exp8.log:llama_print_timings: prompt eval time = 1363583.54 ms / 288768 tokens ( 4.72 ms per token, 211.77 tokens per second) ./ppl-sweep-q2/ppl-q8-exp9.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.0172 +/- 0.03415 ./ppl-sweep-q2/ppl-q8-exp9.log:llama_print_timings: prompt eval time = 1465058.93 ms / 288768 tokens ( 5.07 ms per token, 197.10 tokens per second) ./ppl-sweep-q2/ppl-q8-exp10.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.0686 +/- 0.03473 ./ppl-sweep-q2/ppl-q8-exp10.log:llama_print_timings: prompt eval time = 1499887.42 ms / 288768 tokens ( 5.19 ms per token, 192.53 tokens per second) ./ppl-sweep-q2/ppl-q8-exp11.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.1340 +/- 0.03545 ./ppl-sweep-q2/ppl-q8-exp11.log:llama_print_timings: prompt eval time = 1483429.66 ms / 288768 tokens ( 5.14 ms per token, 194.66 tokens per second) ./ppl-sweep-q2/ppl-q8-exp12.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.2011 +/- 0.03616 ./ppl-sweep-q2/ppl-q8-exp12.log:llama_print_timings: prompt eval time = 1550807.06 ms / 288768 tokens ( 5.37 ms per token, 186.20 tokens per second) ./ppl-sweep-q2/ppl-q8-exp13.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.2771 +/- 0.03692 ./ppl-sweep-q2/ppl-q8-exp13.log:llama_print_timings: prompt eval time = 1457987.97 ms / 288768 tokens ( 5.05 ms per token, 198.06 tokens per second) ./ppl-sweep-q2/ppl-q8-exp14.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.3478 +/- 0.03761 ./ppl-sweep-q2/ppl-q8-exp14.log:llama_print_timings: prompt eval time = 1783733.79 ms / 288768 tokens ( 6.18 ms per token, 161.89 tokens per second) ./ppl-sweep-q2/ppl-q8-exp15.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.4200 +/- 0.03833 ./ppl-sweep-q2/ppl-q8-exp15.log:llama_print_timings: prompt eval time = 1572684.34 ms / 288768 tokens ( 5.45 ms per token, 183.61 tokens per second) ./ppl-sweep-q2/ppl-q8-exp16.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.4931 +/- 0.03906 ./ppl-sweep-q2/ppl-q8-exp16.log:llama_print_timings: prompt eval time = 1549611.13 ms / 288768 tokens ( 5.37 ms per token, 186.35 tokens per second) ./ppl-sweep-q2/ppl-q8-exp17.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.5707 +/- 0.03979 ./ppl-sweep-q2/ppl-q8-exp17.log:llama_print_timings: prompt eval time = 1760978.83 ms / 288768 tokens ( 6.10 ms per token, 163.98 tokens per second) ./ppl-sweep-q2/ppl-q8-exp18.log:Final estimate: PPL over 282 chunks for n_ctx=1024 = 5.6521 +/- 0.04055 ./ppl-sweep-q2/ppl-q8-exp18.log:llama_print_timings: prompt eval time = 1652291.80 ms / 288768 tokens ( 5.72 ms per token, 174.77 tokens per second)

[saood06]

@magikRUKKOLA @hunterx2591

You both may find this comment interesting about changing certain model weights to accommodate higher expert counts for models that use sigmoid gating.

[magikRUKKOLA]

I just retested all the different deepseek versions, kimi-k2 versions, qwen-coder-480b and glm-4.7 with temperature=0 and all end up with misaligned sun rays still.

I just tried with the DeepSeek-V32-Speciale (with 0.5 temp). I do not see the any misaligned sun rays.

[saood06]

This is a really fascinating discussion

I was debating whether or not to @ you, I thought linking your comment wouldn't send you a notification. Either way glad to see you, and glad I correctly interpreted your remark.

[hunterx2591]

I just retested all the different deepseek versions, kimi-k2 versions, qwen-coder-480b and glm-4.7 with temperature=0 and all end up with misaligned sun rays still.

glm-4.7 with --temp 1.0 --mirostat 2 --mirostat-ent 1.5 --mirostat-lr 0.076:

It's only 1 run, but very interesting!

wow.. what generated the particle system in the image? and what q size?

[hunterx2591]

Kimi K2.5 online :

[jukofyork]

I just retested all the different deepseek versions, kimi-k2 versions, qwen-coder-480b and glm-4.7 with temperature=0 and all end up with misaligned sun rays still.
glm-4.7 with --temp 1.0 --mirostat 2 --mirostat-ent 1.5 --mirostat-lr 0.076:

It's only 1 run, but very interesting!

wow.. what generated the particle system in the image? and what q size?

Q4_K for non-shared experts and Q6_K for everything else.

[FNsi]

Without much test, I found qwen next coder (udq3kl) can better understand my prompt example to solve related problems if I reduce the activated experts to 9. Increased to 11, 12 didn't feel good. Sorry only plain text here.

[magikRUKKOLA]

Without much test, I found qwen next coder (udq3kl) can better understand my prompt example to solve related problems if I reduce the activated experts to 9.

As you can see, #1099 (comment) , lowering the experts by 1 or even 2 can help too.

[FNsi]

I tied set to 8 (default -2) and it's not good as 9 so I kept 9 in my case. Should notice that I didn't know how to change the shared experts number. Indeed I'd like to test 2 shared + 18 activated, or might 3 + 27 🤔

[magikRUKKOLA]

I tied set to 8 (default -2) and it's not good as 9 so I kept 9 in my case. Should notice that I didn't know how to change the shared experts number. Indeed I'd like to test 2 shared + 18 activated, or might 3 + 27 🤔

Qwen3-Next-Coder ?

Mixture of Experts:
Number of Experts: 512
Number of Activated Experts: 10
Number of Shared Experts: 1
Expert Intermediate Dimension: 512

You're saying that the default is 8? I believe the default is -1 (that is, to grab the number from the LLM's metadata). No?

[FNsi]

It's my fault for the vagueness. as we knew the model default is 10, thus 10-2=8; btw do you know how to change the shared experts number without touch the code?