Skip to content

respect -fitc from llama-bench instead of using the benchmark context size - #28331

Open
MartinEmrich wants to merge 1 commit into
ggml-org:masterfrom
MartinEmrich:fix-fitc-llama-bench
Open

respect -fitc from llama-bench instead of using the benchmark context size#28331
MartinEmrich wants to merge 1 commit into
ggml-org:masterfrom
MartinEmrich:fix-fitc-llama-bench

Conversation

@MartinEmrich

Copy link
Copy Markdown

Overview

While tweaking the optimal settings for running Qwen 3.6 MoE on a small mobile GPU plus System RAM, I was wondering why what llama-fit-params used with llama-bench produced different results that using -fitc with llama-bench directly.

Disclosure: I used AI (Qwen 3.6, of course via llama-server) for the analysis and the fix. These words here are my own though.

Additional information

Turns out that llama-bench does not use my -fitc 196608 parameter (I wanted to benchmark for) for fitting, but strictly the actual benchmark context size (pp512 plus tg128, makes only 640 tokens). All that -fitc does is enable fitting.
I then tried absurdly different -fitc 16384 as well as -fitc 1638400 parameters to verify, and all produced the same benchmark result (higher, as more layers and parameters fit on the GPU).

At the changed code, if n_ctx_auto is false (which is always the case with llama-bench), it now clamps the context size for fit calculation to the passed -fitc parameter via fit_min_ctx if it is bigger (most likely in practice) than the benchmark's token requirements.

This way the benchmark runs with the same settings I get from llama-fit-params with the actual context I plan to run llama-server with.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, for analysis and implementing the change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant