Feature request: support model specific inference parameters

[rzerres]

Model specific inference speedup

I'd like to improve the inference speed of the gemma 4 models.
Following e.g. Venelin Valkov who talks about significant improvements
inside llama.cpp running gemma 4, if you adapt flags:

parameter	value	category
spec-type	draft-mtp	speculative decoding
spec-type-draft-n-max	2	speculative decoding
fit	off	general

I have seen, that you already published llama-cpp-bindings v0.10.0.

Proposed improvement

Paddler might offer model specific "inference options"

I'm aware of the fact, what the llama.cpp stack is continously growing. Supporting everything inside paddlers UI will get cluttered
and overwhelming. Some are obviously model specific and do not improve general operation.

As an alternative, the extended options could be incorporated via the api calls (curl X "PUT" ...). The interested audience can read up
available options inside the API documentation.

• custom flags

  There might be a workaround/solution introducing a "custom flags" field.
  Thus, an educated user/developer can add the needed flags.

Even if the backend (paddler) might not be able to formaly verify the syntax of a given flag, this can be handled when forwarding to
llama.cpp.

• model flags vs general flags

  Given UI is using general flags. To keep the interface clean, this can be enrichted with a model specific section.
  Well known models can be handled with a formal syntax analize inside the crate.

[mcharytoniuk]

Makes sense, will implement. :) Will let you know here when we will have this scheduled by commenting in this issue. Probably next week or so. Thanks for opening the issue.

[rzerres]

Probably it is a mid- to longterm decision. So investigation to implement a solution as a trait that can be extended when needed?

[mcharytoniuk]

Probably it is a mid- to longterm decision. So investigation to implement a solution as a trait that can be extended when needed?

I want to keep Paddler as user-friendly as possible; currently it exposes a single, simple list of parameters, but I never planned on how to best tie them to a specific model (for example, when model switches, it should not carry those model-specific parameters either).

I just need a few days to figure out how to best deal with this, and how to present those parameters in the web admin panel, so it will still be readable. So that's not just about code for me, I also want to keep good DX/UX. I will post some ideas in this isse next week likely.

[marcjwojtowicz]

I have a similar need, except I only plan to load a single model and could pass the llama.cpp parameters via environment variables as I would if I was running llama-server. For example, it would be great to be able to run::

export LLAMA_ARG_SPEC_DRAFT_N_GPU_LAYERS=99
export LLAMA_ARG_SPEC_TYPE="draft-mpt"
export LLAMA_ARG_SPEC_DRAFT_N_MAX=2
paddler agent --management-addr 192.168.1.10 --slots 1 --name ai1

However, I cannot tell if these environment variables are being used by llama.cpp wrapped within paddler. My best guess is that they are not used, since I see no difference in model performance during a prompt and no difference in Paddler agent output when loading a model.

Many of the llama.cpp command line options have corresponding environment variables as an alternative way of setting these values. Command line options and their corresponding environment variables (where available) are shown on https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md.

If this is implemented as the original poster envisioned (loading values per model), then perhaps the paddler API used for loading models is the right place to put this. If llama-server is running as a sub-process of paddler, maybe it can be invoked with command line parameters specified in a json parameter within the request to load a model in Padder?

[malzag]

I have a similar need, except I only plan to load a single model and could pass the llama.cpp parameters via environment variables as I would if I was running llama-server. For example, it would be great to be able to run::

export LLAMA_ARG_SPEC_DRAFT_N_GPU_LAYERS=99
export LLAMA_ARG_SPEC_TYPE="draft-mpt"
export LLAMA_ARG_SPEC_DRAFT_N_MAX=2
paddler agent --management-addr 192.168.1.10 --slots 1 --name ai1
However, I cannot tell if these environment variables are being used by llama.cpp wrapped within paddler. My best guess is that they are not used, since I see no difference in model performance during a prompt and no difference in Paddler agent output when loading a model.

@marcjwojtowicz Paddler does not support those env variables, and we do not have those parameters exposed. We are working on it though. :) We will likely make a release next week.

Many of the llama.cpp command line options have corresponding environment variables as an alternative way of setting these values. Command line options and their corresponding environment variables (where available) are shown on https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md.

If this is implemented as the original poster envisioned (loading values per model), then perhaps the paddler API used for loading models is the right place to put this. If llama-server is running as a sub-process of paddler, maybe it can be invoked with command line parameters specified in a json parameter within the request to load a model in Padder?

llama-server is not a subprocess, we do not use llama-server at all in fact, we compiled llama.cpp inference library into Paddler, and reimplemented the infrastructure layer to make it scalable, so to support the parameters is on us now. :)

[marcjwojtowicz]

@malzag Thanks for the clarifications and information. Indeed, I knew older versions of Paddler relied on llama-server, and I wasn't sure how llama.cpp got used within Paddler in the newer versions. I look forward to seeing what you come up with for llama.cpp parameter support, and will keep an eye out for updates over the next couple weeks.

[malzag]

@marcjwojtowicz No problem. :) Thanks for the interest in Paddler; yes, indeed initially we we spawning llama-server, but eventually we reimplemented it. We will let you know in this thread.