Gallery install writes sampling parameters where the config loader cannot read them

[jimmykarily]

What happens

When a model is installed from the gallery, LocalAI writes the recommended sampling parameters into the model YAML as top level keys:

name: qwen3.5-122b-a10b
min_p: 0
presence_penalty: 1.5
repeat_penalty: 1
temperature: 0.7
top_k: 20
top_p: 0.8
parameters:
  model: llama-cpp/models/Qwen3.5-122B-A10B-Q4_K_M-00001-of-00003.gguf

The config loader does not read them there. It expects them inside parameters:. Every one of those lines is inert.

Write side, six fields copied to top level keys:

LocalAI/core/gallery/models.go

Lines 624 to 654 in aaec1d6

	// Merge inference defaults into configMap so they are persisted without losing unknown fields.
	if modelConfig.Temperature != nil {
	if _, exists := configMap["temperature"]; !exists {
	configMap["temperature"] = *modelConfig.Temperature
	}
	}
	if modelConfig.TopP != nil {
	if _, exists := configMap["top_p"]; !exists {
	configMap["top_p"] = *modelConfig.TopP
	}
	}
	if modelConfig.TopK != nil {
	if _, exists := configMap["top_k"]; !exists {
	configMap["top_k"] = *modelConfig.TopK
	}
	}
	if modelConfig.MinP != nil {
	if _, exists := configMap["min_p"]; !exists {
	configMap["min_p"] = *modelConfig.MinP
	}
	}
	if modelConfig.RepeatPenalty != 0 {
	if _, exists := configMap["repeat_penalty"]; !exists {
	configMap["repeat_penalty"] = modelConfig.RepeatPenalty
	}
	}
	if modelConfig.PresencePenalty != 0 {
	if _, exists := configMap["presence_penalty"]; !exists {
	configMap["presence_penalty"] = modelConfig.PresencePenalty
	}
	}

Read side, the struct that holds those fields is nested under parameters:

LocalAI/core/config/model_config.go

Line 44 in aaec1d6

schema.PredictionOptions `yaml:"parameters,omitempty" json:"parameters,omitempty"`

So LocalAI writes a file it cannot read back.

Why it is easy to miss

The same defaults are applied again when the model loads:

LocalAI/core/config/model_config.go

Line 1363 in aaec1d6

ApplyInferenceDefaults(cfg, cfg.Name, cfg.Model)

That second pass fills in any value that is still unset, from the same table the install used. The numbers come out correct either way, so nothing looks broken. /api/models/config-json/<model> also shows them correctly nested, because it reports the loaded config rather than the file on disk.

When it actually causes a problem

• Editing one of those top level values to override a default does nothing, and nothing warns you.
• If a model family is missing from inference_defaults.json, the values are written at install and then lost on every load, with nothing to restore them.

How we ran into it

A 122B model would not load. We wanted to pin the batch size, so we added batch: 512 at the top level of the model YAML, matching where the sampling parameters already sat in that same file. It had no effect, and the log kept reporting the auto tuned value instead. Moving it under parameters: fixed it straight away.

That is when we noticed the sampling parameters above it were sitting in the same dead position, and that we had not put them there ourselves. The gallery install had written them.

Suggested fix

Write the defaults into the nested parameters map instead of as top level keys, so they land where the loader reads them.

Version

Seen on v4.7.1 (localai/localai:latest-gpu-vulkan). Code links point at master (aaec1d6), where the same code is present.

[jimmykarily]

Writing a test for this turned up a second failure mode in the same block, which the description above misses.

The YAML that eventually gets written is marshalled from configMap before the defaults are merged into it:

https://github.com/mudler/LocalAI/blob/704b87dc/core/gallery/models.go#L609

and the only re-marshal sits behind the artifact binding:

https://github.com/mudler/LocalAI/blob/704b87dc/core/gallery/models.go#L660-L683

An entry that carries files: never reaches that branch. For those entries the defaults are computed and then dropped before anything is written, so nothing lands in the YAML at all, not even in the inert top level position.

So there are two paths, and neither persists anything usable:

• entry with no files: that binds a primary artifact: the defaults are written, at the top level, where the loader ignores them. This is the path the model in the description took.
• entry with files:: the defaults never reach the file.

Both defeat the stated intent of the block. A fix needs to nest the keys under parameters and re-marshal after the merge; doing only the first still loses them on the files: path.

Happy to open a PR with that plus a regression spec if the approach looks right.

[localai-org-maint-bot]

I’m picking this up. I’ll reproduce both gallery install paths with focused regression specs, persist recommended defaults under the nested parameters map after merging, and open a small PR from latest master.