Continue doesn't allow setting penalty parameters for Ollama provider

[0rzech]

Before submitting your bug report

• I believe this is a bug.
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- Continue version: v0.8.58

Description

Ollama allows setting repeat_penalty, presence_penalty and frequency_penalty when doing API calls [1], neither of which can be used with it within Continue [2].

Also, available Ollama options [3] seem to be modelled after Model File [4] (which AFAIU is used to create models [5], not to make API calls), rather than API options from [1].

This is important, because Ollama sets repeat_penalty to 1.1 by default and instruction-tuned models should not penalize repetitions [6][7]. So to use these with Continue and Ollama, one has to create custom models just to hardcode this parameter.

1. https://github.com/ollama/ollama/blob/a820d2b2673f7f8035e3a2a6f93c83af465f841c/docs/api.md#request-7

2. continue/core/llm/llms/Ollama.ts

   Lines 184 to 197 in 2c59776

   	private _getModelFileParams(options: CompletionOptions): ModelFileParams {
   	return {
   	temperature: options.temperature,
   	top_p: options.topP,
   	top_k: options.topK,
   	num_predict: options.maxTokens,
   	stop: options.stop,
   	num_ctx: this.contextLength,
   	mirostat: options.mirostat,
   	num_thread: options.numThreads,
   	use_mmap: options.useMmap,
   	min_p: options.minP,
   	};
   	}

3. continue/core/llm/llms/Ollama.ts

   Lines 15 to 36 in 2c59776

   	// See https://github.com/ollama/ollama/blob/main/docs/modelfile.md for details on each parameter
   	interface ModelFileParams {
   	mirostat?: number;
   	mirostat_eta?: number;
   	mirostat_tau?: number;
   	num_ctx?: number;
   	repeat_last_n?: number;
   	repeat_penalty?: number;
   	temperature?: number;
   	seed?: number;
   	stop?: string | string[];
   	tfs_z?: number;
   	num_predict?: number;
   	top_k?: number;
   	top_p?: number;
   	min_p?: number;
   	// deprecated?
   	num_thread?: number;
   	use_mmap?: boolean;
   	num_gqa?: number;
   	num_gpu?: number;
   	}

4. https://github.com/ollama/ollama/blob/a820d2b2673f7f8035e3a2a6f93c83af465f841c/docs/modelfile.md#valid-parameters-and-values
5. https://github.com/ollama/ollama/blob/a820d2b2673f7f8035e3a2a6f93c83af465f841c/docs/modelfile.md#ollama-model-file
6. https://huggingface.co/google/gemma-2-2b-it-GGUF#gemma-model-card
7. https://huggingface.co/google/gemma-7b-it/discussions/38#65d7b14adb51f7c160769fa1

To reproduce

No response

Log output

No response

[github-actions]

This issue hasn't been updated in 90 days and will be closed after an additional 10 days without activity. If it's still important, please leave a comment and share any new information that would help us address the issue.

[Anisotropic]

Bump. Trying to use the newest QwQ and get a constant loop. Was unable to pass repeat_penalty to ollama host

[0rzech]

Alright, here are the things I've learned in the meantime:

• config.json is going to be replaced with config.yaml, so it's better to make a switch
• one can pass ollama params directly, for example:

  models:
    - name: Some model
      provider: ollama
      model: some-model
      requestOptions:
        extraBodyProperties:
          options:
            repeat_penalty: 1
            presence_penalty: 1
            frequency_penalty: 1
            use_mmap: false
          keep_alive: 60m

• requestOptions.extraBodyProperties.options completely overwrite whatever can be set in defaultCompletionOptions