Does it support the Ollama service?

[Leroy-X]

Does this plugin support Ollama?

I attempted to modify the URL to http://localhost:11434/v1/chat/completions, but it failed to function.

What should I do? Thank you immensely.

[Krazal]

Hi!

Thank you for your question and your patience!

Unfortunately the plugin NppOpenAI doesn't support Ollama directly, but you can use a PHP proxy to handle NppOpenAI requests and Ollama responses. For example you can create an ollama_proxy.php:

<?php
header('Content-Type: application/json; charset=utf-8');

// Set up some variables
$ollama_url = 'http://localhost:11434/v1/chat/completions';
$postfields = [ # 'prompt' and (optional) 'system' will be added later
	'model'   => 'latest', # PLEASE SET UP MODEL HERE! (additional models: 'llama2', 'orca-mini:3b-q4_1', 'llama2:13b' etc.)
	'stream'  => false,
	'options' => [
		'temperature' => 0.7, # Default -- will be overwritten by NppOpenAI.ini
		'top_k '      => 40,  # Default value
		'top_p'       => 0.9, # Default -- will be overwritten by NppOpenAI.ini
		// For additional options see: https://github.com/ollama/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values
	],
];

// Let's get started
try {

	// Check PHP Input
	$input = json_decode(file_get_contents('php://input'), true);
	if (!$input)
	{
		throw new \Exception("Non-JSON input received");
	}

	// Check system/prompt related elements
	$is_system_message = (@$input['messages'][0]['role'] == 'system');
	if (!isset($input['messages'][0]['content']))
	{
		throw new \Exception("No message received");
	}
	if ($is_system_message && !isset($input['messages'][1]['content']))
	{
		throw new \Exception("No message received, only instructions (system message)");
	}

	// Add system message
	if ($is_system_message)
	{
		$postfields['system'] = $input['messages'][0]['content'];
	}

	// Add prompt
	$postfields['prompt'] = !$is_system_message
		? $input['messages'][0]['content']
		: $input['messages'][1]['content'];

	// Update some options, if possible
	// $postfields['model']               = $input['model']       ?? $postfields['model']; # Use the model above to support system messages
	$postfields['options']['temperature'] = $input['temperature'] ?? $postfields['options']['temperature'];
	$postfields['options']['top_p']       = $input['top_p']       ?? $postfields['options']['top_p'];

	// Call Ollama
	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, $ollama_url);
	curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); # OK on localhost
	curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0); # OK on localhost
	curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); # It may come in handy
	curl_setopt($ch, CURLOPT_MAXREDIRS, 10);     # It may come in handy
	curl_setopt($ch, CURLOPT_TIMEOUT, 60);       # Increase if necessary
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); # Required for output
	curl_setopt($ch, CURLOPT_POST, 1);
	curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($postfields));
	$curl_data   = curl_exec($ch);
	$curl_errno = curl_errno($ch);
	if ($curl_errno)
	{
		$curl_err = curl_error($ch) ?: $curl_errno;
		curl_close($ch);
		throw new \Exception("CURL error: {$curl_err}");
	}

	// Handle Ollama's response
	$response = json_decode($curl_data, true);
	if (!$response)
	{
		throw new \Exception("Non-JSON response received: {$curl_data}");
	}
	if (!isset($response['response']))
	{
		throw new \Exception("Missing response; Ollama's answer: " . print_r($response, true));
	}

	// Convert/Print output
	$output = [
		'usage'   => [
			'total_tokens' => (int)@$response['prompt_eval_count'] + (int)@$response['eval_count'], # Total token usage
		],
		'choices' => [ [
			'message' => [
				'content'       => $response['response'],
				'finish_reason' => 'stop',
			],
		] ],
	];
	echo json_encode($output);

// Handle errors
} catch (Exception $e) {
	echo json_encode([
		'error' => [
			'message' => $e->getMessage(),
		],
	]);
}

?>

You may install a WAMPServer and save the script above to C:\wamp64\www\ollama_proxy.php or similar directory.

Then you should update the API URL in NppOpenAI.ini (Plugins » NppOpenAI » Edit Config), like:

api_url=http://localhost/ollama_proxy.php?endpoint=
temperature=0.7
top_p=0.8

Please leave the model unchanged (e.g. model=gpt-4), and update this in the ollama_proxy.php file if necessary (look for the 'latest' keyword).

After saving the NppOpenAI.ini please don't forget to click Plugins » NppOpenAI » Load Config.

The script above should work, but it has been tested in simulated environment only, without Ollama. If you have any question, feel free to write a new Comment.

[Leroy-X]

Thank you very much for your reply!