local-ai.md

LocalAI

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including your locally hosted models through LocalAI.

Portkey SDK Integration with LocalAI

1. Install the Portkey SDK

{% tabs %} {% tab title="NodeJS" %}

npm install --save portkey-ai

{% endtab %}

{% tab title="Python" %}

pip install portkey-ai

{% endtab %} {% endtabs %}

2. Initialize Portkey with LocalAI URL

First, ensure that your API is externally accessible. If you're running the API on http://localhost, consider using a tool like ngrok to create a public URL. Then, instantiate the Portkey client by adding your LocalAI URL (along with the version identifier) to the customHost property, and add the provider name as openai.

{% hint style="success" %} Note: Don't forget to include the version identifier (e.g., /v1) in the customHost URL {% endhint %}

{% tabs %} {% tab title="NodeJS SDK" %}

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    provider: "openai",
    customHost: "https://7cc4-3-235-157-146.ngrok-free.app/v1" // Your LocalAI ngrok URL
})

{% endtab %}

{% tab title="Python SDK" %}

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="openai",
    custom_host="https://7cc4-3-235-157-146.ngrok-free.app/v1" # Your LocalAI ngrok URL    
)

{% endtab %} {% endtabs %}

{% hint style="info" %} Portkey currently supports all endpoints that adhere to the OpenAI specification. This means, you can access and observe any of your LocalAI models that are exposed through OpenAI-compliant routes.

List of supported endpoints here. {% endhint %}

3. Invoke Chat Completions

Use the Portkey SDK to invoke chat completions from your LocalAI model, just as you would with any other provider.

{% tabs %} {% tab title="NodeJS SDK" %}

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'ggml-koala-7b-model-q4_0-r2.bin',
});

console.log(chatCompletion.choices);

{% endtab %}

{% tab title="Python SDK" %}

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'ggml-koala-7b-model-q4_0-r2.bin'
)

print(completion)

{% endtab %} {% endtabs %}

LocalAI Endpoints Supported

Endpoint	Portkey API Reference
/chat/completions (Chat, Vision, Tools support)	Doc
/images/generations	Doc
/embeddings	Doc
/audio/transcriptions	Doc

Next Steps

Explore the complete list of features supported in the SDK:

{% content-ref url="../../api-reference/portkey-sdk-client.md" %} portkey-sdk-client.md {% endcontent-ref %}

---

You'll find more information in the relevant sections:

1. Add metadata to your requests
2. Add gateway configs to your Ollama requests
3. Tracing Ollama requests
4. Setup a fallback from OpenAI to Ollama APIs