.Net: Allow Modifying the OpenAI endpoint

[239573049]

How to change the endpoint of sk's openai? Instead of forcing the use of official endpoints

[MaxAkbar]

I found the following:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/KernelSyntaxExamples/Example41_HttpClientUsage.cs. It seems we can override the URI using a custom HttpClient. Can you confirm? I plan to test it next week.

EDIT:
So, I tested this, and it's not using my HttpClient Base Address :(.

[RogerBarreto]

@MaxAkbar Try using the IDelegatingHandlerFactory

This example it uses the IDelegatingHandlerFactory to make a retry but you can create your own to alter the Url at the Task<HttpResponseMessage> SendAsync override

[MaxAkbar]

That worked, calling my custom endpoint, and is a workaround. I hope this ticket addresses the issue of us reusing the OpenAI standard for accessing our own LLMs, thank you.

[MaxAkbar]

It seems that the latest code no longer includes IDelegatingHandlerFactory. Has it been replaced by another interface or another way of doing this?

Thanks in advance,
-Max

[stephentoub]

Has it been replaced by another interface or another way of doing this?

Yes. SK's IDelegatingHandlerFactory was a bespoke mechanism for doing this, when .NET already has such a mechanism, so SK's custom thing was removed in favor of just using what .NET already provides. In particular, HttpClient is already built around a pipeline of HttpMessageHandlers, and the core libraries provide a DelegatingHandler that makes it easy to create your own that wraps another. So, for example, if want to create a delegating handler that wraps the core HttpClientHandler but changes the host of any request Uris to be localhost, I can write:

sealed class MyRedirectingHandler() : DelegatingHandler(new HttpClientHandler())
{
    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.RequestUri = new UriBuilder(request.RequestUri!) { Host = "localhost" }.Uri;
        return base.SendAsync(request, cancellationToken);
    }
}

(this is using C# 12 syntax for primary constructors just to simplify things ab it). Then, if I had my SK usage like:

IKernelBuilder builder = 
    Kernel.CreateBuilder()
    .AddOpenAIChatCompletion("gpt-4", apikey);

Kernel kernel = builder.Build();

Console.WriteLine(await kernel.InvokePromptAsync("What color is the sky?"));

I could add one line before the call to Build:

builder.Services.ConfigureHttpClientDefaults(b =>
    b.ConfigurePrimaryHttpMessageHandler(() => new MyRedirectingHandler()));

and now any HttpClient instances created via this kernel or its services (like the one created as part of the AddOpenAIChatCompletion method) will automatically pick up that handler. This approach also means you can take advantage of the rich ecosystem around such handlers. For example, if you want resiliency in the form of retries, automatically respecting Retry-After in response headers, etc., you can add a reference to the Microsoft.Extensions.Http.Reslience package, and then change the line above to be:

builder.Services.ConfigureHttpClientDefaults(b =>
{
    b.AddStandardResilienceHandler();
    b.ConfigurePrimaryHttpMessageHandler(() => new MyRedirectingHandler());
});

and now in addition to using my custom handler, it'll inject into the handler pipeline support for retries and circuit breakers and so on (which can be customized further).

[sangyuxiaowu]

Here is a temporary solution to modify private variables through reflection, hoping to help friends with needs. Of course, other content can also be modified through this method.

I also submitted a modification Azure/azure-sdk-for-net#41242

using Azure.AI.OpenAI;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.OpenAI;
using System.Reflection;

var builder = Kernel.CreateBuilder();
var aiclietn =  new OpenAIClient(
            new Uri("http://127.0.0.1:8000/v1"),
            new Azure.AzureKeyCredential("empty")
        );
// 获取_isConfiguredForAzureOpenAI字段的引用
var field = typeof(OpenAIClient).GetField("_isConfiguredForAzureOpenAI", BindingFlags.NonPublic | BindingFlags.Instance);
// 修改_isConfiguredForAzureOpenAI字段的值
if (field != null)
{
    field.SetValue(aiclietn, false);
}

builder.AddOpenAIChatCompletion("qwen",aiclietn);
// ... 省略部分代码

[markwallace-microsoft]

All .Net issues prior to 1-Dec-2023 are being closed. Please re-open, if this issue is still relevant to the .Net Semantic Kernel 1.x release. In the future all issues that are inactive for more than 90 days will be labelled as 'stale' and closed 14 days later.