Using groq as a chat completion service in semantic kernel #9623
-
What would be the recommended way in semantic kernel to consume groq api for chatCompletion ? but i also discovered Microsoft.Extensions.AI that recommends using an IChatClient : What would be the recommended way in SK. @RogerBarreto |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
@giannik Keep in mind that we don't have an official connector for groq, and normally for API's that follows the OpenAI Chat Completion Standard, for simple tasks using the OpenAI connector may work as expected. Important Semantic Kernel is very extensible and there's not a definitive answer, you can use below approaches OpenAI Custom EndpointIn the first example you don't necessarily need a We also have some UnitTest's ensuring this, see: Using a custom endpoint: var chatCompletion = new OpenAIChatCompletionService(
modelId: "any",
apiKey: "api-key",
endpoint: new Uri("https://api.groq.com/openai/v1")) AI AbstractionsUsing the example provided in the blog you can convert the using Microsoft.SemanticKernel;
// <ProviderChatClient> here can be any, OllamaChatClient, OpenAIChatClient, etc... for groq, OpenAI may work best.
var service = new <ProviderChatClient>(...)
.AsChatCompletionService();
// your SK code Http HandlerFollow the approach mentioned in the link HttpClient with a custom delegate handler |
Beta Was this translation helpful? Give feedback.
-
Following the IChatClient approach, I wrote an adapter by taking advantage of Groq's compatibility with OpenAI's API. Codeusing System.ClientModel;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Options;
using OpenAI;
namespace Adapters;
public class GroqChatClientAdapter : IChatClient
{
private readonly IChatClient _client;
public GroqChatClientAdapter(IOptions<GroqConfiguration> configuration)
{
var groqConfiguration = configuration.Value;
var options = new OpenAIClientOptions
{
Endpoint = new Uri(groqConfiguration.Endpoint)
};
var client =
new OpenAIClient(
new ApiKeyCredential(groqConfiguration.ApiKey),
options)
.AsChatClient(groqConfiguration.Model);
_client = client;
}
public void Dispose()
{
_client.Dispose();
}
public Task<ChatResponse> GetResponseAsync(IList<ChatMessage> chatMessages, ChatOptions? options = null,
CancellationToken cancellationToken = new()) =>
_client.GetResponseAsync(chatMessages, options, cancellationToken);
public IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(IList<ChatMessage> chatMessages,
ChatOptions? options = null,
CancellationToken cancellationToken = new()) =>
_client.GetStreamingResponseAsync(chatMessages, options, cancellationToken);
public object? GetService(Type serviceType, object? serviceKey = null) =>
_client.GetService(serviceType, serviceKey);
} Dependency Injection public static void AddGroqChatClient(this IServiceCollection services, IConfiguration configuration)
{
services.AddChatClient(c =>
ActivatorUtilities.CreateInstance<GroqChatClientAdapter>(c));
services.Configure<GroqConfiguration>(configuration.GetSection("GroqConfiguration"));
} Configuration classpublic class GroqConfiguration
{
public string Endpoint { get; init; } = string.Empty;
public string ApiKey { get; init; } = string.Empty;
public string Model { get; init; } = string.Empty;
} |
Beta Was this translation helpful? Give feedback.
@giannik Keep in mind that we don't have an official connector for groq, and normally for API's that follows the OpenAI Chat Completion Standard, for simple tasks using the OpenAI connector may work as expected.
Important
Semantic Kernel is very extensible and there's not a definitive answer, you can use below approaches
OpenAI Custom Endpoint
In the first example you don't necessarily need a
httphandler
, as the OpenAI SDK allows custom endpoints to be passed.We also have some UnitTest's ensuring this, see:
semantic-kernel/dotnet/src/Connectors/Connectors.OpenAI.UnitTests/Services/OpenAIChatCompletionServiceTests.cs
Line 87 in ffac88a