.Net: Bug: Chat Completion Agent with Open AI o1-preview is not working out of the box

[fgauna12]

Describe the bug
Can't use Open AI o1-preview model with ChatCompletionAgent because it fails with the following exception.

Microsoft.SemanticKernel.HttpOperationException: HTTP 400 (invalid_request_error: unsupported_value)

Microsoft.SemanticKernel.HttpOperationException
HTTP 400 (invalid_request_error: unsupported_value)
Parameter: messages[0].role

Unsupported value: 'messages[0].role' does not support 'system' with this model.

To Reproduce
Steps to reproduce the behavior:

[Fact]
    public async Task ThisFailsAndISad()
    {
        var instructions =
            "You are a cat. You're an expert meower.";

        var configuration = new ConfigurationBuilder()
            .AddUserSecrets<SamplePromptsIntegrationTests>()
            .Build();

        var services = new ServiceCollection();

        services.AddLogging();

        var kernel = Kernel.CreateBuilder()
            .AddOpenAIChatCompletion("o1-preview", configuration["OpenAI:APIKey"]!)
            .Build();

#pragma warning disable SKEXP0110
        var agent = new ChatCompletionAgent()
        {
            Kernel = kernel,
            Instructions = instructions
        };

        ChatHistory chat =
        [
            new ChatMessageContent(AuthorRole.User, "Hello")
        ];

        await foreach (ChatMessageContent response in agent.InvokeAsync(chat))
        {
            response.Content?.ToLowerInvariant().Should().Contain("meow");
        }
#pragma warning restore SKEXP0110
    }

Expected behavior
It should just work. 🙃

Screenshots
Nah.

Platform

• OS: Windows 11
• IDE: Rider
• Language: C#
• Source:

        <PackageReference Include="Microsoft.SemanticKernel" Version="1.34.0" />
        <PackageReference Include="Microsoft.SemanticKernel.Agents.Core" Version="1.34.0-alpha" />

Additional context
Nerp.

[markwallace-microsoft]

@fgauna12 My understanding is that o1 models do not support the system message, take a look here: https://community.openai.com/t/o1-models-do-not-support-system-role-in-chat-completion/953880

And general guidance here: https://platform.openai.com/docs/guides/reasoning

[markwallace-microsoft]

@RogerBarreto we should add a sample which shows the correct usage of an o1 model.

[fgauna12]

@markwallace-microsoft Hi yes, I'm aware. I'm using the Chat Completion Agent, so I think the agent is creating a System prompt from the "Instructions."

[crickman]

@fgauna12 - Instructions are optional for an agent. If you don't define any instructions, you may be able to use the o1 model with a ChatCompletionAgent. While this may seem to limit the ability of an agent to specialize by role, I'd assert that specialization may still be accomplished according to the plugins/tools defined for that specific agent. That is, an agent that has access to a git-hub plugin will have different abilities than one that has access to an accounting database or datawarehouse.

If you use the IChatCompletionService directly after calling ChatHistory.AddSystemMessage, you'll get the same runtime error.

In the meantime, we'll explore how to support specifying a role allowed by the model.

[evmin]

Looks like I get a similar issue with o3-mini model, which supposed to support "system".
Yet I am still getting the same error:

 File "/src/backend/.venv/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/open_ai_handler.py", line 99, in _send_completion_request
    raise ServiceResponseException(
semantic_kernel.exceptions.service_exceptions.ServiceResponseException: ("<class 'semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion.AzureChatCompletion'> service failed to complete the prompt", BadRequestError('Error code: 400 - {\'error\': {\'message\': "Unsupported value: \'messages[0].role\' does not support \'system\' with this model.", \'type\': \'invalid_request_error\', \'param\': \'messages[0].role\', \'code\': \'unsupported_value\'}}'))

Using ChatCompletionAgent class, python.

[RogerBarreto]

I have stumbled upon this error myself when attempting to run with Azure Connectors on API's against o1-mini where neither system or developer where working, only user (No way to do a system prompt).

AFAIK Azure currently don't support system and you would need to specify AddDeveloperMessage in the ChatHistory or avoid any system/developer message if the API is not supporting at the moment.

[crickman]

@fgauna12 - The InstructionsRole property has been added to allow for this case. You can assign AuthorRole.Developer when targeting an o-series model.

#10601