Skip to content

Foundry returns 500 status code when using ResponseFormat (aka structured response) in ChatClient options #81

@tomlm

Description

@tomlm

Activity

jonwis

jonwis commented on May 21, 2025

@jonwis
Member

Can you include what model you were using? The output of (await manager.ListLoadedModelsAsync()).Select(_ => _.ModelId); for instance?

tomlm

tomlm commented on May 23, 2025

@tomlm
Author

Here's repo code:

   public static async Task Main(string[] args)
   {
       var app = new TestApp(); // Create an instance of TestApp  

       var modelName = "Phi-4-mini-instruct-cuda-gpu";
       var manager = new FoundryManager();
       await manager.StartServiceAsync();
       var modelInfo = await manager.GetModelInfoAsync(modelName);
       Console.WriteLine($"Model: {modelInfo.Alias} ({modelInfo.ModelId}) {modelInfo.Version}");
       await manager.LoadModelAsync(modelName);

       var schema =
           $$"""
           {
             "type": "object",
             "additionalProperties": false,
             "required": [
               "Explanation",
               "Result"
             ],
             "properties": {
               "Explanation": {
                 "type": [
                   "string",
                   "null"
                 ],
                 "description": "Explain your reasoning"
               },
               "Result": {
                 "type": [
                   "array",
                   "null"
                 ],
                 "items": {
                   "type": "string"
                 },
                 "description": "The result"
               }
             }
           }
           """;
       var responseFormat = ChatResponseFormat.CreateJsonSchemaFormat("result", 
           jsonSchema: BinaryData.FromString(schema), 
           jsonSchemaFormatDescription: "An JSON object with two properties: Explanation and Result",
           jsonSchemaIsStrict: true);

       ApiKeyCredential key = new ApiKeyCredential(manager.ApiKey);
       OpenAIClient client = new OpenAIClient(key, new OpenAIClientOptions
       {
           Endpoint = manager.Endpoint
       });

       var chatClient = client.GetChatClient(modelInfo.ModelId);

       var result = chatClient.CompleteChat(["List five common male first names?"], new ChatCompletionOptions()
       {
           ResponseFormat = responseFormat 
       });
       Console.WriteLine(result.Value.Content[0].Text);
       Console.WriteLine("Press any key to exit...");
       Console.ReadKey(true);
   }
  1. If you comment out the ResponseFormat = responseFormat line the local model returns text response correctly
  2. If you change it to call OpenAI with following changes:
  • set modelName to "gpt-4o"
  • configure a real "OpenAI" key
  • comment out Endpoint = manager.Endpoint
    The OpenAI response correctly returns a JSON object matching the schema.

This could be because either:
a. the phi-4 model doesn't support structured output OR
b. the schema is blowing up foundry.

tomlm

tomlm commented on May 23, 2025

@tomlm
Author

Also this seems to happen with

  • phi-4-mini
  • qwen2.5-0.5b
  • qwen2.5-1.5b
    I've not tried more models than that.
changed the title [-]Foundry returns 500 status code when using ResponseFormat =JsonSchema in ChatClient options[/-] [+]Foundry returns 500 status code when using ResponseFormat (aka structured response) in ChatClient options[/+] on May 23, 2025
samuel100

samuel100 commented on Jun 6, 2025

@samuel100
Contributor

Duplicate of #112

marked this as a duplicate of Structured output #112 on Jun 6, 2025
added
duplicateThis issue or pull request already exists
papercutThis issue relates to user experience papercut
on Jun 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ADOduplicateThis issue or pull request already existspapercutThis issue relates to user experience papercut

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Participants

    @samuel100@tomlm@jonwis

    Issue actions

      Foundry returns 500 status code when using ResponseFormat (aka structured response) in ChatClient options · Issue #81 · microsoft/Foundry-Local