Lack of determinism when making API calls through the AzureOpenAI object versus vanilla curl requests with postman

[Moetasim97]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

I have some feature that basically consists of making a multimodal API call to one of the chat completions models to parse Information found in Images. After empirically evaluating the responses in terms of factual accuracy and completeness, I have discovered that the performance is much poorer compared to making the same api call with the exact same hyperparamters but as an http curl request with postman. (Case in point; the content length in the screenshot attached)

P.S., I also tried making the API call but with the requests library instead of the openai client, and the responses were equally poor

To Reproduce

1. Get the base64 encoding of the attached sample image.
2. Populate the necessary vars in the code snippet below.
3. Make the api call for an x number of times (Try 10)
4. Validate completeness by comparing the outputs with the image (Based on our extensive testing, the shortcut to this is to just check whether the value of 0.781 is present for a total of 6 times in the output)

Code snippets

import requests
from utility_modules.util_funcs import encode_image
import json

# Azure OpenAI endpoint configuration
url = "https://{ENDPOINT_DNS}.openai.azure.com/openai/deployments/{model_deployment}/chat/completions"
api_version = "{{API_VERSION}}"
api_key = "{{OPENAI_API_KEY}}"
img_file_name = "{{IMG_FILE}}"
# Headers
headers = {
    "Content-Type": "application/json",
    "api-key": api_key
}
def populate_request_dict(body_dict,encoded_img,model_deployment="gpt-4.1"):
    body_dict["model"]=body_dict["model"].format(model_deployment=model_deployment)
    body_dict["messages"][0]["content"][0]["image_url"]["url"]=body_dict["messages"][0]["content"][0]["image_url"]["url"].format(encoded_image=encoded_img)
    return body_dict
# Request payload
data = {
    "model": "{model_deployment}",
    "max_tokens": 3000,
    "temperature": 0.0,
    "seed": 42,
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64,{encoded_image}"
                    }
                },
                {
                    "type": "text",
                    "text": """What is the information in this table?\n    Return the information found in this image of a table based on the following specifications:\n    - Return the table information in the form of key value pairs separated by a comma.\n    - Each key corresponds to a column in the table, and each value corresponds to the cell value.\n    - Make sure to include all the information in the table in your final output.\n    - You must treat each row as a completely independent entity - if multiple rows have the same value in a column, you must repeat that value for each row explicitly.\n    - Never combine, merge or deduplicate values across rows, even if they are identical.\n    - For tables with repeated values:\n        * Each row must have all its values specified explicitly\n        * Do not use any shorthand notations like \"same as above\" or \"ditto\"\n        * If a value appears in multiple rows, repeat it fully in each row\n        * Never reference other rows or use relative references\n    - You may encounter nested columns; if so, take the time to deduce the right way to format them as key value pairs.\n    - Return HTML tags instead of plain text to preserve the original style of the text, whenever needed.\n\t- Merged cells: Repeat the merged cell value in ALL cells that are part of the merge area\n    - Do not leave any row empty unless the original table cell is empty.\n    - Each row's output must be self-contained and complete, regardless of what appears in other rows.\n    - If a cell is empty in the original table, leave it empty in your output for that row.\n    - Lastly, If the image that you see is not an image of a table, return an empty string."""
                }
            ]
        }
    ]
}

# Parameters for the API version
params = {
    "api-version": api_version
}
populated_data_dict = populate_request_dict(data,encoded_img=encode_image(img_file_name))
model = "gpt-4.1"

parse_model_txt_resp = lambda resp_obj: resp_obj['choices'][0]['message']['content']
def make_multimodal_curl_call(model_deployment="gpt-4.1",data=populated_data_dict):
    # Make the API request
    response = requests.post(url.format(model_deployment=model_deployment), headers=headers, params=params, json=data)
    
    # Check if the request was successful
    response.raise_for_status()
    
    # Parse the JSON response
    result = response.json()
    return result.

OS

Windows

Python version

3.11.4

Library version

openai v1.12.1

[LuminaX-alt]

Since you’ve matched model, seed, temperature, max tokens, and prompt, the two likely root causes are:

Serialization differences between Python’s json module and Postman’s payload.

Base64 encoding differences — sometimes Python’s base64 adds newlines unless you use .replace("\n", "").

Azure API gateway behavior when content-length or chunked transfer encoding changes.
The below code can work i do not have free api key for things but if you try with your api it will give you the result

!pip install requests

import os
import base64
import requests

=============================

CONFIG — UPDATE THESE VALUES

=============================

AZURE_OPENAI_API_KEY = "YOUR_API_KEY"
AZURE_OPENAI_ENDPOINT = "YOUR_ENDPOINT" # e.g. https://myresource.openai.azure.com
AZURE_OPENAI_API_VERSION = "2024-05-01-preview" # or your version
MODEL_DEPLOYMENT = "gpt-4.1" # name of your deployment
IMG_FILE_PATH = "sample.png" # name of uploaded image

=============================

Upload image file in Colab

from google.colab import files
uploaded = files.upload()
IMG_FILE_PATH = list(uploaded.keys())[0] # take first uploaded file

Encode image to base64 (no newlines)

with open(IMG_FILE_PATH, "rb") as f:
encoded_img = base64.b64encode(f.read()).decode("utf-8").replace("\n", "")

Prepare request payload exactly as Postman

data = {
"model": MODEL_DEPLOYMENT,
"max_tokens": 3000,
"temperature": 0.0,
"seed": 42,
"messages": [
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{encoded_img}"
}
},
{
"type": "text",
"text": """What is the information in this table?
Return the information found in this image of a table based on the following specifications:

• Return the table information in the form of key value pairs separated by a comma.
• Each key corresponds to a column in the table, and each value corresponds to the cell value.
• Make sure to include all the information in the table in your final output.
• You must treat each row as a completely independent entity - if multiple rows have the same value in a column, you must repeat that value for each row explicitly.
• Never combine, merge or deduplicate values across rows, even if they are identical.
• For tables with repeated values:
  • Each row must have all its values specified explicitly
  • Do not use any shorthand notations like "same as above" or "ditto"
  • If a value appears in multiple rows, repeat it fully in each row
  • Never reference other rows or use relative references
• You may encounter nested columns; if so, take the time to deduce the right way to format them as key value pairs.
• Return HTML tags instead of plain text to preserve the original style of the text, whenever needed.
• Merged cells: Repeat the merged cell value in ALL cells that are part of the merge area
• Do not leave any row empty unless the original table cell is empty.
• Each row's output must be self-contained and complete, regardless of what appears in other rows.
• If a cell is empty in the original table, leave it empty in your output for that row.
• Lastly, If the image that you see is not an image of a table, return an empty string."""
  }
  ]
  }
  ]
  }

Request headers & params

headers = {
"Content-Type": "application/json",
"api-key": AZURE_OPENAI_API_KEY
}
params = {"api-version": AZURE_OPENAI_API_VERSION}

Run multiple times

results_array = []
for i in range(10):
resp = requests.post(
f"{AZURE_OPENAI_ENDPOINT}/openai/deployments/{MODEL_DEPLOYMENT}/chat/completions",
headers=headers,
params=params,
json=data
)
resp.raise_for_status()
content = resp.json()['choices'][0]['message']['content']
results_array.append(content)
print(f"Run {i+1}:\n{content}\n{'='*40}")

Optional: check number of times 0.781 appears

count_0781 = sum(r.count("0.781") for r in results_array)
print(f"\n'0.781' appeared {count_0781} times across all runs")