Possible to return BOTH structured (application/json) and unstructured (markdown) output?

[khteh]

Is it possible for Gemini request to return BOTH structured (application/json) and unstructured (markdown) output in a single API call? The use case:

(1) Structured output for backend business logic process and DB persistence.

(2) Unstructured (markdown) to show it on the UI, maybe for human-in-the-loop validation of the data returned by the model.

For example, I have the following string / object returned:

{"message":"{\n  \"date_str\": \"28-07-2017\",\n  \"vendor\": \"Walmart\",\n  \"currency\": \"USD\",\n  \"items\": [\n    {\n      \"name\": \"PET TOY\",\n      \"amount\": 1.97\n    },\n    {\n      \"name\": \"FLOPPY PUPPY\",\n      \"amount\": 1.97\n    },\n    {\n      \"name\": \"SSSUPREME S\",\n      \"amount\": 4.97\n    },\n    {\n      \"name\": \"2.5 SQUEAK\",\n      \"amount\": 5.92\n    },\n    {\n      \"name\": \"MUNCHY DMBEL\",\n      \"amount\": 3.77\n    },\n    {\n      \"name\": \"DOG TREAT\",\n      \"amount\": 2.92\n    },\n    {\n      \"name\": \"PED PCH 1\",\n      \"amount\": 0.50\n    },\n    {\n      \"name\": \"PED PCH 1\",\n      \"amount\": 0.50\n    },\n    {\n      \"name\": \"COUPON 23100\",\n      \"amount\": 1.00\n    },\n    {\n      \"name\": \"HNYMD SMORES\",\n      \"amount\": 3.98\n    },\n    {\n      \"name\": \"FRENCH DRSNG\",\n      \"amount\": 1.98\n    },\n    {\n      \"name\": \"3 ORANGES\",\n      \"amount\": 5.47\n    },\n    {\n      \"name\": \"BABY CARROTS\",\n      \"amount\": 1.48\n    },\n    {\n      \"name\": \"COLLARDS\",\n      \"amount\": 1.24\n    },\n    {\n      \"name\": \"CALZONE\",\n      \"amount\": 2.50\n    },\n    {\n      \"name\": \"MM RVW MNT\",\n      \"amount\": 19.77\n    },\n    {\n      \"name\": \"STKOBRLPLABL\",\n      \"amount\": 1.97\n    },\n    {\n      \"name\": \"STKOBRLPLABL\",\n      \"amount\": 1.97\n    },\n    {\n      \"name\": \"STKO SUNFLWR\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"STKO SUNFLWR\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"STKO SUNFLWR\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"STKO SUNFLWR\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"BLING BEADS\",\n      \"amount\": 0.97\n    },\n    {\n      \"name\": \"GREAT VALUE\",\n      \"amount\": 9.97\n    },\n    {\n      \"name\": \"LIPTON\",\n      \"amount\": 4.48\n    },\n    {\n      \"name\": \"DRY DOG\",\n      \"amount\": 12.44\n    }\n  ],\n  \"tax\": 4.59,\n  \"total\": 98.21\n}"}

and I try to use showdown javascript library to convert it to markdown and makeHtml but to no avail. What I get on the UI is only an unformatted json string and it is difficult to verify if the model has missed anything or introduced any hallunication in it's reasoning.

[hkt74]

Hi @khteh thanks for raising this question.

Could you try define your own schema with pydantic model?

See the sample code from README

you can utilize the response.parsed to get the object as well.

from pydantic import BaseModel
from google.genai import types


class CountryInfo(BaseModel):
    name: str
    population: int
    capital: str
    continent: str
    gdp: int
    official_language: str
    total_area_sq_mi: int
    short_description: str


response = client.models.generate_content(
    model='gemini-2.5-flash-preview-05-20',
    contents='Give me information for the United States. For the short_description, could you make it return in markdown format?',
    config=types.GenerateContentConfig(
        response_mime_type='application/json',
        response_schema=CountryInfo,
    ),
)
country_info = response.parsed
print(country_info.short_description)

[khteh]

That's exactly what I do. While I definitely get the structured parsed model, I don't get the markdown text.

[hkt74]

Could you try using gemini-2.5-pro-preview-05-06 and specify the field to use markdown in the prompt?

I tested with below code and got markdown text from the short_description

from IPython.display import Markdown, display

class CountryInfo(BaseModel):
    name: str
    population: int
    capital: str
    continent: str
    gdp: int
    official_language: str
    total_area_sq_mi: int
    short_description: str

response = client.models.generate_content(
    model='gemini-2.5-pro-preview-05-06',
    contents='Give me information for the United States. For the short_description, could you make it return in markdown format?',
    config=types.GenerateContentConfig(
        response_mime_type='application/json',
        response_schema=CountryInfo,
    ),
)
country_info = response.parsed

display(Markdown(country_info.short_description)) 

[khteh]

This requires baking a free-text field/property/attribute into the schema which is not used in the backend business logic but only to show to the user. What I want is to show the json string/object verbatim to the man-in-the-loop verification as part of the image OCR.

[hkt74]

Understood. Currently, returning both structured and unstructured responses together is not supported. A workaround is to include the free-text content as a field within your schema definition. When processing the function output, you'll need to extract or handle this field separately.

[khteh]

Would you consider supporting this in the future?