gemini/controlled-generation/intro_controlled_generation.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2024 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Intro to Controlled Generation with the Gemini API\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/gemini/controlled-generation/intro_controlled_generation.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2FGoogleCloudPlatform%2Fgenerative-ai%2Fmain%2Fgemini%2Fcontrolled-generation%2Fintro_controlled_generation.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/gemini/controlled-generation/intro_controlled_generation.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/controlled-generation/intro_controlled_generation.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "84f0f73a0f76"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Eric Dong](https://github.com/gericdong)|"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "### Gemini\n",
        "\n",
        "Gemini is a family of generative AI models developed by Google DeepMind that is designed for multimodal use cases.\n",
        "\n",
        "### Controlled Generation\n",
        "\n",
        "Depending on your application, you may want the model response to a prompt to be returned in a structured data format, particularly if you are using the responses for downstream processes, such as downstream modules that expect a specific format as input. The Gemini API provides the controlled generation capability to constraint the model output to a structured format.\n",
        "\n",
        "\n",
        "### Objectives\n",
        "\n",
        "In this tutorial, you learn how to use the controlled generation capability in the Vertex AI Gemini API to generate model responses in a JSON object with specific fields.\n",
        "\n",
        "You will complete the following tasks:\n",
        "\n",
        "- Using `response_mime_type` with the Gemini 1.5 Flash models\n",
        "- Using `response_mime_type` and `response_schema` with the Gemini 1.5 Pro models\n",
        "- Using controlled generation in use cases requiring output constraints\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "61RBz8LLbxCR"
      },
      "source": [
        "## Get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "No17Cw5hgx12"
      },
      "source": [
        "### Install Vertex AI SDK and other required packages\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tFy3H3aPgx12"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --user --quiet google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel.\n",
        "\n",
        "The restart might take a minute or longer. After it's restarted, continue to the next step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Wait until it's finished before continuing to the next step. ⚠️</b>\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmWOrTJ3gx13"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you're running this notebook on Google Colab, run the cell below to authenticate your environment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NyKGtVQjgx13"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DF4l8DTdWgPY"
      },
      "source": [
        "### Set Google Cloud project information and initialize Vertex AI SDK\n",
        "\n",
        "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n",
        "\n",
        "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nqwi-5ufWp_B"
      },
      "outputs": [],
      "source": [
        "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "\n",
        "import vertexai\n",
        "\n",
        "vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EdvJRUWRNGHE"
      },
      "source": [
        "## Code Examples"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "09720c707f1c"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e45ea9a28734"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "from vertexai import generative_models\n",
        "from vertexai.generative_models import GenerationConfig, GenerativeModel, Part"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "74badac24b3e"
      },
      "source": [
        "### Using `response_mime_type` with the Gemini 1.5 Flash models\n",
        "\n",
        "You can have the model output in certain format by setting the `response_mime_type` configuration option in `generation_config`, and in the prompt, describe the format you want in response."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4a9c4ebc507b"
      },
      "outputs": [],
      "source": [
        "model = GenerativeModel(\n",
        "    model_name=\"gemini-1.5-flash\",\n",
        "    generation_config={\"response_mime_type\": \"application/json\"},\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a63b746a44cf"
      },
      "source": [
        "In the prompt, describe the format you want in response."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "37292b0e4ef6"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"\n",
        "    List a few popular cookie recipes using this JSON schema:\n",
        "    Recipe = {\"recipe_name\": str}\n",
        "    Return: list[Recipe]\n",
        "\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "09e3f92c710c"
      },
      "source": [
        "Generate the content and parse the response string to JSON."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "fee244ad523e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{'recipe_name': 'Chocolate Chip Cookies'}, {'recipe_name': 'Oatmeal Raisin Cookies'}, {'recipe_name': 'Snickerdoodles'}, {'recipe_name': 'Sugar Cookies'}, {'recipe_name': 'Peanut Butter Cookies'}]\n"
          ]
        }
      ],
      "source": [
        "response = model.generate_content(prompt)\n",
        "\n",
        "json_response = json.loads(response.text)\n",
        "print(json_response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "52aeea15a479"
      },
      "source": [
        "### Using `response_mime_type` and `response_schema` with the Gemini 1.5 Pro models\n",
        "\n",
        "While Gemini 1.5 Flash models only accept a text description of the schema you want returned, the Gemini 1.5 Pro models let you pass a data structure in the `response_schema` parameter in `generation_config`, and the model output will strictly follow that schema.\n",
        "\n",
        "Note that when `response_schema` is specified, the `response_mime_type` has to be set to `application/json`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "81cbb6bd51d8"
      },
      "outputs": [],
      "source": [
        "model = GenerativeModel(\"gemini-1.5-pro\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "766346c046f9"
      },
      "source": [
        "Following the previous example, define the data structure for the model output. Note that all of the fields in the JSON are optional by default unless specified in the `required` field."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "af3fa1fbff4f"
      },
      "outputs": [],
      "source": [
        "response_schema = {\n",
        "    \"type\": \"ARRAY\",\n",
        "    \"items\": {\n",
        "        \"type\": \"OBJECT\",\n",
        "        \"properties\": {\n",
        "            \"recipe_name\": {\n",
        "                \"type\": \"STRING\",\n",
        "            },\n",
        "        },\n",
        "        \"required\": [\"recipe_name\"],\n",
        "    },\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "82033e70bf6e"
      },
      "source": [
        "When prompting the model to generate the content, pass the schema to the `response_schema` field of the `generation_config`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "5db8b91d5be0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{\"recipe_name\": \"Classic Chocolate Chip Cookies\"}, {\"recipe_name\": \"Peanut Butter Cookies\"}, {\"recipe_name\": \"Snickerdoodles\"}, {\"recipe_name\": \"Oatmeal Raisin Cookies\"}, {\"recipe_name\": \"Shortbread Cookies\"}] \n"
          ]
        }
      ],
      "source": [
        "response = model.generate_content(\n",
        "    \"List a few popular cookie recipes\",\n",
        "    generation_config=GenerationConfig(\n",
        "        response_mime_type=\"application/json\", response_schema=response_schema\n",
        "    ),\n",
        ")\n",
        "\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ca9af4346be7"
      },
      "source": [
        "You can parse the response string to JSON."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "76b5284016c0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{'recipe_name': 'Classic Chocolate Chip Cookies'}, {'recipe_name': 'Peanut Butter Cookies'}, {'recipe_name': 'Snickerdoodles'}, {'recipe_name': 'Oatmeal Raisin Cookies'}, {'recipe_name': 'Shortbread Cookies'}]\n"
          ]
        }
      ],
      "source": [
        "json_response = json.loads(response.text)\n",
        "print(json_response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "69450c61bc07"
      },
      "source": [
        "### Using controlled generation in use cases requiring output constraints\n",
        "\n",
        "Controlled generation can be used to ensure that model outputs adhere to a specific structure (e.g., JSON), instruct the model to perform pure multiple choices (e.g., sentiment classification), or follow certain style or guidelines.\n",
        "\n",
        "Let's use controlled generation with the Gemini 1.5 Pro models in the following use cases that require output constraints."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "64f95263b81d"
      },
      "outputs": [],
      "source": [
        "model = GenerativeModel(\"gemini-1.5-pro\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eba9ef4d4b50"
      },
      "source": [
        "#### **Example**: Generate game character profile\n",
        "\n",
        "In this example, you instruct the model to create a game character profile with some specific requirements, and constraint the model output to a structured format. This example also demonstrates how to configure the `response_schema` and `response_mime_type` fields in `generative_config` in conjunction with `safety_settings`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "1411f729f2f7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            " [{\n",
            "        \"age\": 42,\n",
            "        \"children\": [\n",
            "          {\n",
            "            \"age\": 21,\n",
            "            \"name\": \"Merida\"\n",
            "          },\n",
            "          {\n",
            "            \"age\": 18,\n",
            "            \"name\": \"Fergus\"\n",
            "          },\n",
            "          {\n",
            "            \"age\": 18,\n",
            "            \"name\": \"Harris\"\n",
            "          }\n",
            "        ],\n",
            "        \"name\": \"Eleanor\",\n",
            "        \"occupation\": \"Queen\",\n",
            "        \"background\": \"Eleanor, the beloved ruler of a prosperous kingdom, is known for her wisdom, grace, and unwavering strength.  After the untimely death of her husband, she has successfully navigated countless challenges, earning her the admiration of both her people and neighboring rulers.  However, a new threat emerges, one that will test Eleanor's mettle and force her to confront her past\",\n",
            "        \"playable\": false\n",
            "      },\n",
            "      {\n",
            "        \"age\": 25,\n",
            "        \"children\": [],\n",
            "        \"name\": \"Kaelen\",\n",
            "        \"occupation\": \"Hunter\",\n",
            "        \"background\": \"Kaelen is a skilled hunter and tracker who lives off the land, relying on his instincts and knowledge of the wilderness. He is fiercely independent and wary of outsiders, but his loyalty to those he trusts is unwavering. Haunted by a tragic event from his past, Kaelen struggles to balance his desire for revenge with his inherent sense of justice\",\n",
            "        \"playable\": true\n",
            "      }\n",
            "] \n"
          ]
        }
      ],
      "source": [
        "response_schema = {\n",
        "    \"type\": \"ARRAY\",\n",
        "    \"items\": {\n",
        "        \"type\": \"OBJECT\",\n",
        "        \"properties\": {\n",
        "            \"name\": {\"type\": \"STRING\"},\n",
        "            \"age\": {\"type\": \"INTEGER\"},\n",
        "            \"occupation\": {\"type\": \"STRING\"},\n",
        "            \"background\": {\"type\": \"STRING\"},\n",
        "            \"playable\": {\"type\": \"BOOLEAN\"},\n",
        "            \"children\": {\n",
        "                \"type\": \"ARRAY\",\n",
        "                \"items\": {\n",
        "                    \"type\": \"OBJECT\",\n",
        "                    \"properties\": {\n",
        "                        \"name\": {\"type\": \"STRING\"},\n",
        "                        \"age\": {\"type\": \"INTEGER\"},\n",
        "                    },\n",
        "                    \"required\": [\"name\", \"age\"],\n",
        "                },\n",
        "            },\n",
        "        },\n",
        "        \"required\": [\"name\", \"age\", \"occupation\", \"children\"],\n",
        "    },\n",
        "}\n",
        "\n",
        "prompt = \"\"\"\n",
        "    Generate a character profile for a video game, including the character's name, age, occupation, background, names of their\n",
        "    three children, and whether they can be controlled by the player.\n",
        "\"\"\"\n",
        "\n",
        "response = model.generate_content(\n",
        "    prompt,\n",
        "    generation_config=GenerationConfig(\n",
        "        response_mime_type=\"application/json\", response_schema=response_schema\n",
        "    ),\n",
        "    safety_settings={\n",
        "        generative_models.HarmCategory.HARM_CATEGORY_HATE_SPEECH: generative_models.HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE,\n",
        "        generative_models.HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: generative_models.HarmBlockThreshold.BLOCK_ONLY_HIGH,\n",
        "        generative_models.HarmCategory.HARM_CATEGORY_HARASSMENT: generative_models.HarmBlockThreshold.BLOCK_LOW_AND_ABOVE,\n",
        "        generative_models.HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: generative_models.HarmBlockThreshold.BLOCK_NONE,\n",
        "    },\n",
        ")\n",
        "\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e02769d61054"
      },
      "source": [
        "#### **Example**: Extract errors from log data\n",
        "\n",
        "In this example, you use the model to pull out specific error messages from unstructured log data, extract key information, and constraint the model output to a structured format.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "007c0394cadc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[{\"error_code\": 308, \"error_message\": \"Could not process image upload: Unsupported file format.\" , \"timestamp\": \"15:43:28\"}, {\"error_code\": 5522, \"error_message\": \"Service dependency unavailable (payment gateway). Retrying...\" , \"timestamp\": \"15:45:02\"}, {\"error_code\": 9001, \"error_message\": \"Application crashed due to out-of-memory exception.\" , \"timestamp\": \"15:45:33\"}] \n"
          ]
        }
      ],
      "source": [
        "response_schema = {\n",
        "    \"type\": \"ARRAY\",\n",
        "    \"items\": {\n",
        "        \"type\": \"OBJECT\",\n",
        "        \"properties\": {\n",
        "            \"timestamp\": {\"type\": \"STRING\"},\n",
        "            \"error_code\": {\"type\": \"INTEGER\"},\n",
        "            \"error_message\": {\"type\": \"STRING\"},\n",
        "        },\n",
        "        \"required\": [\"timestamp\", \"error_message\", \"error_code\"],\n",
        "    },\n",
        "}\n",
        "\n",
        "prompt = \"\"\"\n",
        "[15:43:28] ERROR: Could not process image upload: Unsupported file format. (Error Code: 308)\n",
        "[15:44:10] INFO: Search index updated successfully.\n",
        "[15:45:02] ERROR: Service dependency unavailable (payment gateway). Retrying... (Error Code: 5522)\n",
        "[15:45:33] ERROR: Application crashed due to out-of-memory exception. (Error Code: 9001)\n",
        "\"\"\"\n",
        "\n",
        "response = model.generate_content(\n",
        "    prompt,\n",
        "    generation_config=GenerationConfig(\n",
        "        response_mime_type=\"application/json\", response_schema=response_schema\n",
        "    ),\n",
        ")\n",
        "\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a74594893037"
      },
      "source": [
        "#### **Example**: Analyze product review data\n",
        "\n",
        "In this example, you instruct the model to analyze product review data, extract key entities, perform sentiment classification (multiple choices), provide additional explanation, and output the results in JSON format."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "9a3b8b9800f9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[\n",
            "  [\n",
            "    {\n",
            "      \"explanation\": \"Strong positive sentiment with superlative language (\\\"best ever\\\")\",\n",
            "      \"flavor\": \"Strawberry Cheesecake\",\n",
            "      \"rating\": 4,\n",
            "      \"sentiment\": \"POSITIVE\"\n",
            "    }\n",
            "  ],\n",
            "  [\n",
            "    {\n",
            "      \"explanation\": \"Mixed sentiment - acknowledges positive aspects (\\\"quite good\\\") but expresses a negative preference (\\\"too sweet\\\")\",\n",
            "      \"flavor\": \"Mango Tango\",\n",
            "      \"rating\": 1,\n",
            "      \"sentiment\": \"NEGATIVE\"\n",
            "    }\n",
            "  ]\n",
            "] \n"
          ]
        }
      ],
      "source": [
        "response_schema = {\n",
        "    \"type\": \"ARRAY\",\n",
        "    \"items\": {\n",
        "        \"type\": \"ARRAY\",\n",
        "        \"items\": {\n",
        "            \"type\": \"OBJECT\",\n",
        "            \"properties\": {\n",
        "                \"rating\": {\"type\": \"INTEGER\"},\n",
        "                \"flavor\": {\"type\": \"STRING\"},\n",
        "                \"sentiment\": {\n",
        "                    \"type\": \"STRING\",\n",
        "                    \"enum\": [\"POSITIVE\", \"NEGATIVE\", \"NEUTRAL\"],\n",
        "                },\n",
        "                \"explanation\": {\"type\": \"STRING\"},\n",
        "            },\n",
        "            \"required\": [\"rating\", \"flavor\", \"sentiment\", \"explanation\"],\n",
        "        },\n",
        "    },\n",
        "}\n",
        "\n",
        "prompt = \"\"\"\n",
        "  Analyze the following product reviews, output the sentiment classification and give an explanation.\n",
        "  \n",
        "  - \"Absolutely loved it! Best ice cream I've ever had.\" Rating: 4, Flavor: Strawberry Cheesecake\n",
        "  - \"Quite good, but a bit too sweet for my taste.\" Rating: 1, Flavor: Mango Tango\n",
        "\"\"\"\n",
        "\n",
        "response = model.generate_content(\n",
        "    prompt,\n",
        "    generation_config=GenerationConfig(\n",
        "        response_mime_type=\"application/json\", response_schema=response_schema\n",
        "    ),\n",
        ")\n",
        "\n",
        "print(response.text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "10971b23afcf"
      },
      "source": [
        "#### Example: Detect objects in images\n",
        "\n",
        "You can also use controlled generation in multimodality use cases. In this example, you instruct the model to detect objects in the images and output the results in JSON format. These images are stored in a Google Storage bucket.\n",
        "\n",
        "- [office-desk.jpeg](https://storage.googleapis.com/cloud-samples-data/generative-ai/image/office-desk.jpeg)\n",
        "- [gardening-tools.jpeg](https://storage.googleapis.com/cloud-samples-data/generative-ai/image/gardening-tools.jpeg)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "1f3e9935e2da"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[\n",
            "    [{\"object\": \"globe\"}, {\"object\": \"tablet\"}, {\"object\": \"shopping cart\"}, {\"object\": \"eiffel tower\"}, {\"object\": \"airplane\"}, {\"object\": \"passport\"}, {\"object\": \"keyboard\"}, {\"object\": \"computer mouse\"}, {\"object\": \"sunglasses\"}, {\"object\": \"money\"}, {\"object\": \"notebook\"}, {\"object\": \"pen\"}, {\"object\": \"coffee cup\"}],\n",
            "    [{\"object\": \"watering can\"}, {\"object\": \"plant\"}, {\"object\": \"flower pot\"}, {\"object\": \"flower pot\"}, {\"object\": \"garden gloves\"}, {\"object\": \"garden trowel\"}, {\"object\": \"garden hand tool\"}]\n",
            "] \n"
          ]
        }
      ],
      "source": [
        "response_schema = {\n",
        "    \"type\": \"ARRAY\",\n",
        "    \"items\": {\n",
        "        \"type\": \"ARRAY\",\n",
        "        \"items\": {\n",
        "            \"type\": \"OBJECT\",\n",
        "            \"properties\": {\n",
        "                \"object\": {\"type\": \"STRING\"},\n",
        "            },\n",
        "        },\n",
        "    },\n",
        "}\n",
        "\n",
        "prompt = \"Generate a list of objects in the images.\"\n",
        "\n",
        "response = model.generate_content(\n",
        "    [\n",
        "        Part.from_uri(\n",
        "            \"gs://cloud-samples-data/generative-ai/image/office-desk.jpeg\",\n",
        "            \"image/jpeg\",\n",
        "        ),\n",
        "        Part.from_uri(\n",
        "            \"gs://cloud-samples-data/generative-ai/image/gardening-tools.jpeg\",\n",
        "            \"image/jpeg\",\n",
        "        ),\n",
        "        prompt,\n",
        "    ],\n",
        "    generation_config=GenerationConfig(\n",
        "        response_mime_type=\"application/json\", response_schema=response_schema\n",
        "    ),\n",
        ")\n",
        "\n",
        "print(response.text)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "intro_controlled_generation.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}