language/prompts/examples/text_extraction.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur8xi4C7S06n"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAPoU8Sm5E6e"
      },
      "source": [
        "# Text Extraction with Generative Models on Vertex AI\n",
        "\n",
        "> **NOTE:** This notebook uses the PaLM generative model, which will reach its [discontinuation date in October 2024](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text#model_versions). Please refer to [this updated notebook](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/examples/text_extraction.ipynb) for a version which uses the latest Gemini model.\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/language/prompts/examples/text_extraction.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/language/prompts/examples/text_extraction.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/language/prompts/examples/text_extraction.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8e71fd522e56"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Polong Lin](https://github.com/polong-lin) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tvgnzT1CKxrO"
      },
      "source": [
        "## Overview\n",
        "\n",
        "Text extraction is a process of extracting text from a document. This can be done manually or automatically. Manual text extraction is the process of reading the document and copying the text into a new document. Automatic text extraction is the process of using software to extract the text from the document.\n",
        "\n",
        "Text extraction can be used for a variety of purposes. One common purpose is to convert documents into a machine-readable format. This can be useful for storing documents in a database or for processing documents with software. Another common purpose is to extract information from documents. This can be useful for finding specific information in a document or for summarizing the content of a document.\n",
        "\n",
        "Large language models (LLMs) are good for text extraction because they are trained on massive datasets of text and code, which allows them to learn the relationships between words and phrases. They can also understand the context of text and generate text, which allows them to extract information that is not explicitly stated or fill in the gaps in text that is missing information. The answers from LLMs can also be further improved through methods like few-shot prompting.\n",
        "\n",
        "Learn more about extraction prompts in the [official documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/text/extraction-prompts)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d975e698c9a4"
      },
      "source": [
        "### Objective\n",
        "\n",
        "In this tutorial, you will learn how to use generative models to extract the information from text by working through the following examples:\n",
        "- Google Pixel technical specifications extraction\n",
        "- WiFi troubleshooting with constraints\n",
        "- Respond to inquiries in character\n",
        "- Converting an ingredients list to JSON format\n",
        "- Organizing the results of a text extraction\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aed92deeb4a0"
      },
      "source": [
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* Vertex AI Generative AI Studio\n",
        "\n",
        "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
        "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xoMiNbiCDM2w"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2a5AEr0lkLKD"
      },
      "source": [
        "### Install Vertex AI SDK"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "148dd6321946"
      },
      "outputs": [],
      "source": [
        "%pip install google-cloud-aiplatform --upgrade --user"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FLVWFKFwkLKE"
      },
      "source": [
        "**Colab only:** Uncomment the following cell to restart the kernel or use the button to restart the kernel. For Vertex AI Workbench you can restart the terminal using the button on top. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_Hsqwn4hkLKE"
      },
      "outputs": [],
      "source": [
        "# # Automatically restart kernel after installs so that your environment can access the new packages\n",
        "# import IPython\n",
        "\n",
        "# app = IPython.Application.instance()\n",
        "# app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xe7OuYuGkLKF"
      },
      "source": [
        "### Authenticating your notebook environment\n",
        "* If you are using **Colab** to run this notebook, uncomment the cell below and continue.\n",
        "* If you are using **Vertex AI Workbench**, check out the setup instructions [here](https://github.com/GoogleCloudPlatform/generative-ai/tree/main/setup-env)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U9Gx2SAZkLKF"
      },
      "outputs": [],
      "source": [
        "# from google.colab import auth\n",
        "# auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "960505627ddf"
      },
      "source": [
        "### Import libraries\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1db41b42c089"
      },
      "source": [
        "**Colab only:** Uncomment the following cell to initialize the Vertex AI SDK. For Vertex AI Workbench, you don't need to run this.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9100a53f80c2"
      },
      "outputs": [],
      "source": [
        "# import vertexai\n",
        "\n",
        "# PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "# vertexai.init(project=PROJECT_ID, location=\"us-central1\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyQmSRbKA8r-"
      },
      "outputs": [],
      "source": [
        "from vertexai.language_models import TextGenerationModel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UP76a2la7O-a"
      },
      "source": [
        "### Import models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7isig7e07O-a"
      },
      "outputs": [],
      "source": [
        "generation_model = TextGenerationModel.from_pretrained(\"text-bison\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wbll-sxLD58J"
      },
      "source": [
        "## Text Extraction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H_uxqUPADM2z"
      },
      "source": [
        "### Google Pixel technical specifications extraction\n",
        "\n",
        "In this example, you try to extract the technical specifications of a Pixel phone from text in JSON format using the PaLM API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-T5drTkzDM2z"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"\n",
        "Extract the technical specifications from the text below in JSON format.\n",
        "\n",
        "Text: Google Nest WiFi, network speed up to 1200Mpbs, 2.4GHz and 5GHz frequencies, WP3 protocol\n",
        "JSON: {\n",
        "  \"product\":\"Google Nest WiFi\",\n",
        "  \"speed\":\"1200Mpbs\",\n",
        "  \"frequencies\": [\"2.4GHz\", \"5GHz\"],\n",
        "  \"protocol\":\"WP3\"\n",
        "}\n",
        "\n",
        "Text: Google Pixel 7, 5G network, 8GB RAM, Tensor G2 processor, 128GB of storage, Lemongrass\n",
        "JSON:\n",
        "\"\"\"\n",
        "\n",
        "print(\n",
        "    generation_model.predict(\n",
        "        prompt, temperature=0.2, max_output_tokens=1024, top_k=40, top_p=0.8\n",
        "    ).text\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rpspa8CzDM20"
      },
      "source": [
        "### WiFi troubleshooting with constraints\n",
        "\n",
        "In this example, you ask the generative model to answer a question about troubleshooting a Google WiFi router based on the description of the different status lights on the router. The model will only be able to respond with the text that was provided, which helps to prevent it from generating potentially harmful or incorrect answers. Here is how you can do this using the PaLM API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RmzlnDMBDM20"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"\n",
        "Answer the question using the text below. Respond with only the text provided.\n",
        "Question: What should I do to fix my disconnected WiFi? The light on my Google WiFi router is yellow and blinking slowly.\n",
        "\n",
        "Text:\n",
        "Color: No light\n",
        "What it means: Router has no power or the light was dimmed in the app.\n",
        "What to do:\n",
        "Check that the power cable is properly connected to your router and to a working wall outlet.\n",
        "If your device is already set up and the light appears off, check your light brightness settings in the app.\n",
        "If there's still no light, contact WiFi customer support.\n",
        "\n",
        "Color: Solid white, no light, solid white\n",
        "What it means: Device is booting up.\n",
        "What to do:\n",
        "Wait for the device to boot up. This takes about a minute. When it's done, it will slowly pulse white, letting you know it's ready for setup.\n",
        "\n",
        "Color: Slow-pulsing white\n",
        "What it means: Device is ready for set up.\n",
        "What to do:\n",
        "Use the Google Home app to set up your router.\n",
        "\n",
        "Color: Solid white\n",
        "What it means: Router is online and all is well.\n",
        "What to do:\n",
        "You're online. Enjoy!\n",
        "\n",
        "Color: Slowly pulsing yellow\n",
        "What it means: There is a network error.\n",
        "What to do:\n",
        "Check that the Ethernet cable is connected to both your router and your modem and both devices are turned on. You might need to unplug and plug in each device again.\n",
        "\n",
        "Color: Fast blinking yellow\n",
        "What it means: You are holding down the reset button and are factory resetting this device.\n",
        "What to do:\n",
        "If you keep holding down the reset button, after about 12 seconds, the light will turn solid yellow. Once it is solid yellow, let go of the factory reset button.\n",
        "\n",
        "Color: Solid yellow\n",
        "What it means: Router is factory resetting.\n",
        "What to do:\n",
        "This can take up to 10 minutes. When it's done, the device will reset itself and start pulsing white, letting you know it's ready for setup.\n",
        "Image Solid red light Solid red Something is wrong. Critical failure. Factory reset the router. If the light stays red, contact WiFi customer support.\n",
        "\"\"\"\n",
        "\n",
        "print(\n",
        "    generation_model.predict(\n",
        "        prompt, temperature=0.2, max_output_tokens=256, top_k=1, top_p=0.8\n",
        "    ).text\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZtL1SdHqDM21"
      },
      "source": [
        "### Respond to inquiries in character\n",
        "\n",
        "Now, you instruct the generative model to pretend to be Klara, a person. You will also tell the model about Klara's personality traits. Then, you will ask the model to answer a question as Klara would answer it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kvT8WVNnDM21"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"\n",
        "You are Klara.\n",
        "Klara is an investment manager.\n",
        "Klara only answers if Klara is sure it is correct.\n",
        "Klara answers the user question based on the summaries of the pages below.\n",
        "Klara outputs the Reference ID where Klara found the answer for each sentence in the answer.\n",
        "Klara also summarizes the part where the information is found.\n",
        "\n",
        "Summaries of the pages: ['Reference ID 1. UBS wants to be seen as a global bank with Swiss roots, not just a European bank . New board chairman Colm Kelleher and CEO Ralph Hamers have held a series of meetings with influential U.S. fund managers to increase their stakes in the bank . UBS is one of the most valuable European banks with a price-to-book ratio of 1 .', 'Reference ID 2. Credit Suisse sells 30 percent stake in Swiss asset manager Energy Infrastructure Partners . EIP specializes in long-term equity investments for large-scale renewable and energy infrastructure assets . The transaction may be part of an ongoing effort by the Swiss bank to further close the capital gap .', 'Reference ID 3. Saudi Arabia's Crown Prince Mohammed bin Salman is preparing to invest in Credit Suisse Group AG's investment bank . Prince Mohammed would inject about $500 million in the spinoff of CS First Boston . Other investors could include former Barclays chief Bob Diamonds Atlas Merchant Capital . Saudi National Bank already has a 9. 9 percent stake in the troubled Swiss institution .', 'Reference ID 4. US-based Apollo Global Management is among a group of financial firms in talks with Credit Suisse about a stake in the revamped investment bank . The Wall Street Journal has reported that Apollo is also said to be interested in investing in CS First Boston . The investment banks investment banking division is set to be spun off into a new unit .', 'Reference ID 5. Geneva-based private bank Pictet has signed up a new team in its billion-dollar private market investment business . Edmund Buckley, Nikolaus Hubmann, Sean Howard, Jan Dreesen and Hugo Hickson will work in the area of direct investments in private equity in the future . The commitment of Buckleys team can be seen as a coup for Pictet partner Elif Aktuğ .', 'Reference ID 6. UBS wants to be seen as a global bank with Swiss roots, not just a European bank . New board chairman Colm Kelleher and CEO Ralph Hamers have held a series of meetings with influential U.S. fund managers to increase their stakes in the bank . UBS is one of the most valuable European banks with a price-to-book ratio of 1 .', 'Reference ID 7. US-based Apollo Global Management is among a group of financial firms in talks with Credit Suisse about a stake in the revamped investment bank . The Wall Street Journal has reported that Apollo is also said to be interested in investing in CS First Boston . The investment banks investment banking division is set to be spun off into a new unit .', 'Reference ID 8. Luxembourg-based Apex subsidiary European Depositary Bank (EDB) and securitization specialist GenTwo enter into partnership . EDB and GenTwo will offer paying agent and banking services to third-party investors around the world . GenTwo develops platforms for asset managers, banks, family offices, and venture capitalists .', 'Reference ID 9. SNB has been pursuing plans to acquire stakes in European and American financial institutions for some time . Credit Suisse, Julius Baer, Standard Chartered and the Asian DBS Group have also been mentioned as possible targets .', 'Reference ID 10. Zuercher Kantonalbank is aiming to expand its private banking business, including abroad . CEO Urs Baumann sees Credit Suisse as a reliable partner for the state-owned bank . Baumann will continue to focus on reducing the banks dependence on the interest differential business in an attempt to diversify its business .']\n",
        "\n",
        "User question: Are global banks investing into ESG initiatives?\n",
        "Klara's answer:\n",
        "\"\"\"\n",
        "\n",
        "print(\n",
        "    generation_model.predict(\n",
        "        prompt, temperature=0.2, max_output_tokens=256, top_k=40, top_p=0.8\n",
        "    ).text\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CJTjCPMNDM22"
      },
      "source": [
        "###  Converting an ingredients list to JSON format\n",
        "\n",
        "Suppose that you want to itemize ingredients in recipes to enter into a database, which requires a well-formatted output like JSON. This can be done using a generative model in the following way:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fqrWfuZaDM22"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"\n",
        "Extract the ingredients from the following recipe. Return the ingredients in JSON format with keys: ingredient, quantity, type.\n",
        "\n",
        "Ingredients:\n",
        "* 1 tablespoon olive oil\n",
        "* 1 onion, chopped\n",
        "* 2 carrots, chopped\n",
        "* 2 celery stalks, chopped\n",
        "* 1 teaspoon ground cumin\n",
        "* 1/2 teaspoon ground coriander\n",
        "* 1/4 teaspoon turmeric powder\n",
        "* 1/4 teaspoon cayenne pepper (optional)\n",
        "* Salt and pepper to taste\n",
        "* 1 (15 ounce) can black beans, rinsed and drained\n",
        "* 1 (15 ounce) can kidney beans, rinsed and drained\n",
        "* 1 (14.5 ounce) can diced tomatoes, undrained\n",
        "* 1 (10 ounce) can diced tomatoes with green chilies, undrained\n",
        "* 4 cups vegetable broth\n",
        "* 1 cup chopped fresh cilantro\n",
        "\"\"\"\n",
        "\n",
        "print(\n",
        "    generation_model.predict(\n",
        "        prompt, temperature=0.2, max_output_tokens=1024, top_k=40, top_p=0.8\n",
        "    ).text\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4aOG3EnKDM23"
      },
      "source": [
        "### Organizing the results of a text extraction\n",
        "\n",
        "In this section, you extract the information you want from a block of text and organize it in a structured way, such as separating it by commas. Here you use few-shot prompting to guide the model to format your outputs to be separated by commas.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LKHhBL9fDM23"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"\n",
        "Message: Rachel Green (Jennifer Aniston), a sheltered but friendly woman, flees her wedding day and wealthy yet unfulfilling life and finds childhood friend Monica Geller (Courteney Cox), a tightly wound but caring chef.\n",
        "Rachel becomes a waitress at West Village coffee house Central Perk after she moves into Monica\\'s apartment above Central Perk and joins Monica\\'s group of single friends in their mid-20s:\n",
        "previous roommate Phoebe Buffay (Lisa Kudrow), an odd masseuse and musician; neighbor Joey Tribbiani (Matt LeBlanc), a dim-witted yet loyal struggling actor; Joey\\'s roommate Chandler Bing (Matthew Perry),\n",
        "a sarcastic, self-deprecating data processor; and Monica\\'s older brother and Chandler\\'s college roommate Ross Geller (David Schwimmer), a sweet-natured but insecure paleontologist.\n",
        "\n",
        "Extract the characters and the actors who played them from above message:\n",
        "Rachel Green - Jennifer Aniston, Monica Geller - Courteney Cox, Phoebe Buffay - Lisa Kudrow, Joey Tribbiani - Matt LeBlanc, Chandler Bing - Matthew Perry, Ross Geller - David Schwimmer\n",
        "\n",
        "Message: Games such as chess, poker, Go, and many video games have always been fertile ground for AI research. Diplomacy is a seven-player game of negotiation and alliance formation, played on an old map of Europe partitioned\n",
        "into provinces, where each player controls multiple units (rules of Diplomacy). In the standard version of the game, called Press Diplomacy, each turn includes a negotiation phase, after which all players reveal their\n",
        "chosen moves simultaneously. The heart of Diplomacy is the negotiation phase, where players try to agree on their next moves. For example, one unit may support another unit, allowing it to overcome resistance by other units,\n",
        "as illustrated here: Computational approaches to Diplomacy have been researched since the 1980s, many of which were explored on a simpler version of the game called No-Press Diplomacy, where strategic communication between\n",
        "players is not allowed. Researchers have also proposed computer-friendly negotiation protocols, sometimes called \\342\\200\\234Restricted-Press\\342\\200\\235.\n",
        "\n",
        "Extract the definition of Diplomacy:\n",
        "A seven-player game of negotiation and alliance formation\n",
        "\n",
        "\n",
        "Message: Back in 2016, when we weren\\'t using simulation and were using a small lab-configuration of industrial robots to learn how to grasp small objects like toys, keys and everyday household items, it took the equivalent of\n",
        "four months for one robot to learn how to perform a simple grasp with a 75%% success rate. Today, a single robot learns how to perform a complex task such as opening doors with a 90%% success rate with less than a day\n",
        "of real-world learning. Even more excitingly, we\\'ve shown that we can build on the algorithms and learnings from door opening and apply them to a new task: straightening up chairs in our cafes. This progress gives us\n",
        "hope that our moonshot for building general purpose learning robots might just be possible.\n",
        "\n",
        "Extract the success rates of the robots in 2016 and today, respectively:\n",
        "75%, 90%\n",
        "\n",
        "Message: CapitalG was founded a decade ago to empower entrepreneurs with Alphabet and Google\\'s unparalleled expertise in growth.\n",
        "We are privileged to share the lessons learned from helping to scale Google, Stripe, Airbnb, CrowdStrike, Databricks, and Zscaler with the next wave of generational tech companies-perhaps including yours.\n",
        "Alphabet is our sole LP and provides patient, long-term capital. As an independent growth fund, our priorities align with our entrepreneurs. CapitalG companies have achieved product-market fit and are ready to scale. We maintain a small, concentrated portfolio so every company receives substantial capital and hands-on support.\n",
        "\n",
        "Extract the companies funded by CapitalG:\n",
        "\"\"\"\n",
        "\n",
        "print(\n",
        "    generation_model.predict(\n",
        "        prompt, temperature=0.2, max_output_tokens=256, top_k=1, top_p=0.8\n",
        "    ).text\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2032a159249c"
      },
      "source": [
        "As you can see in the output above, based on the few-shot prompt, you should see the names of companies funded by CapitalG."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "text_extraction.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}