Product_Data_Enrichment_with_Vertex_AI.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qk95204JX-c7"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hj20Ixwkd4jU"
      },
      "source": [
        "# Overview\n",
        "\n",
        "This notebook demonstrates how to enrich your data using Generative AI with Vertex AI on Google Cloud.\n",
        "\n",
        "The specific example is a retail use case for improving product description metadata. Better product descriptions lead to more user engagement and higher conversion rates.\n",
        "\n",
        "\n",
        "The workflow includes:\n",
        "* Importing the data\n",
        "* Analyzing the product metadata\n",
        "* Enriching the data with an LLM\n",
        "* Updating the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rSCowhwJopyI"
      },
      "source": [
        "# Getting Started"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9rp8GGz-ongv"
      },
      "outputs": [],
      "source": [
        "# Install the Vertex AI SDK\n",
        "\n",
        "# This step currently includes an additional dependency, to avoid a conflict\n",
        "\n",
        "!pip install --upgrade google-cloud-aiplatform shapely\"<2\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Hp6xeaoosMB"
      },
      "source": [
        "**Restart** the runtime to use these package versions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lgbvew1r5YJe"
      },
      "source": [
        "# Import data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5s7gnElv5a9D"
      },
      "source": [
        "This notebook requires downloading the [Flipkart Products](https://www.kaggle.com/datasets/PromptCloudHQ/flipkart-products?resource=download) dataset from Kaggle.\n",
        "\n",
        "After downloading the data, upload the CSV file to the notebook using the Files feature of Colab."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6qgo2oLp30Va"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Read the data from the CSV file\n",
        "df = pd.read_csv(\"flipkart_com-ecommerce_sample.csv\", engine=\"python\")\n",
        "\n",
        "# Filter out blank rows\n",
        "df = df.dropna(how=\"all\")\n",
        "\n",
        "# Print the first five rows of the data\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uTmRVv_Zc7z5"
      },
      "source": [
        "# Data Analysis\n",
        "\n",
        "Let's now look at the product description field. To keep things simple, we'll use the length of the description as a proxy for data quality.\n",
        "\n",
        "We'll look for short descriptions, where it's likely that useful details are omitted."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1nF1kOCy4OLF"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from scipy.stats import gaussian_kde\n",
        "\n",
        "def get_num_chars_in_description(dataframe: pd.DataFrame) -> pd.Series:\n",
        "  # Get the number of characters in the description field\n",
        "  num_chars_in_description = dataframe[\"description\"].str.len()\n",
        "\n",
        "  # Try converting the number of characters to numerics before plotting the distribution\n",
        "  try:\n",
        "    # If it is string type, remove all characters except numbers, \".\", \"+\" or \"-\".\n",
        "    if pd.api.types.is_string_dtype(num_chars_in_description):\n",
        "      num_chars_in_description = num_chars_in_description.str.replace(r\"[^\\d\\-+\\.]\", \"\")\n",
        "    num_chars_in_description = pd.to_numeric(num_chars_in_description)\n",
        "  except:\n",
        "    pass\n",
        "\n",
        "  # Drop the rows with NA values\n",
        "  num_chars_in_description = num_chars_in_description.dropna()\n",
        "\n",
        "  # Convert the number of characters to integers\n",
        "  num_chars_in_description = num_chars_in_description.astype(int)\n",
        "\n",
        "  return num_chars_in_description\n",
        "\n",
        "def plot_distribution(series: pd.Series) -> None:\n",
        "  # Create a kernel density estimate (KDE) object\n",
        "  kde = gaussian_kde(series)\n",
        "\n",
        "  # Evaluate the PDF at a range of points\n",
        "  x = np.linspace(0, max(series), 1000)\n",
        "  y = kde(x)\n",
        "\n",
        "  # Plot the smoothed distribution function\n",
        "  plt.xscale(\"log\")\n",
        "  plt.plot(x, y)\n",
        "  plt.xlabel(\"Number of characters in description\")\n",
        "  plt.ylabel(\"Probability density\")\n",
        "  plt.title(\"Product Description Length KDE Plot\")\n",
        "  plt.show()\n",
        "\n",
        "num_chars_in_description = get_num_chars_in_description(df)\n",
        "plot_distribution(num_chars_in_description)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "caHZEyFq9gJs"
      },
      "source": [
        "Let's now set a threshold to improve the shortest 0.05% of our descriptions. We see that 93 characters should be that threshold, and there are 13 descriptions of that length or less."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0Yre716JDXm2"
      },
      "outputs": [],
      "source": [
        "threshold = int(num_chars_in_description.quantile(0.0005))\n",
        "\n",
        "threshold"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SgJnUXnj9COm"
      },
      "outputs": [],
      "source": [
        "def get_rows_under_threshold(series: pd.Series, threshold) -> pd.Series:\n",
        "  return num_chars_in_description.loc[series <= threshold]\n",
        "\n",
        "rows_with_description_under_threshold = get_rows_under_threshold(num_chars_in_description, threshold)\n",
        "\n",
        "print(rows_with_description_under_threshold.shape[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "52Cf3r7pdBVN"
      },
      "source": [
        "# Data Transformation\n",
        "\n",
        "Next, let's collect the relevant details from each row with short descriptions. We'll put that into a JSON structure that we'll pass into the large language model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VY08ro3VCmLz"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "# Create a JSON object for each row\n",
        "json_objects = []\n",
        "for index, row in df.loc[rows_with_description_under_threshold.index].iterrows():\n",
        "  row = row[[\"product_name\", \"description\", \"brand\", \"product_category_tree\", \"pid\"]]\n",
        "  json_object = {}\n",
        "  for column in row.index:\n",
        "    json_object[column] = row[column]\n",
        "  json_objects.append(json_object)\n",
        "\n",
        "# Create a JSON array\n",
        "json_array = json.dumps(json_objects)\n",
        "\n",
        "# Print the JSON array\n",
        "print(json_array)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7tkufA1e-MXV"
      },
      "source": [
        "We can now include this JSON array into a prompt to query the LLM with."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KEH2zIAWkCiV"
      },
      "outputs": [],
      "source": [
        "prompt = f\"\"\"\n",
        "Generate a compelling and accurate product description\n",
        "for each of the products provided in the JSON data structure below.\n",
        "This description should be included in the output.\n",
        "\n",
        "The output should be a JSON array consisting only of the\n",
        "original `pid` and updated `description` fields for each product.\n",
        "===\n",
        "{json_array}\n",
        "\"\"\"\n",
        "\n",
        "print(prompt)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qkRWhaRJjdfV"
      },
      "source": [
        "# Query LLM"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YpAmBjhd-Z7U"
      },
      "source": [
        "In this step, we'll connect to a Vertex AI LLM with our prompt and return a result.\n",
        "\n",
        "First, let's define some project parameters:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VPyuwWuEj4Bd"
      },
      "outputs": [],
      "source": [
        "project_id = 'YOUR_PROJECT_ID' # @param {type:\"string\"}\n",
        "location = 'us-central1' # @param {type:\"string\"}'\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eVljyCbiAtcq"
      },
      "outputs": [],
      "source": [
        "# Authenticate to use the SDK\n",
        "\n",
        "import google.colab.auth\n",
        "google.colab.auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GngwIaWD_u59"
      },
      "outputs": [],
      "source": [
        "import vertexai\n",
        "from vertexai.language_models import TextGenerationModel\n",
        "\n",
        "# Initialize the client\n",
        "vertexai.init(project=project_id, location=location)\n",
        "\n",
        "# Use the text-bison model from the Vertex Model Garden\n",
        "model = TextGenerationModel.from_pretrained(\"text-bison@001\")\n",
        "\n",
        "# Update the default max_output_tokens\n",
        "parameters = {\"max_output_tokens\": 1024}\n",
        "\n",
        "# Query the model\n",
        "response = model.predict(prompt, **parameters)\n",
        "\n",
        "# Print the result\n",
        "response.text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P0xZYLK3_Goo"
      },
      "source": [
        "# Update Data\n",
        "\n",
        "With these new descriptions, let's update the original dataframe and analyze the result."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qCs40eu5BWkp"
      },
      "outputs": [],
      "source": [
        "# Load the string response into a JSON object\n",
        "\n",
        "def clean_array(string):\n",
        "  \"\"\"Remove whitespace and any trailing/leading brackets.\"\"\"\n",
        "  string = string.strip()\n",
        "  if string[0] == '{':\n",
        "    string = string[1:]\n",
        "  if string[-1] == '}':\n",
        "    string = string[:-1]\n",
        "  return string\n",
        "\n",
        "products = json.loads(clean_array(response.text))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "olG6cz98nhsK"
      },
      "outputs": [],
      "source": [
        "# Create a mapping between product IDs and updated descriptions\n",
        "\n",
        "pid_to_description = {}\n",
        "for product in products:\n",
        "    pid_to_description[product['pid']] = product['description']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CSVZz7AGTtEB"
      },
      "outputs": [],
      "source": [
        "# Create a boolean mask to indicate which of the original products have an updated description\n",
        "mask = df['pid'].isin(pid_to_description.keys())\n",
        "\n",
        "# Update the descriptions in a new data frame\n",
        "updated_df = df.copy()\n",
        "updated_df.loc[mask, 'description'] = df.loc[mask, 'pid'].apply(lambda pid: pid_to_description.get(pid))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WTob6EK0YCQr"
      },
      "outputs": [],
      "source": [
        "# Create a new series containing the product description lengths\n",
        "updated_num_chars_in_description = get_num_chars_in_description(updated_df)\n",
        "\n",
        "# Calculate how many rows are now under the threshold\n",
        "print(get_rows_under_threshold(updated_num_chars_in_description, threshold).shape[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YBLue2XGn8LR"
      },
      "source": [
        "Finally, let's plot our result!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qsVKtYwZiKz4"
      },
      "outputs": [],
      "source": [
        "# Define the bin width and xlimit\n",
        "bin_width = 10\n",
        "xlim_left = 60\n",
        "xlim_right = 120\n",
        "\n",
        "# Create two histograms\n",
        "fig, ax = plt.subplots()\n",
        "ax.hist(num_chars_in_description, bins=np.arange(0, xlim_right + bin_width, bin_width), alpha=0.5, label='Original')\n",
        "ax.hist(updated_num_chars_in_description, bins=np.arange(0, xlim_right + bin_width, bin_width), alpha=0.5, label='Updated')\n",
        "\n",
        "# Set the x-axis limits\n",
        "ax.set_xlim(xlim_left, xlim_right)\n",
        "\n",
        "# Add a legend\n",
        "ax.legend()\n",
        "\n",
        "# Set the title and labels\n",
        "ax.set_title('Product Description Length')\n",
        "ax.set_xlabel('Number of Characters')\n",
        "ax.set_ylabel('Frequency')\n",
        "\n",
        "# Show the plot\n",
        "plt.show()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vnKzQMX7nHUf"
      },
      "source": [
        "Thanks for walking through this tutorial. I hope you've learned some new things, and enjoyed the experience!"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}