examples/Tag_caption_images_with_GPT4V.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fd7e6164",
   "metadata": {},
   "source": [
    "# Using GPT-4V to tag & caption images\n",
    "\n",
    "This notebook explores how to leverage GPT-4V to tag & caption images. \n",
    "\n",
    "We can leverage the multimodal capabilities of GPT-4V to provide input images along with additional context on what they represent, and prompt the model to output tags or image descriptions. The image descriptions can then be further refined with a language model (in this notebook, we'll use GPT-4-turbo) to generate captions. \n",
    "\n",
    "Generating text content from images can be useful for multiple use cases, especially use cases involving search.  \n",
    "We will illustrate a search use case in this notebook by using generated keywords and product captions to search for products - both from a text input and an image input.\n",
    "\n",
    "As an example, we will use a dataset of Amazon furniture items, tag them with relevant keywords and generate short, descriptive captions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a00b5267",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "176a7836",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install dependencies if needed\n",
    "%pip install openai\n",
    "%pip install scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5b4c63de",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image, display\n",
    "import pandas as pd\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "import numpy as np\n",
    "from openai import OpenAI\n",
    "\n",
    "# Initializing OpenAI client - see https://platform.openai.com/docs/quickstart?context=python\n",
    "client = OpenAI()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1b0fa62d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>asin</th>\n",
       "      <th>url</th>\n",
       "      <th>title</th>\n",
       "      <th>brand</th>\n",
       "      <th>price</th>\n",
       "      <th>availability</th>\n",
       "      <th>categories</th>\n",
       "      <th>primary_image</th>\n",
       "      <th>images</th>\n",
       "      <th>upc</th>\n",
       "      <th>...</th>\n",
       "      <th>color</th>\n",
       "      <th>material</th>\n",
       "      <th>style</th>\n",
       "      <th>important_information</th>\n",
       "      <th>product_overview</th>\n",
       "      <th>about_item</th>\n",
       "      <th>description</th>\n",
       "      <th>specifications</th>\n",
       "      <th>uniq_id</th>\n",
       "      <th>scraped_at</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B0CJHKVG6P</td>\n",
       "      <td>https://www.amazon.com/dp/B0CJHKVG6P</td>\n",
       "      <td>GOYMFK 1pc Free Standing Shoe Rack, Multi-laye...</td>\n",
       "      <td>GOYMFK</td>\n",
       "      <td>$24.99</td>\n",
       "      <td>Only 13 left in stock - order soon.</td>\n",
       "      <td>['Home &amp; Kitchen', 'Storage &amp; Organization', '...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/416WaLx10j...</td>\n",
       "      <td>['https://m.media-amazon.com/images/I/416WaLx1...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>White</td>\n",
       "      <td>Metal</td>\n",
       "      <td>Modern</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'Brand': ' GOYMFK '}, {'Color': ' White '}, ...</td>\n",
       "      <td>['Multiple layers: Provides ample storage spac...</td>\n",
       "      <td>multiple shoes, coats, hats, and other items E...</td>\n",
       "      <td>['Brand: GOYMFK', 'Color: White', 'Material: M...</td>\n",
       "      <td>02593e81-5c09-5069-8516-b0b29f439ded</td>\n",
       "      <td>2024-02-02 15:15:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>B0B66QHB23</td>\n",
       "      <td>https://www.amazon.com/dp/B0B66QHB23</td>\n",
       "      <td>subrtex Leather ding Room, Dining Chairs Set o...</td>\n",
       "      <td>subrtex</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['Home &amp; Kitchen', 'Furniture', 'Dining Room F...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/31SejUEWY7...</td>\n",
       "      <td>['https://m.media-amazon.com/images/I/31SejUEW...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Black</td>\n",
       "      <td>Sponge</td>\n",
       "      <td>Black Rubber Wood</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['【Easy Assembly】: Set of 2 dining room chairs...</td>\n",
       "      <td>subrtex Dining chairs Set of 2</td>\n",
       "      <td>['Brand: subrtex', 'Color: Black', 'Product Di...</td>\n",
       "      <td>5938d217-b8c5-5d3e-b1cf-e28e340f292e</td>\n",
       "      <td>2024-02-02 15:15:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B0BXRTWLYK</td>\n",
       "      <td>https://www.amazon.com/dp/B0BXRTWLYK</td>\n",
       "      <td>Plant Repotting Mat MUYETOL Waterproof Transpl...</td>\n",
       "      <td>MUYETOL</td>\n",
       "      <td>$5.98</td>\n",
       "      <td>In Stock</td>\n",
       "      <td>['Patio, Lawn &amp; Garden', 'Outdoor Décor', 'Doo...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/41RgefVq70...</td>\n",
       "      <td>['https://m.media-amazon.com/images/I/41RgefVq...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Green</td>\n",
       "      <td>Polyethylene</td>\n",
       "      <td>Modern</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'Brand': ' MUYETOL '}, {'Size': ' 26.8*26.8 ...</td>\n",
       "      <td>['PLANT REPOTTING MAT SIZE: 26.8\" x 26.8\", squ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['Brand: MUYETOL', 'Size: 26.8*26.8', 'Item We...</td>\n",
       "      <td>b2ede786-3f51-5a45-9a5b-bcf856958cd8</td>\n",
       "      <td>2024-02-02 15:15:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>B0C1MRB2M8</td>\n",
       "      <td>https://www.amazon.com/dp/B0C1MRB2M8</td>\n",
       "      <td>Pickleball Doormat, Welcome Doormat Absorbent ...</td>\n",
       "      <td>VEWETOL</td>\n",
       "      <td>$13.99</td>\n",
       "      <td>Only 10 left in stock - order soon.</td>\n",
       "      <td>['Patio, Lawn &amp; Garden', 'Outdoor Décor', 'Doo...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/61vz1Igler...</td>\n",
       "      <td>['https://m.media-amazon.com/images/I/61vz1Igl...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>A5589</td>\n",
       "      <td>Rubber</td>\n",
       "      <td>Modern</td>\n",
       "      <td>[]</td>\n",
       "      <td>[{'Brand': ' VEWETOL '}, {'Size': ' 16*24INCH ...</td>\n",
       "      <td>['Specifications: 16x24 Inch ', \" High-Quality...</td>\n",
       "      <td>The decorative doormat features a subtle textu...</td>\n",
       "      <td>['Brand: VEWETOL', 'Size: 16*24INCH', 'Materia...</td>\n",
       "      <td>8fd9377b-cfa6-5f10-835c-6b8eca2816b5</td>\n",
       "      <td>2024-02-02 15:15:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>B0CG1N9QRC</td>\n",
       "      <td>https://www.amazon.com/dp/B0CG1N9QRC</td>\n",
       "      <td>JOIN IRON Foldable TV Trays for Eating Set of ...</td>\n",
       "      <td>JOIN IRON Store</td>\n",
       "      <td>$89.99</td>\n",
       "      <td>Usually ships within 5 to 6 weeks</td>\n",
       "      <td>['Home &amp; Kitchen', 'Furniture', 'Game &amp; Recrea...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/41p4d4VJnN...</td>\n",
       "      <td>['https://m.media-amazon.com/images/I/41p4d4VJ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Grey Set of 4</td>\n",
       "      <td>Iron</td>\n",
       "      <td>X Classic Style</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['Includes 4 Folding Tv Tray Tables And one Co...</td>\n",
       "      <td>Set of Four Folding Trays With Matching Storag...</td>\n",
       "      <td>['Brand: JOIN IRON', 'Shape: Rectangular', 'In...</td>\n",
       "      <td>bdc9aa30-9439-50dc-8e89-213ea211d66a</td>\n",
       "      <td>2024-02-02 15:15:11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         asin                                   url  \\\n",
       "0  B0CJHKVG6P  https://www.amazon.com/dp/B0CJHKVG6P   \n",
       "1  B0B66QHB23  https://www.amazon.com/dp/B0B66QHB23   \n",
       "2  B0BXRTWLYK  https://www.amazon.com/dp/B0BXRTWLYK   \n",
       "3  B0C1MRB2M8  https://www.amazon.com/dp/B0C1MRB2M8   \n",
       "4  B0CG1N9QRC  https://www.amazon.com/dp/B0CG1N9QRC   \n",
       "\n",
       "                                               title            brand   price  \\\n",
       "0  GOYMFK 1pc Free Standing Shoe Rack, Multi-laye...           GOYMFK  $24.99   \n",
       "1  subrtex Leather ding Room, Dining Chairs Set o...          subrtex     NaN   \n",
       "2  Plant Repotting Mat MUYETOL Waterproof Transpl...          MUYETOL   $5.98   \n",
       "3  Pickleball Doormat, Welcome Doormat Absorbent ...          VEWETOL  $13.99   \n",
       "4  JOIN IRON Foldable TV Trays for Eating Set of ...  JOIN IRON Store  $89.99   \n",
       "\n",
       "                          availability  \\\n",
       "0  Only 13 left in stock - order soon.   \n",
       "1                                  NaN   \n",
       "2                             In Stock   \n",
       "3  Only 10 left in stock - order soon.   \n",
       "4    Usually ships within 5 to 6 weeks   \n",
       "\n",
       "                                          categories  \\\n",
       "0  ['Home & Kitchen', 'Storage & Organization', '...   \n",
       "1  ['Home & Kitchen', 'Furniture', 'Dining Room F...   \n",
       "2  ['Patio, Lawn & Garden', 'Outdoor Décor', 'Doo...   \n",
       "3  ['Patio, Lawn & Garden', 'Outdoor Décor', 'Doo...   \n",
       "4  ['Home & Kitchen', 'Furniture', 'Game & Recrea...   \n",
       "\n",
       "                                       primary_image  \\\n",
       "0  https://m.media-amazon.com/images/I/416WaLx10j...   \n",
       "1  https://m.media-amazon.com/images/I/31SejUEWY7...   \n",
       "2  https://m.media-amazon.com/images/I/41RgefVq70...   \n",
       "3  https://m.media-amazon.com/images/I/61vz1Igler...   \n",
       "4  https://m.media-amazon.com/images/I/41p4d4VJnN...   \n",
       "\n",
       "                                              images  upc  ...          color  \\\n",
       "0  ['https://m.media-amazon.com/images/I/416WaLx1...  NaN  ...          White   \n",
       "1  ['https://m.media-amazon.com/images/I/31SejUEW...  NaN  ...          Black   \n",
       "2  ['https://m.media-amazon.com/images/I/41RgefVq...  NaN  ...          Green   \n",
       "3  ['https://m.media-amazon.com/images/I/61vz1Igl...  NaN  ...          A5589   \n",
       "4  ['https://m.media-amazon.com/images/I/41p4d4VJ...  NaN  ...  Grey Set of 4   \n",
       "\n",
       "       material              style important_information  \\\n",
       "0         Metal             Modern                    []   \n",
       "1        Sponge  Black Rubber Wood                    []   \n",
       "2  Polyethylene             Modern                    []   \n",
       "3        Rubber             Modern                    []   \n",
       "4          Iron    X Classic Style                    []   \n",
       "\n",
       "                                    product_overview  \\\n",
       "0  [{'Brand': ' GOYMFK '}, {'Color': ' White '}, ...   \n",
       "1                                                NaN   \n",
       "2  [{'Brand': ' MUYETOL '}, {'Size': ' 26.8*26.8 ...   \n",
       "3  [{'Brand': ' VEWETOL '}, {'Size': ' 16*24INCH ...   \n",
       "4                                                NaN   \n",
       "\n",
       "                                          about_item  \\\n",
       "0  ['Multiple layers: Provides ample storage spac...   \n",
       "1  ['【Easy Assembly】: Set of 2 dining room chairs...   \n",
       "2  ['PLANT REPOTTING MAT SIZE: 26.8\" x 26.8\", squ...   \n",
       "3  ['Specifications: 16x24 Inch ', \" High-Quality...   \n",
       "4  ['Includes 4 Folding Tv Tray Tables And one Co...   \n",
       "\n",
       "                                         description  \\\n",
       "0  multiple shoes, coats, hats, and other items E...   \n",
       "1                     subrtex Dining chairs Set of 2   \n",
       "2                                                NaN   \n",
       "3  The decorative doormat features a subtle textu...   \n",
       "4  Set of Four Folding Trays With Matching Storag...   \n",
       "\n",
       "                                      specifications  \\\n",
       "0  ['Brand: GOYMFK', 'Color: White', 'Material: M...   \n",
       "1  ['Brand: subrtex', 'Color: Black', 'Product Di...   \n",
       "2  ['Brand: MUYETOL', 'Size: 26.8*26.8', 'Item We...   \n",
       "3  ['Brand: VEWETOL', 'Size: 16*24INCH', 'Materia...   \n",
       "4  ['Brand: JOIN IRON', 'Shape: Rectangular', 'In...   \n",
       "\n",
       "                                uniq_id           scraped_at  \n",
       "0  02593e81-5c09-5069-8516-b0b29f439ded  2024-02-02 15:15:08  \n",
       "1  5938d217-b8c5-5d3e-b1cf-e28e340f292e  2024-02-02 15:15:09  \n",
       "2  b2ede786-3f51-5a45-9a5b-bcf856958cd8  2024-02-02 15:15:09  \n",
       "3  8fd9377b-cfa6-5f10-835c-6b8eca2816b5  2024-02-02 15:15:10  \n",
       "4  bdc9aa30-9439-50dc-8e89-213ea211d66a  2024-02-02 15:15:11  \n",
       "\n",
       "[5 rows x 25 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Loading dataset\n",
    "dataset_path =  \"data/amazon_furniture_dataset.csv\"\n",
    "df = pd.read_csv(dataset_path)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4d862fa",
   "metadata": {},
   "source": [
    "## Tag images\n",
    "\n",
    "In this section, we'll use GPT-4V to generate relevant tags for our products.\n",
    "\n",
    "We'll use a simple zero-shot approach to extract keywords, and deduplicate those keywords using embeddings to avoid having multiple keywords that are too similar.\n",
    "\n",
    "We will use a combination of an image and the product title to avoid extracting keywords for other items that are depicted in the image - sometimes there are multiple items used in the scene and we want to focus on just the one we want to tag."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20d7a5c8",
   "metadata": {},
   "source": [
    "### Extract keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "752118c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = '''\n",
    "    You are an agent specialized in tagging images of furniture items, decorative items, or furnishings with relevant keywords that could be used to search for these items on a marketplace.\n",
    "    \n",
    "    You will be provided with an image and the title of the item that is depicted in the image, and your goal is to extract keywords for only the item specified. \n",
    "    \n",
    "    Keywords should be concise and in lower case. \n",
    "    \n",
    "    Keywords can describe things like:\n",
    "    - Item type e.g. 'sofa bed', 'chair', 'desk', 'plant'\n",
    "    - Item material e.g. 'wood', 'metal', 'fabric'\n",
    "    - Item style e.g. 'scandinavian', 'vintage', 'industrial'\n",
    "    - Item color e.g. 'red', 'blue', 'white'\n",
    "    \n",
    "    Only deduce material, style or color keywords when it is obvious that they make the item depicted in the image stand out.\n",
    "\n",
    "    Return keywords in the format of an array of strings, like this:\n",
    "    ['desk', 'industrial', 'metal']\n",
    "    \n",
    "'''\n",
    "\n",
    "def analyze_image(img_url, title):\n",
    "    response = client.chat.completions.create(\n",
    "    model=\"gpt-4-vision-preview\",\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": system_prompt\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": [\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": img_url,\n",
    "                },\n",
    "            ],\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": title\n",
    "        }\n",
    "    ],\n",
    "        max_tokens=300,\n",
    "        top_p=0.1\n",
    "    )\n",
    "\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf75e59c",
   "metadata": {},
   "source": [
    "#### Testing with a few examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "eb6a8159",
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = df.iloc[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b729129b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/416WaLx10jL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['shoe rack', 'free standing', 'multi-layer', 'metal', 'white']\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/31SejUEWY7L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['dining chairs', 'set of 2', 'leather', 'black']\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41RgefVq70L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['plant repotting mat', 'waterproof', 'portable', 'foldable', 'easy to clean', 'green']\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/61vz1IglerL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['doormat', 'absorbent', 'non-slip', 'brown']\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41p4d4VJnNL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['tv tray table set', 'foldable', 'iron', 'grey']\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, ex in examples.iterrows():\n",
    "    url = ex['primary_image']\n",
    "    img = Image(url=url)\n",
    "    display(img)\n",
    "    result = analyze_image(url, ex['title'])\n",
    "    print(result)\n",
    "    print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00b727b5",
   "metadata": {},
   "source": [
    "### Looking up existing keywords\n",
    "\n",
    "Using embeddings to avoid duplicates (synonyms) and/or match pre-defined keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f73d1b59",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feel free to change the embedding model here\n",
    "def get_embedding(value, model=\"text-embedding-3-large\"): \n",
    "    embeddings = client.embeddings.create(\n",
    "      model=model,\n",
    "      input=value,\n",
    "      encoding_format=\"float\"\n",
    "    )\n",
    "    return embeddings.data[0].embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "651c99ce",
   "metadata": {},
   "source": [
    "#### Testing with example keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f0262f03",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Existing keywords\n",
    "keywords_list = ['industrial', 'metal', 'wood', 'vintage', 'bed']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "afc489d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>keyword</th>\n",
       "      <th>embedding</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>industrial</td>\n",
       "      <td>[-0.026137426, 0.021297162, -0.007273361, -0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>metal</td>\n",
       "      <td>[-0.020530997, 0.004478126, -0.011049379, -0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>wood</td>\n",
       "      <td>[0.013877833, 0.02955235, 0.0006239023, -0.035...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>vintage</td>\n",
       "      <td>[-0.05235507, 0.008213689, -0.015532949, 0.002...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>bed</td>\n",
       "      <td>[-0.011677503, 0.023275835, 0.0026937425, -0.0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      keyword                                          embedding\n",
       "0  industrial  [-0.026137426, 0.021297162, -0.007273361, -0.0...\n",
       "1       metal  [-0.020530997, 0.004478126, -0.011049379, -0.0...\n",
       "2        wood  [0.013877833, 0.02955235, 0.0006239023, -0.035...\n",
       "3     vintage  [-0.05235507, 0.008213689, -0.015532949, 0.002...\n",
       "4         bed  [-0.011677503, 0.023275835, 0.0026937425, -0.0..."
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_keywords = pd.DataFrame(keywords_list, columns=['keyword'])\n",
    "df_keywords['embedding'] = df_keywords['keyword'].apply(lambda x: get_embedding(x))\n",
    "df_keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f549e537",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_keyword(keyword):\n",
    "    embedded_value = get_embedding(keyword)\n",
    "    df_keywords['similarity'] = df_keywords['embedding'].apply(lambda x: cosine_similarity(np.array(x).reshape(1,-1), np.array(embedded_value).reshape(1, -1)))\n",
    "    most_similar = df_keywords.sort_values('similarity', ascending=False).iloc[0]\n",
    "    return most_similar\n",
    "\n",
    "def replace_keyword(keyword, threshold = 0.6):\n",
    "    most_similar = compare_keyword(keyword)\n",
    "    if most_similar['similarity'] > threshold:\n",
    "        print(f\"Replacing '{keyword}' with existing keyword: '{most_similar['keyword']}'\")\n",
    "        return most_similar['keyword']\n",
    "    return keyword"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "08c0ec9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Replacing 'bed frame' with existing keyword: 'bed'\n",
      "Replacing 'wooden' with existing keyword: 'wood'\n",
      "Replacing 'vintage' with existing keyword: 'vintage'\n",
      "Replacing 'metal' with existing keyword: 'metal'\n",
      "Replacing 'metallic' with existing keyword: 'metal'\n",
      "Replacing 'woody' with existing keyword: 'wood'\n",
      "Final keywords: {'table', 'desk', 'bed', 'old', 'vintage', 'metal', 'wood', 'old school'}\n"
     ]
    }
   ],
   "source": [
    "# Example keywords to compare to our list of existing keywords\n",
    "example_keywords = ['bed frame', 'wooden', 'vintage', 'old school', 'desk', 'table', 'old', 'metal', 'metallic', 'woody']\n",
    "final_keywords = []\n",
    "\n",
    "for k in example_keywords:\n",
    "    final_keywords.append(replace_keyword(k))\n",
    "    \n",
    "final_keywords = set(final_keywords)\n",
    "print(f\"Final keywords: {final_keywords}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11b0afa1",
   "metadata": {},
   "source": [
    "## Generate captions\n",
    "\n",
    "In this section, we'll use GPT-4V to generate an image description and then use a few-shot examples approach with GPT-4-turbo to generate captions from the images.\n",
    "\n",
    "If few-shot examples are not enough for your use case, consider fine-tuning a model to get the generated captions to match the style & tone you are targeting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c67a434b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>primary_image</th>\n",
       "      <th>style</th>\n",
       "      <th>material</th>\n",
       "      <th>color</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GOYMFK 1pc Free Standing Shoe Rack, Multi-laye...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/416WaLx10j...</td>\n",
       "      <td>Modern</td>\n",
       "      <td>Metal</td>\n",
       "      <td>White</td>\n",
       "      <td>https://www.amazon.com/dp/B0CJHKVG6P</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>subrtex Leather ding Room, Dining Chairs Set o...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/31SejUEWY7...</td>\n",
       "      <td>Black Rubber Wood</td>\n",
       "      <td>Sponge</td>\n",
       "      <td>Black</td>\n",
       "      <td>https://www.amazon.com/dp/B0B66QHB23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Plant Repotting Mat MUYETOL Waterproof Transpl...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/41RgefVq70...</td>\n",
       "      <td>Modern</td>\n",
       "      <td>Polyethylene</td>\n",
       "      <td>Green</td>\n",
       "      <td>https://www.amazon.com/dp/B0BXRTWLYK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pickleball Doormat, Welcome Doormat Absorbent ...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/61vz1Igler...</td>\n",
       "      <td>Modern</td>\n",
       "      <td>Rubber</td>\n",
       "      <td>A5589</td>\n",
       "      <td>https://www.amazon.com/dp/B0C1MRB2M8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>JOIN IRON Foldable TV Trays for Eating Set of ...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/41p4d4VJnN...</td>\n",
       "      <td>X Classic Style</td>\n",
       "      <td>Iron</td>\n",
       "      <td>Grey Set of 4</td>\n",
       "      <td>https://www.amazon.com/dp/B0CG1N9QRC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title  \\\n",
       "0  GOYMFK 1pc Free Standing Shoe Rack, Multi-laye...   \n",
       "1  subrtex Leather ding Room, Dining Chairs Set o...   \n",
       "2  Plant Repotting Mat MUYETOL Waterproof Transpl...   \n",
       "3  Pickleball Doormat, Welcome Doormat Absorbent ...   \n",
       "4  JOIN IRON Foldable TV Trays for Eating Set of ...   \n",
       "\n",
       "                                       primary_image              style  \\\n",
       "0  https://m.media-amazon.com/images/I/416WaLx10j...             Modern   \n",
       "1  https://m.media-amazon.com/images/I/31SejUEWY7...  Black Rubber Wood   \n",
       "2  https://m.media-amazon.com/images/I/41RgefVq70...             Modern   \n",
       "3  https://m.media-amazon.com/images/I/61vz1Igler...             Modern   \n",
       "4  https://m.media-amazon.com/images/I/41p4d4VJnN...    X Classic Style   \n",
       "\n",
       "       material          color                                   url  \n",
       "0         Metal          White  https://www.amazon.com/dp/B0CJHKVG6P  \n",
       "1        Sponge          Black  https://www.amazon.com/dp/B0B66QHB23  \n",
       "2  Polyethylene          Green  https://www.amazon.com/dp/B0BXRTWLYK  \n",
       "3        Rubber          A5589  https://www.amazon.com/dp/B0C1MRB2M8  \n",
       "4          Iron  Grey Set of 4  https://www.amazon.com/dp/B0CG1N9QRC  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Cleaning up dataset columns\n",
    "selected_columns = ['title', 'primary_image', 'style', 'material', 'color', 'url']\n",
    "df = df[selected_columns].copy()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb5c4fa",
   "metadata": {},
   "source": [
    "### Describing images with GPT-4V"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "561912fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "describe_system_prompt = '''\n",
    "    You are a system generating descriptions for furniture items, decorative items, or furnishings on an e-commerce website.\n",
    "    Provided with an image and a title, you will describe the main item that you see in the image, giving details but staying concise.\n",
    "    You can describe unambiguously what the item is and its material, color, and style if clearly identifiable.\n",
    "    If there are multiple items depicted, refer to the title to understand which item you should describe.\n",
    "    '''\n",
    "\n",
    "def describe_image(img_url, title):\n",
    "    response = client.chat.completions.create(\n",
    "    model=\"gpt-4-vision-preview\",\n",
    "    temperature=0.2,\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": describe_system_prompt\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": [\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": img_url,\n",
    "                },\n",
    "            ],\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": title\n",
    "        }\n",
    "    ],\n",
    "    max_tokens=300,\n",
    "    )\n",
    "\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f46f6486",
   "metadata": {},
   "source": [
    "#### Testing on a few examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a0351489",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me... - https://www.amazon.com/dp/B0CJHKVG6P :\n",
      "\n",
      "This is a free-standing shoe rack featuring a multi-layer design, constructed from metal for durability. The rack is finished in a clean white color, which gives it a modern and versatile look, suitable for various home decor styles. It includes several horizontal shelves dedicated to organizing shoes, providing ample space for multiple pairs.\n",
      "\n",
      "Additionally, the rack is equipped with 8 double hooks, which are integrated into the frame above the shoe shelves. These hooks offer extra functionality, allowing for the hanging of accessories such as hats, scarves, or bags. The design is space-efficient and ideal for placement in living rooms, bathrooms, hallways, or entryways, where it can serve as a practical storage solution while contributing to the tidiness and aesthetic of the space.\n",
      "--------------------------\n",
      "\n",
      "subrtex Leather ding Room, Dining Chairs Set of 2,... - https://www.amazon.com/dp/B0B66QHB23 :\n",
      "\n",
      "This image showcases a set of two dining chairs. The chairs are upholstered in black leather, featuring a sleek and modern design. They have a high back with subtle stitching details that create vertical lines, adding an element of elegance to the overall appearance. The legs of the chairs are also black, maintaining a consistent color scheme and enhancing the sophisticated look. These chairs would make a stylish addition to any contemporary dining room setting.\n",
      "--------------------------\n",
      "\n",
      "Plant Repotting Mat MUYETOL Waterproof Transplanti... - https://www.amazon.com/dp/B0BXRTWLYK :\n",
      "\n",
      "This is a square plant repotting mat designed for indoor gardening activities such as transplanting or changing soil for plants. The mat measures 26.8 inches by 26.8 inches, providing ample space for gardening tasks. It is made from a waterproof material, which is likely a durable, easy-to-clean fabric, in a vibrant green color. The edges of the mat are raised with corner fastenings to keep soil and water contained, making the workspace tidy and preventing messes. The mat is also foldable, which allows for convenient storage when not in use. This mat is suitable for a variety of gardening tasks, including working with succulents and other small plants, and it can be a practical gift for garden enthusiasts.\n",
      "--------------------------\n",
      "\n",
      "Pickleball Doormat, Welcome Doormat Absorbent Non-... - https://www.amazon.com/dp/B0C1MRB2M8 :\n",
      "\n",
      "This is a rectangular doormat featuring a playful design that caters to pickleball enthusiasts. The mat's background is a natural coir color, which is a fibrous material made from coconut husks, known for its durability and excellent scraping properties. Emblazoned across the mat in bold, black letters is the phrase \"it's a good day to play PICKLEBALL,\" with the word \"PICKLEBALL\" being prominently displayed in larger font size. Below the text, there are two crossed pickleball paddles in black, symbolizing the sport.\n",
      "\n",
      "The doormat measures approximately 16x24 inches, making it a suitable size for a variety of entryways. Its design suggests that it has an absorbent quality, which would be useful for wiping shoes and preventing dirt from entering the home. Additionally, the description implies that the doormat has a non-slip feature, which is likely due to a backing material that helps keep the mat in place on various floor surfaces. This mat would be a welcoming addition to the home of any pickleball player or sports enthusiast, offering both functionality and a touch of personal interest.\n",
      "--------------------------\n",
      "\n",
      "JOIN IRON Foldable TV Trays for Eating Set of 4 wi... - https://www.amazon.com/dp/B0CG1N9QRC :\n",
      "\n",
      "This image features a set of four foldable TV trays with a stand, designed for eating or as snack tables. The tables are presented in a sleek grey finish, which gives them a modern and versatile look, suitable for a variety of home decor styles. Each tray table has a rectangular top with a wood grain texture, supported by a sturdy black iron frame that folds easily for compact storage. The accompanying stand allows for neat organization and easy access when the tables are not in use. These tables are ideal for small spaces where multifunctional furniture is essential, offering a convenient surface for meals, work, or leisure activities.\n",
      "--------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, row in examples.iterrows():\n",
    "    print(f\"{row['title'][:50]}{'...' if len(row['title']) > 50 else ''} - {row['url']} :\\n\")\n",
    "    img_description = describe_image(row['primary_image'], row['title'])\n",
    "    print(f\"{img_description}\\n--------------------------\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d63f628f",
   "metadata": {},
   "source": [
    "### Turning descriptions into captions\n",
    "Using a few-shot examples approach to turn a long description into a short image caption"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "66b5bc53",
   "metadata": {},
   "outputs": [],
   "source": [
    "caption_system_prompt = '''\n",
    "Your goal is to generate short, descriptive captions for images of furniture items, decorative items, or furnishings based on an image description.\n",
    "You will be provided with a description of an item image and you will output a caption that captures the most important information about the item.\n",
    "Your generated caption should be short (1 sentence), and include the most relevant information about the item.\n",
    "The most important information could be: the type of the item, the style (if mentioned), the material if especially relevant and any distinctive features.\n",
    "'''\n",
    "\n",
    "few_shot_examples = [\n",
    "    {\n",
    "        \"description\": \"This is a multi-layer metal shoe rack featuring a free-standing design. It has a clean, white finish that gives it a modern and versatile look, suitable for various home decors. The rack includes several horizontal shelves dedicated to organizing shoes, providing ample space for multiple pairs. Above the shoe storage area, there are 8 double hooks arranged in two rows, offering additional functionality for hanging items such as hats, scarves, or bags. The overall structure is sleek and space-saving, making it an ideal choice for placement in living rooms, bathrooms, hallways, or entryways where efficient use of space is essential.\",\n",
    "        \"caption\": \"White metal free-standing shoe rack\"\n",
    "    },\n",
    "    {\n",
    "        \"description\": \"The image shows a set of two dining chairs in black. These chairs are upholstered in a leather-like material, giving them a sleek and sophisticated appearance. The design features straight lines with a slight curve at the top of the high backrest, which adds a touch of elegance. The chairs have a simple, vertical stitching detail on the backrest, providing a subtle decorative element. The legs are also black, creating a uniform look that would complement a contemporary dining room setting. The chairs appear to be designed for comfort and style, suitable for both casual and formal dining environments.\",\n",
    "        \"caption\": \"Set of 2 modern black leather dining chairs\"\n",
    "    },\n",
    "    {\n",
    "        \"description\": \"This is a square plant repotting mat designed for indoor gardening tasks such as transplanting and changing soil for plants. It measures 26.8 inches by 26.8 inches and is made from a waterproof material, which appears to be a durable, easy-to-clean fabric in a vibrant green color. The edges of the mat are raised with integrated corner loops, likely to keep soil and water contained during gardening activities. The mat is foldable, enhancing its portability, and can be used as a protective surface for various gardening projects, including working with succulents. It's a practical accessory for garden enthusiasts and makes for a thoughtful gift for those who enjoy indoor plant care.\",\n",
    "        \"caption\": \"Waterproof square plant repotting mat\"\n",
    "    }\n",
    "]\n",
    "\n",
    "formatted_examples = [[{\n",
    "    \"role\": \"user\",\n",
    "    \"content\": ex['description']\n",
    "},\n",
    "{\n",
    "    \"role\": \"assistant\", \n",
    "    \"content\": ex['caption']\n",
    "}]\n",
    "    for ex in few_shot_examples\n",
    "]\n",
    "\n",
    "formatted_examples = [i for ex in formatted_examples for i in ex]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b5ac5262",
   "metadata": {},
   "outputs": [],
   "source": [
    "def caption_image(description, model=\"gpt-4-turbo-preview\"):\n",
    "    messages = formatted_examples\n",
    "    messages.insert(0, \n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": caption_system_prompt\n",
    "        })\n",
    "    messages.append(\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": description\n",
    "        })\n",
    "    response = client.chat.completions.create(\n",
    "    model=model,\n",
    "    temperature=0.2,\n",
    "    messages=messages\n",
    "    )\n",
    "\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5661152",
   "metadata": {},
   "source": [
    "#### Testing on a few examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "2d8f3427",
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = df.iloc[5:8]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "b426d22d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LOVMOR 30'' Bathroom Vanity Sink Base Cabine, Stor... - https://www.amazon.com/dp/B0C9WYYFLB :\n",
      "\n",
      "This is a LOVMOR 30'' Bathroom Vanity Sink Base Cabinet featuring a classic design with a rich brown finish. The cabinet is designed to provide ample storage with three drawers on the left side, offering organized space for bathroom essentials. The drawers are likely to have smooth glides for easy operation. Below the drawers, there is a large cabinet door that opens to reveal additional storage space, suitable for larger items. The paneling on the drawers and door features a raised, framed design, adding a touch of elegance to the overall appearance. This vanity base is versatile and can be used not only in bathrooms but also in kitchens, laundry rooms, and other areas where extra storage is needed. The construction material is not specified, but it appears to be made of wood or a wood-like composite. Please note that the countertop and sink are not included and would need to be purchased separately.\n",
      "--------------------------\n",
      "\n",
      "LOVMOR 30'' classic brown bathroom vanity base cabinet with three drawers and additional storage space.\n",
      "--------------------------\n",
      "\n",
      "Folews Bathroom Organizer Over The Toilet Storage,... - https://www.amazon.com/dp/B09NZY3R1T :\n",
      "\n",
      "This is a 4-tier bathroom organizer designed to fit over a standard toilet, providing a space-saving storage solution. The unit is constructed with a sturdy metal frame in a black finish, which offers both durability and a sleek, modern look. The design includes four shelves that offer ample space for bathroom essentials, towels, and decorative items. Two of the shelves are designed with a metal grid pattern, while the other two feature a solid metal surface for stable storage.\n",
      "\n",
      "Additionally, the organizer includes adjustable baskets, which can be positioned according to your storage needs, allowing for customization and flexibility. The freestanding structure is engineered to maximize the unused vertical space above the toilet, making it an ideal choice for small bathrooms or for those looking to declutter countertops and cabinets.\n",
      "\n",
      "The overall design is minimalist and functional, with clean lines that can complement a variety of bathroom decors. The open shelving concept ensures that items are easily accessible and visible. Installation is typically straightforward, with no need for wall mounting, making it a convenient option for renters or those who prefer not to drill into walls.\n",
      "--------------------------\n",
      "\n",
      "Modern 4-tier black metal bathroom organizer with adjustable shelves and baskets, designed to fit over a standard toilet for space-saving storage.\n",
      "--------------------------\n",
      "\n",
      "GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me... - https://www.amazon.com/dp/B0CJHKVG6P :\n",
      "\n",
      "This is a multi-functional free-standing shoe rack featuring a sturdy metal construction with a white finish. It is designed with multiple layers, providing ample space to organize and store shoes. The rack includes four tiers dedicated to shoe storage, each tier capable of holding several pairs of shoes.\n",
      "\n",
      "Above the shoe storage area, there is an additional shelf that can be used for placing bags, small decorative items, or other accessories. At the top, the rack is equipped with 8 double hooks, which are ideal for hanging hats, scarves, coats, or umbrellas, making it a versatile piece for an entryway, living room, bathroom, or hallway.\n",
      "\n",
      "The overall design is sleek and modern, with clean lines that would complement a variety of home decor styles. The vertical structure of the rack makes it a space-saving solution for keeping footwear and accessories organized in areas with limited floor space.\n",
      "--------------------------\n",
      "\n",
      "Multi-layer white metal shoe rack with additional shelf and 8 double hooks for versatile storage in entryways or hallways.\n",
      "--------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, row in examples.iterrows():\n",
    "    print(f\"{row['title'][:50]}{'...' if len(row['title']) > 50 else ''} - {row['url']} :\\n\")\n",
    "    img_description = describe_image(row['primary_image'], row['title'])\n",
    "    print(f\"{img_description}\\n--------------------------\\n\")\n",
    "    img_caption = caption_image(img_description)\n",
    "    print(f\"{img_caption}\\n--------------------------\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4b5e705",
   "metadata": {},
   "source": [
    "## Image search\n",
    "\n",
    "In this section, we will use generated keywords and captions to search items that match a given input, either text or image.\n",
    "\n",
    "We will leverage our embeddings model to generate embeddings for the keywords and captions and compare them to either input text or the generated caption from an input image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "302d251d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Df we'll use to compare keywords\n",
    "df_keywords = pd.DataFrame(columns=['keyword', 'embedding'])\n",
    "df['keywords'] = ''\n",
    "df['img_description'] = ''\n",
    "df['caption'] = ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "9285bb14",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to replace a keyword with an existing keyword if it's too similar\n",
    "def get_keyword(keyword, df_keywords, threshold = 0.6):\n",
    "    embedded_value = get_embedding(keyword)\n",
    "    df_keywords['similarity'] = df_keywords['embedding'].apply(lambda x: cosine_similarity(np.array(x).reshape(1,-1), np.array(embedded_value).reshape(1, -1)))\n",
    "    sorted_keywords = df_keywords.copy().sort_values('similarity', ascending=False)\n",
    "    if len(sorted_keywords) > 0 :\n",
    "        most_similar = sorted_keywords.iloc[0]\n",
    "        if most_similar['similarity'] > threshold:\n",
    "            print(f\"Replacing '{keyword}' with existing keyword: '{most_similar['keyword']}'\")\n",
    "            return most_similar['keyword']\n",
    "    new_keyword = {\n",
    "        'keyword': keyword,\n",
    "        'embedding': embedded_value\n",
    "    }\n",
    "    df_keywords = pd.concat([df_keywords, pd.DataFrame([new_keyword])], ignore_index=True)\n",
    "    return keyword"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a4051ae",
   "metadata": {},
   "source": [
    "### Preparing the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "297bbc45",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ast\n",
    "\n",
    "def tag_and_caption(row):\n",
    "    keywords = analyze_image(row['primary_image'], row['title'])\n",
    "    try:\n",
    "        keywords = ast.literal_eval(keywords)\n",
    "        mapped_keywords = [get_keyword(k, df_keywords) for k in keywords]\n",
    "    except Exception as e:\n",
    "        print(f\"Error parsing keywords: {keywords}\")\n",
    "        mapped_keywords = []\n",
    "    img_description = describe_image(row['primary_image'], row['title'])\n",
    "    caption = caption_image(img_description)\n",
    "    return {\n",
    "        'keywords': mapped_keywords,\n",
    "        'img_description': img_description,\n",
    "        'caption': caption\n",
    "    }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "e62b8708",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(312, 9)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d59b4b98",
   "metadata": {},
   "source": [
    "Processing all 312 lines of the dataset will take a while.\n",
    "To test out the idea, we will only run it on the first 50 lines: this takes ~20 mins. \n",
    "Feel free to skip this step and load the already processed dataset (see below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43ec629b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Running on first 50 lines\n",
    "for index, row in df[:50].iterrows():\n",
    "    print(f\"{index} - {row['title'][:50]}{'...' if len(row['title']) > 50 else ''}\")\n",
    "    updates = tag_and_caption(row)\n",
    "    df.loc[index, updates.keys()] = updates.values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "85febcd2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>primary_image</th>\n",
       "      <th>style</th>\n",
       "      <th>material</th>\n",
       "      <th>color</th>\n",
       "      <th>url</th>\n",
       "      <th>keywords</th>\n",
       "      <th>img_description</th>\n",
       "      <th>caption</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GOYMFK 1pc Free Standing Shoe Rack, Multi-laye...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/416WaLx10j...</td>\n",
       "      <td>Modern</td>\n",
       "      <td>Metal</td>\n",
       "      <td>White</td>\n",
       "      <td>https://www.amazon.com/dp/B0CJHKVG6P</td>\n",
       "      <td>[shoe rack, free standing, multi-layer, metal,...</td>\n",
       "      <td>This is a free-standing shoe rack featuring a ...</td>\n",
       "      <td>White metal free-standing shoe rack with multi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>subrtex Leather ding Room, Dining Chairs Set o...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/31SejUEWY7...</td>\n",
       "      <td>Black Rubber Wood</td>\n",
       "      <td>Sponge</td>\n",
       "      <td>Black</td>\n",
       "      <td>https://www.amazon.com/dp/B0B66QHB23</td>\n",
       "      <td>[dining chairs, set of 2, leather, black]</td>\n",
       "      <td>This image features a set of two black dining ...</td>\n",
       "      <td>Set of 2 sleek black faux leather dining chair...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Plant Repotting Mat MUYETOL Waterproof Transpl...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/41RgefVq70...</td>\n",
       "      <td>Modern</td>\n",
       "      <td>Polyethylene</td>\n",
       "      <td>Green</td>\n",
       "      <td>https://www.amazon.com/dp/B0BXRTWLYK</td>\n",
       "      <td>[plant repotting mat, waterproof, portable, fo...</td>\n",
       "      <td>This is a square plant repotting mat designed ...</td>\n",
       "      <td>Waterproof green square plant repotting mat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pickleball Doormat, Welcome Doormat Absorbent ...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/61vz1Igler...</td>\n",
       "      <td>Modern</td>\n",
       "      <td>Rubber</td>\n",
       "      <td>A5589</td>\n",
       "      <td>https://www.amazon.com/dp/B0C1MRB2M8</td>\n",
       "      <td>[doormat, absorbent, non-slip, brown]</td>\n",
       "      <td>This is a rectangular doormat featuring a play...</td>\n",
       "      <td>Pickleball-themed coir doormat with playful de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>JOIN IRON Foldable TV Trays for Eating Set of ...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/41p4d4VJnN...</td>\n",
       "      <td>X Classic Style</td>\n",
       "      <td>Iron</td>\n",
       "      <td>Grey Set of 4</td>\n",
       "      <td>https://www.amazon.com/dp/B0CG1N9QRC</td>\n",
       "      <td>[tv tray table set, foldable, iron, grey]</td>\n",
       "      <td>This image showcases a set of two foldable TV ...</td>\n",
       "      <td>Set of two foldable TV trays with grey wood gr...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title  \\\n",
       "0  GOYMFK 1pc Free Standing Shoe Rack, Multi-laye...   \n",
       "1  subrtex Leather ding Room, Dining Chairs Set o...   \n",
       "2  Plant Repotting Mat MUYETOL Waterproof Transpl...   \n",
       "3  Pickleball Doormat, Welcome Doormat Absorbent ...   \n",
       "4  JOIN IRON Foldable TV Trays for Eating Set of ...   \n",
       "\n",
       "                                       primary_image              style  \\\n",
       "0  https://m.media-amazon.com/images/I/416WaLx10j...             Modern   \n",
       "1  https://m.media-amazon.com/images/I/31SejUEWY7...  Black Rubber Wood   \n",
       "2  https://m.media-amazon.com/images/I/41RgefVq70...             Modern   \n",
       "3  https://m.media-amazon.com/images/I/61vz1Igler...             Modern   \n",
       "4  https://m.media-amazon.com/images/I/41p4d4VJnN...    X Classic Style   \n",
       "\n",
       "       material          color                                   url  \\\n",
       "0         Metal          White  https://www.amazon.com/dp/B0CJHKVG6P   \n",
       "1        Sponge          Black  https://www.amazon.com/dp/B0B66QHB23   \n",
       "2  Polyethylene          Green  https://www.amazon.com/dp/B0BXRTWLYK   \n",
       "3        Rubber          A5589  https://www.amazon.com/dp/B0C1MRB2M8   \n",
       "4          Iron  Grey Set of 4  https://www.amazon.com/dp/B0CG1N9QRC   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  [shoe rack, free standing, multi-layer, metal,...   \n",
       "1          [dining chairs, set of 2, leather, black]   \n",
       "2  [plant repotting mat, waterproof, portable, fo...   \n",
       "3              [doormat, absorbent, non-slip, brown]   \n",
       "4          [tv tray table set, foldable, iron, grey]   \n",
       "\n",
       "                                     img_description  \\\n",
       "0  This is a free-standing shoe rack featuring a ...   \n",
       "1  This image features a set of two black dining ...   \n",
       "2  This is a square plant repotting mat designed ...   \n",
       "3  This is a rectangular doormat featuring a play...   \n",
       "4  This image showcases a set of two foldable TV ...   \n",
       "\n",
       "                                             caption  \n",
       "0  White metal free-standing shoe rack with multi...  \n",
       "1  Set of 2 sleek black faux leather dining chair...  \n",
       "2        Waterproof green square plant repotting mat  \n",
       "3  Pickleball-themed coir doormat with playful de...  \n",
       "4  Set of two foldable TV trays with grey wood gr...  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "b98b0ec8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving locally for later\n",
    "data_path = \"data/items_tagged_and_captioned.csv\"\n",
    "df.to_csv(data_path, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9f0de2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional: load data from saved file\n",
    "df = pd.read_csv(data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9336b03c",
   "metadata": {},
   "source": [
    "### Embedding captions and keywords\n",
    "We can now use the generated captions and keywords to match relevant content to an input text query or caption. \n",
    "To do this, we will embed a combination of keywords + captions.\n",
    "Note: creating the embeddings will take ~3 mins to run. Feel free to load the pre-processed dataset (see below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "9cc54beb",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_search = df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "d9810f84",
   "metadata": {},
   "outputs": [],
   "source": [
    "def embed_tags_caption(x):\n",
    "    if x['caption'] != '':\n",
    "        keywords_string = \",\".join(k for k in x['keywords']) + '\\n'\n",
    "        content = keywords_string + x['caption']\n",
    "        embedding = get_embedding(content)\n",
    "        return embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "6780488b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_search['embedding'] = df_search.apply(lambda x: embed_tags_caption(x), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "52241091",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>primary_image</th>\n",
       "      <th>style</th>\n",
       "      <th>material</th>\n",
       "      <th>color</th>\n",
       "      <th>url</th>\n",
       "      <th>keywords</th>\n",
       "      <th>img_description</th>\n",
       "      <th>caption</th>\n",
       "      <th>embedding</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GOYMFK 1pc Free Standing Shoe Rack, Multi-laye...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/416WaLx10j...</td>\n",
       "      <td>Modern</td>\n",
       "      <td>Metal</td>\n",
       "      <td>White</td>\n",
       "      <td>https://www.amazon.com/dp/B0CJHKVG6P</td>\n",
       "      <td>['shoe rack', 'free standing', 'multi-layer', ...</td>\n",
       "      <td>This is a free-standing shoe rack featuring a ...</td>\n",
       "      <td>White metal free-standing shoe rack with multi...</td>\n",
       "      <td>[-0.06596625, -0.026769113, -0.013789515, -0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>subrtex Leather ding Room, Dining Chairs Set o...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/31SejUEWY7...</td>\n",
       "      <td>Black Rubber Wood</td>\n",
       "      <td>Sponge</td>\n",
       "      <td>Black</td>\n",
       "      <td>https://www.amazon.com/dp/B0B66QHB23</td>\n",
       "      <td>['dining chairs', 'set of 2', 'leather', 'black']</td>\n",
       "      <td>This image features a set of two black dining ...</td>\n",
       "      <td>Set of 2 sleek black faux leather dining chair...</td>\n",
       "      <td>[-0.0077859573, -0.010376813, -0.01928079, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Plant Repotting Mat MUYETOL Waterproof Transpl...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/41RgefVq70...</td>\n",
       "      <td>Modern</td>\n",
       "      <td>Polyethylene</td>\n",
       "      <td>Green</td>\n",
       "      <td>https://www.amazon.com/dp/B0BXRTWLYK</td>\n",
       "      <td>['plant repotting mat', 'waterproof', 'portabl...</td>\n",
       "      <td>This is a square plant repotting mat designed ...</td>\n",
       "      <td>Waterproof green square plant repotting mat</td>\n",
       "      <td>[-0.023248248, 0.005370147, -0.0048999498, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Pickleball Doormat, Welcome Doormat Absorbent ...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/61vz1Igler...</td>\n",
       "      <td>Modern</td>\n",
       "      <td>Rubber</td>\n",
       "      <td>A5589</td>\n",
       "      <td>https://www.amazon.com/dp/B0C1MRB2M8</td>\n",
       "      <td>['doormat', 'absorbent', 'non-slip', 'brown']</td>\n",
       "      <td>This is a rectangular doormat featuring a play...</td>\n",
       "      <td>Pickleball-themed coir doormat with playful de...</td>\n",
       "      <td>[-0.028953036, -0.026369056, -0.011363288, 0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>JOIN IRON Foldable TV Trays for Eating Set of ...</td>\n",
       "      <td>https://m.media-amazon.com/images/I/41p4d4VJnN...</td>\n",
       "      <td>X Classic Style</td>\n",
       "      <td>Iron</td>\n",
       "      <td>Grey Set of 4</td>\n",
       "      <td>https://www.amazon.com/dp/B0CG1N9QRC</td>\n",
       "      <td>['tv tray table set', 'foldable', 'iron', 'grey']</td>\n",
       "      <td>This image showcases a set of two foldable TV ...</td>\n",
       "      <td>Set of two foldable TV trays with grey wood gr...</td>\n",
       "      <td>[-0.030723095, -0.0051356032, -0.027088132, 0....</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title  \\\n",
       "0  GOYMFK 1pc Free Standing Shoe Rack, Multi-laye...   \n",
       "1  subrtex Leather ding Room, Dining Chairs Set o...   \n",
       "2  Plant Repotting Mat MUYETOL Waterproof Transpl...   \n",
       "3  Pickleball Doormat, Welcome Doormat Absorbent ...   \n",
       "4  JOIN IRON Foldable TV Trays for Eating Set of ...   \n",
       "\n",
       "                                       primary_image              style  \\\n",
       "0  https://m.media-amazon.com/images/I/416WaLx10j...             Modern   \n",
       "1  https://m.media-amazon.com/images/I/31SejUEWY7...  Black Rubber Wood   \n",
       "2  https://m.media-amazon.com/images/I/41RgefVq70...             Modern   \n",
       "3  https://m.media-amazon.com/images/I/61vz1Igler...             Modern   \n",
       "4  https://m.media-amazon.com/images/I/41p4d4VJnN...    X Classic Style   \n",
       "\n",
       "       material          color                                   url  \\\n",
       "0         Metal          White  https://www.amazon.com/dp/B0CJHKVG6P   \n",
       "1        Sponge          Black  https://www.amazon.com/dp/B0B66QHB23   \n",
       "2  Polyethylene          Green  https://www.amazon.com/dp/B0BXRTWLYK   \n",
       "3        Rubber          A5589  https://www.amazon.com/dp/B0C1MRB2M8   \n",
       "4          Iron  Grey Set of 4  https://www.amazon.com/dp/B0CG1N9QRC   \n",
       "\n",
       "                                            keywords  \\\n",
       "0  ['shoe rack', 'free standing', 'multi-layer', ...   \n",
       "1  ['dining chairs', 'set of 2', 'leather', 'black']   \n",
       "2  ['plant repotting mat', 'waterproof', 'portabl...   \n",
       "3      ['doormat', 'absorbent', 'non-slip', 'brown']   \n",
       "4  ['tv tray table set', 'foldable', 'iron', 'grey']   \n",
       "\n",
       "                                     img_description  \\\n",
       "0  This is a free-standing shoe rack featuring a ...   \n",
       "1  This image features a set of two black dining ...   \n",
       "2  This is a square plant repotting mat designed ...   \n",
       "3  This is a rectangular doormat featuring a play...   \n",
       "4  This image showcases a set of two foldable TV ...   \n",
       "\n",
       "                                             caption  \\\n",
       "0  White metal free-standing shoe rack with multi...   \n",
       "1  Set of 2 sleek black faux leather dining chair...   \n",
       "2        Waterproof green square plant repotting mat   \n",
       "3  Pickleball-themed coir doormat with playful de...   \n",
       "4  Set of two foldable TV trays with grey wood gr...   \n",
       "\n",
       "                                           embedding  \n",
       "0  [-0.06596625, -0.026769113, -0.013789515, -0.0...  \n",
       "1  [-0.0077859573, -0.010376813, -0.01928079, -0....  \n",
       "2  [-0.023248248, 0.005370147, -0.0048999498, -0....  \n",
       "3  [-0.028953036, -0.026369056, -0.011363288, 0.0...  \n",
       "4  [-0.030723095, -0.0051356032, -0.027088132, 0....  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_search.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "f4a3558f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(49, 10)\n"
     ]
    }
   ],
   "source": [
    "# Keep only the lines where we have embeddings\n",
    "df_search = df_search.dropna(subset=['embedding'])\n",
    "print(df_search.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "5d0bc2df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving locally for later\n",
    "data_embeddings_path = \"data/items_tagged_and_captioned_embeddings.csv\"\n",
    "df_search.to_csv(data_embeddings_path, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "1794c6fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional: load data from saved file\n",
    "from ast import literal_eval\n",
    "df_search = pd.read_csv(data_embeddings_path)\n",
    "df_search[\"embedding\"] = df_search.embedding.apply(literal_eval).apply(np.array)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c255a1bc",
   "metadata": {},
   "source": [
    "### Search from input text    \n",
    "\n",
    "We can compare the input text from a user directly to the embeddings we just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "1c957f85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Searching for N most similar results\n",
    "def search_from_input_text(query, n = 2):\n",
    "    embedded_value = get_embedding(query)\n",
    "    df_search['similarity'] = df_search['embedding'].apply(lambda x: cosine_similarity(np.array(x).reshape(1,-1), np.array(embedded_value).reshape(1, -1)))\n",
    "    most_similar = df_search.sort_values('similarity', ascending=False).iloc[:n]\n",
    "    return most_similar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "94d48746",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_inputs = ['shoe storage', 'black metal side table', 'doormat', 'step bookshelf', 'ottoman']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "828e6adf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: shoe storage\n",
      "\n",
      "GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me... (https://www.amazon.com/dp/B0CJHKVG6P) - Similarity: 0.62\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/416WaLx10jL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me... (https://www.amazon.com/dp/B0CJHKVG6P) - Similarity: 0.57\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/416WaLx10jL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: black metal side table\n",
      "\n",
      "FLYJOE Narrow Side Table with PU Leather Magazine ... (https://www.amazon.com/dp/B0CHYDTQKN) - Similarity: 0.59\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41Hsse9SYsL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "HomePop Metal Accent Table Triangle Base Round Mir... (https://www.amazon.com/dp/B08N5H868H) - Similarity: 0.57\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41cG70UIWTL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: doormat\n",
      "\n",
      "Pickleball Doormat, Welcome Doormat Absorbent Non-... (https://www.amazon.com/dp/B0C1MRB2M8) - Similarity: 0.59\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/61vz1IglerL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Caroline's Treasures PPD3013JMAT Enchanted Garden ... (https://www.amazon.com/dp/B08Q5KDSQK) - Similarity: 0.57\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/51Zn-AivGrL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: step bookshelf\n",
      "\n",
      "Leick Home 70007-WTGD Mixed Metal and Wood Stepped... (https://www.amazon.com/dp/B098KNRNLQ) - Similarity: 0.61\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/31XhtLE1F1L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Wildkin Kids Canvas Sling Bookshelf with Storage f... (https://www.amazon.com/dp/B07GBVFZ1Y) - Similarity: 0.47\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/51-GsdoM+IS._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: ottoman\n",
      "\n",
      "HomePop Home Decor | K2380-YDQY-2 | Luxury Large F... (https://www.amazon.com/dp/B0B94T1TZ1) - Similarity: 0.53\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/416lZwKs-SL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Moroccan Leather Pouf Ottoman for Living Room - Ro... (https://www.amazon.com/dp/B0CP45784G) - Similarity: 0.51\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/51UKACPPL9L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in user_inputs:\n",
    "    print(f\"Input: {i}\\n\")\n",
    "    res = search_from_input_text(i)\n",
    "    for index, row in res.iterrows():\n",
    "        similarity_score = row['similarity']\n",
    "        if isinstance(similarity_score, np.ndarray):\n",
    "            similarity_score = similarity_score[0][0]\n",
    "        print(f\"{row['title'][:50]}{'...' if len(row['title']) > 50 else ''} ({row['url']}) - Similarity: {similarity_score:.2f}\")\n",
    "        img = Image(url=row['primary_image'])\n",
    "        display(img)\n",
    "        print(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13d20f4d",
   "metadata": {},
   "source": [
    "### Search from image\n",
    "\n",
    "If the input is an image, we can find similar images by first turning images into captions, and embedding those captions to compare them to the already created embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "bdae7409",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll take a mix of images: some we haven't seen and some that are already in the dataset\n",
    "example_images = df.iloc[306:]['primary_image'].to_list() + df.iloc[5:10]['primary_image'].to_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "78845a30",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/31dCSKQ14YL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mimoglad Office Chair, High Back Ergonomic Desk Ch... (https://www.amazon.com/dp/B0C2YQZS69) - Similarity: 0.63\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/414jZL4tnaL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41CPL03Y-WL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CangLong Mid Century Modern Side Chair with Wood L... (https://www.amazon.com/dp/B08RTLBD1T) - Similarity: 0.51\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/31wnzq-TwKL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/31qQ2tZPv-L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAEPA RV Shoe Storage for Bedside - 8 Extra Large ... (https://www.amazon.com/dp/B0C4PL1R3F) - Similarity: 0.61\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/31bcwiowcBL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41TkLI3K2-L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Chief Mfg.Swing-Arm Wall Mount Hardware Mount Blac... (https://www.amazon.com/dp/B007E40Z5K) - Similarity: 0.63\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41HxUoRXloL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/21Uq9uJEE5L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HomePop Home Decor | K2380-YDQY-2 | Luxury Large F... (https://www.amazon.com/dp/B0B94T1TZ1) - Similarity: 0.63\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/416lZwKs-SL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41f8WNXejUL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CangLong Mid Century Modern Side Chair with Wood L... (https://www.amazon.com/dp/B08RTLBD1T) - Similarity: 0.58\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/31wnzq-TwKL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41zMuj2wvvL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LOVMOR 30'' Bathroom Vanity Sink Base Cabine, Stor... (https://www.amazon.com/dp/B0C9WYYFLB) - Similarity: 0.69\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41zMuj2wvvL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41ixgM73DgL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Folews Bathroom Organizer Over The Toilet Storage,... (https://www.amazon.com/dp/B09NZY3R1T) - Similarity: 0.82\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41ixgM73DgL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/416WaLx10jL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GOYMFK 1pc Free Standing Shoe Rack, Multi-layer Me... (https://www.amazon.com/dp/B0CJHKVG6P) - Similarity: 0.69\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/416WaLx10jL._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/31SejUEWY7L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "subrtex Leather ding Room, Dining Chairs Set of 2,... (https://www.amazon.com/dp/B0B66QHB23) - Similarity: 0.87\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/31SejUEWY7L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "Input: \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41RgefVq70L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Plant Repotting Mat MUYETOL Waterproof Transplanti... (https://www.amazon.com/dp/B0BXRTWLYK) - Similarity: 0.69\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://m.media-amazon.com/images/I/41RgefVq70L._SS522_.jpg\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in example_images:\n",
    "    img_description = describe_image(i, '')\n",
    "    caption = caption_image(img_description)\n",
    "    img = Image(url=i)\n",
    "    print('Input: \\n')\n",
    "    display(img)\n",
    "    res = search_from_input_text(caption, 1).iloc[0]\n",
    "    similarity_score = res['similarity']\n",
    "    if isinstance(similarity_score, np.ndarray):\n",
    "        similarity_score = similarity_score[0][0]\n",
    "    print(f\"{res['title'][:50]}{'...' if len(res['title']) > 50 else ''} ({res['url']}) - Similarity: {similarity_score:.2f}\")\n",
    "    img_res = Image(url=res['primary_image'])\n",
    "    display(img_res)\n",
    "    print(\"\\n\\n\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f3f316d",
   "metadata": {},
   "source": [
    "## Wrapping up\n",
    "\n",
    "\n",
    "In this notebook, we explored how to leverage the multimodal capabilities of GPT-4V to tag and caption images. By providing images along with contextual information to the model, we were able to generate tags and descriptions that can be further refined using a language model like GPT-4-turbo to create captions. This process has practical applications in various scenarios, particularly in enhancing search functionalities.\n",
    "\n",
    "The search use case illustrated can be directly applied to applications such as recommendation systems, but the techniques covered in this notebook can be extended beyond items search and used in multiple use cases, for example RAG applications leveraging unstructured image data.\n",
    "\n",
    "As a next step, you could explore using a combination of rule-based filtering with keywords and embeddings search with captions to retrieve more relevant results."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}