exercises/movie_classification_unsupervised_incl_recommendations_exercise.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Embeddings for unsupervised clustering with named clusters (and other fun things)\n",
    "\n",
    "In this notebook, we use a subset of the [Top 10000 Popular Movies Dataset](https://www.kaggle.com/datasets/db55ac3dfd0098a0cf96dd542807f9253a16587ff233e06baef372bccfd09942) to calculate embeddings on movie descriptions and then apply kmeans to find similar clusters. Once we have these clusters, we'll use a prompt to extract the topics from each cluster. \n",
    "\n",
    "Fill out the missing pieces in the source source to get everything working (indicated by `#FIXME`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import tiktoken\n",
    "import openai\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from dotenv import load_dotenv\n",
    "from openai.embeddings_utils import cosine_similarity\n",
    "from tenacity import retry, wait_random_exponential, stop_after_attempt\n",
    "\n",
    "# Load environment variables\n",
    "load_dotenv()\n",
    "\n",
    "# Configure Azure OpenAI Service API\n",
    "openai.api_type = \"azure\"\n",
    "openai.api_version = \"2022-12-01\"\n",
    "openai.api_base = os.getenv('OPENAI_API_BASE')\n",
    "openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n",
    "\n",
    "# Define embedding model and encoding\n",
    "EMBEDDING_MODEL = 'text-embedding-ada-002'\n",
    "COMPLETION_MODEL = 'text-davinci-003'\n",
    "encoding = tiktoken.get_encoding('cl100k_base')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load `movies.csv`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('../data/movies/movies.csv')\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's create a new column and calculate how many tokens each embedding would cost. This allows us to get an estimate how much we'd pay to create embeddings on the whole dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add a new column to the dataframe where you put the token count of the review\n",
    "#FIXME\n",
    "\n",
    "# print the first 5 rows of the dataframe, then also the total number of tokens\n",
    "total_tokens = #FIXME\n",
    "\n",
    "cost_for_embeddings = total_tokens / 1000 * 0.0004\n",
    "print(f\"Test would cost ${cost_for_embeddings} for embeddings\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define our embedding method. Please note the use of tenacity for having an automated retry mechanism, in case we hit the TPS limits of Azure OpenAI Service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(10))\n",
    "def get_embedding(text) -> list[float]:\n",
    "    text = text.replace(\"\\n\", \" \")\n",
    "    return #FIXME"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's creating the embeddings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = #FIXME\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's create clusters on the embeddings using KMeans. In this case, we'll go for 5 clusters, knowing that this might be wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train k-means on df embeddings\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "n_clusters = 5\n",
    "#FIXME\n",
    "df = df.assign(cluster=kmeans.labels_)\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a cluster per row, let's use t-SNE to project our embeddings into 2d space and visualize the clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "tsne = TSNE(\n",
    "    n_components=2, perplexity=15, random_state=42, init=\"random\", learning_rate=200\n",
    ")\n",
    "\n",
    "matrix = np.vstack(df.embedding.values)\n",
    "print(matrix.shape)\n",
    "vis_dims2 = tsne.fit_transform(matrix)\n",
    "\n",
    "x = [x for x, y in vis_dims2]\n",
    "y = [y for x, y in vis_dims2]\n",
    "\n",
    "for category, color in enumerate([\"purple\", \"green\", \"red\", \"blue\",\"yellow\", 'black', 'orange', 'brown', 'pink', 'grey']):\n",
    "    xs = np.array(x)[df.cluster == category]\n",
    "    ys = np.array(y)[df.cluster == category]\n",
    "    plt.scatter(xs, ys, color=color, alpha=0.3)\n",
    "\n",
    "    avg_x = xs.mean()\n",
    "    avg_y = ys.mean()\n",
    "\n",
    "    plt.scatter(avg_x, avg_y, marker=\"x\", color=color, s=100)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ugh ok, that does not look great, but was somehow expected. We have all kinds of movies and with only 5 clusters, this might not be ideal. However, if you look closely, you can make up some rough shape that resembles a cluster. Also, movies might fall into two or more categories, so it kind of makes sense.\n",
    "\n",
    "Lastly, let's take a few examples from each cluster, send them to Azure OpenAI Service and get the common theses extracted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take 10 movies from each cluster and write a prompt that asks what these have in common\n",
    "# ideally you would use more movies than 10, but this is just a demo\n",
    "for i in range(n_clusters):\n",
    "    reviews = df[df['cluster'] == i]['overview'].sample(10)\n",
    "    reviews = \"\\n\".join(reviews.values.tolist())\n",
    "    \n",
    "    prompt = #FIXME write a prompt that find common topics in the reviews\n",
    "    response = #FIXME\n",
    "    print(f\"Cluster {i} topics: {response}\")\n",
    "    movies = df[df['cluster'] == i]['original_title'].sample(25)\n",
    "    print(f\"Movies from cluster {i}: {', '.join(movies.values.tolist())}\")\n",
    "    print(\"================\")    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doesn't look too bad. But again, 10 samples for each class might not be enough, given the low number of clusters. Anyway, looking at the movie titles, some of the topics actually make fairly ok sense."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bonus: Using the embeddings to build a simple recommendation system"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another thing we can do is use the embeddings for building a very simple recommendation system. So let's try it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie = \"Frozen\"\n",
    "\n",
    "# get embedding for movie\n",
    "e = #FIXME\n",
    "\n",
    "# get cosine similarity between movie and all other movies\n",
    "similarities = #FIXME\n",
    "\n",
    "# get most similar movies\n",
    "movies = #FIXME\n",
    "movies[1:6]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides the last one, this actually does not look too bad...probably would have been useful if we added movie categories and age ratings to our recommendations... :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai-qna-env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "c4ee1bbf3137c7ea9420c4fd488a55642063e5739fe2a7286130d9ba47405b69"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}