notebooks/BM25-POC.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "068262b4-edde-478f-96f2-ecb6042648eb",
   "metadata": {},
   "source": [
    "## Introdution - BM25 POC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95597d47-c460-41e5-ba49-9e3aaf8ab191",
   "metadata": {},
   "source": [
    "BM25, which stands for \"Best Matching 25,\" is an information retrieval algorithm used in search engines and text retrieval systems. It is an improved version of the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm, designed to address some of its limitations.\n",
    "\n",
    "Here's a brief description of BM25 and its primary purpose:\n",
    "\n",
    "1. **Relevance Scoring**: BM25 is used to calculate the relevance score of a document or web page to a specific query. This score helps search engines rank documents in order of relevance when a user performs a search.\n",
    "\n",
    "2. **Term Frequency and Inverse Document Frequency**: Like TF-IDF, BM25 takes into account the frequency of terms (words) in a document (Term Frequency) and the rarity of those terms across all documents (Inverse Document Frequency). However, BM25 uses a different formula for calculating these values.\n",
    "\n",
    "3. **Tuning Parameters**: BM25 introduces tuning parameters, such as \"k1\" and \"b,\" which allow system administrators to adjust the algorithm's sensitivity to term frequency and document length. This makes BM25 more flexible and adaptable to different types of documents and search scenarios.\n",
    "\n",
    "4. **Non-linear Relationship**: Unlike TF-IDF, BM25 incorporates a non-linear relationship between term frequency and relevance. This means that as a term appears more frequently in a document, its contribution to the relevance score saturates, preventing documents with excessive keyword repetition from receiving disproportionately high scores.\n",
    "\n",
    "5. **Improved Retrieval Performance**: BM25 has been found to perform well in various information retrieval tasks, such as document retrieval, web search, and text classification. It often yields more accurate and contextually relevant results compared to simple TF-IDF-based approaches.\n",
    "\n",
    "In summary, BM25 is a relevance scoring algorithm used to rank documents in search engines and information retrieval systems. It overcomes some of the limitations of TF-IDF by introducing tuning parameters and a non-linear relationship between term frequency and relevance, resulting in improved retrieval performance and more accurate search results."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bbea588-d591-48a8-bd25-1619a0df4c12",
   "metadata": {},
   "source": [
    "# POC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "29e7426a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "import numpy as np\n",
    "from rank_bm25 import BM25Okapi\n",
    "\n",
    "from movielens_ai_playground.io.read_data import read_movies_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5cfb6d61",
   "metadata": {},
   "outputs": [],
   "source": [
    "MOVIES_PATH = \"../data/movielens-100k/u.item\"\n",
    "movies_df = read_movies_data(path=MOVIES_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "983e4a74-bc14-48c9-ab82-47e8adbf31ac",
   "metadata": {},
   "source": [
    "## 1. Build Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cef00a9f-fec2-4888-b9f0-77ca6c255a6f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start building bm25 with titles\n"
     ]
    }
   ],
   "source": [
    "print(\"Start building bm25 with titles\")\n",
    "bm25_title = BM25Okapi(movies_df.title)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e8d18af-6297-428d-9b00-766550d584b2",
   "metadata": {},
   "source": [
    "## 2. Ask query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d456d352-d435-4c5c-9ab7-be7b17765799",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Toy\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dc2bd3cb-c89b-4399-bc65-04726daeb2c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_scores = bm25_title.get_scores(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38ca4f79-1abc-402b-b159-08d3d19fa99e",
   "metadata": {},
   "source": [
    "## 3. Get Indexes to be shown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4e2296d8-37d0-4355-b8cc-393568cdd008",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   0  789 1182 ...  516  525  472]\n"
     ]
    }
   ],
   "source": [
    "# Order by max and get the movie index\n",
    "sorted_indices = np.argsort(doc_scores)[::-1]\n",
    "print(sorted_indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7b6f647-3578-4702-9eb3-9fa0c39d0b9c",
   "metadata": {},
   "source": [
    "## 4. Retrieve movie_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "37271547-42ee-4a11-9b11-29157e3e52aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "bm25_df = movies_df.iloc[sorted_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5f9f1717-e4a7-404e-8f4c-f749ce05cb27",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   1  790 1183  328 1484]\n"
     ]
    }
   ],
   "source": [
    "# get results\n",
    "hits=  5\n",
    "bm25_top = bm25_df.movieId[: int(hits)].values\n",
    "print(bm25_top)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}