How_to_calc_BM25_by_python.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# PythonでBM25を計算する方法（tf-idfの場合との比較）"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "本コンテンツは、以下の記事に付属するJupyterノートブックです。もちろんColabなどでも実行可能です。[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/isshiki/MLnotebooks/blob/master/How_to_calc_BM25_by_python.ipynb)\n",
        "\n",
        "- [記事「BM25／Okapi BM25（情報検索のアルゴリズム）とは？：AI・機械学習の用語辞典 - ＠IT」](https://atmarkit.itmedia.co.jp/ait/articles/2404/22/news021.html)に付属するノートブックです。\n",
        "\n",
        "下記の記事の説明をベースに実装しています。最終的には各文書の類似度を算出しています。なお、実装はChatGPTを使って少し楽をしました。\n",
        "\n",
        "- [記事「tf-idf（term frequency - inverse document frequency）とは？：AI・機械学習の用語辞典 - ＠IT」](https://atmarkit.itmedia.co.jp/ait/articles/2112/23/news028.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R9p5Q_ex4iYj"
      },
      "source": [
        "## 表1　各文書における各単語の出現回数（BoW：Bag of Words）"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "lIYdn1woOS1n",
        "outputId": "9ce66cf1-a916-49a7-89d1-5bd3111f8fe2"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>単語</th>\n",
              "      <th>イヌ</th>\n",
              "      <th>キジ</th>\n",
              "      <th>キツネ</th>\n",
              "      <th>サル</th>\n",
              "      <th>タヌキ</th>\n",
              "      <th>ネコ</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>文書A</th>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書B</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書C</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "単語   イヌ  キジ  キツネ  サル  タヌキ  ネコ\n",
              "文書                           \n",
              "文書A   3   1    0   1    0   0\n",
              "文書B   1   0    1   0    0   2\n",
              "文書C   1   0    1   0    1   0"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import pandas as pd\n",
        "from collections import Counter\n",
        "\n",
        "# 各文書のデータ\n",
        "documents = {\n",
        "    '文書A': 'イヌ イヌ イヌ サル キジ',\n",
        "    '文書B': 'イヌ ネコ ネコ キツネ',\n",
        "    '文書C': 'イヌ タヌキ キツネ'\n",
        "}\n",
        "\n",
        "# 単語の出現回数をカウントしDataFrameに変換する\n",
        "data = {}\n",
        "for doc, text in documents.items():\n",
        "    word_count = Counter(text.split())\n",
        "    data[doc] = word_count\n",
        "\n",
        "# DataFrameを作成し、NaNを0に変換（単語が出現しなかった場合は0とする）\n",
        "df = pd.DataFrame(data).fillna(0).astype(int).T\n",
        "\n",
        "# 列の順序を指定\n",
        "columns_order = ['イヌ', 'キジ', 'キツネ', 'サル', 'タヌキ', 'ネコ']\n",
        "df = df[columns_order]\n",
        "\n",
        "# 列と行の名前を設定\n",
        "df.columns.name = '単語'\n",
        "df.index.name = '文書'\n",
        "\n",
        "# 表示\n",
        "df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2pU4LAjO4lYZ"
      },
      "source": [
        "## 表2　各文書における各単語の出現頻度（tf：term frequency）"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 192
        },
        "id": "FmQepaOJ3pVW",
        "outputId": "eb26bdc7-b45b-429f-92f5-3047540d4484"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf値（tf-idf）\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>単語</th>\n",
              "      <th>イヌ</th>\n",
              "      <th>キジ</th>\n",
              "      <th>キツネ</th>\n",
              "      <th>サル</th>\n",
              "      <th>タヌキ</th>\n",
              "      <th>ネコ</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>文書A</th>\n",
              "      <td>0.60</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書B</th>\n",
              "      <td>0.25</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書C</th>\n",
              "      <td>0.33</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.33</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.33</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "単語     イヌ   キジ   キツネ   サル   タヌキ   ネコ\n",
              "文書                                  \n",
              "文書A  0.60  0.2  0.00  0.2  0.00  0.0\n",
              "文書B  0.25  0.0  0.25  0.0  0.00  0.5\n",
              "文書C  0.33  0.0  0.33  0.0  0.33  0.0"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# tf値を計算（各要素を行の合計で割る）\n",
        "tf = df.div(df.sum(axis=1), axis=0)\n",
        "\n",
        "# 列と行の名前を設定\n",
        "tf.columns.name = '単語'\n",
        "tf.index.name = '文書'\n",
        "\n",
        "# 表示（＋少数第2位まで丸める）\n",
        "print('tf値（tf-idf）')\n",
        "tf.round(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7MK3652p5GZI"
      },
      "source": [
        "### Okapi BM25の正規化されたtf値の場合"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 192
        },
        "id": "eJmez6xN5Iqf",
        "outputId": "b4012108-27a2-47d9-e8df-573c60e58b9b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf値（BM25）\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>単語</th>\n",
              "      <th>イヌ</th>\n",
              "      <th>キジ</th>\n",
              "      <th>キツネ</th>\n",
              "      <th>サル</th>\n",
              "      <th>タヌキ</th>\n",
              "      <th>ネコ</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>文書A</th>\n",
              "      <td>1.67</td>\n",
              "      <td>0.89</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.89</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書B</th>\n",
              "      <td>1.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>1.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>1.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書C</th>\n",
              "      <td>1.14</td>\n",
              "      <td>0.00</td>\n",
              "      <td>1.14</td>\n",
              "      <td>0.00</td>\n",
              "      <td>1.14</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "単語     イヌ    キジ   キツネ    サル   タヌキ   ネコ\n",
              "文書                                    \n",
              "文書A  1.67  0.89  0.00  0.89  0.00  0.0\n",
              "文書B  1.00  0.00  1.00  0.00  0.00  1.5\n",
              "文書C  1.14  0.00  1.14  0.00  1.14  0.0"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# パラメータ\n",
        "k1 = 2.0\n",
        "b = 0.75\n",
        "\n",
        "# 各文書の長さ（単語数）と平均文書長\n",
        "document_lengths = df.sum(axis=1)\n",
        "average_length = document_lengths.mean()\n",
        "\n",
        "# BM25のtf値を各単語に対して計算\n",
        "k_values = k1 * (1 - b + b * document_lengths / average_length)\n",
        "bm25_tf = (df * (k1 + 1)) / (df + k_values.values[:, None])  # 各単語のtf値を計算\n",
        "\n",
        "# 表示\n",
        "print('tf値（BM25）')\n",
        "bm25_tf.round(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6HBA17Zh7k4Y"
      },
      "source": [
        "## 表3　各単語を含む文書の数"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "id": "_Vjup63N7lk4",
        "outputId": "a0007acd-12e0-48c4-cbae-4ea513c73266"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>単語</th>\n",
              "      <th>イヌ</th>\n",
              "      <th>キジ</th>\n",
              "      <th>キツネ</th>\n",
              "      <th>サル</th>\n",
              "      <th>タヌキ</th>\n",
              "      <th>ネコ</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>全文書（3個）中での文書の数</th>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "単語              イヌ  キジ  キツネ  サル  タヌキ  ネコ\n",
              "全文書（3個）中での文書の数   3   1    2   1    1   1"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# 各単語が含まれる文書の数を計算\n",
        "document_count = (df > 0).sum(axis=0)\n",
        "\n",
        "# 単一行のDataFrameを作成\n",
        "document_count = pd.DataFrame([document_count], index=[\"全文書（3個）中での文書の数\"])\n",
        "\n",
        "# 表示\n",
        "document_count"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7LLxCSSi8Pkr"
      },
      "source": [
        "## 表4　各単語の文書間でのレア度（idf：inverse document frequency）"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "pzkvV7XS8RO0",
        "outputId": "df5e2202-8463-47c5-b17e-a2deaf575d5e"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>単語</th>\n",
              "      <th>イヌ</th>\n",
              "      <th>キジ</th>\n",
              "      <th>キツネ</th>\n",
              "      <th>サル</th>\n",
              "      <th>タヌキ</th>\n",
              "      <th>ネコ</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>idf値（tf-idf）</th>\n",
              "      <td>0.0</td>\n",
              "      <td>1.1</td>\n",
              "      <td>0.41</td>\n",
              "      <td>1.1</td>\n",
              "      <td>1.1</td>\n",
              "      <td>1.1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "単語             イヌ   キジ   キツネ   サル  タヌキ   ネコ\n",
              "idf値（tf-idf）  0.0  1.1  0.41  1.1  1.1  1.1"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import numpy as np\n",
        "\n",
        "# 文書の総数\n",
        "N = len(df)\n",
        "\n",
        "# 各単語が含まれる文書の数を計算\n",
        "document_count = (df > 0).sum(axis=0)\n",
        "\n",
        "# idf値を計算\n",
        "idf = np.log(N / document_count)\n",
        "\n",
        "# 単一行のDataFrameとしてidf値を表示\n",
        "idf = pd.DataFrame([idf], index=[\"idf値（tf-idf）\"])\n",
        "\n",
        "# 表示\n",
        "idf.round(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SVU81NLu9BAI"
      },
      "source": [
        "### Okapi BM25のidf値の場合"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "id": "aZnBKPIf9Hx2",
        "outputId": "7ba1fed7-21af-474b-9834-833e1b7dc6c6"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>単語</th>\n",
              "      <th>イヌ</th>\n",
              "      <th>キジ</th>\n",
              "      <th>キツネ</th>\n",
              "      <th>サル</th>\n",
              "      <th>タヌキ</th>\n",
              "      <th>ネコ</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>idf値（BM25）</th>\n",
              "      <td>0.13</td>\n",
              "      <td>0.98</td>\n",
              "      <td>0.47</td>\n",
              "      <td>0.98</td>\n",
              "      <td>0.98</td>\n",
              "      <td>0.98</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "単語            イヌ    キジ   キツネ    サル   タヌキ    ネコ\n",
              "idf値（BM25）  0.13  0.98  0.47  0.98  0.98  0.98"
            ]
          },
          "execution_count": 16,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "# 文書の総数\n",
        "N = len(df)\n",
        "\n",
        "# 各単語が含まれる文書の数を計算\n",
        "document_count = (df > 0).sum(axis=0)\n",
        "\n",
        "# BM25のidf値を計算\n",
        "idf_bm25 = np.log(((N - document_count + 0.5) / (document_count + 0.5)) + 1)\n",
        "\n",
        "# 負の値を避けるために、この例では「最小値を0に制限」\n",
        "idf_bm25 = np.maximum(idf_bm25, 0)\n",
        "\n",
        "# 単一行のDataFrameとしてidf値を表示\n",
        "bm25_idf = pd.DataFrame([idf_bm25], index=[\"idf値（BM25）\"])\n",
        "\n",
        "# 表示\n",
        "bm25_idf.round(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idQzi7Rl-Qel"
      },
      "source": [
        "## 表5　各文書における各単語の重要度（tf-idf）"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "GhKH11sK-t7w",
        "outputId": "65479647-90dc-43cc-95dc-e325f83c2ea3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf-idf値\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>単語</th>\n",
              "      <th>イヌ</th>\n",
              "      <th>キジ</th>\n",
              "      <th>キツネ</th>\n",
              "      <th>サル</th>\n",
              "      <th>タヌキ</th>\n",
              "      <th>ネコ</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>文書A</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.22</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.22</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書B</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.10</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.55</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書C</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.14</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.37</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "単語    イヌ    キジ   キツネ    サル   タヌキ    ネコ\n",
              "文書                                    \n",
              "文書A  0.0  0.22  0.00  0.22  0.00  0.00\n",
              "文書B  0.0  0.00  0.10  0.00  0.00  0.55\n",
              "文書C  0.0  0.00  0.14  0.00  0.37  0.00"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# idf値のDataFrameから必要なデータを取得\n",
        "idf_series = idf.loc[\"idf値（tf-idf）\"]\n",
        "\n",
        "# tf値とidf値を掛け合わせてtf-idf値を計算\n",
        "tf_idf = tf.mul(idf_series, axis=1)\n",
        "\n",
        "# 表示\n",
        "print('tf-idf値')\n",
        "tf_idf.round(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rSbCvIvG_cAO"
      },
      "source": [
        "### Okapi BM25の場合"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "qklAP-yE_dV0",
        "outputId": "5eb2ddfe-bbc5-48c6-8f0a-8c5705ce7a0a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "BM25スコア\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>単語</th>\n",
              "      <th>イヌ</th>\n",
              "      <th>キジ</th>\n",
              "      <th>キツネ</th>\n",
              "      <th>サル</th>\n",
              "      <th>タヌキ</th>\n",
              "      <th>ネコ</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>文書A</th>\n",
              "      <td>0.22</td>\n",
              "      <td>0.87</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.87</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書B</th>\n",
              "      <td>0.13</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.47</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>1.47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書C</th>\n",
              "      <td>0.15</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.54</td>\n",
              "      <td>0.00</td>\n",
              "      <td>1.12</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "単語     イヌ    キジ   キツネ    サル   タヌキ    ネコ\n",
              "文書                                     \n",
              "文書A  0.22  0.87  0.00  0.87  0.00  0.00\n",
              "文書B  0.13  0.00  0.47  0.00  0.00  1.47\n",
              "文書C  0.15  0.00  0.54  0.00  1.12  0.00"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# idf値のDataFrameから必要なデータを取得\n",
        "bm25_idf_series = bm25_idf.loc[\"idf値（BM25）\"]\n",
        "\n",
        "# 正規化されたtf値とidf値を掛け合わせてBM25スコアを計算\n",
        "bm25 = bm25_tf.mul(bm25_idf_series, axis=1)\n",
        "\n",
        "# 表示\n",
        "print('BM25スコア')\n",
        "bm25.round(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TNeRyggAAVqY"
      },
      "source": [
        "## 表6　各文書の特徴ベクトル同士で計算したコサイン類似度の一覧表\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "3k7hpzfkAX6O",
        "outputId": "38037e7c-0aed-4963-c9d6-5175ba04d6fb"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "類似度（tf-idf）\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>文書</th>\n",
              "      <th>文書A</th>\n",
              "      <th>文書B</th>\n",
              "      <th>文書C</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>文書A</th>\n",
              "      <td>1.0</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書B</th>\n",
              "      <td>0.0</td>\n",
              "      <td>1.00</td>\n",
              "      <td>0.06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書C</th>\n",
              "      <td>0.0</td>\n",
              "      <td>0.06</td>\n",
              "      <td>1.00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "文書   文書A   文書B   文書C\n",
              "文書                  \n",
              "文書A  1.0  0.00  0.00\n",
              "文書B  0.0  1.00  0.06\n",
              "文書C  0.0  0.06  1.00"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "\n",
        "# tf-idf値のDataFrameを用いてコサイン類似度を計算\n",
        "cosine_sim_matrix = cosine_similarity(tf_idf)\n",
        "\n",
        "# コサイン類似度の結果をDataFrameに変換\n",
        "cosine_sim = pd.DataFrame(cosine_sim_matrix, index=tf_idf.index, columns=tf_idf.index)\n",
        "\n",
        "# 表示\n",
        "print('類似度（tf-idf）')\n",
        "cosine_sim.round(2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-EfHtwLMAgVH"
      },
      "source": [
        "### Okapi BM25の場合"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "Ub6Ww5JdAhW_",
        "outputId": "ba18a022-1dc9-4cf2-ec3d-adc1ecd9aa45"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "類似度（BM25）\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>文書</th>\n",
              "      <th>文書A</th>\n",
              "      <th>文書B</th>\n",
              "      <th>文書C</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>文書A</th>\n",
              "      <td>1.00</td>\n",
              "      <td>0.02</td>\n",
              "      <td>0.02</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書B</th>\n",
              "      <td>0.02</td>\n",
              "      <td>1.00</td>\n",
              "      <td>0.14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>文書C</th>\n",
              "      <td>0.02</td>\n",
              "      <td>0.14</td>\n",
              "      <td>1.00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "文書    文書A   文書B   文書C\n",
              "文書                   \n",
              "文書A  1.00  0.02  0.02\n",
              "文書B  0.02  1.00  0.14\n",
              "文書C  0.02  0.14  1.00"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "\n",
        "# BM25スコアのDataFrameを用いてコサイン類似度を計算\n",
        "bm25_cosine_sim_matrix = cosine_similarity(bm25)\n",
        "\n",
        "# コサイン類似度の結果をDataFrameに変換\n",
        "bm25_cosine_sim = pd.DataFrame(bm25_cosine_sim_matrix, index=bm25.index, columns=bm25.index)\n",
        "\n",
        "# 表示\n",
        "print('類似度（BM25）')\n",
        "bm25_cosine_sim.round(2)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "How_to_calc_BM25_by_python.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}