Document Similarities/Document_Similarities.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "nayjN8Vrodod",
    "outputId": "a07fee49-de2d-4166-9ff5-28eb70e9e537"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from nltk.corpus import stopwords\n",
    "import nltk\n",
    "import re\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer \n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from sklearn.metrics.pairwise import euclidean_distances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wv6pYEhdodoi"
   },
   "outputs": [],
   "source": [
    "# Sample corpus\n",
    "documents = ['Machine learning is the study of computer algorithms that improve automatically through experience.\\\n",
    "Machine learning algorithms build a mathematical model based on sample data, known as training data.\\\n",
    "The discipline of machine learning employs various approaches to teach computers to accomplish tasks \\\n",
    "where no fully satisfactory algorithm is available.',\n",
    "'Machine learning is closely related to computational statistics, which focuses on making predictions using computers.\\\n",
    "The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.',\n",
    "'Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. \\\n",
    "It involves computers learning from data provided so that they carry out certain tasks.',\n",
    "'Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"\\\n",
    "or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement',\n",
    "'Software engineering is the systematic application of engineering approaches to the development of software.\\\n",
    "Software engineering is a computing discipline.',\n",
    "'A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concerned\\\n",
    "about the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.\\\n",
    "Developing a machine learning application is more iterative and explorative process than software engineering.'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hZNxg33Yodok"
   },
   "outputs": [],
   "source": [
    "pd.set_option('display.max_colwidth', 0)\n",
    "pd.set_option('display.max_columns', 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4915Wqjiodon"
   },
   "outputs": [],
   "source": [
    "documents_df=pd.DataFrame(documents,columns=['documents'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "obVfuYPFodop"
   },
   "source": [
    "### Corpus Dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 320
    },
    "colab_type": "code",
    "id": "r8QnxTXkodoq",
    "outputId": "b42ff44d-8866-4340-c6c1-210cbe51488c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>documents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                        documents\n",
       "0  Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.               \n",
       "1  Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.                                                                                                                                \n",
       "2  Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.                                                                                                                                                              \n",
       "3  Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement                                                                                                                                                          \n",
       "4  Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.                                                                                                                                                                                                                   \n",
       "5  A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cgea0X8Nodot"
   },
   "outputs": [],
   "source": [
    "# removing special characters and stop words from the text\n",
    "stop_words_l=stopwords.words('english')\n",
    "documents_df['documents_cleaned']=documents_df.documents.apply(lambda x: \" \".join(re.sub(r'[^a-zA-Z]',' ',w).lower() for w in x.split() if re.sub(r'[^a-zA-Z]',' ',w).lower() not in stop_words_l) )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M6cklpWTodow"
   },
   "source": [
    "#### Tf-idf vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QKIsEwYYodow"
   },
   "outputs": [],
   "source": [
    "tfidfvectoriser=TfidfVectorizer(max_features=64)\n",
    "tfidfvectoriser.fit(documents_df.documents_cleaned)\n",
    "tfidf_vectors=tfidfvectoriser.transform(documents_df.documents_cleaned)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uHOrtNQQodoz",
    "outputId": "0807c38d-7fa8-400e-bda8-fdcddb36eb92"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6, 64)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Every vector is already normalised to have unit L2 norm\n",
    "np.linalg.norm(tfidf_vectors[0],ord=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0ZfJLWirodo2",
    "outputId": "2c9fe5a4-aaf4-4e8b-cdb0-9080a65bbfeb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.20860612 0.41721224 0.         0.14442061 0.17106    0.17106\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.17106    0.14442061 0.         0.         0.         0.\n",
      " 0.28884121 0.17106    0.         0.         0.         0.\n",
      " 0.32062347 0.32062347 0.         0.17106    0.         0.20860612\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.20860612 0.20860612 0.         0.         0.         0.\n",
      " 0.         0.17106    0.         0.         0.         0.17106\n",
      " 0.20860612 0.17106    0.         0.         0.         0.20860612\n",
      " 0.         0.         0.         0.        ]\n"
     ]
    }
   ],
   "source": [
    "tfidf_vectors=tfidf_vectors.toarray()\n",
    "print (tfidf_vectors[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xvB33Dg6odo4"
   },
   "source": [
    "##### Every document has been converted into a 64 dimensional vector. As we set the max_features=64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GguRnIYsodo4"
   },
   "source": [
    "### Pairwise similarity\n",
    "\n",
    "##### Pairwise cosine similarity would just be the dot product of the vectors becasue tf-idf vectors from sklearn are already normalized and L2 norm of these vectors is 1. So denominator of cosine similiarity formula is 1 in this case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1GFWHXK1odo5"
   },
   "outputs": [],
   "source": [
    "pairwise_similarities=np.dot(tfidf_vectors,tfidf_vectors.T)\n",
    "pairwise_differences=euclidean_distances(tfidf_vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "colab_type": "code",
    "id": "zOFiIAfRodo7",
    "outputId": "f5445de6-0a9e-4f84-b79f-95d87b7f177a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.20860612 0.41721224 0.         0.14442061 0.17106    0.17106\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.17106    0.14442061 0.         0.         0.         0.\n",
      " 0.28884121 0.17106    0.         0.         0.         0.\n",
      " 0.32062347 0.32062347 0.         0.17106    0.         0.20860612\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.         0.         0.         0.         0.         0.\n",
      " 0.20860612 0.20860612 0.         0.         0.         0.\n",
      " 0.         0.17106    0.         0.         0.         0.17106\n",
      " 0.20860612 0.17106    0.         0.         0.         0.20860612\n",
      " 0.         0.         0.         0.        ]\n",
      "(6, 6)\n",
      "[1.         0.30335642 0.29899126 0.20763548 0.06056832 0.16004863]\n"
     ]
    }
   ],
   "source": [
    "print (tfidf_vectors[0])\n",
    "print (pairwise_similarities.shape)\n",
    "print (pairwise_similarities[0][:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x8oqU9Mwodo-"
   },
   "source": [
    "##### similarity is highest, 1 at index 0 becasue they are the same documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m487mu9_odo-"
   },
   "outputs": [],
   "source": [
    "def most_similar(doc_id,similarity_matrix,matrix):\n",
    "    print (f'Document: {documents_df.iloc[doc_id][\"documents\"]}')\n",
    "    print ('\\n')\n",
    "    print (f'Similar Documents using {matrix}:')\n",
    "    if matrix=='Cosine Similarity':\n",
    "        similar_ix=np.argsort(similarity_matrix[doc_id])[::-1]\n",
    "    elif matrix=='Euclidean Distance':\n",
    "        similar_ix=np.argsort(similarity_matrix[doc_id])\n",
    "    for ix in similar_ix:\n",
    "        if ix==doc_id:\n",
    "            continue\n",
    "        print('\\n')\n",
    "        print (f'Document: {documents_df.iloc[ix][\"documents\"]}')\n",
    "        print (f'{matrix} : {similarity_matrix[doc_id][ix]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 445
    },
    "colab_type": "code",
    "id": "OJsEh4QaodpA",
    "outputId": "5a4a5168-fa8d-4133-f81e-f0e765ae7afe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents using Cosine Similarity:\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Cosine Similarity : 0.30335642341823865\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Cosine Similarity : 0.29899125782686603\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Cosine Similarity : 0.2076354806540131\n",
      "\n",
      "\n",
      "Document: A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering.\n",
      "Cosine Similarity : 0.16004862981345694\n",
      "\n",
      "\n",
      "Document: Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.\n",
      "Cosine Similarity : 0.060568322052320635\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_similarities,'Cosine Similarity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KwNyW4XqodpD",
    "outputId": "37b0d957-cd6f-4f6a-c422-722400edd094"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents using Euclidean Distance:\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Euclidean Distance : 1.180375852499331\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Euclidean Distance : 1.184068192439214\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Euclidean Distance : 1.258860214119095\n",
      "\n",
      "\n",
      "Document: A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering.\n",
      "Euclidean Distance : 1.29611062042292\n",
      "\n",
      "\n",
      "Document: Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.\n",
      "Euclidean Distance : 1.3707163659544446\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_differences,'Euclidean Distance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J3H7zfoTodpF"
   },
   "source": [
    "#### word2vec embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "pEbPb6BJodpF",
    "outputId": "18b4c7b9-2a9d-4bad-f584-f50261ff86b7"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/varunchaudhary/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "import gensim\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from nltk.tokenize import word_tokenize\n",
    "import nltk\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "id": "cHlL6XtFodpI",
    "outputId": "2c6a6ce2-43ff-4113-81bf-5959630c2b16"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 2  1 10 11 12 20 21 22  2  1 12 23 13 24 14 25  4 26 27  4 15 16  2  1\n",
      " 28 29  7 30  5 31  8 32 33 34 17  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0]\n"
     ]
    }
   ],
   "source": [
    "# tokenize and pad every document to make them of the same size\n",
    "tokenizer=Tokenizer()\n",
    "tokenizer.fit_on_texts(documents_df.documents_cleaned)\n",
    "tokenized_documents=tokenizer.texts_to_sequences(documents_df.documents_cleaned)\n",
    "tokenized_paded_documents=pad_sequences(tokenized_documents,maxlen=64,padding='post')\n",
    "vocab_size=len(tokenizer.word_index)+1\n",
    "\n",
    "print (tokenized_paded_documents[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 395
    },
    "colab_type": "code",
    "id": "SA2tdICzodpL",
    "outputId": "e82e8160-8775-4b1e-bc4f-482229198c55"
   },
   "outputs": [],
   "source": [
    "# loading pre-trained embeddings, each word is represented as a 300 dimensional vector\n",
    "import gensim\n",
    "W2V_PATH=\"/Users/varunchaudhary/Documents/Varun Docs/Medium/GoogleNews-vectors-negative300.bin.gz\"\n",
    "model_w2v = gensim.models.KeyedVectors.load_word2vec_format(W2V_PATH, binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "POofZ14codpO"
   },
   "outputs": [],
   "source": [
    "# creating embedding matrix, every row is a vector representation from the vocabulary indexed by the tokenizer index. \n",
    "embedding_matrix=np.zeros((vocab_size,300))\n",
    "for word,i in tokenizer.word_index.items():\n",
    "    if word in model_w2v:\n",
    "        embedding_matrix[i]=model_w2v[word]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Xstx71L9odpR",
    "outputId": "7610ba3f-598d-48c9-b383-e8c5a22440f1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4_LEe-S6odpU",
    "outputId": "803c926c-92ea-4daa-e12b-9fa55fe9af1b",
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 2.55859375e-01, -2.20947266e-02,  2.90527344e-02,  5.44433594e-02,\n",
       "       -7.42187500e-02,  3.53515625e-01, -6.34765625e-02,  1.44531250e-01,\n",
       "        7.22656250e-02,  1.00097656e-01, -1.82617188e-01, -2.28515625e-01,\n",
       "        2.20947266e-02, -2.20703125e-01,  1.91406250e-01,  1.91406250e-01,\n",
       "       -1.66992188e-01,  1.67968750e-01,  2.94921875e-01, -1.80664062e-01,\n",
       "       -1.45263672e-02,  1.07421875e-01, -1.65039062e-01,  2.98828125e-01,\n",
       "        1.29882812e-01, -1.17187500e-01, -1.67968750e-01,  1.01562500e-01,\n",
       "        4.49218750e-02, -1.20605469e-01,  7.47070312e-02, -3.47656250e-01,\n",
       "       -1.01074219e-01,  3.80859375e-01, -2.06054688e-01, -7.47070312e-02,\n",
       "       -1.08398438e-01,  1.86523438e-01,  2.01171875e-01, -2.12402344e-02,\n",
       "        2.85156250e-01, -9.27734375e-02,  1.39648438e-01,  5.78613281e-02,\n",
       "        2.67578125e-01, -1.50390625e-01, -8.54492188e-02,  1.92382812e-01,\n",
       "        8.00781250e-02,  6.39648438e-02, -7.47070312e-02, -8.59375000e-02,\n",
       "        5.10253906e-02, -3.51562500e-01, -1.50390625e-01,  7.42187500e-02,\n",
       "        3.37890625e-01, -1.39648438e-01, -6.78710938e-02, -1.54296875e-01,\n",
       "       -4.24804688e-02, -2.99072266e-02, -1.12304688e-01, -9.91210938e-02,\n",
       "        4.05273438e-02, -1.42578125e-01, -2.28515625e-01, -7.66601562e-02,\n",
       "       -1.20117188e-01,  4.56542969e-02,  5.12695312e-02,  5.20019531e-02,\n",
       "        5.78613281e-02,  3.88183594e-02, -8.78906250e-02, -1.14746094e-01,\n",
       "        1.69921875e-01, -1.60156250e-01,  2.15148926e-03, -2.19726562e-01,\n",
       "        1.31835938e-01,  1.53320312e-01,  1.17675781e-01, -4.15039062e-02,\n",
       "        1.53320312e-01,  2.63977051e-03, -9.08203125e-02,  3.32031250e-01,\n",
       "        3.12500000e-01,  1.00097656e-01, -1.14746094e-01,  2.69531250e-01,\n",
       "       -3.49426270e-03,  3.95507812e-02, -1.39648438e-01,  4.78515625e-02,\n",
       "        8.88671875e-02, -9.76562500e-02,  2.65625000e-01,  5.10253906e-02,\n",
       "        1.32812500e-01, -2.79296875e-01, -8.34960938e-02,  1.34277344e-02,\n",
       "       -9.47265625e-02,  8.93554688e-02, -8.05664062e-02, -8.93554688e-02,\n",
       "        2.12890625e-01, -1.39648438e-01,  9.13085938e-02, -1.36718750e-01,\n",
       "       -7.95898438e-02, -4.83398438e-02,  6.83593750e-02, -6.71386719e-03,\n",
       "       -4.05273438e-02, -1.32446289e-02,  4.24804688e-02, -1.39648438e-01,\n",
       "       -6.98242188e-02, -5.15136719e-02,  5.46264648e-03, -3.97949219e-02,\n",
       "       -1.18652344e-01,  9.37500000e-02, -4.97436523e-03,  3.69262695e-03,\n",
       "        2.39257812e-01,  1.36718750e-01,  2.63671875e-01, -1.32446289e-02,\n",
       "       -1.29882812e-01, -1.85546875e-02,  1.07421875e-01, -2.53295898e-03,\n",
       "        1.04003906e-01, -1.09863281e-03, -1.54296875e-01, -3.80859375e-02,\n",
       "        1.35742188e-01, -2.00195312e-01,  1.29882812e-01,  1.78710938e-01,\n",
       "        6.29882812e-02, -1.05468750e-01,  1.15722656e-01, -1.66992188e-01,\n",
       "        4.54101562e-02,  1.10351562e-01,  5.95703125e-02, -5.15136719e-02,\n",
       "        6.78710938e-02,  1.74804688e-01, -6.43920898e-03,  1.96289062e-01,\n",
       "       -2.59765625e-01,  1.53320312e-01, -1.83593750e-01, -8.93554688e-02,\n",
       "       -1.77001953e-02, -6.54296875e-02, -3.29589844e-02, -1.33789062e-01,\n",
       "       -5.83496094e-02, -4.12109375e-01,  1.80664062e-01, -1.78710938e-01,\n",
       "        2.92968750e-02, -3.58581543e-03, -1.41601562e-01, -6.78710938e-02,\n",
       "       -4.34570312e-02, -3.07617188e-02, -2.98828125e-01, -1.67846680e-04,\n",
       "        1.24511719e-01, -1.32812500e-01, -1.92871094e-02,  5.56640625e-02,\n",
       "       -4.90722656e-02, -1.50390625e-01,  1.46484375e-01,  4.83398438e-02,\n",
       "        1.30859375e-01,  6.00585938e-02,  7.66601562e-02,  1.12304688e-01,\n",
       "       -2.14843750e-01,  1.30859375e-01,  8.78906250e-02,  3.00781250e-01,\n",
       "       -2.61230469e-02,  2.36328125e-01,  1.41601562e-01, -1.10351562e-01,\n",
       "       -6.39648438e-02, -3.63769531e-02, -1.67968750e-01, -2.59765625e-01,\n",
       "        1.75781250e-02,  1.93359375e-01, -1.19140625e-01,  2.50000000e-01,\n",
       "        1.66015625e-01, -1.02050781e-01,  8.34960938e-02,  1.06933594e-01,\n",
       "        1.01928711e-02, -3.43750000e-01,  1.30859375e-01,  9.13085938e-02,\n",
       "       -2.81250000e-01,  1.85546875e-01,  1.42578125e-01, -2.53906250e-01,\n",
       "       -9.13085938e-02,  1.76757812e-01, -3.04687500e-01,  1.47460938e-01,\n",
       "       -2.81982422e-02,  3.93066406e-02,  8.20312500e-02,  2.25830078e-02,\n",
       "       -6.54296875e-02, -2.52685547e-02, -1.75781250e-01,  1.45507812e-01,\n",
       "       -4.58984375e-02, -2.96875000e-01,  3.14941406e-02, -2.90527344e-02,\n",
       "       -1.02539062e-01, -2.12890625e-01, -3.20312500e-01,  7.81250000e-02,\n",
       "       -1.81640625e-01,  5.85937500e-02,  9.17968750e-02, -2.67028809e-03,\n",
       "        1.69921875e-01, -3.14453125e-01, -3.02734375e-01, -1.23046875e-01,\n",
       "        1.17187500e-01, -4.90722656e-02, -7.29370117e-03,  1.30859375e-01,\n",
       "        2.36816406e-02, -1.63269043e-03, -1.63085938e-01,  7.29370117e-03,\n",
       "        7.42187500e-02, -1.85546875e-01, -6.20117188e-02, -2.46093750e-01,\n",
       "        1.22558594e-01,  3.23486328e-03, -1.29882812e-01, -3.10546875e-01,\n",
       "        1.41601562e-01, -3.83300781e-02, -1.04492188e-01, -3.14941406e-02,\n",
       "       -1.52343750e-01, -3.43322754e-03, -5.83496094e-02,  1.17675781e-01,\n",
       "       -1.64062500e-01,  1.47460938e-01,  9.13085938e-02, -1.68945312e-01,\n",
       "       -2.67333984e-02, -1.10839844e-01, -1.70898438e-01, -5.88378906e-02,\n",
       "       -2.94189453e-02, -2.25585938e-01, -6.73828125e-02,  2.21679688e-01,\n",
       "       -3.93066406e-02,  1.32812500e-01,  2.11914062e-01, -1.58691406e-02,\n",
       "       -9.03320312e-02, -3.02734375e-01,  9.52148438e-02, -1.47460938e-01,\n",
       "        1.95312500e-01,  2.00195312e-01, -2.57812500e-01, -1.01562500e-01,\n",
       "       -2.87109375e-01, -8.20312500e-02, -1.34765625e-01, -3.33984375e-01,\n",
       "        1.21093750e-01,  4.54101562e-02, -3.39843750e-01, -8.15429688e-02])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix[tokenizer.word_index['machine']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ze3NNOLYodpW"
   },
   "outputs": [],
   "source": [
    "# creating document-word embeddings\n",
    "document_word_embeddings=np.zeros((len(tokenized_paded_documents),64,300))\n",
    "\n",
    "for i in range(len(tokenized_paded_documents)):\n",
    "    for j in range(len(tokenized_paded_documents[0])):\n",
    "        document_word_embeddings[i][j]=embedding_matrix[tokenized_paded_documents[i][j]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "I4JSjhQKodpc",
    "outputId": "7ed03bee-2b78-41df-c661-cada1e8e36f0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6, 64, 300)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document_word_embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AdtxAgA4odpf",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# tf-idf vectors do not keep the original sequence of words, converting them into actual word sequences from the documents\n",
    "\n",
    "document_embeddings=np.zeros((len(tokenized_paded_documents),300))\n",
    "words=tfidfvectoriser.get_feature_names()\n",
    "\n",
    "for i in range(len(document_word_embeddings)):\n",
    "    for j in range(len(words)):\n",
    "        document_embeddings[i]+=embedding_matrix[tokenizer.word_index[words[j]]]*tfidf_vectors[i][j]\n",
    "        \n",
    "document_embeddings=document_embeddings/np.sum(tfidf_vectors,axis=1).reshape(-1,1)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QHQl0X7modpj"
   },
   "outputs": [],
   "source": [
    "pairwise_similarities=cosine_similarity(document_embeddings)\n",
    "pairwise_differences=euclidean_distances(document_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "W-xnkF3Rodpl",
    "outputId": "b8ebae3e-3ec5-4454-8729-661e67888cfd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents using Cosine Similarity:\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Cosine Similarity : 0.8397965442591557\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Cosine Similarity : 0.7794695868906795\n",
      "\n",
      "\n",
      "Document: A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering.\n",
      "Cosine Similarity : 0.723246424523851\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Cosine Similarity : 0.6470561876580082\n",
      "\n",
      "\n",
      "Document: Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.\n",
      "Cosine Similarity : 0.6437153703829421\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_similarities,'Cosine Similarity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SqfLobmZodpn",
    "outputId": "e9c1bf58-c618-49da-b6d9-5ab31a22068c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents using Euclidean Distance:\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Euclidean Distance : 0.6964967402409898\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Euclidean Distance : 0.8474825706357123\n",
      "\n",
      "\n",
      "Document: A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering.\n",
      "Euclidean Distance : 0.9145357051542284\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Euclidean Distance : 0.9828969563779665\n",
      "\n",
      "\n",
      "Document: Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.\n",
      "Euclidean Distance : 1.2883277472134371\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_differences,'Euclidean Distance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L7oG1uuGodpp"
   },
   "source": [
    "#### Glove embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Dyl2etaWodpq"
   },
   "outputs": [],
   "source": [
    "# reading Glove word embeddings into a dictionary with \"word\" as key and values as word vectors\n",
    "\n",
    "embeddings_index = dict()\n",
    "\n",
    "with open('glove.6B.100d.txt') as file:\n",
    "    for line in file:\n",
    "        values = line.split()\n",
    "        word = values[0]\n",
    "        coefs = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = coefs\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "27-WY53Yodps"
   },
   "outputs": [],
   "source": [
    "# creating embedding matrix, every row is a vector representation from the vocabulary indexed by the tokenizer index. \n",
    "\n",
    "embedding_matrix=np.zeros((vocab_size,100))\n",
    "\n",
    "for word,i in tokenizer.word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "elSMrWDKodpu"
   },
   "outputs": [],
   "source": [
    "# tf-idf vectors do not keep the original sequence of words, converting them into actual word sequences from the documents\n",
    "\n",
    "document_embeddings=np.zeros((len(tokenized_paded_documents),100))\n",
    "words=tfidfvectoriser.get_feature_names()\n",
    "\n",
    "for i in range(documents_df.shape[0]):\n",
    "    for j in range(len(words)):\n",
    "        document_embeddings[i]+=embedding_matrix[tokenizer.word_index[words[j]]]*tfidf_vectors[i][j]\n",
    "        \n",
    "document_embeddings=document_embeddings/np.sum(tfidf_vectors,axis=1).reshape(-1,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tg2Q706Uodpw",
    "outputId": "c2da518f-b258-4e6c-a92f-627dc1c4f2cf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6, 100)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "document_embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yo6yjjnyodpy"
   },
   "outputs": [],
   "source": [
    "pairwise_similarities=cosine_similarity(document_embeddings)\n",
    "pairwise_differences=euclidean_distances(document_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dzmfvsAlodp0",
    "outputId": "48be743e-e7d2-4277-a6ff-30dfd1fc4c13"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents using Cosine Similarity:\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Cosine Similarity : 0.9496266898343505\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Cosine Similarity : 0.9304119257167628\n",
      "\n",
      "\n",
      "Document: A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering.\n",
      "Cosine Similarity : 0.9090188747978709\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Cosine Similarity : 0.8851106707072046\n",
      "\n",
      "\n",
      "Document: Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.\n",
      "Cosine Similarity : 0.8480924216003729\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_similarities,'Cosine Similarity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EnrtapNtodp1",
    "outputId": "36594abc-21a4-4d82-ab83-54c737796c1b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents using Euclidean Distance:\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Euclidean Distance : 4.928958963932309\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Euclidean Distance : 6.351243879370724\n",
      "\n",
      "\n",
      "Document: A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering.\n",
      "Euclidean Distance : 6.528270473844955\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Euclidean Distance : 7.494544315272435\n",
      "\n",
      "\n",
      "Document: Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.\n",
      "Euclidean Distance : 8.91017330727787\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_differences,'Euclidean Distance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hecViuQRodp3"
   },
   "source": [
    "#### Doc2vec model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MDLKLZXPodp4"
   },
   "outputs": [],
   "source": [
    "from gensim.models.doc2vec import Doc2Vec, TaggedDocument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5THng5QPodp5"
   },
   "outputs": [],
   "source": [
    "tagged_data = [TaggedDocument(words=word_tokenize(doc), tags=[i]) for i, doc in enumerate(documents_df.documents_cleaned)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YURtty-Lodp7",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_d2v = Doc2Vec(vector_size=100,alpha=0.025, min_count=1)\n",
    "  \n",
    "model_d2v.build_vocab(tagged_data)\n",
    "\n",
    "for epoch in range(100):\n",
    "    model_d2v.train(tagged_data,\n",
    "                total_examples=model_d2v.corpus_count,\n",
    "                epochs=model_d2v.epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bJfa1EYWodp9"
   },
   "outputs": [],
   "source": [
    "document_embeddings=np.zeros((documents_df.shape[0],100))\n",
    "\n",
    "for i in range(len(document_embeddings)):\n",
    "    document_embeddings[i]=model_d2v.docvecs[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GtkQELbxodp_"
   },
   "outputs": [],
   "source": [
    "pairwise_similarities=cosine_similarity(document_embeddings)\n",
    "pairwise_differences=euclidean_distances(document_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bZEEOxDvodqC",
    "outputId": "e2255efb-1969-49af-a9b1-e53214d01a84"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents using Cosine Similarity:\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Cosine Similarity : 0.5079113179325461\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Cosine Similarity : 0.3959149048125277\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Cosine Similarity : 0.26970311587628926\n",
      "\n",
      "\n",
      "Document: Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.\n",
      "Cosine Similarity : 0.14681892748399228\n",
      "\n",
      "\n",
      "Document: A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering.\n",
      "Cosine Similarity : 0.09664196835865298\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_similarities,'Cosine Similarity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "a81ZJ4btodqF",
    "outputId": "9f5d15a1-a188-4784-f6f2-7c4b721fd545"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents using Euclidean Distance:\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Euclidean Distance : 6.697156076504221\n",
      "\n",
      "\n",
      "Document: Software engineering is the systematic application of engineering approaches to the development of software.Software engineering is a computing discipline.\n",
      "Euclidean Distance : 7.701093950940338\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Euclidean Distance : 7.9476063448058705\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Euclidean Distance : 8.847275563142073\n",
      "\n",
      "\n",
      "Document: A software engineer creates programs based on logic for the computer to execute. A software engineer has to be more concernedabout the correctness of the program in all the cases. Meanwhile, a data scientist is comfortable with uncertainty and variability.Developing a machine learning application is more iterative and explorative process than software engineering.\n",
      "Euclidean Distance : 10.652200663950957\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_differences,'Euclidean Distance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aVJRh-cSodqH"
   },
   "source": [
    "#### BERT model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TY_ReGIRodqH"
   },
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "IVXJOehRodqK",
    "outputId": "f50b4618-d579-4036-f980-0d4999aae37c"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 405M/405M [05:40<00:00, 1.19MB/s]\n"
     ]
    }
   ],
   "source": [
    "sbert_model = SentenceTransformer('bert-base-nli-mean-tokens')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JNvfJ6ZnodqM"
   },
   "outputs": [],
   "source": [
    "document_embeddings = sbert_model.encode(documents_df['documents_cleaned'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FgMmPQx7o5ny"
   },
   "outputs": [],
   "source": [
    "pairwise_similarities=cosine_similarity(document_embeddings)\n",
    "pairwise_differences=euclidean_distances(document_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "colab_type": "code",
    "id": "K8jlwGz3o7mv",
    "outputId": "b2abaa6a-c94f-43ff-b039-bdd4760002e8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents:\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Cosine Similarity : 0.8365411758422852\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Cosine Similarity : 0.6118108034133911\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Cosine Similarity : 0.47647824883461\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_similarities,'Cosine Similarity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "colab_type": "code",
    "id": "neuyIUs3o9Sc",
    "outputId": "b6156ea5-4cea-4bb9-de18-693754c76635"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document: Machine learning is the study of computer algorithms that improve automatically through experience.Machine learning algorithms build a mathematical model based on sample data, known as training data.The discipline of machine learning employs various approaches to teach computers to accomplish tasks where no fully satisfactory algorithm is available.\n",
      "\n",
      "\n",
      "Similar Documents:\n",
      "\n",
      "\n",
      "Document: Machine learning is closely related to computational statistics, which focuses on making predictions using computers.The study of mathematical optimization delivers methods, theory and application domains to the field of machine learning.\n",
      "Euclidean Distance : 9.360410690307617\n",
      "\n",
      "\n",
      "Document: Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so. It involves computers learning from data provided so that they carry out certain tasks.\n",
      "Euclidean Distance : 14.060439109802246\n",
      "\n",
      "\n",
      "Document: Machine learning approaches are traditionally divided into three broad categories, depending on the nature of the \"signal\"or \"feedback\" available to the learning system: Supervised, Unsupervised and Reinforcement\n",
      "Euclidean Distance : 16.09708023071289\n"
     ]
    }
   ],
   "source": [
    "most_similar(0,pairwise_differences,'Euclidean Distance')"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "J3H7zfoTodpF",
    "L7oG1uuGodpp"
   ],
   "name": "Document Similarities.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}