analysis/code/lines-similarity/cosine_simil.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8f96b5fb-8208-45b7-b6ac-235b9a217fd5",
   "metadata": {},
   "source": [
    "# Cosine Similarity (tf-idf vectorizer & TF-IDF)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c83eb829-7dac-4a11-a674-dc69a0884b31",
   "metadata": {},
   "source": [
    "### Description of the Code\n",
    "\n",
    "The script below performs text processing and analysis on a collection of Latin texts, specifically focusing on identifying and comparing sentences for similarity. The workflow involves several steps, which are outlined below:\n",
    "\n",
    "1. **Import Required Libraries**:\n",
    "    - The script imports necessary libraries for file handling (`os`), CSV writing (`csv`), string manipulation (`string`), Latin text processing (`cltk`), and text similarity analysis (`sklearn`).\n",
    "\n",
    "2. **Setup and Initialize**:\n",
    "    - The script downloads and imports Latin language models using CLTK (Classical Language Toolkit) and initializes a sentence tokenizer for Latin.\n",
    "\n",
    "3. **Define Directories and Texts**:\n",
    "    - It sets up paths for the directory containing the corpus of texts and a results directory where the output will be saved.\n",
    "    - A list of disputed texts (plays) is specified.\n",
    "\n",
    "4. **Function Definitions**:\n",
    "    - **`read_file(file_path)`**: Reads the content of a file given its path.\n",
    "    - **`tokenize_sentences(text)`**: Tokenizes a given text into sentences, removes punctuation, and converts sentences to lowercase.\n",
    "    - **`compare_sentences(play1, play2, threshold=0.6)`**: Converts sentences to TF-IDF vectors and computes cosine similarity between them. Sentences with similarity above the specified threshold are identified.\n",
    "\n",
    "5. **Process Senecan Plays**:\n",
    "    - The script iterates through the files in the specified directory, processes each Senecan play (excluding disputed texts), and tokenizes the sentences. The sentences are stored in a dictionary.\n",
    "\n",
    "6. **Process and Compare Disputed Plays**:\n",
    "    - For each disputed text, the script reads and tokenizes the content.\n",
    "    - It then compares the sentences of the disputed text with each Senecan play using cosine similarity.\n",
    "    - Similar sentences are identified and the results are saved in CSV files for detailed examination.\n",
    "\n",
    "7. **Modify Disputed Texts**:\n",
    "    - Sentences in the disputed texts that are similar to those in Senecan plays are removed.\n",
    "    - The percentage of sentences removed is calculated and printed.\n",
    "    - The modified versions of the disputed texts, with similar sentences removed, are saved to a specified directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9be4826e-b910-42d4-abe0-8c8284dc3da1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Examples for Sen oct vs Sen ag:\n",
      "Examples for Sen oct vs Sen thy:\n",
      "Examples for Sen oct vs Sen her f:\n",
      "Examples for Sen oct vs Sen phaed:\n",
      "Examples for Sen oct vs Sen phoen:\n",
      "Similarity: 0.7437\n",
      "Disputed Sentence: et hoc sat est\n",
      "Senecan Sentence: nec hoc sat est\n",
      "Examples for Sen oct vs Sen oed:\n",
      "Examples for Sen oct vs Sen med:\n",
      "Similarity: 0.6417\n",
      "Disputed Sentence: parere dubitas\n",
      "Senecan Sentence: profugere dubitas\n",
      "Examples for Sen oct vs Sen tro:\n",
      "Total Sentences in Sen oct: 422\n",
      "Removed Sentences: 2\n",
      "Percentage Removed: 0.47%\n",
      "Modified disputed text saved to: ../../corpora/corpus_imposters_cento/Sen oct.txt\n",
      "Examples for Sen her o vs Sen ag:\n",
      "Similarity: 1.0000\n",
      "Disputed Sentence: scelus occupandum est\n",
      "Senecan Sentence: scelus occupandum est\n",
      "Similarity: 0.7694\n",
      "Disputed Sentence: peractum est\n",
      "Senecan Sentence: habet peractum est\n",
      "Similarity: 0.6146\n",
      "Disputed Sentence: habet peractum est quas petis poenas dedit\n",
      "Senecan Sentence: habet peractum est\n",
      "Similarity: 0.7574\n",
      "Disputed Sentence: heu quid hoc\n",
      "Senecan Sentence: eheu quid hoc est\n",
      "Examples for Sen her o vs Sen thy:\n",
      "Similarity: 0.7503\n",
      "Disputed Sentence: quid hoc\n",
      "Senecan Sentence: sed quid hoc\n",
      "Similarity: 0.7919\n",
      "Disputed Sentence: nunc odia ponis\n",
      "Senecan Sentence: sic odia ponis\n",
      "Similarity: 0.7503\n",
      "Disputed Sentence: quid hoc\n",
      "Senecan Sentence: sed quid hoc\n",
      "Similarity: 0.7503\n",
      "Disputed Sentence: quid hoc\n",
      "Senecan Sentence: sed quid hoc\n",
      "Examples for Sen her o vs Sen her f:\n",
      "Similarity: 0.8289\n",
      "Disputed Sentence: inveniam viam\n",
      "Senecan Sentence: mortis inveniam viam\n",
      "Similarity: 0.8124\n",
      "Disputed Sentence: nescio quod animus grande praesagit malum\n",
      "Senecan Sentence: nescio quod mihi nescio quod animus grande praesagit malum ubi es parens\n",
      "Similarity: 1.0000\n",
      "Disputed Sentence: sic sic agendum est\n",
      "Senecan Sentence: sic sic agendum est\n",
      "Similarity: 0.8102\n",
      "Disputed Sentence: misera pergis\n",
      "Senecan Sentence: quo misera pergis\n",
      "Similarity: 0.8078\n",
      "Disputed Sentence: quid diem questu tero\n",
      "Senecan Sentence: cur diem questu tero\n",
      "Similarity: 0.6399\n",
      "Disputed Sentence: complectere ossa\n",
      "Senecan Sentence: complectere aras\n",
      "Examples for Sen her o vs Sen phaed:\n",
      "Similarity: 0.6115\n",
      "Disputed Sentence: quid stupes segnis furor\n",
      "Senecan Sentence: anime quid segnis stupes\n",
      "Similarity: 0.7810\n",
      "Disputed Sentence: en locus ab omni tutus arbitrio vacat\n",
      "Senecan Sentence: en locus ab omni liber arbitrio vacat\n",
      "Similarity: 0.7788\n",
      "Disputed Sentence: quam bene excideras dolor\n",
      "Senecan Sentence: quam bene excideram mihi\n",
      "Examples for Sen her o vs Sen phoen:\n",
      "Examples for Sen her o vs Sen oed:\n",
      "Similarity: 1.0000\n",
      "Disputed Sentence: quid quaeris ultra\n",
      "Senecan Sentence: quid quaeris ultra\n",
      "Similarity: 0.7170\n",
      "Disputed Sentence: vitam relinques\n",
      "Senecan Sentence: viduam relinques\n",
      "Similarity: 1.0000\n",
      "Disputed Sentence: quid quaeris ultra\n",
      "Senecan Sentence: quid quaeris ultra\n",
      "Examples for Sen her o vs Sen med:\n",
      "Similarity: 0.6676\n",
      "Disputed Sentence: numquid timemur\n",
      "Senecan Sentence: timemur\n",
      "Similarity: 0.6317\n",
      "Disputed Sentence: altrix fatebor\n",
      "Senecan Sentence: fatebor\n",
      "Similarity: 1.0000\n",
      "Disputed Sentence: excidimus tibi\n",
      "Senecan Sentence: excidimus tibi\n",
      "Examples for Sen her o vs Sen tro:\n",
      "Total Sentences in Sen her o: 1149\n",
      "Removed Sentences: 25\n",
      "Percentage Removed: 2.18%\n",
      "Modified disputed text saved to: ../../corpora/corpus_imposters_cento/Sen her o.txt\n",
      "CPU times: user 5.61 s, sys: 1.14 s, total: 6.75 s\n",
      "Wall time: 7.85 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "import os\n",
    "import csv\n",
    "import string\n",
    "from cltk.data.fetch import FetchCorpus\n",
    "from cltk.sentence.lat import LatinPunktSentenceTokenizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "# import the Latin model for the sentence tokenizer\n",
    "corpus_downloader = FetchCorpus(language='lat')\n",
    "corpus_downloader.import_corpus('lat_models_cltk')\n",
    "corpus_downloader.import_corpus('latin_training_set_sentence_cltk')\n",
    "\n",
    "# initialize CLTK sentence tokenizer for Latin\n",
    "sentence_tokenizer = LatinPunktSentenceTokenizer(strict=True)\n",
    "\n",
    "# directory containing the corpus of texts\n",
    "directory_path = '../../corpora/corpus_imposters/'\n",
    "\n",
    "# directory to write the results\n",
    "results_directory = os.path.join('..', 'lines-similarity', 'results_line_sim_cosine')\n",
    "os.makedirs(results_directory, exist_ok=True)\n",
    "\n",
    "# list of disputed texts\n",
    "disputed_texts = ['sen_oct.txt', 'sen_her_o.txt']\n",
    "\n",
    "# dictionary to save sentences for each play\n",
    "play_sentences = {}\n",
    "\n",
    "# dictionary to store the count of removed sentences for each disputed play\n",
    "removed_sentence_count = {}\n",
    "\n",
    "# function to read files\n",
    "def read_file(file_path):\n",
    "    with open(file_path, 'r', encoding='utf-8') as file:\n",
    "        return file.read()\n",
    "\n",
    "# function to tokenize sentences\n",
    "def tokenize_sentences(text):\n",
    "    sentences = sentence_tokenizer.tokenize(text)\n",
    "    sentences_without_punct = [sentence.translate(str.maketrans('', '', string.punctuation)).lower() for sentence in sentences]\n",
    "    return sentences_without_punct\n",
    "\n",
    "# function to compare sentences using cosine similarity\n",
    "def compare_sentences(play1, play2, threshold=0.6):\n",
    "    \"\"\"\n",
    "    Convert sentences into vectors and compare them using cosine similarity.\n",
    "    - strip_accents='unicode': removes accents from characters using Unicode\n",
    "    - lowercase=True: converts all characters to lowercase\n",
    "    - analyzer='char': analyzes the input as a sequence of characters\n",
    "    - ngram_range=(4, 4): considers 4-grams as features\n",
    "    \"\"\"\n",
    "    vectorizer = TfidfVectorizer(strip_accents='unicode', lowercase=True, analyzer='char', ngram_range=(4, 4))\n",
    "    vectors = vectorizer.fit_transform(play1 + play2)\n",
    "\n",
    "    # compute cosine similarity by comparing the vectors\n",
    "    similarities = cosine_similarity(vectors[:len(play1)], vectors[len(play1):])\n",
    "\n",
    "    # find similar sentences\n",
    "    similar_lines = []\n",
    "    for i in range(len(play1)):\n",
    "        for j in range(len(play2)):\n",
    "            if similarities[i, j] >= threshold:\n",
    "                similar_lines.append((i, j, similarities[i, j], play1[i], play2[j]))\n",
    "\n",
    "    return similar_lines\n",
    "\n",
    "# process each file in the directory\n",
    "for filename in os.listdir(directory_path):\n",
    "    file_path = os.path.join(directory_path, filename)\n",
    "\n",
    "    # check if the file is a Senecan play and not disputed\n",
    "    if filename.startswith(\"sen_\") and filename not in disputed_texts:\n",
    "        # clean filename to make the result more readable\n",
    "        clean_filename = os.path.splitext(os.path.basename(filename))[0].replace('_', ' ').capitalize()\n",
    "        # read the file\n",
    "        text = read_file(file_path)\n",
    "        # split into sentences\n",
    "        sentences = tokenize_sentences(text)\n",
    "        # save the results into the dictionary\n",
    "        play_sentences[clean_filename] = sentences\n",
    "\n",
    "# process disputed plays\n",
    "for disputed_text in disputed_texts:\n",
    "    # clean filename to make results more readable\n",
    "    disputed_text_name = os.path.splitext(os.path.basename(disputed_text))[0].replace('_', ' ').capitalize()\n",
    "    disputed_path = os.path.join(directory_path, disputed_text)\n",
    "    # read the contents of the file\n",
    "    disputed_text_content = read_file(disputed_path)\n",
    "    # split into sentences\n",
    "    disputed_sentences = tokenize_sentences(disputed_text_content)\n",
    "\n",
    "    # compare sentences with each Senecan play using cosine similarity\n",
    "    for senecan_play, senecan_sentences in play_sentences.items():\n",
    "        similar_lines = compare_sentences(disputed_sentences, senecan_sentences)\n",
    "\n",
    "        # save results to CSV for closer examination\n",
    "        clean_senecan_play = os.path.splitext(os.path.basename(senecan_play))[0]\n",
    "        csv_filename = os.path.join(results_directory, f'similarity_{disputed_text_name}_vs_{clean_senecan_play}_res.csv')\n",
    "        with open(csv_filename, 'w', encoding='utf-8', newline='') as csvfile:\n",
    "            csv_writer = csv.writer(csvfile)\n",
    "            csv_writer.writerow(['Disputed Sentence Index', 'Senecan Sentence Index', 'Similarity', 'Disputed Sentence', 'Senecan Sentence'])\n",
    "            for i, j, similarity, disputed_sentence, senecan_sentence in similar_lines:\n",
    "                csv_writer.writerow([i, j, similarity, disputed_sentence, senecan_sentence])\n",
    "\n",
    "        print(f\"Examples for {disputed_text_name} vs {clean_senecan_play}:\")\n",
    "        for i, j, similarity, disputed_sentence, senecan_sentence in similar_lines:\n",
    "            print(f\"Similarity: {similarity:.4f}\")\n",
    "            print(f\"Disputed Sentence: {disputed_sentence}\")\n",
    "            print(f\"Senecan Sentence: {senecan_sentence}\")\n",
    "\n",
    "        # count and store the number of removed sentences\n",
    "        removed_sentence_count[(disputed_text_name, clean_senecan_play)] = len(similar_lines)\n",
    "\n",
    "        # remove similar sentences from disputed text\n",
    "        for i, _, _, _, _ in similar_lines:\n",
    "            disputed_sentences[i] = \"\"  # Replace similar sentences with an empty string\n",
    "\n",
    "    # calculate and print the percentage of removed sentences\n",
    "    total_sentences = len(disputed_sentences)\n",
    "    removed_sentences = sum(removed_sentence_count.values())\n",
    "    percentage_removed = (removed_sentences / total_sentences) * 100\n",
    "    print(f\"Total Sentences in {disputed_text_name}: {total_sentences}\")\n",
    "    print(f\"Removed Sentences: {removed_sentences}\")\n",
    "    print(f\"Percentage Removed: {percentage_removed:.2f}%\")\n",
    "\n",
    "    # write modified disputed text to the directory that has the two disputed plays with the lines removed\n",
    "    modified_disputed_path = os.path.join('../../corpora/corpus_imposters_cento/', f'{disputed_text_name}.txt')\n",
    "    with open(modified_disputed_path.lower().replace(\" \", \"_\"), 'w', encoding='utf-8') as modified_file:\n",
    "        modified_file.write('\\n'.join([sentence for sentence in disputed_sentences if sentence]))\n",
    "\n",
    "    print(f\"Modified disputed text saved to: {modified_disputed_path}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}