biblical_intertextuality_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Biblical intertextuality\n",
    "This jupyter notebook serves as a descriptive tool, explaining functions and process as are in [biblical_intertextuality_package.py](https://github.com/DigilabNLCR/BibleCitations/blob/main/biblical_intertextuality_package.py)\n",
    "\n",
    "- The process here describes the process of initial search, the evaluation of these preliminary results is describes in [evaluation.ipynb](https://github.com/DigilabNLCR/BibleCitations/blob/main/evaluation.ipynb)\n",
    "\n",
    "- version this notebook describes: v 1.0.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing packages and setting paths\n",
    "- Due to the large amount of data we search for and search in, I have used structure of ```from package import function (as fce)``` because the dot operations take a bit longer. Within the data we explore these additins add up to hours of search time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import joblib\n",
    "\n",
    "from os import listdir as os_listdir\n",
    "from os.path import isdir as os_path_isdir\n",
    "from os.path import exists as os_exists\n",
    "from os import remove as os_remove\n",
    "from shutil import copyfile as shutil_copyfile\n",
    "from json import load as json_load\n",
    "from re import sub as re_sub\n",
    "from re import split as re_split\n",
    "from os.path import join as join_path\n",
    "from unidecode import unidecode\n",
    "from time import time\n",
    "from Levenshtein import distance\n",
    "from gensim import corpora\n",
    "from collections import defaultdict\n",
    "from nltk import word_tokenize, sent_tokenize\n",
    "from math import ceil, isnan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" DEFINING_PATHS------------------------------------------------------------------------------------------------- \"\"\"\n",
    "ROOT_PATH = os.getcwd()\n",
    "\n",
    "BIBLES_PATH = join_path(ROOT_PATH, 'Bible_files')\n",
    "DATASETS_PATH = join_path(ROOT_PATH, 'datasets')\n",
    "DICTS_PATH = join_path(ROOT_PATH, 'dictionaries')\n",
    "CORPUS_PATH = join_path(ROOT_PATH, 'corpuses')\n",
    "RESULTS_PATH = join_path(ROOT_PATH, 'results')\n",
    "ALL_JSONS_PATH = join_path(ROOT_PATH, 'query_jsons')\n",
    "\n",
    "JOURNAL_FULLDATA_PATH = join_path(ROOT_PATH, 'journals_fulldata.joblib')\n",
    "\n",
    "BATCHES_FILE_PATH = join_path(ROOT_PATH, 'batches.csv')\n",
    "BATCH_RESULTS_FILE_PATH = join_path(RESULTS_PATH, 'batch_results.csv')\n",
    "\n",
    "STOP_WORDS_PATH = join_path(ROOT_PATH, 'stop_words.txt')\n",
    "STOP_SUBVERSES_PATH = join_path(ROOT_PATH, 'stop_subverses_21.txt')\n",
    "EXCLUSIVES_PATH = join_path(ROOT_PATH, 'exclusives.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the biblical dataset\n",
    "The following section include functions and classes that are used to convert the Bible files into a dataset that is then used for citations detection.\n",
    "\n",
    "### Summary\n",
    "- Bible files are stored in 'Bible_files' directory. Each TXT file is structured as a python dictionary file and includes one Biblical book in one translation (e.g., file bible_BKR_Gn.txt includes book of Genesis in Bible Kralická translation).\n",
    "- Translations available in this repository are summarized in [README](https://github.com/DigilabNLCR/BibleCitations/blob/main/README.md)\n",
    "- Each verse is split into subverses (because verses are rarely cited in their entirety)\n",
    "- Each subverse is split into n-grams (by default of length up to 4 characters). This partially eliminates the OCR problems of the journals dataset. (The search process itself is described below). [Stop-words](https://github.com/DigilabNLCR/BibleCitations/blob/main/stop_words.txt) are skipped in the process.\n",
    "- These n-grammed subverses are then vectorized (using [gensim](https://radimrehurek.com/gensim/) package), hand in hand creating a dictionary of n-grams.\n",
    "- Several files that are then used for the search are created within the process: dataset (contains all subverses assigned to verse IDs), corpus (contains vectorized subverses) and dictionary (contains n-gram IDs assigner to n-grams)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def word_tokenize_no_punctuation(text):\n",
    "    \"\"\" This function serves to return tokenized text to words and removes punctuation form it. To count only words for the split_verse function. \"\"\"\n",
    "    punctuation = '\\!|\\\"|\\„|\\#|\\$|\\%|\\&|\\'|\\(|\\)|\\*|\\+|\\-|\\–|\\.|\\/|\\:|\\;|\\<|\\=|\\>|\\?|\\[|\\\\|\\]|\\^|\\_|\\`|\\{|\\}|\\~|\\...|\\>>|\\<<|\\»|\\«|\\||\\,|\\’|\\‘'\n",
    "    text = re_sub(punctuation, '', text)\n",
    "    tokenized_text = word_tokenize(text)\n",
    "\n",
    "    return tokenized_text\n",
    "\n",
    "\n",
    "def normalize_string(string_to_be_normalized:str) -> str:\n",
    "    \"\"\" This function normalizes a string. It removes punctuation and diacritics. \"\"\"\n",
    "    string_to_be_normalized = unidecode(string_to_be_normalized)\n",
    "    punctuation = '\\!|\\\"|\\„|\\#|\\$|\\%|\\&|\\'|\\(|\\)|\\*|\\+|\\-|\\–|\\.|\\/|\\:|\\;|\\<|\\=|\\>|\\?|\\[|\\\\|\\]|\\^|\\_|\\`|\\{|\\}|\\~|\\...|\\>>|\\<<|\\»|\\«|\\||\\,|\\’|\\‘'\n",
    "    string_to_be_normalized = re_sub(punctuation, '', string_to_be_normalized)\n",
    "\n",
    "    return string_to_be_normalized.lower().strip()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" Define stop words in the file stop_words.txt \"\"\"\n",
    "with open(STOP_WORDS_PATH, 'r', encoding='utf-8') as sw_f:\n",
    "    stop_words = sw_f.read()\n",
    "    stop_words = unidecode(stop_words)\n",
    "    stop_words = stop_words.replace('\\n', ' ')\n",
    "    stop_words = stop_words.split(', ')\n",
    "    stop_words.append(',')\n",
    "\n",
    "\"\"\" Define stop subverses in the file stop_subverses_21.txt (for subverses of tolerance length 21 characters).\n",
    "Should you wish different subverse lenght tolerance, you have to change the stop verses, too. \n",
    "And rename path to it (above). \"\"\"\n",
    "with open(STOP_SUBVERSES_PATH, 'r', encoding='utf-8') as ssv_f:\n",
    "    stop_subverses = ssv_f.read()\n",
    "    stop_subverses = stop_subverses.split('\\n')\n",
    "\n",
    "\"\"\" List of all translations. \"\"\"\n",
    "# all_translations = ['BKR', 'BSV', 'HEJCL', 'SYK', 'ZP']\n",
    "# NOTE: translation Bible svatováclavská is unfortunately not public. See README\n",
    "all_translations = ['BKR', 'HEJCL', 'SYK', 'ZP']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split verse\n",
    "Following section include functions that ensure the spliting of verse into smaller sections - \"subverses\". By the nature of the simple approach, the verse split is not very smart and it is one of the parts that await further improvement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_text_by_delimiters(input_text: str):\n",
    "    \"\"\" This function splits input text (sentence) by following delimiters: ',', ';', ':'. Used in split_verse(). \"\"\"\n",
    "    delimiters = ',|;|:'\n",
    "    parts = re_split(delimiters, input_text)\n",
    "    parts = [item.strip() for item in parts]\n",
    "\n",
    "    return parts\n",
    "\n",
    "\n",
    "def join_short_part(list_of_parts: list, current_position: int, tole_len: int):\n",
    "    \"\"\" This function connects shorter passages to its neighbouring parts.\n",
    "    Preference is set to connect with following parts, then to previous. Used in split_verse(). \"\"\"\n",
    "    output = list_of_parts[current_position]\n",
    "\n",
    "    if current_position == -1:\n",
    "        # if the part is at input at the end (used in join_short_sents()), we have to proceed differently:\n",
    "        minus_pos = 1\n",
    "        while len(output) < tole_len:\n",
    "            try:\n",
    "                output = f'{list_of_parts[current_position - minus_pos]} {output}'\n",
    "                minus_pos += 1\n",
    "            except IndexError:\n",
    "                # This should not happen, but if there is no next nor previous position long enough, return joined original list\n",
    "                return ' '.join(list_of_parts)\n",
    "\n",
    "    plus_pos = 1\n",
    "    minus_pos = 1\n",
    "    while len(output) < tole_len:\n",
    "        try:\n",
    "            output += f' {list_of_parts[current_position + plus_pos]}'\n",
    "            plus_pos += 1\n",
    "        except IndexError:\n",
    "            # If there is no next part, connect the previous one.\n",
    "            try:\n",
    "                output = f'{list_of_parts[current_position - minus_pos]} {output}'\n",
    "                minus_pos += 1\n",
    "            except IndexError:\n",
    "                # This should not happen, but if there is no next nor previous position long enough, return joined original list\n",
    "                return ' '.join(list_of_parts)\n",
    "\n",
    "    return output\n",
    "\n",
    "\n",
    "def join_short_sents(list_of_sents: list, current_position: int, tole_len: int):\n",
    "    \"\"\" This function connects short sentences to passages of neigbouring sentences.\n",
    "    Preference is set for the following sentence. Used in split_verse(). \"\"\"\n",
    "    output = list_of_sents[current_position]\n",
    "\n",
    "    plus_sents = 1\n",
    "    minus_sents = 1\n",
    "    while len(output) < tole_len:\n",
    "        try:\n",
    "            parts_of_the_following = split_text_by_delimiters(list_of_sents[current_position + plus_sents])\n",
    "            all_parts = [output]\n",
    "            all_parts.extend(parts_of_the_following)\n",
    "            output = join_short_part(all_parts, 0, tole_len=tole_len)\n",
    "            plus_sents += 1\n",
    "        except IndexError:\n",
    "            try:\n",
    "                parts_of_the_previous = split_text_by_delimiters(list_of_sents[current_position - minus_sents])\n",
    "                parts_of_the_previous.append(output)\n",
    "                output = join_short_part(parts_of_the_previous, -1, tole_len=tole_len)\n",
    "                minus_sents += 1\n",
    "            except IndexError:\n",
    "                # If there is no more previous or following sentence long enough, return False.\n",
    "                return False\n",
    "\n",
    "    return output\n",
    "\n",
    "\n",
    "def split_long_sent(input_sent_text: str, tole_len: int):\n",
    "    \"\"\" This function splits a long string into two or three parts with a bit of overlap. Used in split_verse(). \"\"\"\n",
    "    subverses = []\n",
    "    if len(input_sent_text) <= 3.5 * tole_len:\n",
    "        words_in_sub = word_tokenize_no_punctuation(input_sent_text)\n",
    "        sub_w_len = ceil(len(words_in_sub) / 2) + 1\n",
    "        subverses.append(' '.join(words_in_sub[0:sub_w_len]))\n",
    "        subverses.append(' '.join(words_in_sub[-sub_w_len:]))\n",
    "        return subverses\n",
    "    else:\n",
    "        words_in_sub = word_tokenize_no_punctuation(input_sent_text)\n",
    "        sub_w_len = ceil(len(words_in_sub) / 3) + 1\n",
    "        subverses.append(' '.join(words_in_sub[0:sub_w_len]))\n",
    "        subverses.append(\n",
    "            ' '.join(words_in_sub[ceil(len(words_in_sub) / 3 - 1):ceil(len(words_in_sub) / 3 + sub_w_len)]))\n",
    "        subverses.append(' '.join(words_in_sub[-sub_w_len:]))\n",
    "        return subverses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def split_verse(input_text:str, tole_len=21, return_shorts=True, short_limit=9) -> list:\n",
    "    \"\"\" This function ensures verse splitting into smaller subverses.\n",
    "    :param input_text: text of verse that is to be split.\n",
    "    :param tole_len: minimal length of subverse in characters.\n",
    "    \"\"\"\n",
    "    if len(input_text) < tole_len:\n",
    "        if return_shorts:\n",
    "            if len(input_text) >= short_limit:\n",
    "                return [input_text]\n",
    "            else:\n",
    "                return []\n",
    "        else:\n",
    "            return []\n",
    "    \n",
    "    else:\n",
    "        subverses = []    \n",
    "        sentences = sent_tokenize(input_text)\n",
    "\n",
    "        for s, sent in enumerate(sentences):\n",
    "            sent_len = len(sent)\n",
    "            if sent_len >= tole_len and sent_len <= 2.5*tole_len:\n",
    "                # If the sentence is in tolerance but not too long, append it\n",
    "                subverses.append(sent)\n",
    "            elif sent_len > 2.5*tole_len:\n",
    "                sent_parts = split_text_by_delimiters(sent)\n",
    "                if len(sent_parts) == 1:\n",
    "                    # If there is only one part (there are no delimiters), split it to two or three parts with overlap.\n",
    "                    subverses.extend(split_long_sent(sent, tole_len=tole_len))\n",
    "                else:\n",
    "                    for sp, sent_part in enumerate(sent_parts):\n",
    "                        subverses.append(join_short_part(sent_parts, sp, tole_len=tole_len))\n",
    "            else:\n",
    "                # If the sentence is too short, we need to append it to neighbour parts.\n",
    "                subverses.append(join_short_sents(sentences, s, tole_len=tole_len))\n",
    "        \n",
    "    out_subverses = []\n",
    "    for sub in subverses:\n",
    "        # strip the blank spaces in subverse strings\n",
    "        out_subverses.append(sub.strip())\n",
    "    \n",
    "    # return the subverses, set() to remove any duplicates.\n",
    "    return list(set(out_subverses))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function and classes that work with Bible files and datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_book_id(verse_id:str) -> str:\n",
    "    \"\"\" Gets book's ID from a verse_id (e.g. \"Gn 1:1\"). \"\"\"\n",
    "    book_id = verse_id.split(' ')[0]\n",
    "    return book_id\n",
    "\n",
    "\n",
    "def get_verse_text(translation:str, verse_id:str, print_exceptions=True) -> str:\n",
    "    \"\"\" This function gets text selected verse.\n",
    "    :param translation: ID of Bible translation (select from ['BKR', 'BSV', 'CEP', 'COL', 'JB', 'KLP', 'SYK'])\n",
    "    :param verse_id: ID of verse, e.g. \"Gn 1:1\"\n",
    "    \"\"\"\n",
    "    book = get_book_id(verse_id)\n",
    "    bible_filename = f'bible_{translation}_{book}.txt'\n",
    "    try:\n",
    "        with open(join_path(BIBLES_PATH, bible_filename), 'r', encoding='utf-8') as book_file:\n",
    "            dict_of_verses = eval(book_file.read())\n",
    "            verse_text = dict_of_verses[verse_id]\n",
    "    except KeyError:\n",
    "        if print_exceptions:\n",
    "            print(f'Verse does not exist in {translation} translation.')\n",
    "        return ''\n",
    "    except FileNotFoundError:\n",
    "        if print_exceptions:\n",
    "            print(f'Book {book} does not exist in {translation} translation.')\n",
    "        return ''\n",
    "\n",
    "    return verse_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class bibleDataset:\n",
    "    def __init__(self, data, target):\n",
    "        # NOTE: data = texts of a subverse\n",
    "        self.data = data\n",
    "        # NOTE: target = verse ID\n",
    "        self.target = target\n",
    "\n",
    "        self.target_names = sorted(set(target))\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def check_valid(self):\n",
    "        return (len(self.data) == len(self.target))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_dataset(dataset, dataset_name='fullBibleDataset'):\n",
    "    \"\"\" Saving dataset has default name, because there is supposedly only one version of it. \"\"\"\n",
    "    joblib.dump(dataset, join_path(DATASETS_PATH, f'{dataset_name}.joblib'))\n",
    "\n",
    "\n",
    "def load_dataset(dataset_name='fullBibleDataset'):\n",
    "    \"\"\" Loading dataset has default name, because there is supposedly only one version of it. \"\"\"\n",
    "    dataset = joblib.load(join_path(DATASETS_PATH, f'{dataset_name}.joblib'))\n",
    "    return dataset\n",
    "\n",
    "\n",
    "def save_dictionary(dictionary, dictionary_name:str):\n",
    "    joblib.dump(dictionary, join_path(DICTS_PATH, f'{dictionary_name}.joblib'))\n",
    "\n",
    "\n",
    "def load_dictionary(dictionary_name:str):\n",
    "    dictionary = joblib.load(join_path(DICTS_PATH, f'{dictionary_name}.joblib'))\n",
    "    return dictionary\n",
    "\n",
    "\n",
    "def save_corpus(bongrammed_corpus, corpus_name:str):\n",
    "    corpora.MmCorpus.serialize(join_path(CORPUS_PATH, f'{corpus_name}.mm'), bongrammed_corpus)\n",
    "\n",
    "\n",
    "def load_corpus(corpus_name:str):\n",
    "    corpus = corpora.MmCorpus(join_path(CORPUS_PATH, f'{corpus_name}.mm'))\n",
    "    return corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creation of the bibleDataset object\n",
    "\n",
    "- bibleDataset is prepared by using all available translations. Each verse is split by the split_verse function. Stop-subverses may be applied here, but I suggest not using them for the creation of the dataset, but only filtering lone stop-subverses during the evaluation process. (Because even the stop-subverses may be part of the broader citation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bible_to_dataset(save=True, ignore_stop_subs=False, dataset_prefix='fullBible', return_shorts=True,  tole_len=21, short_limit=9):\n",
    "    \"\"\" This function prepares dataset into bibleDataset class.\n",
    "\n",
    "    :param ignore_stop_subs: if True, the stop subverses defined in stop_subverses_21 are ignored.\n",
    "    :param save: set False if you do not want to save the dataset.\n",
    "    :param dataset_prefix: prefix for the filename of the saved dataset (prefixDataset.joblib).\n",
    "    :param return_shorts: should verses shorter than tole_len characters be returned?\n",
    "    :param tole_len: minimal length for verse split\n",
    "    :param short_limit: minimal characters needed for the verse to be included in the dataset.\n",
    "    \"\"\"\n",
    "    data = []\n",
    "    targets = []\n",
    "\n",
    "    bible_files = os_listdir(BIBLES_PATH)\n",
    "\n",
    "    for bible_file in bible_files:\n",
    "        with open(join_path(BIBLES_PATH, bible_file), 'r', encoding='utf-8') as bible_f:\n",
    "            bible_data = bible_f.read()\n",
    "            try:\n",
    "                verses_dict = eval(bible_data)\n",
    "            except:\n",
    "                print('There is some error in:', bible_file)\n",
    "            for verse_id in verses_dict:\n",
    "                subverses = split_verse(verses_dict[verse_id], return_shorts=return_shorts, tole_len=tole_len, short_limit=short_limit)\n",
    "                for sub in subverses:\n",
    "                    if ignore_stop_subs:\n",
    "                        if sub in stop_subverses:\n",
    "                            continue\n",
    "                        else:\n",
    "                            data.append(sub)\n",
    "                            targets.append(verse_id)\n",
    "                    else:\n",
    "                        data.append(sub)\n",
    "                        targets.append(verse_id)\n",
    "\n",
    "    bible_dataset = bibleDataset(data, targets)\n",
    "\n",
    "    if save:\n",
    "        save_dataset(bible_dataset, dataset_name=f'{dataset_prefix}Dataset')\n",
    "\n",
    "    return bible_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N-gramming data\n",
    "The following functions ensure the splitting of text into n-grams (e.g., 'Hello World' becomes ['hell', 'ello', 'worl', 'orld'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ngramming(tokens: list, ngram_size=4, use_stop_words=True) -> list:\n",
    "    \"\"\" Core of nrgamming functions. Used in text_to_ngrams(). \"\"\"\n",
    "    ngrammed_text = []\n",
    "    if use_stop_words:\n",
    "        for token in tokens:\n",
    "            if token not in stop_words:\n",
    "                if len(token) > ngram_size:\n",
    "                    for i in range(len(token) - (ngram_size - 1)):\n",
    "                        ngram = token[i:i + ngram_size]\n",
    "                        if ngram not in stop_words:\n",
    "                            ngrammed_text.append(ngram)\n",
    "                        else:\n",
    "                            continue\n",
    "                else:\n",
    "                    ngrammed_text.append(token)\n",
    "            else:\n",
    "                continue\n",
    "\n",
    "    else:\n",
    "        for token in tokens:\n",
    "            if len(token) > ngram_size:\n",
    "                for i in range(len(token) - (ngram_size - 1)):\n",
    "                    ngram = token[i:i + ngram_size]\n",
    "                    ngrammed_text.append(ngram)\n",
    "            else:\n",
    "                ngrammed_text.append(token)\n",
    "\n",
    "    return ngrammed_text\n",
    "\n",
    "\n",
    "def text_to_ngrams(input_text: str, ngram_size=4, use_stop_words=True) -> list:\n",
    "    \"\"\" This function ngrams input string. Used in tokenize_to_ngrams(). \"\"\"\n",
    "    tokens = word_tokenize_no_punctuation(normalize_string(input_text))\n",
    "    ngrammed_text = ngramming(tokens, ngram_size=ngram_size, use_stop_words=use_stop_words)\n",
    "\n",
    "    return ngrammed_text\n",
    "\n",
    "\n",
    "def tokenize_to_ngrams(input_documents: list, ngram_size=4, use_stop_words=True) -> list:\n",
    "    \"\"\" This function serves to ngram documents for bow (or rather \"bag of n-grams\") creation. \"\"\"\n",
    "    ngrammed_docs = []\n",
    "\n",
    "    for doc in input_documents:\n",
    "        ngrams = text_to_ngrams(doc, ngram_size=ngram_size, use_stop_words=use_stop_words)\n",
    "        ngrammed_docs.append(ngrams)\n",
    "\n",
    "    return ngrammed_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating corpus and dictionary out of dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_corpus(dataset: bibleDataset, cut_off_value=0, ngram_size=4):\n",
    "    \"\"\"\n",
    "    This function creates gensim dictionary from input dataset (class bibleDataset).\n",
    "\n",
    "    :param dataset: input dataset (class bibleDataset)\n",
    "    :param cut_off_value: int; sets minimal token appearance\n",
    "    :param ngram_size: int; size of ngrams (in characters)\n",
    "    :return dictionary, processed_corpus\n",
    "    \"\"\"\n",
    "    ngrammed_verses = tokenize_to_ngrams(dataset.data, ngram_size=ngram_size)\n",
    "\n",
    "    frequency = defaultdict(int)\n",
    "    for verse in ngrammed_verses:\n",
    "        for ngram in verse:\n",
    "            frequency[ngram] += 1\n",
    "\n",
    "    processed_corpus = [[token for token in text if frequency[token] > cut_off_value] for text in ngrammed_verses]\n",
    "\n",
    "    dictionary = corpora.Dictionary(processed_corpus)\n",
    "\n",
    "    return dictionary, processed_corpus\n",
    "\n",
    "\n",
    "def create_corpus(dictionary, processed_corpus):\n",
    "    \"\"\" This function creates corpus from input dictionary and processed corpus. \"\"\"\n",
    "    return [dictionary.doc2bow(text) for text in processed_corpus]\n",
    "\n",
    "\n",
    "def transfer_corpus_to_simple_token_vectors(corpus):\n",
    "    \"\"\" This function extracts from corpus simple vectors consisting of only token IDs\"\"\"\n",
    "    return [[token for token, token_count in subverse] for subverse in corpus]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating and loading datasets, corpuses and dictionaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_necessary_objects(ngram_size=4, skip_dataset=True, dataset=None, save_objects=True, objects_name='fullBible'):\n",
    "    \"\"\"\n",
    "    This function creates all necessary objects for the search. Run this function if you are starting the process or if\n",
    "    you have changed the dataset or functions that create it, otherwise it is not necessary.\n",
    "\n",
    "    :param ngram_size: size of ngrams (in characters) to which everything is parsed; According to it,\n",
    "        other objects for search are loaded.\n",
    "    :param skip_dataset: if the dataset exists, it is loaded instead of created.\n",
    "    :param dataset: dataset can be also loaded externaly, so it does not have to be loaded for every iteration.\n",
    "    :param save_objects: if True, objects are saved, if False, objects are only returned.\n",
    "    \"\"\"\n",
    "\n",
    "    if dataset:\n",
    "        bible_dataset = dataset\n",
    "    else:\n",
    "        if skip_dataset:\n",
    "            print('Dataset already exists --> loaded.')\n",
    "            bible_dataset = load_dataset(f'{objects_name}Dataset')\n",
    "        else:\n",
    "            start_ = time()\n",
    "            print('Creating bible dataset...')\n",
    "            bible_dataset = bible_to_dataset()\n",
    "            save_dataset(bible_dataset, f'{objects_name}Dataset')\n",
    "            end_ = time()\n",
    "            print(f'Dataset has been created in {round((end_-start_)/60, 2)} minutes. Saved as {objects_name}Dataset.joblib')\n",
    "\n",
    "    start_ = time()\n",
    "    print('Processing corpus and creating dictionary...')\n",
    "    dictionary, processed_corpus = process_corpus(bible_dataset, ngram_size=ngram_size)\n",
    "    if save_objects:\n",
    "        save_dictionary(dictionary, f'n{ngram_size}_{objects_name}Dict')\n",
    "    end_ = time()\n",
    "    print(f'Dictionary has been created in {round((end_-start_), 2)} seconds. Saved as n{ngram_size}_{objects_name}Dict.joblib')\n",
    "\n",
    "    start_ = time()\n",
    "    print('Creating corpus...')\n",
    "    corpus = create_corpus(dictionary, processed_corpus)\n",
    "    if save_objects:\n",
    "        save_corpus(corpus, f'n{ngram_size}_{objects_name}Corpus')\n",
    "    end_ = time()\n",
    "    print(f'Corpus has been created in {round((end_-start_), 2)} seconds. Saved as n{ngram_size}_{objects_name}Corpus.mm')\n",
    "\n",
    "    return corpus, dictionary\n",
    "\n",
    "\n",
    "def load_necessary_objects(ngram_size=4, objects_name='fullBible'):\n",
    "    \"\"\"\n",
    "    This function loads all necessary objects for the search.\n",
    "\n",
    "    :param ngram_size: int; size of ngrams to which everything is parsed; According to it,\n",
    "        other objects for search are loaded.\n",
    "    \"\"\"\n",
    "    dataset = load_dataset(f'{objects_name}Dataset')\n",
    "    corpus = load_corpus(f'n{ngram_size}_{objects_name}Corpus')\n",
    "    dictionary = load_dictionary(f'n{ngram_size}_{objects_name}Dict')\n",
    "\n",
    "    subverses = transfer_corpus_to_simple_token_vectors(corpus)\n",
    "\n",
    "    return dataset, corpus, dictionary, subverses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Query documents\n",
    "The following section includes functions that work with query documents. The query documents in our case are JSON files of individual pages of various journals. The files include both the text of these pages (with quite low-quality OCR) and metadata (journal name, date, uuid, and page number).\n",
    "\n",
    "- These JSON files are for the sake of faster search transformed into a jobib file that contains a python dictionary object of all the files at once. The size of this file was in our case around 2 GB.\n",
    "- These function ensure that the query strings are n-grammed and vectorized in the same way as the used bible objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def query_to_bongrams(query_doc:str, dictionary, ngram_size=4):\n",
    "    \"\"\" Transfers text into bag of ngrams vector according to Bible dictionary (defined above!)\"\"\"\n",
    "    query_ngrams = text_to_ngrams(query_doc, ngram_size=ngram_size)\n",
    "    query_bon = dictionary.doc2bow(query_ngrams)\n",
    "\n",
    "    return query_bon\n",
    "\n",
    "\n",
    "def load_json_data(json_path: str) -> dict:\n",
    "    \"\"\" This function loads JSON data so that we do not have to write the open statement over and over again. \"\"\"\n",
    "    with open(json_path, encoding='utf-8') as json_file:\n",
    "        data = json_load(json_file)\n",
    "\n",
    "    return data\n",
    "\n",
    "\n",
    "def split_query(input_text:str, window_len=4, overlap=1):\n",
    "    \"\"\"\n",
    "    This function is used to split query document (text) into smaller parts for comparison with Biblical verses.\n",
    "    :param window_len: int; how many sentences are to be connected\n",
    "    :param overlap: int; how many sentences are set to overlap in the mooving window; must be lower than window_len\n",
    "    \"\"\"\n",
    "    query_sentences = sent_tokenize(input_text)\n",
    "\n",
    "    query_docs = []\n",
    "\n",
    "    constant = window_len-overlap\n",
    "    for i in range(int(ceil(len(query_sentences)/constant))):\n",
    "        query_part = query_sentences[i*constant:i*constant+window_len]\n",
    "        query_docs.append(' '.join(query_part))\n",
    "\n",
    "    return query_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search function and classes\n",
    "The following section includes the core of the whole script - functions and objects that facilitate the search of biblical citations across the query documents.\n",
    "\n",
    "- The search itself is run in smaller batches containing by default up to 40 query documents. After finishing the search of each batch, the results are saved. This ensures that in case of computer collapse, the data are not lost entirely, but only in a small part and at the same time, we are not constantly saving the results that would lead to a considerable slowdown.\n",
    "- The search is not executed on the full query text but its on smaller parts (split by the above split_query function)\n",
    "- The search process has two basic steps:\n",
    "    - comparison of vectors - does the query part include any of the subverse vectors? (with 70 % match needed)\n",
    "    - fuzzy string comparison - is any of the matched subverses present as string in the query part string? Due to typos and bad OCR quality, this is done using [Levenhtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) (how much characters we must change to get the same string) with a needed hit of 85 % of characters.\n",
    "- If these criteria are met, the result is saved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batches functions\n",
    "Following section includes function that create the batch file and work with it (including cganging logs on which batches has already been run and collecting average runtimes on documents within one batch)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def collect_query_jsons():\n",
    "    \"\"\" This function list all json files as stored in folder \"query_jsons\".\"\"\"\n",
    "    folders_in_extracted_query_jsons = os_listdir(ALL_JSONS_PATH)\n",
    "\n",
    "    journals_et_folders = {}\n",
    "\n",
    "    for journal_folder in folders_in_extracted_query_jsons:\n",
    "        journal_folder_path = join_path(ALL_JSONS_PATH, journal_folder)\n",
    "        if not os_path_isdir(journal_folder_path):\n",
    "            continue\n",
    "\n",
    "        jsons_in_journal = os_listdir(journal_folder_path)\n",
    "        journals_et_folders[journal_folder] = jsons_in_journal\n",
    "\n",
    "    return journals_et_folders\n",
    "\n",
    "\n",
    "batches_columns = ['journal', 'json_file', 'batch_id', 'run', 'runtime']\n",
    "\n",
    "\n",
    "def clear_batches_csv(clear: bool):\n",
    "    if clear:\n",
    "        # Delete old batches.csv\n",
    "        os_remove(BATCHES_FILE_PATH)\n",
    "        # create clear batches.csv\n",
    "        empty_df = pd.DataFrame(columns=batches_columns)\n",
    "        pd.DataFrame.to_csv(empty_df, BATCHES_FILE_PATH)\n",
    "\n",
    "\n",
    "def create_batches_csv():\n",
    "    \"\"\" create batches.csv if not exists \"\"\"\n",
    "    if not os_exists(BATCHES_FILE_PATH):\n",
    "        empty_df = pd.DataFrame(columns=batches_columns)\n",
    "        pd.DataFrame.to_csv(empty_df, BATCHES_FILE_PATH)\n",
    "\n",
    "\n",
    "def get_last_batch_id():\n",
    "    \"\"\" This function is used in 'run_biblical_intertextuality.py' to detect the last batch in order to set the right range. \"\"\"\n",
    "    batches_df = pd.read_csv(BATCHES_FILE_PATH)\n",
    "    last_assigned_batch = batches_df['batch_id'].max()\n",
    "\n",
    "    return last_assigned_batch\n",
    "\n",
    "\n",
    "def update_batches_csv(clear=False, max_batch_size=40):\n",
    "    \"\"\" This function creates or updates batches file. Use this e.g. if you have added some new journals to dir. extracted_query_jsons\n",
    "\n",
    "    :param clear: bool; set true if you want to reastart the batches.csv (delete information on runs).\n",
    "    \"\"\"\n",
    "    # create batches.csv if not exists\n",
    "    create_batches_csv()\n",
    "\n",
    "    # clear batches.csv if set\n",
    "    clear_batches_csv(clear)\n",
    "\n",
    "    # create dictionary of all json files for each journal to be ignored while updating\n",
    "    all_jsons_by_journal = collect_query_jsons()\n",
    "\n",
    "    # load batches.csv as pandas dataframe\n",
    "    batches_df = pd.read_csv(BATCHES_FILE_PATH)\n",
    "\n",
    "    existing_jsons = batches_df['json_file'].to_list()\n",
    "    last_assigned_batch = batches_df['batch_id'].max()\n",
    "    if isnan(last_assigned_batch):\n",
    "        last_assigned_batch = -1\n",
    "\n",
    "    # creat dictionary dataframe by individual uuids\n",
    "    batch_id = last_assigned_batch + 1\n",
    "    data = defaultdict(list)\n",
    "    for journal in all_jsons_by_journal:\n",
    "        print(f\"working on {journal}\")\n",
    "        batch_size = 0\n",
    "        for json_f in all_jsons_by_journal[journal]:\n",
    "            if batch_size == max_batch_size:\n",
    "                batch_size = 0\n",
    "                batch_id += 1\n",
    "\n",
    "            if json_f in existing_jsons:\n",
    "                continue\n",
    "\n",
    "            data['journal'].append(journal)\n",
    "            data['json_file'].append(json_f)\n",
    "            data['batch_id'].append(batch_id)\n",
    "            data['run'].append(False)\n",
    "            data['runtime'].append(0.0)\n",
    "\n",
    "            batch_size += 1\n",
    "\n",
    "    # merge old and new dataframes\n",
    "    append_df = pd.DataFrame(data)\n",
    "    batches_df = pd.concat([batches_df, append_df])\n",
    "    batches_df.to_csv(BATCHES_FILE_PATH, index=False, columns=batches_columns)\n",
    "\n",
    "    print(f'The last batch id is: {batch_id}')\n",
    "\n",
    "\n",
    "def change_run_log(batch_id:int, avereage_pre_page:float):\n",
    "    \"\"\" Changes run log in batches.csv. \"\"\"\n",
    "    batches_df = pd.read_csv(BATCHES_FILE_PATH)\n",
    "    batches_df.loc[batches_df['batch_id'] == batch_id, \"run\"] = True\n",
    "    batches_df.loc[batches_df['batch_id'] == batch_id, \"runtime\"] = avereage_pre_page\n",
    "    batches_df.to_csv(BATCHES_FILE_PATH, index=False, columns=batches_columns)\n",
    "\n",
    "\n",
    "def get_query_files_from_batch(batch_id:int):\n",
    "    \"\"\" This fnction gets the list of query files assigned to a batch. \"\"\"\n",
    "    batches_df = pd.read_csv(BATCHES_FILE_PATH)\n",
    "    relevant_data = batches_df.loc[batches_df['batch_id'] == batch_id]\n",
    "    query_files_in_batch = relevant_data['json_file'].to_list()\n",
    "\n",
    "    return query_files_in_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Search functions and classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class bibleObject:\n",
    "    def __init__(self, dataset:bibleDataset, create_anew_other_necessary_objects=False, ngram_size=4, objects_prefix='fullBible'):\n",
    "        \"\"\"\n",
    "        :param dataset: prepared dataset object of bibleDataset class\n",
    "        :param create_anew_necessary_objects: Bool, set True if you want to create and save corpus and dictionary. These may be already created so the default is set to False.\n",
    "        :param ngram_size: (maximum) size of n-grams (in characters)\n",
    "        \"\"\"\n",
    "        if create_anew_other_necessary_objects:\n",
    "            print('Creating necessary objects for bibleObject...')\n",
    "            corpus, dictionary = create_necessary_objects(ngram_size=ngram_size)\n",
    "            subverse_vectors = transfer_corpus_to_simple_token_vectors(corpus)\n",
    "            # NOTE: corpus is then discarded, it is not used anymore, it can be romoved from memory\n",
    "            del corpus\n",
    "            \n",
    "        else:\n",
    "            print('Loading necessary objects for bibleObject...')\n",
    "            corpus = load_corpus(f'n{ngram_size}_{objects_prefix}Corpus')\n",
    "            dictionary = load_dictionary(f'n{ngram_size}_{objects_prefix}Dict')\n",
    "            subverse_vectors = transfer_corpus_to_simple_token_vectors(corpus)\n",
    "            # NOTE: corpus is then discarded, it is not used anymore, it can be romoved from memory\n",
    "            del corpus\n",
    "\n",
    "        attributed_subverses = defaultdict(list)\n",
    "        subverse_lens = defaultdict(int)\n",
    "\n",
    "        for i, sub in enumerate(subverse_vectors):\n",
    "            subverse_lens[i] = len(sub)\n",
    "            for elem in sub:\n",
    "                attributed_subverses[elem].append(i)\n",
    "\n",
    "        self.attr_subs = attributed_subverses\n",
    "        self.sub_lens = subverse_lens\n",
    "\n",
    "        self.data = dataset.data\n",
    "        self.verse_id = dataset.target\n",
    "        self.dictionary = dictionary\n",
    "        self.subverse_vectors = subverse_vectors\n",
    "        self.dataset = dataset\n",
    "\n",
    "        self.verse_ids = sorted(set(dataset.target))\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_vector(query_string:str, attributed_subverses:dict, subverse_lens:dict, dictionary:corpora.Dictionary, tolerance=0.85, ngram_size=4):\n",
    "    \"\"\" \n",
    "    This function implements vector comparison based on preprepared dictionaries of token references to subverses. \n",
    "    \n",
    "    :param query_string:\n",
    "    :param attributed_subverses:\n",
    "    :param subverse_lens:\n",
    "    :param dictionary:\n",
    "    :param tolerance:\n",
    "    :param ngram_size:\n",
    "\n",
    "    \"\"\"\n",
    "    vector_q_bon = query_to_bongrams(query_string, dictionary, ngram_size=ngram_size)\n",
    "    tokens_in_query = [token for token, token_count in vector_q_bon]\n",
    "\n",
    "    subverse_scores = defaultdict(int)\n",
    "    # Get scores of subverses by every token.\n",
    "    for token in tokens_in_query:\n",
    "        for sft in attributed_subverses[token]:\n",
    "            subverse_scores[sft] += 1\n",
    "\n",
    "    # Select those subverses that have scored high enough.    \n",
    "    possible_subverses = []\n",
    "    for sub in subverse_scores:\n",
    "        if subverse_scores[sub] >= subverse_lens[sub]*tolerance:\n",
    "            possible_subverses.append(sub)\n",
    "\n",
    "    return possible_subverses\n",
    "\n",
    "\n",
    "def fuzzy_string_matching_for_implementation(subverse_string:str, query_string:str, tolerance=0.85):\n",
    "    \"\"\" \n",
    "    This function is for implementation of typo similarity detection applied to two strings. It returns bool value of match.\n",
    "\n",
    "    :param subverse_string: string of the biblical subverse we are searching for.\n",
    "    :param query_string: string in which we are searching for the seubverse_string.\n",
    "    :param tolerance: how large proportion of the subverse_string must be present in query_string to consider it a match.\n",
    "    \"\"\"\n",
    "    subverse_string = normalize_string(subverse_string)\n",
    "    subverse_len = len(subverse_string)\n",
    "\n",
    "    query_string = normalize_string(query_string)\n",
    "    query_len = len(query_string)\n",
    "\n",
    "    tolerance = subverse_len * (1-tolerance)\n",
    "\n",
    "    match = False\n",
    " \n",
    "    if subverse_len-tolerance > query_len:\n",
    "        # If subverse is longer than query string, it is not a match by default\n",
    "        return match\n",
    "    elif subverse_len-tolerance <= query_len <= subverse_len+tolerance:\n",
    "        # If subverse is more or les of the same length as query string, just compare them.\n",
    "        if distance(subverse_string, query_string) <= tolerance:\n",
    "            match = True\n",
    "    else:\n",
    "        # Oherwise, compare parts of the query string (always staring with word, so it is quicker; however, some mistakes may be made here.\n",
    "        # NOTE: change in split - skipping by words but len by characters...\n",
    "        char_len_sub = len(subverse_string)\n",
    "        word_len_subv = len(word_tokenize(subverse_string))\n",
    "        words_in_query_string = word_tokenize(query_string)\n",
    "        word_len_query_string = len(words_in_query_string)\n",
    "\n",
    "        for i, cycle in enumerate(range(word_len_subv, word_len_query_string+1)):\n",
    "            gram_str = ' '.join(words_in_query_string[i:])[:char_len_sub]\n",
    "            if distance(subverse_string, gram_str) <= tolerance:\n",
    "                match = True\n",
    "                return match\n",
    "            else:\n",
    "                continue\n",
    "    \n",
    "    return match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def search_for_bible_for_batches_implementation(bible_object:bibleObject, journals_fulldata:dict, batch_id:int, ngram_tolerance=0.7, edit_distance_tolerance=0.85, ngram_size=4, query_window_len=4, query_overlap=1):\n",
    "    \"\"\"    \n",
    "    This function is appropriated for implementation within search  by batches (in run_search_by_batch() function)\n",
    "    This function executes search for Bible quotations within all JSON documents assigned to selected batch using dictionary from 'journals_fulldata.joblib'.\n",
    "\n",
    "    :param bible_object: an input object of bibleObject class (created from bibleDataset).\n",
    "    :param ngram_tolerance: what portion of ngrams of Bible subverse must match to be considered as a match.\n",
    "    :param edit_distance_tolerance: what portion of characters of a subverse must match a sequence from a \"query document\".\n",
    "    :param ngram_size: size of ngrams (in characters) to which everything is parsed; According to it, other objects for search are loaded.\n",
    "    :param query_window_len: How many sentences are put together as smaller \"query documets\". This parameter can influence the speed of the process, depending on the nature of the document.\n",
    "    :param query_overlap: Overlap of the sentences among \"query documents\". Must be higher than query_window_len.\n",
    "    :return: Detected citations, time of search.\n",
    "    \"\"\"\n",
    "    function_start = time()\n",
    "\n",
    "    attributed_subverses = bible_object.attr_subs\n",
    "    subverse_lens = bible_object.sub_lens\n",
    "    dictionary = bible_object.dictionary\n",
    "    dataset = bible_object.dataset\n",
    "    \n",
    "    query_files = get_query_files_from_batch(batch_id)\n",
    "    num_of_query_files = len(query_files)\n",
    "\n",
    "    discovered_citations = defaultdict(list)\n",
    "    \n",
    "    print('Initiating search in query documents...')\n",
    "    for qi, query_file in enumerate(query_files):\n",
    "        query_time_start = time()\n",
    "        print(f'\\tBatch {batch_id} ... Analysing document ({qi+1}/{num_of_query_files}) {query_file}')\n",
    "        \n",
    "        full_query_text = journals_fulldata[query_file]['text']\n",
    "        query_documents = split_query(full_query_text, window_len=query_window_len, overlap=query_overlap)\n",
    "\n",
    "        for i, query_doc in enumerate(query_documents):\n",
    "            # First stage - compare vectors (token = n-gram of ngram_size)\n",
    "            results_by_ngrams = compare_vector(query_doc, attributed_subverses=attributed_subverses, subverse_lens=subverse_lens, dictionary=dictionary, tolerance=ngram_tolerance, ngram_size=ngram_size)\n",
    "\n",
    "            # Second stage - compare by fuzzy string matching\n",
    "            for subverse_id in results_by_ngrams:\n",
    "                try:\n",
    "                    subverse_text = dataset.data[subverse_id]\n",
    "                except IndexError:\n",
    "                    # TODO: tohle zčeknout a vyřadit!\n",
    "                    print('ERRROOROROROR', subverse_id)\n",
    "                match = fuzzy_string_matching_for_implementation(subverse_string=subverse_text, query_string=query_doc, tolerance=edit_distance_tolerance)\n",
    "                if match:\n",
    "                    discovered_citations[subverse_id].append((query_file, i))\n",
    "\n",
    "        query_time_end = time()\n",
    "        print(f'\\t\\tDocument analysed in {round((query_time_end-query_time_start), 2)} seconds.')\n",
    "\n",
    "    function_end = time()\n",
    "    average_per_page = round(((function_end-function_start)/num_of_query_files), 2)\n",
    "    \n",
    "    return discovered_citations, average_per_page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results functions and objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_columns_names = ['verse_id', 'query_file', 'index_query_part', 'batch_id', 'ngram_size', 'query_window_len', 'query_overlap', 'ngram_tolerance', 'edit_distance_tolerance']\n",
    "\n",
    "\n",
    "def create_batches_results_csv():\n",
    "    \"\"\" Create batch_results.csv if not exists. \"\"\"\n",
    "    if not os_exists(BATCH_RESULTS_FILE_PATH):\n",
    "        empty_df = pd.DataFrame(columns=results_columns_names)\n",
    "        pd.DataFrame.to_csv(empty_df, BATCH_RESULTS_FILE_PATH)\n",
    "\n",
    "\n",
    "def save_batch_results(results:dict, dataset:bibleDataset, batch_id:int, ngram_size=4, query_window_len=6, query_overlap=1, ngram_tolerance=0.7, edit_distance_tolerance=0.85):\n",
    "    \"\"\"\n",
    "    This function saves results as generated by search_for_bible() function. Results are append to previously created results.\n",
    "\n",
    "    :param results: results as generated by search_for_bible() function. The structure of results dict is: key = subverse_id, key_content: [(query_file, index_query_part), ...].\n",
    "    :param dataset: bibleDataset object to which current results are associated. When the dataset is changed, the subverse_id will no longer be valid, therefore, we also save verse_id right away.\n",
    "    :param batch_id: id of batch to match with batches.csv.\n",
    "    :param ngram_size: size of n-grams on which the search was based.\n",
    "    :param query_window_len: size of query split by which the search was done.\n",
    "    :param query_overlap: size of query split overlap by which the search was done.\n",
    "    \"\"\"\n",
    "    # If result CSV do not exist, create it:\n",
    "    create_batches_results_csv()\n",
    "\n",
    "    # Load results dataframe:\n",
    "    results_df = pd.read_csv(BATCH_RESULTS_FILE_PATH)\n",
    "\n",
    "    # Create new df from current results:\n",
    "    new_data = defaultdict(list)\n",
    "\n",
    "    for subverse_id in results:\n",
    "            for attr in results[subverse_id]:\n",
    "                new_data['verse_id'].append(dataset.target[subverse_id])\n",
    "                new_data['query_file'].append(attr[0])\n",
    "                new_data['index_query_part'].append(attr[1])\n",
    "                new_data['batch_id'].append(batch_id)\n",
    "                new_data['ngram_size'].append(ngram_size)\n",
    "                new_data['query_window_len'].append(query_window_len)\n",
    "                new_data['query_overlap'].append(query_overlap)\n",
    "                new_data['ngram_tolerance'].append(ngram_tolerance)\n",
    "                new_data['edit_distance_tolerance'].append(edit_distance_tolerance)\n",
    "\n",
    "    # merge old and new dataframes\n",
    "    append_df = pd.DataFrame(new_data)\n",
    "    results_df = pd.concat([results_df, append_df])\n",
    "    results_df.to_csv(BATCH_RESULTS_FILE_PATH, index=False, columns=results_columns_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The search function connecting the above functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_search_by_batch(batch_id:int, journals_fulldata:dict, bible_object:bibleObject, ngram_tolerance=0.7, edit_distance_tolerance=0.85, ngram_size=4, query_window_len=4, query_overlap=1):\n",
    "    \"\"\" This function executes search by a batch_id (as linked to json files in batches.csv) and saves it results to batch_results.csv. \"\"\"\n",
    "    bible_dataset = bible_object.dataset\n",
    "    \n",
    "    # Run search:\n",
    "    print(f'... Initiating search of batch {batch_id}')\n",
    "    batch_results, avg_time_per_page = search_for_bible_for_batches_implementation(bible_object=bible_object, journals_fulldata=journals_fulldata, batch_id=batch_id, ngram_tolerance=ngram_tolerance, edit_distance_tolerance=edit_distance_tolerance, ngram_size=ngram_size, query_window_len=query_window_len, query_overlap=query_overlap)\n",
    "\n",
    "    # Save results:\n",
    "    print(f'... Saving results of batch {batch_id}')\n",
    "    save_batch_results(results=batch_results, dataset=bible_dataset, batch_id=batch_id, ngram_size=ngram_size, query_window_len=query_window_len, query_overlap=query_overlap, ngram_tolerance=ngram_tolerance, edit_distance_tolerance=edit_distance_tolerance)\n",
    "\n",
    "    # Change run log in batches.csv:\n",
    "    print(f'... Changing search log for {batch_id}')\n",
    "    change_run_log(batch_id=batch_id, avereage_pre_page=avg_time_per_page)\n",
    "\n",
    "\n",
    "def search_by_batches(batches_to_run:list, bible_dataset_filename='fullBibleDataset', skip_done=True, ngram_tolerance=0.7, edit_distance_tolerance=0.85, ngram_size=4, query_window_len=4, query_overlap=1):\n",
    "    \"\"\" \n",
    "    NOTE: CHENGE, this function uses journals_fulldata:dict, to possibly make it faster...\n",
    "\n",
    "    This function executes search across a number of batches.\n",
    "    Batches must be prepared in batches.csv (with update_batches_csv() function).\n",
    "    \n",
    "    :param batches_to_run: list of batches IDs that are to be run (batch IDs are int)\n",
    "    :param bible_dataset_filename: filename (without \".joblib\"!) of a dataset that is to be loaded and with which the search is run.\n",
    "    :param skip_done: bool, if True, batches that have already been run are skipped.\n",
    "    :param ngram_tolerance: what portion of ngrams of Bible subverse must match to be considered as a match.\n",
    "    :param edit_distance_tolerance: what portion of characters of a subverse must match a sequence from a \"query document\".\n",
    "    :param ngram_size: size of ngrams (in characters) to which everything is parsed; According to it, other objects for search are loaded.\n",
    "    :param query_window_len: How many sentences are put together as smaller \"query documets\". This parameter can influence the speed of the process, depending on the nature of the document.\n",
    "    :param query_overlap: Overlap of the sentences among \"query documents\". Must be higher than query_window_len.\n",
    "    \"\"\"\n",
    "    bible_dataset = load_dataset(bible_dataset_filename)\n",
    "    object_prefix = bible_dataset_filename.replace('Dataset', '')\n",
    "    bible_object = bibleObject(bible_dataset, ngram_size=ngram_size, objects_prefix=object_prefix)\n",
    "\n",
    "    print('Loading journals_fulldata.joblib')\n",
    "    journals_fulldata = joblib.load(JOURNAL_FULLDATA_PATH)\n",
    "\n",
    "    batches_to_skip = []\n",
    "    if skip_done:\n",
    "        batches_df = pd.read_csv(BATCHES_FILE_PATH)\n",
    "        relevant_data = batches_df.loc[batches_df['run'] == True]\n",
    "        batches_to_skip = list(set(relevant_data['batch_id'].to_list()))\n",
    "\n",
    "    for batch_id in batches_to_run:\n",
    "        if batch_id in batches_to_skip:\n",
    "            print(f'Batch {batch_id} has already been run.')\n",
    "            continue\n",
    "        else:\n",
    "            run_search_by_batch(batch_id=batch_id, journals_fulldata=journals_fulldata, bible_object=bible_object, ngram_tolerance=ngram_tolerance, edit_distance_tolerance=edit_distance_tolerance, ngram_size=ngram_size, query_window_len=query_window_len, query_overlap=query_overlap)\n",
    "\n",
    "    print('SEARCH FINISHED')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# END\n",
    "This has been the description of the search for biblical citations as suggested by this project. Probably not the best approach and many improvements are possible. Maybe, it will be developed further somewhen in the future :-)\n",
    "\n",
    "- For the evaluation of results, see [evaluate.py](https://github.com/DigilabNLCR/BibleCitations/blob/main/evaluate.py) or description in [evaluation.ipynb](https://github.com/DigilabNLCR/BibleCitations/blob/main/evaluation.ipynb) notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.9 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "8427d73f02270cccde5d7b657eb9efbd549974ced0a163dd64bdafb1c87004cd"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}