AllTransQuestAllLangs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bb4e6d30-7229-4e17-b543-0fef804d36d1",
   "metadata": {},
   "source": [
    "# Select Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43ed92da-2cb9-41d5-920b-d88aefc84bba",
   "metadata": {},
   "source": [
    "### You will need to select one Transquest model at a time. The model can be specified with the help of the variable \"tqmodel\" defined below.\n",
    "### The valid values for this variable are: \n",
    "* mono   ->   [for Monolingual TransQuest model(s)].\n",
    "* siamese   ->   [for Siamese Transquest model(s)].\n",
    "* multi   ->   [for Multilingual TransQuest model]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88d87da0-842c-49d1-bad5-c47b9b1252c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "tqmodel = \"mono\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6807c562-e62b-4a1a-a826-90b8aa44bf58",
   "metadata": {},
   "source": [
    "# Select Source Language\n",
    "\n",
    "### Similarly, you will need to select one source language for QE model evaluation. \n",
    "### The source language can be specified with the variable \"source_lang\" defined below. \n",
    "### The valid values for this variable are:\n",
    "\n",
    "* ru   ->   (Russian)\n",
    "* ro   ->   (Romanian)\n",
    "* si   ->   (Sinhala)\n",
    "* et   ->   (Estonian)\n",
    "* ne   ->   (Nepali)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e728b2f5-8abc-4053-9db3-f6c3ca267f86",
   "metadata": {},
   "outputs": [],
   "source": [
    "source_lang = \"ne\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd86a754-2d13-48a5-9015-16ba9c90074b",
   "metadata": {},
   "source": [
    "# Import required libraries and Load Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "583f3148-a3fe-42bc-9049-56ef0cad9b9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import pandas as pd\n",
    "import io, sys, os\n",
    "from functools import reduce\n",
    "import spacy\n",
    "import string\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "from spacy.lang.en import English\n",
    "import numpy as np\n",
    "import random\n",
    "import re\n",
    "nlp = English()\n",
    "tokenizer = nlp.tokenizer\n",
    "from tqdm.notebook import tqdm\n",
    "import plotly.express as px"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b2c0b21-4681-479b-bf2a-61678cfa6736",
   "metadata": {},
   "source": [
    "### Loading the correct model for source-target ([source_lang] -en) pair based on the values selected above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9430bed-1d05-4a18-a81e-1c04b2739c28",
   "metadata": {},
   "outputs": [],
   "source": [
    "if tqmodel == \"siamese\":\n",
    "    from transquest.algo.sentence_level.siamesetransquest.run_model import SiameseTransQuestModel\n",
    "    \n",
    "    if source_lang == \"ru\":\n",
    "        model = SiameseTransQuestModel(\"TransQuest/siamesetransquest-da-\"+source_lang+\"_en-reddit_wikiquotes\")\n",
    "    elif (source_lang == \"et\" or source_lang == \"si\" or source_lang == \"ne\" or source_lang == \"ro\"):\n",
    "        model = SiameseTransQuestModel(\"TransQuest/siamesetransquest-da-\"+source_lang+\"_en-wiki\")\n",
    "    else:\n",
    "        print(\"Source language should be selected correctly for model loading. Use ru, si, et, ro, ne (Russian, Sinhala, Estonian, Romanian, Nepali) as your choice of source language\")\n",
    "        \n",
    "elif tqmodel == \"mono\":\n",
    "    from transquest.algo.sentence_level.monotransquest.run_model import MonoTransQuestModel\n",
    "\n",
    "    if source_lang == \"ru\":\n",
    "        model = MonoTransQuestModel(\"xlmroberta\", \"TransQuest/monotransquest-da-\"+source_lang+\"_en-reddit_wikiquotes\", num_labels=1, use_cuda=torch.cuda.is_available())\n",
    "    elif (source_lang == \"et\" or source_lang == \"si\" or source_lang == \"ne\" or source_lang == \"ro\"):\n",
    "        model = MonoTransQuestModel(\"xlmroberta\", \"TransQuest/monotransquest-da-\"+source_lang+\"_en-wiki\", num_labels=1, use_cuda=torch.cuda.is_available())\n",
    "    else:\n",
    "        print(\"Source language should be selected correctly for model loading. Use ru, si, et, ro, ne (Russian, Sinhala, Estonian, Romanian, Nepali) as your choice of source language\")\n",
    "\n",
    "elif tqmodel == \"multi\":\n",
    "    if (source_lang == \"ru\" or source_lang == \"et\" or source_lang == \"si\" or source_lang == \"ne\" or source_lang == \"ro\"):\n",
    "        from transquest.algo.sentence_level.monotransquest.run_model import MonoTransQuestModel\n",
    "        model = MonoTransQuestModel(\"xlmroberta\", \"TransQuest/monotransquest-da-multilingual\", num_labels=1, use_cuda=torch.cuda.is_available())\n",
    "    else:\n",
    "        print(\"Source language should be selected correctly for model loading. Use ru, si, et, ro, ne (Russian, Sinhala, Estonian, Romanian, Nepali) as your choice of source language\")\n",
    "\n",
    "else:\n",
    "    print(\"Please use the values 'mono', 'siamese', or 'multi' for the correct selection of the TransQuest model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c94ebde1-f0d5-4283-a65d-edc1a5fd5eed",
   "metadata": {},
   "source": [
    "# Data Preprocessing for Dev and Test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31328ea8-6c3e-45a3-a8f2-ce84f908e0b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "dfdev = pd.read_csv(f\"{source_lang}_en_data/dev.{source_lang}en.df.short.tsv\",sep='\\t')\n",
    "dftest = pd.read_csv(f\"{source_lang}_en_data/test20.{source_lang}en.df.short.tsv\", sep='\\t')\n",
    "frames = [dfdev, dftest]\n",
    "final = pd.concat(frames)\n",
    "comb = final.reset_index(drop=True)\n",
    "if(source_lang!=\"ru\"):\n",
    "    devtestComb = comb.drop(columns=['index', 'mean', 'z_scores', 'z_mean', 'model_scores', 'scores'])\n",
    "else:\n",
    "    devtestComb = comb.drop(columns=['segid', 'mean', 'z_scores', 'z_mean', 'model_scores', 'scores'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c53ef10d-3895-4198-9030-1a5e9a6523a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "dtlol = devtestComb.values.tolist()\n",
    "if tqmodel == \"siamese\":\n",
    "    nppredictions = model.predict(dtlol)\n",
    "else:\n",
    "    nppredictions, raw = model.predict(dtlol)\n",
    "\n",
    "df_orig_trans_score = devtestComb\n",
    "df_orig_trans_score['trans_score'] = nppredictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9ed4488-3a65-46ff-94b6-b67b6e2c7480",
   "metadata": {},
   "source": [
    "## Obtaining Human Judgements from the WMT dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84ad005f-3fb9-4b55-9ade-957d0f05cda0",
   "metadata": {},
   "outputs": [],
   "source": [
    "####### BELOW CODE CALCULATES THE AVERAGE SCORE FOR HUMAN DA AND SUBSTITUTES IT IN THE TRANS_SCORE COLUMN INSTEAD OF THE PREDICTED SCORE #########\n",
    "l = comb['scores'].tolist()\n",
    "newlist = []\n",
    "for sublist in l:\n",
    "    sublist = sublist.split(\", \")\n",
    "    sublist = [i.strip('[]') for i in sublist]\n",
    "    sublist = [float(i) for i in sublist]\n",
    "    newlist.append(sublist)\n",
    "\n",
    "newlistnparray = np.array(newlist)\n",
    "newlistmeans = np.mean(newlistnparray, axis=1)\n",
    "newlistmeans = newlistmeans/100\n",
    "newlistmeans = newlistmeans.tolist()\n",
    "df_orig_trans_score['trans_score_hum'] = newlistmeans\n",
    "####### ABOVE CODE CALCULATES THE AVERAGE SCORE FOR HUMAN DA AND SUBSTITUTES IT IN THE TRANS_SCORE COLUMN INSTEAD OF THE PREDICTED SCORE #########\n",
    "\n",
    "def trans_threshold(df, th=0.7):\n",
    "  trans_score_np = np.array(list(df['trans_score_hum'])) ## CHANGE HERE BASED ON WHICH COLUMN SHOULD THE THRESHOLDING BE APPLIED\n",
    "  trans_score_idx_ = np.argwhere(trans_score_np >= th)\n",
    "  trans_score_idx = trans_score_idx_.reshape(trans_score_idx_.shape[0])\n",
    "  return df.iloc[trans_score_idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e53fc708-1ceb-4454-8785-136ce2df783a",
   "metadata": {},
   "source": [
    "### Applying threshold of 0.7 now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9009fabd-3741-4ace-a2bb-7c2e75f2ea39",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_trans_thres = trans_threshold(df_orig_trans_score, 0.7) #Selection of threshold 0.7\n",
    "df_trans_thres = df_trans_thres.drop(columns=['trans_score_hum'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1da64699-099a-4434-9463-ff60cb93dab5",
   "metadata": {},
   "source": [
    "## Generating Vocabulary from the preprocessed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15c28c3e-d4a3-46d8-848d-97b6fd0f3058",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Vocabulary:\n",
    "    def __init__(self, name):\n",
    "        PAD_token = 0   # Used for padding short sentences\n",
    "        SOS_token = 1   # Start-of-sentence token\n",
    "        EOS_token = 2   # End-of-sentence token\n",
    "        self.name = name\n",
    "        self.word2index = {}\n",
    "        self.word2count = {}\n",
    "        self.index2word = {PAD_token: \"PAD\", SOS_token: \"SOS\", EOS_token: \"EOS\"}\n",
    "        self.num_words = 3\n",
    "        self.num_sentences = 0\n",
    "        self.longest_sentence = 0\n",
    "\n",
    "    def add_word(self, word):\n",
    "        if word not in self.word2index:\n",
    "            # First entry of word into vocabulary\n",
    "            self.word2index[word] = self.num_words\n",
    "            self.word2count[word] = 1\n",
    "            self.index2word[self.num_words] = word\n",
    "            self.num_words += 1\n",
    "        else:\n",
    "            # Word exists; increase word count\n",
    "            self.word2count[word] += 1\n",
    "            \n",
    "    def add_sentence(self, sentence):\n",
    "        sentence_len = 0\n",
    "        for word in sentence.split(' '):\n",
    "            sentence_len += 1\n",
    "            self.add_word(word)\n",
    "        if sentence_len > self.longest_sentence:\n",
    "            # This is the longest sentence\n",
    "            self.longest_sentence = sentence_len\n",
    "        # Count the number of sentences\n",
    "        self.num_sentences += 1\n",
    "\n",
    "    def to_word(self, index):\n",
    "        return self.index2word[index]\n",
    "\n",
    "    def to_index(self, word):\n",
    "        return self.word2index[word]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64a17a42-8aac-45df-8977-380fe8df78e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "voc = Vocabulary('corpusVocab')\n",
    "for sentence in devtestComb['translation']:\n",
    "    voc.add_sentence(sentence.translate(str.maketrans('', '', string.punctuation))) ## removing puntuations\n",
    "\n",
    "# vocab with stopwords \n",
    "vocab = []\n",
    "for word in range(voc.num_words):\n",
    "    vocab.append(voc.to_word(word))\n",
    "# vocab without stopwords \n",
    "vocab_wo_stopw = []\n",
    "for word in vocab:\n",
    "    if(word.lower() not in STOP_WORDS):\n",
    "        vocab_wo_stopw.append(word)\n",
    "\n",
    "len(vocab), len(vocab_wo_stopw)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "033c01af-7c60-4c25-ad2a-730893292321",
   "metadata": {},
   "source": [
    "# Perturbations and Scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f05c5fe-d306-4803-8be1-0d3be88b8da0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_map1_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translation based on the removal of negation markers - \"not, no, and n't\"\n",
    "    print(\"Generating sentences for MAP1\")\n",
    "    new_sents_lol = [] \n",
    "    map1_orig_sents_lol = []\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        map1_orig_sents = []\n",
    "        for original, sentence in zip(df['original'], df['translation']):\n",
    "            tempSentVoc = sentence.split()\n",
    "\n",
    "            sentVoc = []\n",
    "            for token in tempSentVoc:\n",
    "                token = token.replace(\"n't\", \"\", 1)\n",
    "                sentVoc.append(token)\n",
    "            \n",
    "            sentVoc = [token for token in sentVoc if (token.lower() != \"not\" and token.lower() != \"no\")]\n",
    "            \n",
    "            \n",
    "            if(sentence != ' '.join(sentVoc)):\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(' '.join(sentVoc))\n",
    "                new_sents.append(tempL)\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(sentence)\n",
    "                map1_orig_sents.append(tempL)\n",
    "        new_sents_lol.append(new_sents)\n",
    "        map1_orig_sents_lol.append(map1_orig_sents)\n",
    "\n",
    "    return new_sents_lol, map1_orig_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82cd97dc-fe69-449f-8cdc-394b40abae1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_map2_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on removal of random content words\n",
    "    print(\"Generating sentences for MAP2\")\n",
    "    puncts = list(set(string.punctuation))\n",
    "    new_sents_lol = [] \n",
    "    vocab_ = vocab_wo_stopw\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        for sentence in df['translation']:\n",
    "            sentVoc = [str(i) for i in tokenizer(sentence)]\n",
    "            sentVoc_wo_punct = []\n",
    "\n",
    "            for v in sentVoc:\n",
    "                if(v not in puncts):\n",
    "                    sentVoc_wo_punct.append(v)\n",
    "\n",
    "            sentVoc = sentVoc_wo_punct\n",
    "            \n",
    "            sentVoc_wo_stop_word = []\n",
    "            for v in sentVoc:\n",
    "                if(v.lower() not in STOP_WORDS):\n",
    "                    sentVoc_wo_stop_word.append(v)\n",
    "            \n",
    "            sentVoc = sentVoc_wo_stop_word\n",
    "       \n",
    "            new_word = \"\" \n",
    "            old_word = random.choice(sentVoc)\n",
    "          \n",
    "            flag = 0\n",
    "            while(flag == 0):\n",
    "                if(new_word not in sentVoc):\n",
    "                    new_sents.append(sentence.replace(old_word, new_word, 1))\n",
    "                    flag = 1\n",
    "                else:\n",
    "                    new_word = random.choice(vocab_)\n",
    "    \n",
    "        new_sents_lol.append(new_sents)\n",
    "    return new_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e21de49-1a62-4a1b-90eb-4a3baaffa520",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_map3_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on duplication of a random content word\n",
    "    print(\"Generating sentences for MAP3\")\n",
    "    puncts = list(set(string.punctuation))\n",
    "    new_sents_lol = []\n",
    "    vocab_ = vocab_wo_stopw\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        for sentence in df['translation']:\n",
    "            sentVoc = [str(i) for i in tokenizer(sentence)]\n",
    "            sentVoc_wo_punct = []\n",
    "\n",
    "            for v in sentVoc:\n",
    "                if(v not in puncts):\n",
    "                    sentVoc_wo_punct.append(v)\n",
    "            sentVoc = sentVoc_wo_punct\n",
    "\n",
    "            sentVoc_wo_stop_word = []\n",
    "            for v in sentVoc:\n",
    "                if(v.lower() not in STOP_WORDS):\n",
    "                    sentVoc_wo_stop_word.append(v)\n",
    "            sentVoc = sentVoc_wo_stop_word\n",
    "        \n",
    "            old_word = random.choice(sentVoc)\n",
    "            new_word = old_word + \" \" + old_word \n",
    "          \n",
    "            flag = 0\n",
    "            while (flag == 0):\n",
    "                if(new_word not in sentVoc):\n",
    "                    new_sents.append(sentence.replace(old_word, new_word, 1))\n",
    "                    flag = 1\n",
    "                else:\n",
    "                    new_word = random.choice(vocab_)\n",
    "    \n",
    "        new_sents_lol.append(new_sents)\n",
    "    return new_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a9d5000-5672-460a-b318-b213ea9853f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_map4_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on insertion of random content word\n",
    "    print(\"Generating sentences for MAP4\")\n",
    "    puncts = list(set(string.punctuation))\n",
    "    new_sents_lol = []\n",
    "    vocab_ = vocab_wo_stopw\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        for sentence in df['translation']:\n",
    "            sentVoc = [str(i) for i in tokenizer(sentence)]\n",
    "            sentVoc_wo_punct = []\n",
    "\n",
    "            for v in sentVoc:\n",
    "                if(v not in puncts):\n",
    "                    sentVoc_wo_punct.append(v)\n",
    "            sentVoc = sentVoc_wo_punct\n",
    "\n",
    "            sentVoc_wo_stop_word = []\n",
    "            for v in sentVoc:\n",
    "                if(v.lower() not in STOP_WORDS):\n",
    "                    sentVoc_wo_stop_word.append(v)\n",
    "            sentVoc = sentVoc_wo_stop_word\n",
    "        \n",
    "            new_word = random.choice(vocab_)\n",
    "            old_word = random.choice(sentVoc)\n",
    "            new_word = old_word + \" \" + new_word \n",
    "          \n",
    "            flag = 0\n",
    "            while (flag == 0):\n",
    "                if(new_word not in sentVoc):\n",
    "                    new_sents.append(sentence.replace(old_word, new_word, 1))\n",
    "                    flag = 1\n",
    "                else:\n",
    "                    new_word = random.choice(vocab_)\n",
    "    \n",
    "        new_sents_lol.append(new_sents)\n",
    "    return new_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f57f9191-e4ff-4244-b57d-f81f90a93ec1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_map5_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on random replacements of words\n",
    "    print(\"Generating sentences for MAP5\")\n",
    "    puncts = list(set(string.punctuation))\n",
    "    new_sents_lol = []\n",
    "    vocab_ = vocab\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        for sentence in df['translation']:\n",
    "            sentVoc = [str(i) for i in tokenizer(sentence)]\n",
    "            sentVoc_wo_punct = []\n",
    "\n",
    "            for v in sentVoc:\n",
    "                if(v not in puncts):\n",
    "                    sentVoc_wo_punct.append(v)\n",
    "            sentVoc = sentVoc_wo_punct\n",
    "\n",
    "            sentVoc_wo_stop_word = []\n",
    "            for v in sentVoc:\n",
    "                if(v.lower() not in STOP_WORDS):\n",
    "                    sentVoc_wo_stop_word.append(v)\n",
    "            sentVoc = sentVoc_wo_stop_word\n",
    "            \n",
    "            new_word = random.choice(vocab_)\n",
    "            old_word = random.choice(sentVoc)\n",
    "          \n",
    "            flag = 0\n",
    "            while (flag == 0):\n",
    "                if(new_word not in sentVoc):\n",
    "                    new_sents.append(sentence.replace(old_word, new_word, 1))\n",
    "                    flag = 1\n",
    "                else:\n",
    "                    new_word = random.choice(vocab_)\n",
    "    \n",
    "        new_sents_lol.append(new_sents)\n",
    "    return new_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b795d09-e493-481b-a4f8-60edb29c4c65",
   "metadata": {},
   "outputs": [],
   "source": [
    "## For NLP Augmentation Library ##\n",
    "# import nlpaug.augmenter.char as nac\n",
    "import nlpaug.augmenter.word as naw\n",
    "# import nlpaug.augmenter.sentence as nas\n",
    "import nlpaug.flow as nafc\n",
    "from nlpaug.util import Action\n",
    "## For NLP Augmentation Library ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "94dfc4c5-20f2-490c-934d-8a997f7c41ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "## For NLP Augmentation Library ##\n",
    "\n",
    "## This takes time ##\n",
    "\n",
    "aug = naw.ContextualWordEmbsAug( model_path='bert-base-uncased', action=\"substitute\" )\n",
    "\n",
    "## For NLP Augmentation Library ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bafcddde-8a67-499f-9424-4317ad9bef9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "## This function uses the NLP Augmentation Library##\n",
    "\n",
    "def generate_map6_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on synonymy replacement using BERT, considered an MAP as BERT-based replacements\n",
    "    ## have been shown to replace important content words, which do change the meaning.\n",
    "    print(\"Generating sentences for MAP6 - This takes some time for all the 20 trials.\")\n",
    "    new_sents_lol = []\n",
    "    map6_orig_sents_lol = []\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        map6_orig_sents = []\n",
    "\n",
    "        for original, sentence in zip(df['original'], df['translation']):    \n",
    "            origSent = sentence.split()\n",
    "            augmented_text = aug.augment(sentence)\n",
    "\n",
    "            if((' '.join(origSent)) != augmented_text):\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(augmented_text)\n",
    "                new_sents.append(tempL)\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(' '.join(origSent))\n",
    "                map6_orig_sents.append(tempL)\n",
    "         \n",
    "        new_sents_lol.append(new_sents)\n",
    "        map6_orig_sents_lol.append(map6_orig_sents)\n",
    "    \n",
    "    return new_sents_lol, map6_orig_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "359752bb-73ab-41b1-b20c-21e7a3ccc5d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "augAntonym = naw.AntonymAug()\n",
    "\n",
    "def generate_map7_sents(df, nTrials, stopwords=True):\n",
    "    ## returns the list of perturbed translations where words are replace by their Antonyms\n",
    "    print(\"Generating sentences for MAP7\")\n",
    "    new_sents_lol = [] \n",
    "    map7_orig_sents_lol = []\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        map7_orig_sents = []\n",
    "\n",
    "        for original, sentence in zip(df['original'], df['translation']):     \n",
    "            origSent = sentence.split()\n",
    "            augmented_text = augAntonym.augment(sentence)\n",
    "            \n",
    "            if((' '.join(origSent)) != augmented_text):  \n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(augmented_text)\n",
    "                new_sents.append(tempL)\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(' '.join(origSent))\n",
    "                map7_orig_sents.append(tempL)\n",
    "        new_sents_lol.append(new_sents)\n",
    "        map7_orig_sents_lol.append(map7_orig_sents)\n",
    "\n",
    "    return new_sents_lol, map7_orig_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71b54164-51fb-4d1e-b1ce-5bc4ab508a44",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_map8_sents(df, nTrials):\n",
    "    # returns a list of perturbed translations where it copyies source sentence instead of target\n",
    "    print(\"Generating sentences for MAP8\")\n",
    "    new_sents_lol = [] \n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        for sentence in df['original']:      \n",
    "            new_sents.append(sentence)\n",
    "        new_sents_lol.append(new_sents)\n",
    "\n",
    "    return new_sents_lol"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fd973b4-b5e4-4e87-b32c-e84763856523",
   "metadata": {},
   "source": [
    "## Generating Perturbed Sentences for MAPs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d97144d-d8fc-420d-b885-20d0af9a2d7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_sents_lol_map1, map1_orig_sents = generate_map1_sents(df_trans_thres, 20) ## Perturbations for removing negation markers\n",
    "new_sents_lol_map2  = generate_map2_sents(df_trans_thres, 20) ## Perturbations for removing random content words\n",
    "new_sents_lol_map3  = generate_map3_sents(df_trans_thres, 20) ## Perturbations for duplicating random content words\n",
    "new_sents_lol_map4  = generate_map4_sents(df_trans_thres, 20) ## Perturbations for inserting random content words\n",
    "new_sents_lol_map5  = generate_map5_sents(df_trans_thres, 20) ## Perturbations for replacing random content words\n",
    "\n",
    "## For MAP6, each trial takes ~1 minute over 16 core machine. This library uses CPU cores, and you are passing 20 trials, so this takes time.\n",
    "\n",
    "new_sents_lol_map6, map6_orig_sents = generate_map6_sents(df_trans_thres, 20) ## Perturbations for replacing target by source\n",
    "\n",
    "## For MAP6, each trial takes ~1 minute over 16 core machine. This library uses CPU cores, and you are passing 20 trials, so this takes time.\n",
    "\n",
    "new_sents_lol_map7, map7_orig_sents = generate_map7_sents(df_trans_thres, 20) ## Perturbations for replacing words by their Antonyms\n",
    "new_sents_lol_map8  = generate_map8_sents(df_trans_thres, 20) ## Perturbations for replacing translation with source sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84a33b0a-cd06-4c43-8d9f-c6dc4761bd3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_mpp1_sents(df, nTrials):\n",
    "    ##returns the list of perturbed translations based removal of punctuations\n",
    "    print(\"Generating sentences for MPP1\")\n",
    "    exclude = set(string.punctuation)\n",
    "    new_sents_lol = [] \n",
    "    mpp1_orig_sents_lol = []\n",
    "    \n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        mpp1_orig_sents = []\n",
    "\n",
    "        for original, sentence in zip(df['original'], df['translation']):\n",
    "            newSentence = ''.join(ch for ch in sentence if ch not in exclude)\n",
    "            sentVoc = newSentence.split()\n",
    "            origSent = sentence.split()\n",
    "      \n",
    "            if((' '.join(origSent)) != (' '.join(sentVoc))):\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(' '.join(sentVoc))\n",
    "                new_sents.append(tempL)\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(sentence)\n",
    "                mpp1_orig_sents.append(tempL)\n",
    "        \n",
    "        new_sents_lol.append(new_sents)\n",
    "        mpp1_orig_sents_lol.append(mpp1_orig_sents)\n",
    "    \n",
    "    return new_sents_lol, mpp1_orig_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "896eecda-62f5-402f-a151-e158326fd818",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_mpp2_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on replacement of punctuations\n",
    "    print(\"Generating sentences for MPP2\")\n",
    "    puncts = list(set(string.punctuation))\n",
    "    new_sents_lol = [] \n",
    "    mpp2_orig_sents_lol = []\n",
    "\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        mpp2_orig_sents = []\n",
    "\n",
    "        for original, sentence in zip(df['original'], df['translation']):\n",
    "            for i in range(len(sentence)):\n",
    "                if(sentence[i] in puncts): #Found punctuation, replacing here.\n",
    "                    new_punct = random.choice(puncts)\n",
    "                    if(new_punct!=sentence[i]):\n",
    "                        newSentence = sentence.replace(sentence[i], new_punct)\n",
    "            sentVoc = newSentence.split()\n",
    "            origSent = sentence.split()\n",
    "      \n",
    "            if((' '.join(origSent)) != (' '.join(sentVoc))):\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(' '.join(sentVoc))\n",
    "                new_sents.append(tempL)\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(sentence)\n",
    "                mpp2_orig_sents.append(tempL)\n",
    "                \n",
    "        new_sents_lol.append(new_sents)\n",
    "        mpp2_orig_sents_lol.append(mpp2_orig_sents)\n",
    "    \n",
    "    return new_sents_lol, mpp2_orig_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e696cd4c-97fd-43be-9576-0628a64834c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Required for MPP3/4 ##\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "## Required for MPP3/4 ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e14f3408-a4ed-420a-8078-7d41a01b1044",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_mpp3_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on removal of determiners\n",
    "    print(\"Generating sentences for MPP3\")\n",
    "    puncts = list(set(string.punctuation))\n",
    "    new_sents_lol = []\n",
    "    mpp3_orig_sents_lol = []\n",
    "    \n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        mpp3_orig_sents = []\n",
    "\n",
    "        for original, sentence in zip(df['original'], df['translation']):\n",
    "            doc = nlp(sentence)\n",
    "            sentVoc = [token.text for token in doc if token.pos_ != \"DET\"]\n",
    "            origSent = sentence.split()\n",
    "            indices = []\n",
    "            for i in range(len(sentVoc)):\n",
    "                if(sentVoc[i] in puncts):\n",
    "                    sentVoc[i-1] = sentVoc[i-1]+sentVoc[i]\n",
    "                    indices.append(i)\n",
    "            iter=0\n",
    "            for idx in indices:\n",
    "                sentVoc.pop(idx-iter)\n",
    "                iter+=1\n",
    "            if((' '.join(origSent)) != (' '.join(sentVoc))):\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(' '.join(sentVoc))\n",
    "                new_sents.append(tempL)\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(sentence)\n",
    "                mpp3_orig_sents.append(tempL)\n",
    "        new_sents_lol.append(new_sents)\n",
    "        mpp3_orig_sents_lol.append(mpp3_orig_sents)\n",
    "    \n",
    "    return new_sents_lol, mpp3_orig_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6974fcb-949d-4ff8-bcbb-3df93dc9ddd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_mpp4_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on replacement of determiners from a list\n",
    "    print(\"Generating sentences for MPP4\")\n",
    "    detList =  [\"the\", \"a\", \"an\", \"this\", \"that\", \"these\", \"those\", \"my\", \"your\", \"his\", \"her\", \"its\", \"our\", \"their\", \"some\", \"any\", \"enough\", \"all\", \"each\", \"every\", \"other\", \"another\", \"such\"]\n",
    "    new_sents_lol = [] \n",
    "    mpp4_orig_sents_lol = []\n",
    "\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        mpp4_orig_sents = []\n",
    "\n",
    "        for original, sentence in zip(df['original'], df['translation']):\n",
    "            doc = nlp(sentence)\n",
    "            sentVoc = []\n",
    "            origSent = sentence.split()\n",
    "            for i in range(len(doc)):\n",
    "                if(doc[i].pos_ == \"DET\"):\n",
    "                    sentVoc.append(random.choice(detList))\n",
    "                else:\n",
    "                    sentVoc.append(str(doc[i]))\n",
    "\n",
    "            if((' '.join(origSent)) != (' '.join(sentVoc))):  \n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(' '.join(sentVoc))\n",
    "                new_sents.append(tempL)\n",
    "                tempL=[]\n",
    "                tempL.append(original)\n",
    "                tempL.append(sentence)\n",
    "                mpp4_orig_sents.append(tempL)\n",
    "\n",
    "        new_sents_lol.append(new_sents)\n",
    "        mpp4_orig_sents_lol.append(mpp4_orig_sents)\n",
    "    \n",
    "    return new_sents_lol, mpp4_orig_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd3d1409-1b7e-4c88-97d1-2a58fc845190",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_mpp5_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on uppercasing words\n",
    "    print(\"Generating sentences for MPP5\")\n",
    "    new_sents_lol = []\n",
    "\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        for sentence in df['translation']:\n",
    "            sentVoc = sentence.split()\n",
    "            for i in range(len(sentVoc)):\n",
    "                wordswithUC = []\n",
    "                wordswithUC.append(str(sentVoc[i]))\n",
    "                wordswithUC.append(str(sentVoc[i].upper()))\n",
    "                sentVoc[i] = random.choice(wordswithUC)\n",
    "          \n",
    "            new_sents.append(' '.join(sentVoc))\n",
    "        new_sents_lol.append(new_sents)\n",
    "\n",
    "    return new_sents_lol"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3c26dbf-2c00-4077-95e5-ababdf0350f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_mpp6_sents(df, nTrials):\n",
    "    ## returns the list of perturbed translations based on lowercasing words\n",
    "    print(\"Generating sentences for MPP6\")\n",
    "    new_sents_lol = []\n",
    "\n",
    "    for i in range(nTrials):\n",
    "        new_sents  = []\n",
    "        for sentence in df['translation']:\n",
    "            sentVoc = sentence.split()\n",
    "            for i in range(len(sentVoc)):\n",
    "                wordswithUC = []\n",
    "                wordswithUC.append(str(sentVoc[i]))\n",
    "                wordswithUC.append(str(sentVoc[i].lower()))\n",
    "                sentVoc[i] = random.choice(wordswithUC)\n",
    "          \n",
    "            new_sents.append(' '.join(sentVoc))\n",
    "        new_sents_lol.append(new_sents)\n",
    "\n",
    "    return new_sents_lol"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e80d279-167e-4df4-972f-484b2e5bc42f",
   "metadata": {},
   "source": [
    "## Generating Perturbed Sentences for MPPs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93a887ba-aa05-4cdd-84fe-f6e2c4169b27",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_sents_lol_mpp1, mpp1_orig_sents = generate_mpp1_sents(df_trans_thres, 20) ## Perturbations for removing punctuations\n",
    "new_sents_lol_mpp2, mpp2_orig_sents = generate_mpp2_sents(df_trans_thres, 20) ## Perturbations for replacing punctuations\n",
    "new_sents_lol_mpp3, mpp3_orig_sents = generate_mpp3_sents(df_trans_thres, 20) ## Perturbations for removing determiners\n",
    "new_sents_lol_mpp4, mpp4_orig_sents = generate_mpp4_sents(df_trans_thres, 20) ## Perturbations for replacing determiners\n",
    "new_sents_lol_mpp5 = generate_mpp5_sents(df_trans_thres, 20) ## Perturbations for random uppercasing\n",
    "new_sents_lol_mpp6 = generate_mpp6_sents(df_trans_thres, 20) ## Perturbations for random lowercasing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc66a083-4cd5-419a-9a94-78c9471dd95e",
   "metadata": {},
   "source": [
    "## Scoring Perturbations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7825fc38-0c06-4c61-9363-25764a9f225f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_scores_1(df, new_sents, nTrials):\n",
    "    scores = np.zeros((nTrials + 1 , df.shape[0]))\n",
    "    scores[0] = df['trans_score'].to_numpy() # storing prediction for original translations at index 0\n",
    "\n",
    "    for i in range(nTrials):\n",
    "        print(\"Computing Scores on Perturbed Sentence Trial \"+str(i+1))\n",
    "        df_perturbed = df.drop(columns=['translation','trans_score'])\n",
    "        df_perturbed['perturbed'] = new_sents[i]\n",
    "\n",
    "        dtlol2 = df_perturbed.values.tolist()\n",
    "        p1predictions, p1raw_outputs = model.predict(dtlol2)\n",
    "\n",
    "        scores[1+i] = p1predictions\n",
    "\n",
    "    return scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cfeb102-b498-4110-b3aa-869498f6eb04",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_scores_2(orig_list_lol, new_sents_lol, nTrials):\n",
    "    x = len(orig_list_lol)\n",
    "    y = max(map(len, orig_list_lol))\n",
    "    scores_orig = np.zeros([x,y])\n",
    "    for i in range(len(orig_list_lol)):\n",
    "        print(\"Original Sentence Pairs - Trial \"+str(i+1))\n",
    "        nppredictions, npraw_outputs = model.predict(orig_list_lol[i])\n",
    "        scores_orig[i,:len(nppredictions)] = nppredictions\n",
    "  \n",
    "    x = len(new_sents_lol)\n",
    "    y = max(map(len, new_sents_lol))\n",
    "    scores_pert = np.zeros([x,y])    \n",
    "    for j in range(len(new_sents_lol)):\n",
    "        print(\"Perturbed Sentence Pairs - Trial \"+str(j+1))\n",
    "        p1predictions, p1raw_outputs = model.predict(new_sents_lol[j])\n",
    "        scores_pert[j,:len(p1predictions)] = p1predictions\n",
    "        \n",
    "    return scores_orig, scores_pert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "062c53c7-2f81-43f5-be5d-1a70d88a8af9",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "## This will take some time, score computation uses the models to get scores for each sentence pair in each trial  ##\n",
    "\n",
    "scores_map1_orig, scores_map1_perts = predict_scores_2(map1_orig_sents, new_sents_lol_map1, 20)\n",
    "scores_map2 = predict_scores_1(df_trans_thres, new_sents_lol_map2, 20)\n",
    "scores_map3 = predict_scores_1(df_trans_thres, new_sents_lol_map3, 20)\n",
    "scores_map4 = predict_scores_1(df_trans_thres, new_sents_lol_map4, 20)\n",
    "scores_map5 = predict_scores_1(df_trans_thres, new_sents_lol_map5, 20)\n",
    "scores_map6_orig, scores_map6_perts = predict_scores_2(map6_orig_sents, new_sents_lol_map6, 20)\n",
    "scores_map7_orig, scores_map7_perts = predict_scores_2(map7_orig_sents, new_sents_lol_map7, 20)\n",
    "scores_map8 = predict_scores_1(df_trans_thres, new_sents_lol_map8, 20)\n",
    "\n",
    "## This will take some time, score computation uses the models to get scores for each sentence pair in each trial  ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b334ba9a-2ff4-415d-938f-daf7c313393c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p generated_data_{source_lang}_en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "221a6d8d-25d6-4d6e-83b5-d2a3a7db4c4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "################## MAP1 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_map1_perts)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_map1_perts_df = pd.DataFrame(list(map(list, zip(*scores_map1_perts))), columns = columnsList)\n",
    "scores_map1_perts_df.to_csv('generated_data_'+source_lang+'_en/scores_map1_perts_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "\n",
    "columnsList = []\n",
    "for i in range(len(scores_map1_orig)):\n",
    "    columnsList.append('orig'+str(i+1))\n",
    "\n",
    "scores_map1_orig_df = pd.DataFrame(list(map(list, zip(*scores_map1_orig))), columns = columnsList)\n",
    "scores_map1_orig_df.to_csv('generated_data_'+source_lang+'_en/scores_map1_orig_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MAP2 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_map2)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_map2_df = pd.DataFrame(list(map(list, zip(*scores_map2))), columns = columnsList)\n",
    "scores_map2_df.to_csv('generated_data_'+source_lang+'_en/scores_map2_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MAP3 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_map3)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_map3_df = pd.DataFrame(list(map(list, zip(*scores_map3))), columns = columnsList)\n",
    "scores_map3_df.to_csv('generated_data_'+source_lang+'_en/scores_map3_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MAP4 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_map4)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_map4_df = pd.DataFrame(list(map(list, zip(*scores_map4))), columns = columnsList)\n",
    "scores_map4_df.to_csv('generated_data_'+source_lang+'_en/scores_map4_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MAP5 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_map5)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_map5_df = pd.DataFrame(list(map(list, zip(*scores_map5))), columns = columnsList)\n",
    "scores_map5_df.to_csv('generated_data_'+source_lang+'_en/scores_map5_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MAP6 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_map6_perts)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_map6_perts_df = pd.DataFrame(list(map(list, zip(*scores_map6_perts))), columns = columnsList)\n",
    "scores_map6_perts_df.to_csv('generated_data_'+source_lang+'_en/scores_map6_perts_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "\n",
    "columnsList = []\n",
    "for i in range(len(scores_map6_orig)):\n",
    "    columnsList.append('orig'+str(i+1))\n",
    "\n",
    "scores_map6_orig_df = pd.DataFrame(list(map(list, zip(*scores_map6_orig))), columns = columnsList)\n",
    "scores_map6_orig_df.to_csv('generated_data_'+source_lang+'_en/scores_map6_orig_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MAP7 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_map7_perts)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_map7_perts_df = pd.DataFrame(list(map(list, zip(*scores_map7_perts))), columns = columnsList)\n",
    "scores_map7_perts_df.to_csv('generated_data_'+source_lang+'_en/scores_map7_perts_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "\n",
    "columnsList = []\n",
    "for i in range(len(scores_map7_orig)):\n",
    "    columnsList.append('orig'+str(i+1))\n",
    "\n",
    "scores_map7_orig_df = pd.DataFrame(list(map(list, zip(*scores_map7_orig))), columns = columnsList)\n",
    "scores_map7_orig_df.to_csv('generated_data_'+source_lang+'_en/scores_map7_orig_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MAP8 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_map8)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_map8_df = pd.DataFrame(list(map(list, zip(*scores_map8))), columns = columnsList)\n",
    "scores_map8_df.to_csv('generated_data_'+source_lang+'_en/scores_map8_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "414e16c5-33e3-4710-92ed-57843dba4fb7",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "## This will take some time, score computation uses the models to get scores for each sentence pair in each trial  ##\n",
    "\n",
    "scores_mpp1_orig, scores_mpp1_perts = predict_scores_2(mpp1_orig_sents, new_sents_lol_mpp1, 20)\n",
    "scores_mpp2_orig, scores_mpp2_perts = predict_scores_2(mpp2_orig_sents, new_sents_lol_mpp2, 20)\n",
    "scores_mpp3_orig, scores_mpp3_perts = predict_scores_2(mpp3_orig_sents, new_sents_lol_mpp3, 20)\n",
    "scores_mpp4_orig, scores_mpp4_perts = predict_scores_2(mpp4_orig_sents, new_sents_lol_mpp4, 20)\n",
    "\n",
    "scores_mpp5 = predict_scores_1(df_trans_thres, new_sents_lol_mpp5, 20)\n",
    "scores_mpp6 = predict_scores_1(df_trans_thres, new_sents_lol_mpp6, 20)\n",
    "\n",
    "## This will take some time, score computation uses the models to get scores for each sentence pair in each trial  ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d914a905-7f43-4c3e-aaf7-0fbc78fa0a1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "################## MPP1 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp1_perts)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_mpp1_perts_df = pd.DataFrame(list(map(list, zip(*scores_mpp1_perts))), columns = columnsList)\n",
    "scores_mpp1_perts_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp1_perts_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp1_orig)):\n",
    "    columnsList.append('orig'+str(i+1))\n",
    "\n",
    "scores_mpp1_orig_df = pd.DataFrame(list(map(list, zip(*scores_mpp1_orig))), columns = columnsList)\n",
    "scores_mpp1_orig_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp1_orig_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MPP2 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp2_perts)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_mpp2_perts_df = pd.DataFrame(list(map(list, zip(*scores_mpp2_perts))), columns = columnsList)\n",
    "scores_mpp2_perts_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp2_perts_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp2_orig)):\n",
    "    columnsList.append('orig'+str(i+1))\n",
    "\n",
    "scores_mpp2_orig_df = pd.DataFrame(list(map(list, zip(*scores_mpp2_orig))), columns = columnsList)\n",
    "scores_mpp2_orig_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp2_orig_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MPP3 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp3_perts)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_mpp3_perts_df = pd.DataFrame(list(map(list, zip(*scores_mpp3_perts))), columns = columnsList)\n",
    "scores_mpp3_perts_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp3_perts_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp3_orig)):\n",
    "    columnsList.append('orig'+str(i+1))\n",
    "\n",
    "scores_mpp3_orig_df = pd.DataFrame(list(map(list, zip(*scores_mpp3_orig))), columns = columnsList)\n",
    "scores_mpp3_orig_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp3_orig_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MPP4 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp4_perts)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_mpp4_perts_df = pd.DataFrame(list(map(list, zip(*scores_mpp4_perts))), columns = columnsList)\n",
    "scores_mpp4_perts_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp4_perts_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp4_orig)):\n",
    "    columnsList.append('orig'+str(i+1))\n",
    "\n",
    "scores_mpp4_orig_df = pd.DataFrame(list(map(list, zip(*scores_mpp4_orig))), columns = columnsList)\n",
    "scores_mpp4_orig_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp4_orig_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MPP5 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp5)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_mpp5_df = pd.DataFrame(list(map(list, zip(*scores_mpp5))), columns = columnsList)\n",
    "scores_mpp5_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp5_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################\n",
    "\n",
    "################## MPP6 - Dumping the score into a dataframe for computation and in a CSV file for retrieving later ##################\n",
    "columnsList = []\n",
    "for i in range(len(scores_mpp6)):\n",
    "    columnsList.append('pert'+str(i+1))\n",
    "\n",
    "scores_mpp6_df = pd.DataFrame(list(map(list, zip(*scores_mpp6))), columns = columnsList)\n",
    "scores_mpp6_df.to_csv('generated_data_'+source_lang+'_en/scores_mpp6_df_'+tqmodel+'.csv', index = False, sep=';')\n",
    "######################################################################################################################################"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0363b137-873a-4f89-bc21-ae3205b047ca",
   "metadata": {},
   "source": [
    "### Once the scores are generated for a particular model and language pair, use these two code blocks below to read the scores from files. \n",
    "### If redoing experiments, this can save the time required to regenerate the scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f97bfaca-3e01-4ef0-8592-c07348f115cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_map1_perts_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map1_perts_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map1_perts = scores_map1_perts_df.transpose().to_numpy()\n",
    "scores_map1_orig_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map1_orig_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map1_orig = scores_map1_orig_df.transpose().to_numpy()\n",
    "\n",
    "scores_map2_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map2_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map2 = scores_map2_df.transpose().to_numpy()\n",
    "\n",
    "scores_map3_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map3_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map3 = scores_map3_df.transpose().to_numpy()\n",
    "\n",
    "scores_map3_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map3_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map3 = scores_map3_df.transpose().to_numpy()\n",
    "\n",
    "scores_map4_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map4_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map4 = scores_map4_df.transpose().to_numpy()\n",
    "\n",
    "scores_map5_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map5_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map5 = scores_map5_df.transpose().to_numpy()\n",
    "\n",
    "scores_map6_perts_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map6_perts_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map6_perts = scores_map6_perts_df.transpose().to_numpy()\n",
    "scores_map6_orig_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map6_orig_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map6_orig = scores_map6_orig_df.transpose().to_numpy()\n",
    "\n",
    "scores_map7_perts_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map7_perts_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map7_perts = scores_map7_perts_df.transpose().to_numpy()\n",
    "scores_map7_orig_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map7_orig_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map7_orig = scores_map7_orig_df.transpose().to_numpy()\n",
    "\n",
    "scores_map8_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_map8_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_map8 = scores_map8_df.transpose().to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ee59b6d-2716-4c67-b3b9-b1f505c1fd68",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_mpp1_perts_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp1_perts_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp1_perts = scores_mpp1_perts_df.transpose().to_numpy()\n",
    "scores_mpp1_orig_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp1_orig_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp1_orig = scores_mpp1_orig_df.transpose().to_numpy()\n",
    "\n",
    "scores_mpp2_perts_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp2_perts_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp2_perts = scores_mpp2_perts_df.transpose().to_numpy()\n",
    "scores_mpp2_orig_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp2_orig_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp2_orig = scores_mpp2_orig_df.transpose().to_numpy()\n",
    "\n",
    "scores_mpp3_perts_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp3_perts_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp3_perts = scores_mpp3_perts_df.transpose().to_numpy()\n",
    "scores_mpp3_orig_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp3_orig_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp3_orig = scores_mpp3_orig_df.transpose().to_numpy()\n",
    "\n",
    "scores_mpp4_perts_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp4_perts_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp4_perts = scores_mpp4_perts_df.transpose().to_numpy()\n",
    "scores_mpp4_orig_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp4_orig_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp4_orig = scores_mpp4_orig_df.transpose().to_numpy()\n",
    "\n",
    "scores_mpp5_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp5_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp5 = scores_mpp5_df.transpose().to_numpy()\n",
    "\n",
    "scores_mpp6_df = pd.read_csv('generated_data_'+source_lang+'_en/scores_mpp6_df_'+tqmodel+'.csv', sep=';')\n",
    "scores_mpp6 = scores_mpp6_df.transpose().to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db0ab26f-a3c2-43a5-99c0-04d31074033e",
   "metadata": {},
   "source": [
    "## Computing Relative Differences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baa9fa25-c5ed-408b-9f7d-d0517c958169",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_mean_map1_perts = np.mean(scores_map1_perts[0:],axis=0) ## Mean of 20 trials of perturbed scores.\n",
    "scores_mean_map1_orig = np.mean(scores_map1_orig[0:],axis=0) ## Mean of 20 trails of original translation scores\n",
    "scores_relative_diff_map1 = scores_mean_map1_orig - scores_mean_map1_perts ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_map2 = np.mean(scores_map2[1:],axis=0) ## Computing means for the 20 perturbed scores.\n",
    "scores_relative_diff_map2 = scores_map2[0] - scores_mean_map2 ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_map3 = np.mean(scores_map3[1:],axis=0) ## Computing means for the 20 perturbed scores.\n",
    "scores_relative_diff_map3 = scores_map3[0] - scores_mean_map3 ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_map4 = np.mean(scores_map4[1:],axis=0) ## Computing means for the 20 perturbed scores.\n",
    "scores_relative_diff_map4 = scores_map4[0] - scores_mean_map4 ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_map5 = np.mean(scores_map5[1:],axis=0) ## Computing means for the 20 perturbed scores.\n",
    "scores_relative_diff_map5 = scores_map5[0] - scores_mean_map5 ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_map6_perts = np.mean(scores_map6_perts[0:],axis=0) ## Mean of 20 trials of perturbed scores.\n",
    "scores_mean_map6_orig = np.mean(scores_map6_orig[0:],axis=0) ## Mean of 20 trails of original translation scores\n",
    "scores_relative_diff_map6 = scores_mean_map6_orig - scores_mean_map6_perts ## Computing relative difference \n",
    "\n",
    "scores_mean_map7_perts = np.mean(scores_map7_perts[0:],axis=0) ## Mean of 20 trials of perturbed scores.\n",
    "scores_mean_map7_orig = np.mean(scores_map7_orig[0:],axis=0) ## Mean of 20 trails of original translation scores\n",
    "scores_relative_diff_map7 = scores_mean_map7_orig - scores_mean_map7_perts ## Computing relative difference \n",
    "\n",
    "scores_mean_map8 = np.mean(scores_map8[1:],axis=0) ## Computing means for the 20 perturbed scores.\n",
    "scores_relative_diff_map8 = scores_map8[0] - scores_mean_map8 ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa1fc895-1047-41af-af2b-d1bcd619d9a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_mean_mpp1_perts = np.mean(scores_mpp1_perts[0:],axis=0) ## Mean of 20 trials of perturbed scores.\n",
    "scores_mean_mpp1_orig = np.mean(scores_mpp1_orig[0:],axis=0) ## Mean of 20 trails of original translation scores\n",
    "scores_relative_diff_mpp1 = scores_mean_mpp1_orig - scores_mean_mpp1_perts ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_mpp2_perts = np.mean(scores_mpp2_perts[0:],axis=0) ## Mean of 20 trials of perturbed scores.\n",
    "scores_mean_mpp2_orig = np.mean(scores_mpp2_orig[0:],axis=0) ## Mean of 20 trails of original translation scores\n",
    "scores_relative_diff_mpp2 = scores_mean_mpp2_orig - scores_mean_mpp2_perts ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_mpp3_perts = np.mean(scores_mpp3_perts[0:],axis=0) ## Mean of 20 trials of perturbed scores.\n",
    "scores_mean_mpp3_orig = np.mean(scores_mpp3_orig[0:],axis=0) ## Mean of 20 trails of original translation scores\n",
    "scores_relative_diff_mpp3 = scores_mean_mpp3_orig - scores_mean_mpp3_perts ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_mpp4_perts = np.mean(scores_mpp4_perts[0:],axis=0) ## Mean of 20 trials of perturbed scores.\n",
    "scores_mean_mpp4_orig = np.mean(scores_mpp4_orig[0:],axis=0) ## Mean of 20 trails of original translation scores\n",
    "scores_relative_diff_mpp4 = scores_mean_mpp4_orig - scores_mean_mpp4_perts ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_mpp5 = np.mean(scores_mpp5[1:],axis=0) ## Computing means for the 20 perturbed scores.\n",
    "scores_relative_diff_mpp5 = scores_mpp5[0] - scores_mean_mpp5 ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations.\n",
    "\n",
    "scores_mean_mpp6 = np.mean(scores_mpp6[1:],axis=0) ## Computing means for the 20 perturbed scores.\n",
    "scores_relative_diff_mpp6 = scores_mpp6[0] - scores_mean_mpp6 ## Computing relative difference among the original translation scores and scores (mean) obtained after perturbations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80bea75d-901a-498a-9f56-1123e9f9c5e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "map1_mean = np.mean(scores_relative_diff_map1)\n",
    "map2_mean = np.mean(scores_relative_diff_map2)\n",
    "map3_mean = np.mean(scores_relative_diff_map3)\n",
    "map4_mean = np.mean(scores_relative_diff_map4)\n",
    "map5_mean = np.mean(scores_relative_diff_map5)\n",
    "map6_mean = np.mean(scores_relative_diff_map6)\n",
    "map7_mean = np.mean(scores_relative_diff_map7)\n",
    "map8_mean = np.mean(scores_relative_diff_map8)\n",
    "\n",
    "mpp1_mean = np.mean(scores_relative_diff_mpp1)\n",
    "mpp2_mean = np.mean(scores_relative_diff_mpp2)\n",
    "mpp3_mean = np.mean(scores_relative_diff_mpp3)\n",
    "mpp4_mean = np.mean(scores_relative_diff_mpp4)\n",
    "mpp5_mean = np.mean(scores_relative_diff_mpp5)\n",
    "mpp6_mean = np.mean(scores_relative_diff_mpp6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3997c25f-fde9-4818-b307-909efcaa2619",
   "metadata": {},
   "outputs": [],
   "source": [
    "map1_var = np.var(scores_relative_diff_map1)\n",
    "map2_var = np.var(scores_relative_diff_map2)\n",
    "map3_var = np.var(scores_relative_diff_map3)\n",
    "map4_var = np.var(scores_relative_diff_map4)\n",
    "map5_var = np.var(scores_relative_diff_map5)\n",
    "map6_var = np.var(scores_relative_diff_map6)\n",
    "map7_var = np.var(scores_relative_diff_map7)\n",
    "map8_var = np.var(scores_relative_diff_map8)\n",
    "\n",
    "mpp1_var = np.var(scores_relative_diff_mpp1)\n",
    "mpp2_var = np.var(scores_relative_diff_mpp2)\n",
    "mpp3_var = np.var(scores_relative_diff_mpp3)\n",
    "mpp4_var = np.var(scores_relative_diff_mpp4)\n",
    "mpp5_var = np.var(scores_relative_diff_mpp5)\n",
    "mpp6_var = np.var(scores_relative_diff_mpp6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ed48a33-1a8f-4a06-93c2-575f792f2737",
   "metadata": {},
   "source": [
    "# Plotting and Dumping Means of relative differenecs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fbd9ec6-b11b-425a-bd33-f67f02b53682",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_mpp = np.array([mpp1_mean, mpp2_mean, mpp3_mean, mpp4_mean, mpp5_mean, mpp6_mean])\n",
    "err_mpp = np.array([mpp1_var, mpp2_var, mpp3_var, mpp4_var, mpp5_var, mpp6_var])\n",
    "rows = [\"RemovePunctuation\", \"ReplacePunctuation\", \"RemoveDeterminers\", \"ReplaceDeterminers\", \"UPPERCASE\", \"lowercase\"]\n",
    "lang = []\n",
    "for i in range(len(X_mpp)):\n",
    "    lang.append(source_lang)\n",
    "\n",
    "dfplot = pd.DataFrame(dict(Perturbations=rows, Differences=X_mpp, error=err_mpp, Language=lang))\n",
    "\n",
    "fig = px.scatter(dfplot, title=\"Plotting Relative Difference in MPPs\", y=\"Differences\", x=\"Perturbations\", error_y=\"error\", color=\"Language\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65cd322b-8c38-4ef1-9136-20e0fa3b1803",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_map = np.array([map1_mean, map2_mean, map3_mean, map4_mean, map5_mean, map6_mean, map7_mean, map8_mean])\n",
    "err_map = np.array([map1_var, map2_var, map3_var, map4_var, map5_var, map6_var, map7_var, map8_var])\n",
    "rows = [\"RemoveNegation\", \"RemoveWords\", \"DuplicateWords\", \"InsertWords\", \"ReplaceWords\", \"BERTSentenceReplace\", \"ReplaceWordsWithAntonyms\", \"SourceAsTarget\"]\n",
    "lang = []\n",
    "for i in range(len(X_map)):\n",
    "    lang.append(source_lang)\n",
    "\n",
    "dfplot = pd.DataFrame(dict(Perturbations=rows, Differences=X_map, error=err_map, Language=lang))\n",
    "\n",
    "fig = px.scatter(dfplot, title=\"Plotting Relative Difference in MAPs\", y=\"Differences\", x=\"Perturbations\", error_y=\"error\", color=\"Language\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b99c831f-ca83-48ed-b154-0475a3f018ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_both = np.array([mpp1_mean, mpp2_mean, mpp3_mean, mpp4_mean, mpp5_mean, mpp6_mean, map1_mean, map2_mean, map3_mean, map4_mean, map5_mean, map6_mean, map7_mean, map8_mean])\n",
    "err_both = np.array([mpp1_var, mpp2_var, mpp3_var, mpp4_var, mpp5_var, mpp6_var, map1_var, map2_var, map3_var, map4_var, map5_var, map6_var, map7_var, map8_var])\n",
    "rows = [\"RemovePunctuation\", \"ReplacePunctuation\", \"RemoveDeterminers\", \"ReplaceDeterminers\", \"UPPERCASE\", \"lowercase\", \n",
    "        \"RemoveNegation\", \"RemoveWords\", \"DuplicateWords\", \"InsertWords\", \"ReplaceWords\", \"BERTSentenceReplace\", \"ReplaceWordsWithAntonyms\", \"SourceAsTarget\"]\n",
    "\n",
    "lang = [\"MPP\", \"MPP\", \"MPP\", \"MPP\", \"MPP\", \"MPP\", \"MAP\", \"MAP\", \"MAP\", \"MAP\", \"MAP\", \"MAP\", \"MAP\", \"MAP\"]\n",
    "\n",
    "    \n",
    "dfplot = pd.DataFrame(dict(Perturbations=rows, Differences=X_both, error=err_both, Language=lang))\n",
    "\n",
    "fig = px.scatter(dfplot, title=\"Plotting Relative Difference in MPPs and MAPs\", y=\"Differences\", x=\"Perturbations\", error_y=\"error\", color=\"Language\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "632c73c1-1681-43df-ae66-4ad719220457",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p commonResults"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e9387b0-55cf-4b77-a5f5-55a5d8303c89",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.save(\"commonResults/\"+source_lang+\"Means_\"+tqmodel, X_both)\n",
    "np.save(\"commonResults/\"+source_lang+\"Errs_\"+tqmodel, err_both)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "173e7b6b-f34c-4677-bd1a-95e4725fbcfe",
   "metadata": {},
   "source": [
    "## From here onwards, you have dumped the relative differences for all the perturbations in terms of mean values and variances. \n",
    "## You should do this for all five language pairs, for each of the three models before being able to generate system rankings via \"getPearsonandRanking.ipynb\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}