Score Exams.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsupervised Machine Scoring of Free Response Answers—Validated Against Law School Final Exams\n",
    "\n",
    "This notebook implements the methods described in [Unsupervised Machine Scoring of Free Response Answers—Validated Against Law School Final Exams](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4040303), presented at the [Computational Legal Studies Conference](https://cclaw.smu.edu.sg/events/computational-legal-studies-2022), March 2022, hosted by the Center for Computational Law at Singapore Management University. Here is a link to the presentation's [slide deck](https://docs.google.com/presentation/d/15fxG3zoZSdUfmxFuVQZd2_MrDe1Z1rcsXwBDIvFIy3U/edit?usp=sharing), including image credits. \n",
    "\n",
    "Some of the data presented here differ slightly from that found in the version presented at the CLS Conference and in the slide deck above. This is due to additional work done in response to feedback received after presenting the paper. They now show the difference in performance between pseudo-random and machine ordering _after_ both machine and human markings are converted into z-scores. The translation of both scores into z-scores allowed the machine score to be compared with the human score using intraclass correlation ICC and Cohen's kappa. The CLS presentation compared the machine scores only to pseudo-random scores without translating the human scoring and after the machine score was transformed into a numerical score based on a standard grading scale (e.g., 90, 80, etc.). Older versions of this notebook with prior results can be found [here](https://github.com/colarusso/free-response-scoring/commits/main/Score%20Exams.ipynb).\n",
    "\n",
    "**Paper Summary**\n",
    "\n",
    "> This paper presents a novel method for unsupervised machine scoring of short answer and essay question responses, relying solely on a sufficiently large set of responses to a common prompt, absent the need for pre-labeled sample answers—given said prompt is of a particular character. That is, for questions where “good” answers look similar, “wrong” answers are likely to be “wrong” in different ways. Consequently, when a collection of text embeddings for responses to a common prompt are placed in an appropriate feature space, the centroid of their placements can stand in for a model answer, providing a lodestar against which to measure individual responses. This paper examines the efficacy of this method and discusses potential applications.\n",
    ">\n",
    ">Current methods for the automated scoring of short answer and essay questions are poorly suited to spontaneous and idiosyncratic assessments. That is, the time saved in grading must be balanced against the time required for the training of a model. This includes tasks such as the creation of pre-labeled sample answers. This limits the utility of machine grading for single classes working with novel assessments. The method described here eliminates the need for the preparation of pre-labeled sample answers. It is the author’s hope that such a method may be leveraged to reduce the time needed to grade free response questions, promoting the increased adoption of formative assessment esp. in contexts like law school instruction which traditionally have relied almost exclusively on summative assessments.\n",
    ">\n",
    ">Ranking by the algorithm is found to be statistically significant when compared to a pseudo-random shuffle. To determine how similar a list’s order was to that produced by a human grader, the lowest number of neighbor swaps needed to transform the ordering of these lists into that of the human ordering was calculated. For a dataset including more than one thousand student answers to a set of thirteen free response questions, drawn from six Suffolk University Law School final exams, taught by five instructors, the p-value for a paired t-test of the two populations’ swaps, with the pseudo-random group acting as the untreated group and the machine-grader acting as the treatment, came to 0.000000334, allowing us to reject the null hypothesis that the machine's ordering is equivalent to a random shuffle. Additionally, the Cohen’s d for the number of swaps between the pseudo-random ordering and machine ordering was found to be large (i.e., 1.03).\n",
    "\n",
    "This notebook was used to obtain the results referenced above. Student exam answers and their associated grade data were acquired and this research conducted after approval was granted by Suffolk University’s Office of Research and Sponsored Programs (ORSP) which oversees all human subject research at Suffolk University.\n",
    "\n",
    "This notebook implements the primary steps of the method described in the paper, namely:  \n",
    "\n",
    "1. Produce an embedding for each answer that captures as much of the relevant information as possible.\n",
    "2. Find the centroid for all of the embeddings in your population of answers and impute the location of a  “correct” model answer.\n",
    "3. Measure the distance between each answer’s embedding and the “correct” answer (e.g., the answers’ centroid or medoid).\n",
    "4. Convert the answers’ distances from the model answer into z-scores for the population of answers.\n",
    "5. Translate these z-scores into some known grading scale.\n",
    "6. Order the answers according to this scale.\n",
    "7. Compare these orderings to random orderings by seeing how many times you have to change their rankings to obtain the same ordering as one would get if they were ordered by their human-assigned grades.\n",
    "\n",
    "_Note: The methods described here are the subject of USPTO Patent Application Serial No. 17/246,563._\n",
    "\n",
    "## Contents\n",
    "\n",
    "_These, and other internal links (e.g., \"back to contents\"), may not work if you are viewing the preview of this notebook on GitHub.com. Links to other notebooks, external sites, and exam questions should, however, be okay._\n",
    "\n",
    "This notebook has three sections:\n",
    "\n",
    "- [Data: Student Exam Answers](#Data:-Student-Exam-Answers)\n",
    "- [Code:Code: Libraries and Novel Functions](#Code:-Libraries-and-Novel-Functions)\n",
    "- [Results](#Results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data: Student Exam Answers\n",
    "\n",
    "More than one thousand exam answers were obtained as PDF files containing answers to thirteen free response questions, drawn from six Suffolk University Law School final exams, taught by five instructors. Each PDF corresponded to a single student exam. These PDFs were parsed to extract their answers, their contents converted into XML of the following format.\n",
    "\n",
    "```\n",
    "<EXAM>\n",
    "    <STUDENT id='ID'>00000000</STUDENT>\n",
    "    <QUESTION id='Q1'>\n",
    "        <![CDATA[\n",
    "            text of written answer to question one\n",
    "        ]]>\n",
    "    </QUESTION>\n",
    "    <QUESTION id='Q2'>\n",
    "        <![CDATA[\n",
    "            text of written answer to question two\n",
    "        ]]>\n",
    "    </QUESTION>\n",
    "    <QUESTION id='Q3'>\n",
    "        <![CDATA[\n",
    "            text of written answer to question three\n",
    "        ]]>\n",
    "    </QUESTION>\n",
    "</EXAM>\n",
    "```\n",
    "\n",
    "These translated files were reviewed by hand and reformatted as needed to correct any formatting errors. Each XML file was then read into a csv file for its associated exam as a single row with their columns corresponding to each of its elements (e.g., ID, Q1, Q2). Additionally, a column stating the number of words contained in each question was appended to the csv file (e.g., size_Q1, size_Q2). E.g., \n",
    "\n",
    "|ID|Q1|Q2|size_Q1|size_Q2|\n",
    "|--|--|--|-------|-------|\n",
    "|00001|text of 1's ans to q1|text of 1's ans to q2|6|6|\n",
    "|00002|text of 2's ans to q1|text of 2's ans to q2|6|6|\n",
    "\n",
    "Instructors also provided scores for each exam question. These were placed in a csv for each exam with the scores on each question associated to the exam ID. E.g., \n",
    "\n",
    "|ID|Q1|Q2|\n",
    "|--|--|--|\n",
    "|00001|96|93|\n",
    "|00002|83|89|\n",
    "\n",
    "A PDF did not extist for every exam as some exams were hand-written. Also, for the _Property Instructor A_ exam, the IDs of two PDFs did not macth any of the IDs provided in the score sheet. To address such issues, you will see that we merge the two data tables above using the default behavior of Panda's merge method (i.e., we kept only the intersection of ID, those found in both files). Consequently, we did not score the exams for which no PDF existed or those for which we could not find a score on the intructor-provided list of scores.\n",
    "\n",
    "To see the code used to assist in this processing see the following notebook: [Prep Exams](Prep%20Exams.ipynb).\n",
    "\n",
    "In keeping with the wishes of those instructors who provided exams, only three of the exams are available here for review (i.e., [property_instructor_A](https://colarusso.github.io/free-response-scoring/data/property_instructor_A/property_instructor_A.docx), [property_instructor_B](https://colarusso.github.io/free-response-scoring/data/property_instructor_B/property_instructor_B.docx), and [crim_instructor_E](https://colarusso.github.io/free-response-scoring/data/crim_instructor_E/crim_instructor_E.docx)). Another two are on file with the author and may be shared upon request and the assent of their authors. In keeping with the author's wishes, the remaining exam will not be shared. Subject to constraints imposed by the Family Educational Rights and Privacy Act (FERPA), at least three of the answer sets may be shared upon request. One of these is linked to an exam requiring instructor assent to be shared. I am in the process of reviewing and cleaning these data for PII. You can see those data that have been cleaned and shared without need for instructor assent [here](https://github.com/colarusso/free-response-scoring/tree/main/data) and track my progress [here](https://twitter.com/Colarusso/status/1494781654237425666?s=20&t=g0Zt5f3wDnVys6oH3nVSGQ).\n",
    "\n",
    "Please note that the exam questiones shared at the links above are docx files and have been redacted to exclude the instructor's name and text that does not include the scored question prompts (e.g., multiple choice questions). \n",
    "\n",
    "To facilitate retrieval of the exam answers which were stored in various folders, a list of dictionaries is defined. Each dictionary defines the folder name where csv files can be found as well as the names of its relevant columns for later consideration (i.e., their ID and those questions to be scored). To avoid sharing of this data publicly, as seen below, these files are not included in this repository and were located outside of this repository's folder (i.e., `../data/`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "exams = [\n",
    "            {\"folder\":\"../data/property_instructor_A\",\"columns\":[\"ID\",\"SHORT_ANS\",\"Q1\",\"Q2\"]},\n",
    "            {\"folder\":\"../data/property_instructor_B\",\"columns\":[\"ID\",\"Q1\",\"Q2\"]},\n",
    "            {\"folder\":\"../data/environ_instructor_B\",\"columns\":[\"ID\",\"Q2\"]},\n",
    "            {\"folder\":\"../data/PR_instructor_C\",\"columns\":[\"ID\",\"Q1\",\"Q2\"]},\n",
    "            {\"folder\":\"../data/contracts_instructor_D\",\"columns\":[\"ID\",\"Q1\",\"Q2\",\"Q3\"]},\n",
    "            {\"folder\":\"../data/crim_instructor_E\",\"columns\":[\"ID\",\"Q1\",\"Q2\"]}\n",
    "          ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to contents](#Contents)\n",
    "\n",
    "## Code: Libraries and Novel Functions\n",
    "\n",
    "### Python Libraries et al.\n",
    "\n",
    "First we'll load the following libraries. These are needed to perform our scoring. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from numpy import var\n",
    "from math import sqrt\n",
    "import spacy\n",
    "from scipy import stats\n",
    "from sklearn.preprocessing import normalize\n",
    "from sklearn.metrics import cohen_kappa_score\n",
    "import pingouin as pg\n",
    "\n",
    "# https://spacy.io/models/en#en_core_web_lg\n",
    "nlp1 = spacy.load('en_core_web_lg') \n",
    "# https://github.com/explosion/spacy-transformers/tree/88814f5f4be7f0d4c784d8500c558d9ba06b9a56\n",
    "nlp2 = spacy.load(\"en_trf_distilbertbaseuncased_lg\") \n",
    "nlp3 = spacy.load(\"en_trf_robertabase_lg\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def score_exams(exams,model='nlp1',normv=0,score=0,goal=\"centroid\",runs=1):\n",
    "\n",
    "    # list of swaps needed to move from machine-scored ordering to that of the graded exams\n",
    "    M = []  \n",
    "    # list of swaps needed to move from machine-scored ordering to that of the graded exams\n",
    "    R = []\n",
    "\n",
    "    # for every exam in the list of exams  \n",
    "    for exam in exams:\n",
    "\n",
    "        print(\"\\n=========================================================\")\n",
    "        print(exam[\"folder\"])\n",
    "        print(\"=========================================================\\n\")\n",
    "\n",
    "        # load this exam's answer texts\n",
    "        texts = pd.read_csv(\"%s/texts_redacted.csv\"%exam[\"folder\"], encoding=\"utf-8\").sample(frac=1)\n",
    "        # load this exam's grades\n",
    "        actual = pd.read_csv(\"%s/actual_redacted.csv\"%exam[\"folder\"], encoding=\"utf-8\").sample(frac=1)\n",
    "        \n",
    "        # print the relevant columns for this exam (i.e., ID and questions to score)\n",
    "        print(exam[\"columns\"])        \n",
    "\n",
    "        # score the questions listed above\n",
    "        output_df = grade(texts[texts[\"ID\"].isin(actual[\"ID\"])][exam[\"columns\"]],model,normv,score,goal)\n",
    "        \n",
    "        if score==2:\n",
    "            # convert human grades to z scores if score==2\n",
    "            print(\"Computing z scores for human scoring\")\n",
    "            actual = z_scores_from_grades(actual)\n",
    "            # create a df of the format needed for ICC\n",
    "            df_output_df = output_df.copy()\n",
    "            df_output_df[\"judge\"] = \"Machine\"\n",
    "            df_actual = actual.copy()\n",
    "            df_actual[\"judge\"] = \"Human\"\n",
    "            df_icc = pd.concat([df_actual,df_output_df], ignore_index=True)\n",
    "\n",
    "        # below we want the average word counts for questions\n",
    "        # so we need a df with only the questions actually used here\n",
    "        # i.e., the inner merge. output_df doesn't have the texts columns\n",
    "        # so we'll do a second merge here real quick to facilitate that count\n",
    "        texts_df = actual.merge(texts, on=\"ID\")\n",
    "\n",
    "        # only include grades for those questions defined above\n",
    "        actual = actual[[x for x in actual.columns if x in exam[\"columns\"]]]\n",
    "\n",
    "        # merge the machine-scored dataframe with that of the actual grades\n",
    "        df_exam = actual.merge(output_df, on=\"ID\")\n",
    "        \n",
    "        # display a preview of the two scores (first few rows of the merged dataframe)\n",
    "        # I'm excluding the ID here to avoid sharing IDs in the cell's output\n",
    "        #display(df_exam[[x for x in list(df_exam.columns.values) if (x not in [\"ID\"])]].head())\n",
    "        display(df_exam.head())\n",
    "\n",
    "        print(\"Number of entries:\",len(df_exam))\n",
    "\n",
    "        # for each of the questions that were scored\n",
    "        for qn in [x for x in exam[\"columns\"] if (x not in [\"ID\"])]:\n",
    "            mean_words = int(round(texts_df[\"size_%s\"%qn].mean()))\n",
    "            print(\"\\n%s\\nMean Words:\"%qn,mean_words)\n",
    "\n",
    "            # create a dataframe with the grade and machine scores for this question\n",
    "            # drop any nans\n",
    "            df = df_exam[[\"ID\", \"%s_x\"%qn, \"%s_y\"%qn]].dropna()\n",
    "            \n",
    "            if score==2:\n",
    "                # get ICC for score==2 (i.e., when both scores are z scores)\n",
    "                df_icc_tmp = df_icc.copy()\n",
    "                df_icc_tmp[\"score\"] = df_icc_tmp[qn]\n",
    "                df_icc_tmp = df_icc_tmp[[\"ID\",\"judge\",\"score\"]]\n",
    "                display(df_icc_tmp.head()) \n",
    "                icc = pg.intraclass_corr(data=df_icc_tmp, targets='ID', raters='judge', ratings='score',nan_policy='omit')\n",
    "                display(icc.set_index('Type'))\n",
    "                \n",
    "                # Kappa isn't meant for continuous scoring, so we're binning the z score \n",
    "                # multipling by ten and rounding the product. \n",
    "                kappa = cohen_kappa_score(round(df[\"%s_x\"%qn]*10), round(df[\"%s_y\"%qn]*10),weights=\"quadratic\")\n",
    "                print(\"Cohen's kappa for rounded(z scores x 10):\",kappa)\n",
    "\n",
    "            # create an list of these scores then count the swaps needed to bring them into \n",
    "            # agreement with the grades, averaged over `runs`. Append this to our list of \n",
    "            # swaps needed to move from machine-scored ordering to that of the graded exams. \n",
    "            # we're shuffling before sorting to address the fact that the original order of\n",
    "            # rows can effect the swaps as the order of actual scores could be different\n",
    "            # within the bins provided by the machine grade \n",
    "            tmp = []\n",
    "            for x in range(runs):\n",
    "                # shuffle df then sort answers by the machine scores\n",
    "                # this is needed becaue the `actual` rows can effect the number \n",
    "                # of swaps, so we want to get a represnative sample over `runs`\n",
    "                # note the use of `sample.` This is where we shuffle the base df\n",
    "                df = df.sample(frac=1).sort_values(by='%s_y'%qn, ascending=True)\n",
    "                \n",
    "                arr_ = df[df.columns[1]].values.copy()\n",
    "                n = len(arr_)\n",
    "                M_o = countSwaps(arr_, n)\n",
    "                tmp.append(M_o)\n",
    "\n",
    "            M_o = np.array(tmp).mean()\n",
    "            M.append(M_o)\n",
    "\n",
    "            # we're going to do the same as the above but for pseudo-random shuffles.\n",
    "            # we'll shuffle and count swaps for the number of times defined in `runs`\n",
    "            # and average the results \n",
    "            tmp = []\n",
    "            for x in range(runs):\n",
    "                # note the use of `sample.` This is where we get our pseudo-random ordering\n",
    "                arr_ = df[df.columns[1]].sample(frac=1).values.copy()\n",
    "                n = len(arr_)\n",
    "                R_o = countSwaps(arr_, n)\n",
    "                tmp.append(R_o)\n",
    "\n",
    "            R_o = np.array(tmp).mean()\n",
    "            R.append(R_o)\n",
    "            \n",
    "            print(\"Algo score, swaps needed:\",M_o)\n",
    "            print(\"Swaps needed for first 10 out of %s pseudo-random runs:\"%runs)\n",
    "            print(tmp[:10])\n",
    "            print(\"Pseudo-random, avergae swaps needed:\",R_o)\n",
    "            \n",
    "    print(\"\\n=========================================================\")\n",
    "    print(\"Swaps (Machine):\")\n",
    "    print(M)\n",
    "    print(\"N:\",len(M),\"\\tMean: \",np.mean(M),\"\\tVar: \",np.var(M))\n",
    "    print(\"Avergae swaps (Pseudo-random):\")\n",
    "    print(R)\n",
    "    print(\"N:\",len(R),\"\\tMean: \",np.mean(R),\"\\tVar: \",np.var(R),\"\\n\")\n",
    "    \n",
    "    \n",
    "    # Print out a summary table of the percent difference between pseudo-random and machine swaps\n",
    "    pdiff = []\n",
    "    i = 0\n",
    "    for x in R:\n",
    "        if x != 0 and M[i] != 0:\n",
    "            pdiff_ = 100*(x-M[i])/((x+M[i])/2)\n",
    "            pdiff.append(pdiff_)\n",
    "            print( i+1,\")\",x,\">\",M[i],\"=>\",pdiff_ )\n",
    "        else:\n",
    "            print(\"error\")\n",
    "        i = i + 1\n",
    "\n",
    "    print(\"\\nAvergae percent difference between swaps needed for pseudo-random and machine sorting:\")\n",
    "    print(np.array(pdiff).mean(),\"\\n\")\n",
    "        \n",
    "    # run t test for the two populations of swaps\n",
    "    sample_n = len(M)\n",
    "    print(stats.ttest_rel(M[:sample_n], R[:sample_n]))\n",
    "    # return the effect size\n",
    "    print(\"Cohen's d: \",cohend(R[:sample_n], M[:sample_n]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def grade(df,model,normv,score=0,goal=\"centroid\"):\n",
    "       \n",
    "    df = df.copy()    \n",
    "    \n",
    "    # exclude the ID column to get only the answer texts\n",
    "    questions = [x for x in list(df.columns.values) if (x not in [\"ID\"])]\n",
    "    # vectorize the text of each answer\n",
    "    for x in questions:\n",
    "        df[\"%s_vec\"%x] = df[x].apply(vectorize, args=(model,normv)) \n",
    "    \n",
    "    # for each question\n",
    "    for x in questions:\n",
    "        \n",
    "        # based on the `goal` parameter grade, against either the centroid or medoid\n",
    "        # that is, set either as the model answer aginst which to grade others\n",
    "        if goal==\"centroid\":\n",
    "            # centroid \n",
    "            best_ans = df[\"%s_vec\"%x].mean()\n",
    "        else:\n",
    "            # medoid \n",
    "            best_ans = df[\"%s_vec\"%x].iloc[best_answer(df,x)]\n",
    "\n",
    "        # measure each answer's distance from the model answer above\n",
    "        df[\"%s_dist\"%x] = df[\"%s_vec\"%x].apply(distance, args=(best_ans,))\n",
    "        \n",
    "        # replace the text of the answer with its z-score\n",
    "        df[x] =  df[\"%s_vec\"%x].apply(z_score, args=(df,x,best_ans))\n",
    "        \n",
    "        # translate the z-score into a standard grade using\n",
    "        # the translation defined by `score`\n",
    "        if score==0:\n",
    "            df[x] =  df[x].apply(score_0)\n",
    "        elif score==1:\n",
    "            df[x] =  df[x].apply(score_1)\n",
    "        elif score==2:\n",
    "            print(\"Leaving z scores in place\")\n",
    "            \n",
    "    # create output df without the vectors or distance measures\n",
    "    output_df = df[[ x for x in list(df.columns.values) if (\"_vec\" not in x) and (\"_dist\" not in x) ]].copy()\n",
    "            \n",
    "    return output_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def vectorize(row,model='nlp1',normv=0): \n",
    "\n",
    "    # use a method for vectorization based on the `model` parameter\n",
    "    # note: some methods can work across the entire text of a qustion\n",
    "    # others requier vectorization at the sentence level\n",
    "    if model=='nlp1':\n",
    "        nlp = nlp1\n",
    "        mode = 0\n",
    "    elif model=='nlp2':\n",
    "        nlp = nlp2\n",
    "        mode = 1\n",
    "    elif model=='nlp3':\n",
    "        nlp = nlp3\n",
    "        mode = 1\n",
    "        \n",
    "    try:\n",
    "        if mode == 1:\n",
    "            # vectorize at the sentence level\n",
    "            doc = nlp(row)\n",
    "            tmp_df = pd.DataFrame({'sent' : []})\n",
    "            i=0 \n",
    "            for sent in doc.sents:\n",
    "                sent_vec = nlp(sent.string.strip()).vector\n",
    "                # if normv=1 normalize each sentence vector\n",
    "                if normv==1:\n",
    "                    sent_vec = norm(sent_vec)\n",
    "                if i==0:\n",
    "                    tmp_df = pd.DataFrame([[sent_vec]],columns=[\"sent\"])\n",
    "                else:\n",
    "                    tmp_df = tmp_df.append(pd.DataFrame([[sent_vec]],columns=[\"sent\"]), ignore_index=True)\n",
    "                i+=1\n",
    "            # average sentence vectors \n",
    "            output = tmp_df[\"sent\"].mean()\n",
    "        else:\n",
    "            # vectorize all input text together\n",
    "            output = nlp(row).vector\n",
    "\n",
    "        # if normv=1 normalize the answer's vector\n",
    "        if normv==1:\n",
    "            output = norm(output)  \n",
    "    \n",
    "        return output\n",
    "    except:\n",
    "        return np.NaN\n",
    "    \n",
    "def norm(row):\n",
    "    # normalize vector\n",
    "    matrix = row.reshape(1,-1).astype(np.float64)\n",
    "    return normalize(matrix, axis=1, norm='l1')[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def best_answer(df,column):\n",
    "    # Return the medoid (like the centroid, but confined to a member of the group) \n",
    "    # i.e., the best answer in the group of answers.\n",
    "    tmp_df = df[[ \"%s_vec\"%column ]].copy()\n",
    "    centroid = tmp_df[\"%s_vec\"%column].mean()\n",
    "    tmp_df[\"%s_dist\"%column] = tmp_df[\"%s_vec\"%column].apply(distance, args=(centroid,))\n",
    "    tmp_df = tmp_df[tmp_df[\"%s_dist\"%column]==tmp_df[\"%s_dist\"%column].min()]\n",
    "    return tmp_df.index[0]\n",
    "\n",
    "def distance(row,centroid):\n",
    "    # calculate the Euclidean distance between centroid and row\n",
    "    # See https://stackoverflow.com/a/1401828\n",
    "    return np.linalg.norm(centroid-row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def z_score(row,df,x,best):\n",
    "    # calculate the z-score for a given distance\n",
    "    return -1*(np.linalg.norm(best-row)-df[\"%s_dist\"%x].mean())/df[\"%s_dist\"%x].std()\n",
    "\n",
    "def z_scores_from_grades(df):\n",
    "    questions = [x for x in list(df.columns.values) if (x not in [\"ID\"])]\n",
    "    for x in questions:\n",
    "        df[x] = df[x].apply(z_score_tradional, args=(df,x))\n",
    "    return df\n",
    "\n",
    "def z_score_tradional(row,df,x):\n",
    "    # calculate the z-score for a given score\n",
    "    return ((row-df[x].mean())/df[x].std())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define grade bin boarders based on z-score\n",
    "# --------------------------------------\n",
    "# There's no standard mapping between z-scores and letter grades\n",
    "# However, it's common for schools to set their own curve which \n",
    "# is in the same spirit. Therefor I have created a mapping where\n",
    "# the average answer recieves a B. Others might choose differently\n",
    "\n",
    "grade_bins = [\n",
    "              2,    # A+\n",
    "              1.5,  # A\n",
    "              1,    # A-\n",
    "              0.5,  # B+\n",
    "              0,    # B\n",
    "              -0.5, # B-\n",
    "              -1,   # C+\n",
    "              -1.5, # C\n",
    "              -2,   # C-\n",
    "              -2.5, # D+\n",
    "              -3,   # D\n",
    "              -3.5  # D-\n",
    "             ]\n",
    "\n",
    "# Assign numeric grades based on z-score\n",
    "# -----------------------------------------\n",
    "# This was the original z-score to percentage grade translation.\n",
    "# I took the above and translated it into numberic grades\n",
    "# with the scale bottoming out at 59 and topping out at 97\n",
    "def score_0(row):\n",
    "    if row >= grade_bins[0]:\n",
    "        grade = 97\n",
    "    elif row >= grade_bins[1]:\n",
    "        grade = 93+((row-grade_bins[1])/(grade_bins[0]-grade_bins[1]))*4\n",
    "    elif row >= grade_bins[2]:\n",
    "        grade = 90+((row-grade_bins[2])/(grade_bins[1]-grade_bins[2]))*3\n",
    "    elif row >= grade_bins[3]:\n",
    "        grade = 87+((row-grade_bins[3])/(grade_bins[2]-grade_bins[3]))*3\n",
    "    elif row >= grade_bins[4]:\n",
    "        grade = 83+((row-grade_bins[4])/(grade_bins[3]-grade_bins[4]))*4\n",
    "    elif row >= grade_bins[5]:\n",
    "        grade = 80+((row-grade_bins[5])/(grade_bins[4]-grade_bins[5]))*3\n",
    "    elif row >= grade_bins[6]:\n",
    "        grade = 77+((row-grade_bins[6])/(grade_bins[5]-grade_bins[6]))*3\n",
    "    elif row >= grade_bins[7]:\n",
    "        grade = 73+((row-grade_bins[7])/(grade_bins[6]-grade_bins[7]))*4\n",
    "    elif row >= grade_bins[8]:\n",
    "        grade = 70+((row-grade_bins[8])/(grade_bins[7]-grade_bins[8]))*3\n",
    "    elif row >= grade_bins[9]:\n",
    "        grade = 67+((row-grade_bins[9])/(grade_bins[8]-grade_bins[9]))*3\n",
    "    elif row >= grade_bins[10]:\n",
    "        grade = 63+((row-grade_bins[10])/(grade_bins[9]-grade_bins[10]))*4\n",
    "    elif row >= grade_bins[11]:\n",
    "        grade = 60+(row-grade_bins[11])*(3/(grade_bins[10]-grade_bins[11]))\n",
    "    else:\n",
    "        grade = 59\n",
    "        \n",
    "    # The final output is rounded to avoid false percision. A better \n",
    "    # approach is to bin all of the scores. This is done in score_1 below\n",
    "    return round(grade)\n",
    "\n",
    "# Assign numeric grades based on z-score\n",
    "# -----------------------------------------\n",
    "# Upon subsequent reflection, I realized that binned scores were preferable\n",
    "# to a continuous score with dropoffs at 59 and 97. The above translation,\n",
    "# however, was used on the first run of this code and so is included here\n",
    "# for compleatness.  \n",
    "def score_1(row):\n",
    "    if row >= grade_bins[0]:\n",
    "        grade = 97\n",
    "    elif row >= grade_bins[1]:\n",
    "        grade = 93\n",
    "    elif row >= grade_bins[2]:\n",
    "        grade = 90\n",
    "    elif row >= grade_bins[3]:\n",
    "        grade = 87\n",
    "    elif row >= grade_bins[4]:\n",
    "        grade = 83\n",
    "    elif row >= grade_bins[5]:\n",
    "        grade = 80\n",
    "    elif row >= grade_bins[6]:\n",
    "        grade = 77\n",
    "    elif row >= grade_bins[7]:\n",
    "        grade = 73\n",
    "    elif row >= grade_bins[8]:\n",
    "        grade = 70\n",
    "    elif row >= grade_bins[9]:\n",
    "        grade = 67\n",
    "    elif row >= grade_bins[10]:\n",
    "        grade = 63\n",
    "    elif row >= grade_bins[11]:\n",
    "        grade = 60\n",
    "    else:\n",
    "        grade = 59\n",
    "    return grade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following functions are used to count the number of adjacent swaps needed\n",
    "# to change one ordering into another. It was written by Shivam Gupta\n",
    "# See https://www.geeksforgeeks.org/number-swaps-sort-adjacent-swapping-allowed/\n",
    "# -----------------------------------------------------\n",
    "# python 3 program to count number of swaps required \n",
    "# to sort an array when only swapping of adjacent \n",
    "# elements is allowed. \n",
    "# include <bits/stdc++.h> \n",
    "\n",
    "#This function merges two sorted arrays and returns inversion count in the arrays.*/ \n",
    "def merge(arr, temp, left, mid, right): \n",
    "    inv_count = 0\n",
    "\n",
    "    i = left #i is index for left subarray*/ \n",
    "    j = mid #i is index for right subarray*/ \n",
    "    k = left #i is index for resultant merged subarray*/ \n",
    "    while ((i <= mid - 1) and (j <= right)): \n",
    "        if (arr[i] <= arr[j]): \n",
    "            temp[k] = arr[i] \n",
    "            k += 1\n",
    "            i += 1\n",
    "        else: \n",
    "            temp[k] = arr[j] \n",
    "            k += 1\n",
    "            j += 1\n",
    "\n",
    "            #this is tricky -- see above explanation/ \n",
    "            # diagram for merge()*/ \n",
    "            inv_count = inv_count + (mid - i) \n",
    "\n",
    "    #Copy the remaining elements of left subarray \n",
    "    # (if there are any) to temp*/ \n",
    "    while (i <= mid - 1): \n",
    "        temp[k] = arr[i] \n",
    "        k += 1\n",
    "        i += 1\n",
    "\n",
    "    #Copy the remaining elements of right subarray \n",
    "    # (if there are any) to temp*/ \n",
    "    while (j <= right): \n",
    "        temp[k] = arr[j] \n",
    "        k += 1\n",
    "        j += 1\n",
    "\n",
    "    # Copy back the merged elements to original array*/ \n",
    "    for i in range(left,right+1,1): \n",
    "        arr[i] = temp[i] \n",
    "\n",
    "    return inv_count \n",
    "\n",
    "#An auxiliary recursive function that sorts the input \n",
    "# array and returns the number of inversions in the \n",
    "# array. */ \n",
    "def _mergeSort(arr, temp, left, right): \n",
    "    inv_count = 0\n",
    "    if (right > left): \n",
    "        # Divide the array into two parts and call \n",
    "        #_mergeSortAndCountInv() \n",
    "        # for each of the parts */ \n",
    "        mid = int((right + left)/2) \n",
    "\n",
    "        #Inversion count will be sum of inversions in \n",
    "        # left-part, right-part and number of inversions \n",
    "        # in merging */ \n",
    "        inv_count = _mergeSort(arr, temp, left, mid) \n",
    "        inv_count += _mergeSort(arr, temp, mid+1, right) \n",
    "\n",
    "        # Merge the two parts*/ \n",
    "        inv_count += merge(arr, temp, left, mid+1, right) \n",
    "\n",
    "    return inv_count \n",
    "\n",
    "#This function sorts the input array and returns the \n",
    "#number of inversions in the array */ \n",
    "def countSwaps(arr, n): \n",
    "    temp = [0 for i in range(n)] \n",
    "    return _mergeSort(arr, temp, 0, n - 1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following function calculates Cohen d (the effect size)\n",
    "# based on two populations. It was written by Jason Brownlee\n",
    "# See https://machinelearningmastery.com/effect-size-measures-in-python/\n",
    "# function to calculate Cohen's d for independent samples\n",
    "def cohend(d1, d2):\n",
    "    # calculate the size of samples\n",
    "    n1, n2 = len(d1), len(d2)\n",
    "    # calculate the variance of the samples\n",
    "    s1, s2 = np.var(d1, ddof=1), np.var(d2, ddof=1)\n",
    "    # calculate the pooled standard deviation\n",
    "    s = np.sqrt(((n1 - 1) * s1 + (n2 - 1) * s2) / (n1 + n2 - 2))\n",
    "    # calculate the means of the samples\n",
    "    u1, u2 = np.mean(d1), np.mean(d2)\n",
    "    # calculate the effect size\n",
    "    return (u1 - u2) / s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to contents](#Contents)\n",
    "\n",
    "# Results\n",
    "\n",
    "A subset of the data presented in the cell below can be found in Table 1 of _Unsupervised Machine Scoring of Free Response Answers—Validated Against Law School Final Exams_. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>f346f20d-4883-4fea-af93-f716f6e99274</td>\n",
       "      <td>1.521905</td>\n",
       "      <td>-0.147495</td>\n",
       "      <td>-0.386848</td>\n",
       "      <td>1.043495</td>\n",
       "      <td>0.453994</td>\n",
       "      <td>0.028075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ed56f2bc-52d8-4323-8f41-2cb80548a1dc</td>\n",
       "      <td>-0.159269</td>\n",
       "      <td>0.264623</td>\n",
       "      <td>0.650466</td>\n",
       "      <td>-0.487243</td>\n",
       "      <td>-1.628164</td>\n",
       "      <td>-0.336026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0</td>\n",
       "      <td>-0.999856</td>\n",
       "      <td>-1.486880</td>\n",
       "      <td>-0.535036</td>\n",
       "      <td>0.636763</td>\n",
       "      <td>0.584401</td>\n",
       "      <td>0.496988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7b622388-4835-41b0-916e-bb4ca373d6c4</td>\n",
       "      <td>1.802101</td>\n",
       "      <td>0.264623</td>\n",
       "      <td>1.539592</td>\n",
       "      <td>0.835584</td>\n",
       "      <td>0.177698</td>\n",
       "      <td>0.284655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dcef7f09-f784-48cb-b65e-59d0293f4195</td>\n",
       "      <td>0.961514</td>\n",
       "      <td>0.676742</td>\n",
       "      <td>2.576906</td>\n",
       "      <td>0.462191</td>\n",
       "      <td>0.062287</td>\n",
       "      <td>0.393672</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x      Q1_x      Q2_x  \\\n",
       "0  f346f20d-4883-4fea-af93-f716f6e99274     1.521905 -0.147495 -0.386848   \n",
       "1  ed56f2bc-52d8-4323-8f41-2cb80548a1dc    -0.159269  0.264623  0.650466   \n",
       "2  ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0    -0.999856 -1.486880 -0.535036   \n",
       "3  7b622388-4835-41b0-916e-bb4ca373d6c4     1.802101  0.264623  1.539592   \n",
       "4  dcef7f09-f784-48cb-b65e-59d0293f4195     0.961514  0.676742  2.576906   \n",
       "\n",
       "   SHORT_ANS_y      Q1_y      Q2_y  \n",
       "0     1.043495  0.453994  0.028075  \n",
       "1    -0.487243 -1.628164 -0.336026  \n",
       "2     0.636763  0.584401  0.496988  \n",
       "3     0.835584  0.177698  0.284655  \n",
       "4     0.462191  0.062287  0.393672  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>f346f20d-4883-4fea-af93-f716f6e99274</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.521905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>065476cf-e25b-4039-a5b0-69bfd27e61dd</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.999856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ed56f2bc-52d8-4323-8f41-2cb80548a1dc</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.159269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.999856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7b622388-4835-41b0-916e-bb4ca373d6c4</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.802101</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  f346f20d-4883-4fea-af93-f716f6e99274  Human  1.521905\n",
       "1  065476cf-e25b-4039-a5b0-69bfd27e61dd  Human -0.999856\n",
       "2  ed56f2bc-52d8-4323-8f41-2cb80548a1dc  Human -0.159269\n",
       "3  ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0  Human -0.999856\n",
       "4  7b622388-4835-41b0-916e-bb4ca373d6c4  Human  1.802101"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.426259</td>\n",
       "      <td>2.485892</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000030</td>\n",
       "      <td>[0.23, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.424240</td>\n",
       "      <td>2.455869</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000041</td>\n",
       "      <td>[0.23, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.421274</td>\n",
       "      <td>2.455869</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000041</td>\n",
       "      <td>[0.22, 0.58]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.597730</td>\n",
       "      <td>2.485892</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000030</td>\n",
       "      <td>[0.38, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.595742</td>\n",
       "      <td>2.455869</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000041</td>\n",
       "      <td>[0.37, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.592812</td>\n",
       "      <td>2.455869</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000041</td>\n",
       "      <td>[0.37, 0.74]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.426259  2.485892   80   81  0.000030   \n",
       "ICC2      Single random raters  0.424240  2.455869   80   80  0.000041   \n",
       "ICC3       Single fixed raters  0.421274  2.455869   80   80  0.000041   \n",
       "ICC1k  Average raters absolute  0.597730  2.485892   80   81  0.000030   \n",
       "ICC2k    Average random raters  0.595742  2.455869   80   80  0.000041   \n",
       "ICC3k     Average fixed raters  0.592812  2.455869   80   80  0.000041   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.23, 0.59]  \n",
       "ICC2   [0.23, 0.59]  \n",
       "ICC3   [0.22, 0.58]  \n",
       "ICC1k  [0.38, 0.74]  \n",
       "ICC2k  [0.37, 0.74]  \n",
       "ICC3k  [0.37, 0.74]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.446784149383303\n",
      "Algo score, swaps needed: 998.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1624, 1536, 1402, 1685, 1434, 1744, 1401, 1249, 1551, 1652]\n",
      "Pseudo-random, avergae swaps needed: 1499.68528\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>f346f20d-4883-4fea-af93-f716f6e99274</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.147495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>065476cf-e25b-4039-a5b0-69bfd27e61dd</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.250525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ed56f2bc-52d8-4323-8f41-2cb80548a1dc</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.264623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.486880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7b622388-4835-41b0-916e-bb4ca373d6c4</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.264623</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  f346f20d-4883-4fea-af93-f716f6e99274  Human -0.147495\n",
       "1  065476cf-e25b-4039-a5b0-69bfd27e61dd  Human -0.250525\n",
       "2  ed56f2bc-52d8-4323-8f41-2cb80548a1dc  Human  0.264623\n",
       "3  ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0  Human -1.486880\n",
       "4  7b622388-4835-41b0-916e-bb4ca373d6c4  Human  0.264623"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.506941</td>\n",
       "      <td>3.056311</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>5.281393e-07</td>\n",
       "      <td>[0.33, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.505419</td>\n",
       "      <td>3.018623</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>7.604948e-07</td>\n",
       "      <td>[0.32, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.502317</td>\n",
       "      <td>3.018623</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>7.604948e-07</td>\n",
       "      <td>[0.32, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.672808</td>\n",
       "      <td>3.056311</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>5.281393e-07</td>\n",
       "      <td>[0.49, 0.79]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.671466</td>\n",
       "      <td>3.018623</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>7.604948e-07</td>\n",
       "      <td>[0.49, 0.79]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.668723</td>\n",
       "      <td>3.018623</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>7.604948e-07</td>\n",
       "      <td>[0.48, 0.79]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.506941  3.056311   80   81  5.281393e-07   \n",
       "ICC2      Single random raters  0.505419  3.018623   80   80  7.604948e-07   \n",
       "ICC3       Single fixed raters  0.502317  3.018623   80   80  7.604948e-07   \n",
       "ICC1k  Average raters absolute  0.672808  3.056311   80   81  5.281393e-07   \n",
       "ICC2k    Average random raters  0.671466  3.018623   80   80  7.604948e-07   \n",
       "ICC3k     Average fixed raters  0.668723  3.018623   80   80  7.604948e-07   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.33, 0.65]  \n",
       "ICC2   [0.32, 0.65]  \n",
       "ICC3   [0.32, 0.65]  \n",
       "ICC1k  [0.49, 0.79]  \n",
       "ICC2k  [0.49, 0.79]  \n",
       "ICC3k  [0.48, 0.79]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.48698613796947476\n",
      "Algo score, swaps needed: 1054.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1651, 1566, 1641, 1851, 1555, 1530, 1517, 1582, 1325, 1377]\n",
      "Pseudo-random, avergae swaps needed: 1566.90241\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>f346f20d-4883-4fea-af93-f716f6e99274</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.386848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>065476cf-e25b-4039-a5b0-69bfd27e61dd</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.386848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ed56f2bc-52d8-4323-8f41-2cb80548a1dc</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.650466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.535036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7b622388-4835-41b0-916e-bb4ca373d6c4</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.539592</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  f346f20d-4883-4fea-af93-f716f6e99274  Human -0.386848\n",
       "1  065476cf-e25b-4039-a5b0-69bfd27e61dd  Human -0.386848\n",
       "2  ed56f2bc-52d8-4323-8f41-2cb80548a1dc  Human  0.650466\n",
       "3  ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0  Human -0.535036\n",
       "4  7b622388-4835-41b0-916e-bb4ca373d6c4  Human  1.539592"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.418784</td>\n",
       "      <td>2.441062</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>[0.22, 0.58]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.416990</td>\n",
       "      <td>2.415478</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.22, 0.58]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.414430</td>\n",
       "      <td>2.415478</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.22, 0.58]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.590342</td>\n",
       "      <td>2.441062</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>[0.36, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.588557</td>\n",
       "      <td>2.415478</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.36, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.586003</td>\n",
       "      <td>2.415478</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.36, 0.73]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.418784  2.441062   80   81  0.000042   \n",
       "ICC2      Single random raters  0.416990  2.415478   80   80  0.000054   \n",
       "ICC3       Single fixed raters  0.414430  2.415478   80   80  0.000054   \n",
       "ICC1k  Average raters absolute  0.590342  2.441062   80   81  0.000042   \n",
       "ICC2k    Average random raters  0.588557  2.415478   80   80  0.000054   \n",
       "ICC3k     Average fixed raters  0.586003  2.415478   80   80  0.000054   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.22, 0.58]  \n",
       "ICC2   [0.22, 0.58]  \n",
       "ICC3   [0.22, 0.58]  \n",
       "ICC1k  [0.36, 0.74]  \n",
       "ICC2k  [0.36, 0.74]  \n",
       "ICC3k  [0.36, 0.73]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.457167253080513\n",
      "Algo score, swaps needed: 930.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1460, 1716, 1458, 1704, 1473, 1771, 1657, 1618, 1575, 1632]\n",
      "Pseudo-random, avergae swaps needed: 1560.20248\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>972e8032-a55b-4c9a-a2ed-9913f124d716</td>\n",
       "      <td>0.196855</td>\n",
       "      <td>0.056514</td>\n",
       "      <td>1.228718</td>\n",
       "      <td>0.572495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0cd9b843-be11-4595-847f-4b89f6644253</td>\n",
       "      <td>0.314968</td>\n",
       "      <td>0.957513</td>\n",
       "      <td>-0.040083</td>\n",
       "      <td>1.398819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9db4494e-5ddd-4a7d-b3e9-031a33003a02</td>\n",
       "      <td>1.259871</td>\n",
       "      <td>1.858512</td>\n",
       "      <td>1.348420</td>\n",
       "      <td>1.511824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6726ea7e-8d3e-46f5-bb70-5aff7c5b6402</td>\n",
       "      <td>-1.811065</td>\n",
       "      <td>-1.144818</td>\n",
       "      <td>-0.192882</td>\n",
       "      <td>-0.551673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>43da532e-504a-485a-8559-92b640834f06</td>\n",
       "      <td>-0.511823</td>\n",
       "      <td>0.056514</td>\n",
       "      <td>-0.365385</td>\n",
       "      <td>-1.464528</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  972e8032-a55b-4c9a-a2ed-9913f124d716  0.196855  0.056514  1.228718   \n",
       "1  0cd9b843-be11-4595-847f-4b89f6644253  0.314968  0.957513 -0.040083   \n",
       "2  9db4494e-5ddd-4a7d-b3e9-031a33003a02  1.259871  1.858512  1.348420   \n",
       "3  6726ea7e-8d3e-46f5-bb70-5aff7c5b6402 -1.811065 -1.144818 -0.192882   \n",
       "4  43da532e-504a-485a-8559-92b640834f06 -0.511823  0.056514 -0.365385   \n",
       "\n",
       "       Q2_y  \n",
       "0  0.572495  \n",
       "1  1.398819  \n",
       "2  1.511824  \n",
       "3 -0.551673  \n",
       "4 -1.464528  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>972e8032-a55b-4c9a-a2ed-9913f124d716</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.196855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0cd9b843-be11-4595-847f-4b89f6644253</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.314968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9db4494e-5ddd-4a7d-b3e9-031a33003a02</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.259871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6726ea7e-8d3e-46f5-bb70-5aff7c5b6402</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.811065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>43da532e-504a-485a-8559-92b640834f06</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.511823</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  972e8032-a55b-4c9a-a2ed-9913f124d716  Human  0.196855\n",
       "1  0cd9b843-be11-4595-847f-4b89f6644253  Human  0.314968\n",
       "2  9db4494e-5ddd-4a7d-b3e9-031a33003a02  Human  1.259871\n",
       "3  6726ea7e-8d3e-46f5-bb70-5aff7c5b6402  Human -1.811065\n",
       "4  43da532e-504a-485a-8559-92b640834f06  Human -0.511823"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.435186</td>\n",
       "      <td>2.540989</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000009</td>\n",
       "      <td>[0.25, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.433440</td>\n",
       "      <td>2.513555</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>[0.25, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.430776</td>\n",
       "      <td>2.513555</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>[0.24, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.606452</td>\n",
       "      <td>2.540989</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000009</td>\n",
       "      <td>[0.4, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.604755</td>\n",
       "      <td>2.513555</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>[0.4, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.602157</td>\n",
       "      <td>2.513555</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>[0.39, 0.74]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.435186  2.540989   87   88  0.000009   \n",
       "ICC2      Single random raters  0.433440  2.513555   87   87  0.000013   \n",
       "ICC3       Single fixed raters  0.430776  2.513555   87   87  0.000013   \n",
       "ICC1k  Average raters absolute  0.606452  2.540989   87   88  0.000009   \n",
       "ICC2k    Average random raters  0.604755  2.513555   87   87  0.000013   \n",
       "ICC3k     Average fixed raters  0.602157  2.513555   87   87  0.000013   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.25, 0.59]  \n",
       "ICC2   [0.25, 0.59]  \n",
       "ICC3   [0.24, 0.59]  \n",
       "ICC1k   [0.4, 0.74]  \n",
       "ICC2k   [0.4, 0.74]  \n",
       "ICC3k  [0.39, 0.74]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.4602594296957241\n",
      "Algo score, swaps needed: 1275.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1756, 1990, 1739, 1622, 1830, 2018, 1796, 1944, 1892, 1513]\n",
      "Pseudo-random, avergae swaps needed: 1838.09328\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>972e8032-a55b-4c9a-a2ed-9913f124d716</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.056514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0cd9b843-be11-4595-847f-4b89f6644253</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.957513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9db4494e-5ddd-4a7d-b3e9-031a33003a02</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.858512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6726ea7e-8d3e-46f5-bb70-5aff7c5b6402</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.144818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>43da532e-504a-485a-8559-92b640834f06</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.056514</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  972e8032-a55b-4c9a-a2ed-9913f124d716  Human  0.056514\n",
       "1  0cd9b843-be11-4595-847f-4b89f6644253  Human  0.957513\n",
       "2  9db4494e-5ddd-4a7d-b3e9-031a33003a02  Human  1.858512\n",
       "3  6726ea7e-8d3e-46f5-bb70-5aff7c5b6402  Human -1.144818\n",
       "4  43da532e-504a-485a-8559-92b640834f06  Human  0.056514"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.403045</td>\n",
       "      <td>2.350336</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>[0.21, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.401305</td>\n",
       "      <td>2.327666</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.21, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.398978</td>\n",
       "      <td>2.327666</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.21, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.574529</td>\n",
       "      <td>2.350336</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>[0.35, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.572759</td>\n",
       "      <td>2.327666</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.35, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.570385</td>\n",
       "      <td>2.327666</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.34, 0.72]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.403045  2.350336   87   88  0.000042   \n",
       "ICC2      Single random raters  0.401305  2.327666   87   87  0.000054   \n",
       "ICC3       Single fixed raters  0.398978  2.327666   87   87  0.000054   \n",
       "ICC1k  Average raters absolute  0.574529  2.350336   87   88  0.000042   \n",
       "ICC2k    Average random raters  0.572759  2.327666   87   87  0.000054   \n",
       "ICC3k     Average fixed raters  0.570385  2.327666   87   87  0.000054   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.21, 0.56]  \n",
       "ICC2   [0.21, 0.56]  \n",
       "ICC3   [0.21, 0.56]  \n",
       "ICC1k  [0.35, 0.72]  \n",
       "ICC2k  [0.35, 0.72]  \n",
       "ICC3k  [0.34, 0.72]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.4202472486054576\n",
      "Algo score, swaps needed: 1224.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1793, 1597, 1899, 1601, 1798, 1784, 1459, 1663, 1785, 1709]\n",
      "Pseudo-random, avergae swaps needed: 1756.86311\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3ad4aed5-023b-47c7-9ddf-3f2284a403b1</td>\n",
       "      <td>-1.853126</td>\n",
       "      <td>-1.931438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a0b1247e-71db-499f-a8e9-5191a179ac6c</td>\n",
       "      <td>-0.063175</td>\n",
       "      <td>-0.617613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>740ba105-1c40-49be-b74d-e78072fdbf34</td>\n",
       "      <td>-0.421165</td>\n",
       "      <td>-1.437531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c1609ab5-b028-447c-8f26-60c711841a33</td>\n",
       "      <td>-1.017815</td>\n",
       "      <td>-2.002584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>47e96791-9e80-4638-b63a-a00903ce8508</td>\n",
       "      <td>0.175485</td>\n",
       "      <td>-0.065810</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q2_x      Q2_y\n",
       "0  3ad4aed5-023b-47c7-9ddf-3f2284a403b1 -1.853126 -1.931438\n",
       "1  a0b1247e-71db-499f-a8e9-5191a179ac6c -0.063175 -0.617613\n",
       "2  740ba105-1c40-49be-b74d-e78072fdbf34 -0.421165 -1.437531\n",
       "3  c1609ab5-b028-447c-8f26-60c711841a33 -1.017815 -2.002584\n",
       "4  47e96791-9e80-4638-b63a-a00903ce8508  0.175485 -0.065810"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3ad4aed5-023b-47c7-9ddf-3f2284a403b1</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.853126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a0b1247e-71db-499f-a8e9-5191a179ac6c</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.063175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>740ba105-1c40-49be-b74d-e78072fdbf34</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.421165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c1609ab5-b028-447c-8f26-60c711841a33</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.017815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>47e96791-9e80-4638-b63a-a00903ce8508</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.175485</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  3ad4aed5-023b-47c7-9ddf-3f2284a403b1  Human -1.853126\n",
       "1  a0b1247e-71db-499f-a8e9-5191a179ac6c  Human -0.063175\n",
       "2  740ba105-1c40-49be-b74d-e78072fdbf34  Human -0.421165\n",
       "3  c1609ab5-b028-447c-8f26-60c711841a33  Human -1.017815\n",
       "4  47e96791-9e80-4638-b63a-a00903ce8508  Human  0.175485"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.568949</td>\n",
       "      <td>3.639821</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.000549</td>\n",
       "      <td>[0.26, 0.77]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.566593</td>\n",
       "      <td>3.550264</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>[0.25, 0.77]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.560465</td>\n",
       "      <td>3.550264</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>[0.24, 0.77]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.725261</td>\n",
       "      <td>3.639821</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.000549</td>\n",
       "      <td>[0.41, 0.87]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.723344</td>\n",
       "      <td>3.550264</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>[0.4, 0.87]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.718331</td>\n",
       "      <td>3.550264</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>[0.39, 0.87]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.568949  3.639821   27   28  0.000549   \n",
       "ICC2      Single random raters  0.566593  3.550264   27   27  0.000783   \n",
       "ICC3       Single fixed raters  0.560465  3.550264   27   27  0.000783   \n",
       "ICC1k  Average raters absolute  0.725261  3.639821   27   28  0.000549   \n",
       "ICC2k    Average random raters  0.723344  3.550264   27   27  0.000783   \n",
       "ICC3k     Average fixed raters  0.718331  3.550264   27   27  0.000783   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.26, 0.77]  \n",
       "ICC2   [0.25, 0.77]  \n",
       "ICC3   [0.24, 0.77]  \n",
       "ICC1k  [0.41, 0.87]  \n",
       "ICC2k   [0.4, 0.87]  \n",
       "ICC3k  [0.39, 0.87]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.5166799574694312\n",
      "Algo score, swaps needed: 117.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[176, 126, 184, 230, 198, 205, 195, 191, 174, 168]\n",
      "Pseudo-random, avergae swaps needed: 182.35202\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28f16501-0f48-4649-9348-caf3c3ce5d09</td>\n",
       "      <td>0.363190</td>\n",
       "      <td>0.880199</td>\n",
       "      <td>0.046741</td>\n",
       "      <td>0.260597</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f7422fbb-df4a-4788-8d19-03ca5ab472f4</td>\n",
       "      <td>-2.134996</td>\n",
       "      <td>-0.498226</td>\n",
       "      <td>-1.464614</td>\n",
       "      <td>0.177870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fa4c4929-ae24-4cf5-a3dd-d77f754d900e</td>\n",
       "      <td>0.696281</td>\n",
       "      <td>-0.038751</td>\n",
       "      <td>1.148659</td>\n",
       "      <td>0.360820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>22e8b506-0aa5-400a-aca0-ad1457dc1f19</td>\n",
       "      <td>0.529736</td>\n",
       "      <td>-1.417176</td>\n",
       "      <td>-0.333251</td>\n",
       "      <td>0.194901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>af7980d7-0f7d-4600-82c5-14f34f16f53a</td>\n",
       "      <td>-1.968450</td>\n",
       "      <td>-1.417176</td>\n",
       "      <td>-0.998740</td>\n",
       "      <td>0.041338</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  28f16501-0f48-4649-9348-caf3c3ce5d09  0.363190  0.880199  0.046741   \n",
       "1  f7422fbb-df4a-4788-8d19-03ca5ab472f4 -2.134996 -0.498226 -1.464614   \n",
       "2  fa4c4929-ae24-4cf5-a3dd-d77f754d900e  0.696281 -0.038751  1.148659   \n",
       "3  22e8b506-0aa5-400a-aca0-ad1457dc1f19  0.529736 -1.417176 -0.333251   \n",
       "4  af7980d7-0f7d-4600-82c5-14f34f16f53a -1.968450 -1.417176 -0.998740   \n",
       "\n",
       "       Q2_y  \n",
       "0  0.260597  \n",
       "1  0.177870  \n",
       "2  0.360820  \n",
       "3  0.194901  \n",
       "4  0.041338  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28f16501-0f48-4649-9348-caf3c3ce5d09</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.363190</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f7422fbb-df4a-4788-8d19-03ca5ab472f4</td>\n",
       "      <td>Human</td>\n",
       "      <td>-2.134996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fa4c4929-ae24-4cf5-a3dd-d77f754d900e</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.696281</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>22e8b506-0aa5-400a-aca0-ad1457dc1f19</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.529736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>af7980d7-0f7d-4600-82c5-14f34f16f53a</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.968450</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  28f16501-0f48-4649-9348-caf3c3ce5d09  Human  0.363190\n",
       "1  f7422fbb-df4a-4788-8d19-03ca5ab472f4  Human -2.134996\n",
       "2  fa4c4929-ae24-4cf5-a3dd-d77f754d900e  Human  0.696281\n",
       "3  22e8b506-0aa5-400a-aca0-ad1457dc1f19  Human  0.529736\n",
       "4  af7980d7-0f7d-4600-82c5-14f34f16f53a  Human -1.968450"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.540244</td>\n",
       "      <td>3.350137</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>2.086765e-07</td>\n",
       "      <td>[0.36, 0.68]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.538824</td>\n",
       "      <td>3.305833</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>3.106074e-07</td>\n",
       "      <td>[0.36, 0.68]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.535514</td>\n",
       "      <td>3.305833</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>3.106074e-07</td>\n",
       "      <td>[0.35, 0.68]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.701505</td>\n",
       "      <td>3.350137</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>2.086765e-07</td>\n",
       "      <td>[0.53, 0.81]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.700306</td>\n",
       "      <td>3.305833</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>3.106074e-07</td>\n",
       "      <td>[0.52, 0.81]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.697504</td>\n",
       "      <td>3.305833</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>3.106074e-07</td>\n",
       "      <td>[0.52, 0.81]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.540244  3.350137   74   75  2.086765e-07   \n",
       "ICC2      Single random raters  0.538824  3.305833   74   74  3.106074e-07   \n",
       "ICC3       Single fixed raters  0.535514  3.305833   74   74  3.106074e-07   \n",
       "ICC1k  Average raters absolute  0.701505  3.350137   74   75  2.086765e-07   \n",
       "ICC2k    Average random raters  0.700306  3.305833   74   74  3.106074e-07   \n",
       "ICC3k     Average fixed raters  0.697504  3.305833   74   74  3.106074e-07   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.36, 0.68]  \n",
       "ICC2   [0.36, 0.68]  \n",
       "ICC3   [0.35, 0.68]  \n",
       "ICC1k  [0.53, 0.81]  \n",
       "ICC2k  [0.52, 0.81]  \n",
       "ICC3k  [0.52, 0.81]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.5102469877573461\n",
      "Algo score, swaps needed: 903.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1418, 1213, 1350, 1136, 1529, 1347, 1411, 1248, 1310, 1204]\n",
      "Pseudo-random, avergae swaps needed: 1323.80177\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28f16501-0f48-4649-9348-caf3c3ce5d09</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.880199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f7422fbb-df4a-4788-8d19-03ca5ab472f4</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.498226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fa4c4929-ae24-4cf5-a3dd-d77f754d900e</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.038751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>22e8b506-0aa5-400a-aca0-ad1457dc1f19</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.417176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>af7980d7-0f7d-4600-82c5-14f34f16f53a</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.417176</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  28f16501-0f48-4649-9348-caf3c3ce5d09  Human  0.880199\n",
       "1  f7422fbb-df4a-4788-8d19-03ca5ab472f4  Human -0.498226\n",
       "2  fa4c4929-ae24-4cf5-a3dd-d77f754d900e  Human -0.038751\n",
       "3  22e8b506-0aa5-400a-aca0-ad1457dc1f19  Human -1.417176\n",
       "4  af7980d7-0f7d-4600-82c5-14f34f16f53a  Human -1.417176"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.177572</td>\n",
       "      <td>1.431825</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.061668</td>\n",
       "      <td>[-0.05, 0.39]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.173036</td>\n",
       "      <td>1.412976</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>[-0.06, 0.39]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.171148</td>\n",
       "      <td>1.412976</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>[-0.06, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.301591</td>\n",
       "      <td>1.431825</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.061668</td>\n",
       "      <td>[-0.1, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.295023</td>\n",
       "      <td>1.412976</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>[-0.12, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.292274</td>\n",
       "      <td>1.412976</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>[-0.12, 0.55]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.177572  1.431825   74   75  0.061668   \n",
       "ICC2      Single random raters  0.173036  1.412976   74   74  0.069672   \n",
       "ICC3       Single fixed raters  0.171148  1.412976   74   74  0.069672   \n",
       "ICC1k  Average raters absolute  0.301591  1.431825   74   75  0.061668   \n",
       "ICC2k    Average random raters  0.295023  1.412976   74   74  0.069672   \n",
       "ICC3k     Average fixed raters  0.292274  1.412976   74   74  0.069672   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.05, 0.39]  \n",
       "ICC2   [-0.06, 0.39]  \n",
       "ICC3   [-0.06, 0.38]  \n",
       "ICC1k   [-0.1, 0.56]  \n",
       "ICC2k  [-0.12, 0.56]  \n",
       "ICC3k  [-0.12, 0.55]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.23952975753122707\n",
      "Algo score, swaps needed: 830.99856\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1286, 1353, 1162, 1355, 1181, 1096, 1154, 1424, 1124, 1215]\n",
      "Pseudo-random, avergae swaps needed: 1219.98775\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>737af13d-d5ba-4cf8-beaa-63bdc403d760</td>\n",
       "      <td>-0.615570</td>\n",
       "      <td>-2.523529</td>\n",
       "      <td>-0.659332</td>\n",
       "      <td>0.363257</td>\n",
       "      <td>-0.771901</td>\n",
       "      <td>-0.055214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>39268066-b798-49dd-8159-eda5299389ef</td>\n",
       "      <td>0.963604</td>\n",
       "      <td>-1.087240</td>\n",
       "      <td>0.022388</td>\n",
       "      <td>0.771376</td>\n",
       "      <td>0.221689</td>\n",
       "      <td>0.074232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>da6a7c2f-35f3-476f-a98e-c2445b5f08f4</td>\n",
       "      <td>1.650202</td>\n",
       "      <td>0.759417</td>\n",
       "      <td>1.385828</td>\n",
       "      <td>0.289398</td>\n",
       "      <td>1.190295</td>\n",
       "      <td>-0.014770</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>30234d01-ef3f-41bf-ae4a-1efaba3522d8</td>\n",
       "      <td>1.718862</td>\n",
       "      <td>1.477562</td>\n",
       "      <td>-0.075000</td>\n",
       "      <td>1.050680</td>\n",
       "      <td>1.379937</td>\n",
       "      <td>0.777783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c1b34a08-528e-4824-85ac-1d2043c2bb32</td>\n",
       "      <td>-2.194745</td>\n",
       "      <td>-2.215753</td>\n",
       "      <td>-2.120159</td>\n",
       "      <td>-0.476800</td>\n",
       "      <td>-0.283172</td>\n",
       "      <td>-0.913744</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q3_x  \\\n",
       "0  737af13d-d5ba-4cf8-beaa-63bdc403d760 -0.615570 -2.523529 -0.659332   \n",
       "1  39268066-b798-49dd-8159-eda5299389ef  0.963604 -1.087240  0.022388   \n",
       "2  da6a7c2f-35f3-476f-a98e-c2445b5f08f4  1.650202  0.759417  1.385828   \n",
       "3  30234d01-ef3f-41bf-ae4a-1efaba3522d8  1.718862  1.477562 -0.075000   \n",
       "4  c1b34a08-528e-4824-85ac-1d2043c2bb32 -2.194745 -2.215753 -2.120159   \n",
       "\n",
       "       Q1_y      Q2_y      Q3_y  \n",
       "0  0.363257 -0.771901 -0.055214  \n",
       "1  0.771376  0.221689  0.074232  \n",
       "2  0.289398  1.190295 -0.014770  \n",
       "3  1.050680  1.379937  0.777783  \n",
       "4 -0.476800 -0.283172 -0.913744  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>737af13d-d5ba-4cf8-beaa-63bdc403d760</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.615570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>39268066-b798-49dd-8159-eda5299389ef</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.963604</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b46d83cb-4d90-49b1-a40a-9a10cc909734</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.096189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>da6a7c2f-35f3-476f-a98e-c2445b5f08f4</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.650202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30234d01-ef3f-41bf-ae4a-1efaba3522d8</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.718862</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  737af13d-d5ba-4cf8-beaa-63bdc403d760  Human -0.615570\n",
       "1  39268066-b798-49dd-8159-eda5299389ef  Human  0.963604\n",
       "2  b46d83cb-4d90-49b1-a40a-9a10cc909734  Human -1.096189\n",
       "3  da6a7c2f-35f3-476f-a98e-c2445b5f08f4  Human  1.650202\n",
       "4  30234d01-ef3f-41bf-ae4a-1efaba3522d8  Human  1.718862"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.277498</td>\n",
       "      <td>1.768157</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.006460</td>\n",
       "      <td>[0.06, 0.47]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.274216</td>\n",
       "      <td>1.746300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.007704</td>\n",
       "      <td>[0.05, 0.47]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.271748</td>\n",
       "      <td>1.746300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.007704</td>\n",
       "      <td>[0.05, 0.47]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.434439</td>\n",
       "      <td>1.768157</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.006460</td>\n",
       "      <td>[0.11, 0.64]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.430408</td>\n",
       "      <td>1.746300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.007704</td>\n",
       "      <td>[0.1, 0.64]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.427361</td>\n",
       "      <td>1.746300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.007704</td>\n",
       "      <td>[0.1, 0.63]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.277498  1.768157   77   78  0.006460   \n",
       "ICC2      Single random raters  0.274216  1.746300   77   77  0.007704   \n",
       "ICC3       Single fixed raters  0.271748  1.746300   77   77  0.007704   \n",
       "ICC1k  Average raters absolute  0.434439  1.768157   77   78  0.006460   \n",
       "ICC2k    Average random raters  0.430408  1.746300   77   77  0.007704   \n",
       "ICC3k     Average fixed raters  0.427361  1.746300   77   77  0.007704   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.06, 0.47]  \n",
       "ICC2   [0.05, 0.47]  \n",
       "ICC3   [0.05, 0.47]  \n",
       "ICC1k  [0.11, 0.64]  \n",
       "ICC2k   [0.1, 0.64]  \n",
       "ICC3k   [0.1, 0.63]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.2949785232758061\n",
      "Algo score, swaps needed: 1176.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1354, 1635, 1289, 1425, 1436, 1514, 1458, 1396, 1537, 1441]\n",
      "Pseudo-random, avergae swaps needed: 1472.10551\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>737af13d-d5ba-4cf8-beaa-63bdc403d760</td>\n",
       "      <td>Human</td>\n",
       "      <td>-2.523529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>39268066-b798-49dd-8159-eda5299389ef</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.087240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b46d83cb-4d90-49b1-a40a-9a10cc909734</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.041273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>da6a7c2f-35f3-476f-a98e-c2445b5f08f4</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.759417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30234d01-ef3f-41bf-ae4a-1efaba3522d8</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.477562</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  737af13d-d5ba-4cf8-beaa-63bdc403d760  Human -2.523529\n",
       "1  39268066-b798-49dd-8159-eda5299389ef  Human -1.087240\n",
       "2  b46d83cb-4d90-49b1-a40a-9a10cc909734  Human  0.041273\n",
       "3  da6a7c2f-35f3-476f-a98e-c2445b5f08f4  Human  0.759417\n",
       "4  30234d01-ef3f-41bf-ae4a-1efaba3522d8  Human  1.477562"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.319052</td>\n",
       "      <td>1.937081</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.002003</td>\n",
       "      <td>[0.11, 0.5]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.316352</td>\n",
       "      <td>1.914873</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.002425</td>\n",
       "      <td>[0.1, 0.5]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.313864</td>\n",
       "      <td>1.914873</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.002425</td>\n",
       "      <td>[0.1, 0.5]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.483759</td>\n",
       "      <td>1.937081</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.002003</td>\n",
       "      <td>[0.19, 0.67]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.480650</td>\n",
       "      <td>1.914873</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.002425</td>\n",
       "      <td>[0.18, 0.67]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.477772</td>\n",
       "      <td>1.914873</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.002425</td>\n",
       "      <td>[0.18, 0.67]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.319052  1.937081   77   78  0.002003   \n",
       "ICC2      Single random raters  0.316352  1.914873   77   77  0.002425   \n",
       "ICC3       Single fixed raters  0.313864  1.914873   77   77  0.002425   \n",
       "ICC1k  Average raters absolute  0.483759  1.937081   77   78  0.002003   \n",
       "ICC2k    Average random raters  0.480650  1.914873   77   77  0.002425   \n",
       "ICC3k     Average fixed raters  0.477772  1.914873   77   77  0.002425   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1    [0.11, 0.5]  \n",
       "ICC2     [0.1, 0.5]  \n",
       "ICC3     [0.1, 0.5]  \n",
       "ICC1k  [0.19, 0.67]  \n",
       "ICC2k  [0.18, 0.67]  \n",
       "ICC3k  [0.18, 0.67]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.3509009258745911\n",
      "Algo score, swaps needed: 1043.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1484, 1394, 1340, 1355, 1287, 1507, 1680, 1411, 1312, 1324]\n",
      "Pseudo-random, avergae swaps needed: 1449.9545\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>737af13d-d5ba-4cf8-beaa-63bdc403d760</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.659332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>39268066-b798-49dd-8159-eda5299389ef</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.022388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b46d83cb-4d90-49b1-a40a-9a10cc909734</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.606719</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>da6a7c2f-35f3-476f-a98e-c2445b5f08f4</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.385828</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30234d01-ef3f-41bf-ae4a-1efaba3522d8</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.075000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  737af13d-d5ba-4cf8-beaa-63bdc403d760  Human -0.659332\n",
       "1  39268066-b798-49dd-8159-eda5299389ef  Human  0.022388\n",
       "2  b46d83cb-4d90-49b1-a40a-9a10cc909734  Human  0.606719\n",
       "3  da6a7c2f-35f3-476f-a98e-c2445b5f08f4  Human  1.385828\n",
       "4  30234d01-ef3f-41bf-ae4a-1efaba3522d8  Human -0.075000"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.557776</td>\n",
       "      <td>3.522595</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>4.057886e-08</td>\n",
       "      <td>[0.38, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.556504</td>\n",
       "      <td>3.477485</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>6.165675e-08</td>\n",
       "      <td>[0.38, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.553321</td>\n",
       "      <td>3.477485</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>6.165675e-08</td>\n",
       "      <td>[0.38, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.716118</td>\n",
       "      <td>3.522595</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>4.057886e-08</td>\n",
       "      <td>[0.56, 0.82]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.715069</td>\n",
       "      <td>3.477485</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>6.165675e-08</td>\n",
       "      <td>[0.55, 0.82]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.712436</td>\n",
       "      <td>3.477485</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>6.165675e-08</td>\n",
       "      <td>[0.55, 0.82]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.557776  3.522595   77   78  4.057886e-08   \n",
       "ICC2      Single random raters  0.556504  3.477485   77   77  6.165675e-08   \n",
       "ICC3       Single fixed raters  0.553321  3.477485   77   77  6.165675e-08   \n",
       "ICC1k  Average raters absolute  0.716118  3.522595   77   78  4.057886e-08   \n",
       "ICC2k    Average random raters  0.715069  3.477485   77   77  6.165675e-08   \n",
       "ICC3k     Average fixed raters  0.712436  3.477485   77   77  6.165675e-08   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.38, 0.69]  \n",
       "ICC2   [0.38, 0.69]  \n",
       "ICC3   [0.38, 0.69]  \n",
       "ICC1k  [0.56, 0.82]  \n",
       "ICC2k  [0.55, 0.82]  \n",
       "ICC3k  [0.55, 0.82]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.6055882812907261\n",
      "Algo score, swaps needed: 783.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1354, 1360, 1544, 1309, 1322, 1425, 1494, 1427, 1371, 1442]\n",
      "Pseudo-random, avergae swaps needed: 1457.60233\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>69b6ce9c-4cdb-4afd-9ace-a993777b08e1</td>\n",
       "      <td>-0.945323</td>\n",
       "      <td>-0.705991</td>\n",
       "      <td>0.137195</td>\n",
       "      <td>1.510483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e824f723-032e-4900-a6e5-a82e279eefc8</td>\n",
       "      <td>-1.281854</td>\n",
       "      <td>-0.705991</td>\n",
       "      <td>-0.251746</td>\n",
       "      <td>-0.178808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b79bcda6-c021-453b-96e9-fd6d58c20469</td>\n",
       "      <td>-0.945323</td>\n",
       "      <td>-1.284442</td>\n",
       "      <td>-0.273292</td>\n",
       "      <td>-0.944506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>54273ba8-efd3-4d24-89aa-bb5f5cda0ee0</td>\n",
       "      <td>-0.608793</td>\n",
       "      <td>0.258094</td>\n",
       "      <td>0.841415</td>\n",
       "      <td>0.545715</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>afb1f54f-6525-497c-9642-32aca074c609</td>\n",
       "      <td>-1.618384</td>\n",
       "      <td>-1.284442</td>\n",
       "      <td>-1.047122</td>\n",
       "      <td>-0.123607</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  69b6ce9c-4cdb-4afd-9ace-a993777b08e1 -0.945323 -0.705991  0.137195   \n",
       "1  e824f723-032e-4900-a6e5-a82e279eefc8 -1.281854 -0.705991 -0.251746   \n",
       "2  b79bcda6-c021-453b-96e9-fd6d58c20469 -0.945323 -1.284442 -0.273292   \n",
       "3  54273ba8-efd3-4d24-89aa-bb5f5cda0ee0 -0.608793  0.258094  0.841415   \n",
       "4  afb1f54f-6525-497c-9642-32aca074c609 -1.618384 -1.284442 -1.047122   \n",
       "\n",
       "       Q2_y  \n",
       "0  1.510483  \n",
       "1 -0.178808  \n",
       "2 -0.944506  \n",
       "3  0.545715  \n",
       "4 -0.123607  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>69b6ce9c-4cdb-4afd-9ace-a993777b08e1</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.945323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e824f723-032e-4900-a6e5-a82e279eefc8</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.281854</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b79bcda6-c021-453b-96e9-fd6d58c20469</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.945323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>54273ba8-efd3-4d24-89aa-bb5f5cda0ee0</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.608793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>afb1f54f-6525-497c-9642-32aca074c609</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.618384</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  69b6ce9c-4cdb-4afd-9ace-a993777b08e1  Human -0.945323\n",
       "1  e824f723-032e-4900-a6e5-a82e279eefc8  Human -1.281854\n",
       "2  b79bcda6-c021-453b-96e9-fd6d58c20469  Human -0.945323\n",
       "3  54273ba8-efd3-4d24-89aa-bb5f5cda0ee0  Human -0.608793\n",
       "4  afb1f54f-6525-497c-9642-32aca074c609  Human -1.618384"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.090017</td>\n",
       "      <td>1.197843</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.194626</td>\n",
       "      <td>[-0.12, 0.29]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.085672</td>\n",
       "      <td>1.185463</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.209323</td>\n",
       "      <td>[-0.12, 0.29]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.084862</td>\n",
       "      <td>1.185463</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.209323</td>\n",
       "      <td>[-0.12, 0.28]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.165166</td>\n",
       "      <td>1.197843</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.194626</td>\n",
       "      <td>[-0.26, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.157823</td>\n",
       "      <td>1.185463</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.209323</td>\n",
       "      <td>[-0.28, 0.44]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.156447</td>\n",
       "      <td>1.185463</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.209323</td>\n",
       "      <td>[-0.28, 0.44]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.090017  1.197843   91   92  0.194626   \n",
       "ICC2      Single random raters  0.085672  1.185463   91   91  0.209323   \n",
       "ICC3       Single fixed raters  0.084862  1.185463   91   91  0.209323   \n",
       "ICC1k  Average raters absolute  0.165166  1.197843   91   92  0.194626   \n",
       "ICC2k    Average random raters  0.157823  1.185463   91   91  0.209323   \n",
       "ICC3k     Average fixed raters  0.156447  1.185463   91   91  0.209323   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.12, 0.29]  \n",
       "ICC2   [-0.12, 0.29]  \n",
       "ICC3   [-0.12, 0.28]  \n",
       "ICC1k  [-0.26, 0.45]  \n",
       "ICC2k  [-0.28, 0.44]  \n",
       "ICC3k  [-0.28, 0.44]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.15220449016453974\n",
      "Algo score, swaps needed: 1705.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2177, 1981, 2158, 1882, 2158, 2116, 2215, 2154, 1748, 1832]\n",
      "Pseudo-random, avergae swaps needed: 2045.71394\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>69b6ce9c-4cdb-4afd-9ace-a993777b08e1</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.705991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e824f723-032e-4900-a6e5-a82e279eefc8</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.705991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b79bcda6-c021-453b-96e9-fd6d58c20469</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.284442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>54273ba8-efd3-4d24-89aa-bb5f5cda0ee0</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.258094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>afb1f54f-6525-497c-9642-32aca074c609</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.284442</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  69b6ce9c-4cdb-4afd-9ace-a993777b08e1  Human -0.705991\n",
       "1  e824f723-032e-4900-a6e5-a82e279eefc8  Human -0.705991\n",
       "2  b79bcda6-c021-453b-96e9-fd6d58c20469  Human -1.284442\n",
       "3  54273ba8-efd3-4d24-89aa-bb5f5cda0ee0  Human  0.258094\n",
       "4  afb1f54f-6525-497c-9642-32aca074c609  Human -1.284442"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.267120</td>\n",
       "      <td>1.728959</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.004700</td>\n",
       "      <td>[0.07, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.264758</td>\n",
       "      <td>1.713932</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.005423</td>\n",
       "      <td>[0.06, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.263062</td>\n",
       "      <td>1.713932</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.005423</td>\n",
       "      <td>[0.06, 0.44]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.421617</td>\n",
       "      <td>1.728959</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.004700</td>\n",
       "      <td>[0.13, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.418669</td>\n",
       "      <td>1.713932</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.005423</td>\n",
       "      <td>[0.12, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.416546</td>\n",
       "      <td>1.713932</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.005423</td>\n",
       "      <td>[0.12, 0.61]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.267120  1.728959   91   92  0.004700   \n",
       "ICC2      Single random raters  0.264758  1.713932   91   91  0.005423   \n",
       "ICC3       Single fixed raters  0.263062  1.713932   91   91  0.005423   \n",
       "ICC1k  Average raters absolute  0.421617  1.728959   91   92  0.004700   \n",
       "ICC2k    Average random raters  0.418669  1.713932   91   91  0.005423   \n",
       "ICC3k     Average fixed raters  0.416546  1.713932   91   91  0.005423   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.07, 0.45]  \n",
       "ICC2   [0.06, 0.45]  \n",
       "ICC3   [0.06, 0.44]  \n",
       "ICC1k  [0.13, 0.62]  \n",
       "ICC2k  [0.12, 0.62]  \n",
       "ICC3k  [0.12, 0.61]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.25302929011688136\n",
      "Algo score, swaps needed: 1589.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1864, 1885, 2070, 2055, 2154, 1972, 1830, 2085, 1889, 1932]\n",
      "Pseudo-random, avergae swaps needed: 1994.7508\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[998.0, 1054.0, 930.0, 1275.0, 1224.0, 117.0, 903.0, 830.99856, 1176.0, 1043.0, 783.0, 1705.0, 1589.0]\n",
      "N: 13 \tMean:  1048.3075815384616 \tVar:  141907.95346760284\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.68528, 1566.90241, 1560.20248, 1838.09328, 1756.86311, 182.35202, 1323.80177, 1219.98775, 1472.10551, 1449.9545, 1457.60233, 2045.71394, 1994.7508]\n",
      "N: 13 \tMean:  1489.8473215384618 \tVar:  198489.0188468986 \n",
      "\n",
      "1 ) 1499.68528 > 998.0 => 40.17201718865076\n",
      "2 ) 1566.90241 > 1054.0 => 39.13937489950265\n",
      "3 ) 1560.20248 > 930.0 => 50.61455725479801\n",
      "4 ) 1838.09328 > 1275.0 => 36.17580517857146\n",
      "5 ) 1756.86311 > 1224.0 => 35.75226975115942\n",
      "6 ) 182.35202 > 117.0 => 43.66232103594958\n",
      "7 ) 1323.80177 > 903.0 => 37.79427299449291\n",
      "8 ) 1219.98775 > 830.99856 => 37.93191481614521\n",
      "9 ) 1472.10551 > 1176.0 => 22.36357342121161\n",
      "10 ) 1449.9545 > 1043.0 => 32.648369635306224\n",
      "11 ) 1457.60233 > 783.0 => 60.21615892901441\n",
      "12 ) 2045.71394 > 1705.0 => 18.1679512461033\n",
      "13 ) 1994.7508 > 1589.0 => 22.643918209938036\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "36.71403881237258 \n",
      "\n",
      "Ttest_relResult(statistic=-10.064631733221875, pvalue=3.3403619563784594e-07)\n",
      "Cohen's d:  1.0282788377360816\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp1',normv=1,score=2,goal=\"centroid\",runs=100000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell presents the output for the parameters used when all of the exam data was first run through the above scoring method, with the exception of `runs`. To avoid biasing parameter selection, prior work on the method did NOT use the exam data used here. \n",
    "\n",
    "Note: the below cell has been re-run since this first run. Additionally, the current version makes use of the average number of swaps over 100,000 instances to provide a better representation of the method's performance. This is because the number of swaps can vary by small amounts for scoring methods where the granularity of the machine scores/grading scale is sufficiently coarse for exams with different human scores to find themselves with the same machine grades. That is, when sorting on the machine grade, one could get different absolute orderings based on the initial order of the list. For example: \n",
    "\n",
    "|ID|Q1_human|Q1_machine|\n",
    "|--|--|--|\n",
    "|00001|4|90|\n",
    "|00002|5|90|\n",
    "\n",
    "and\n",
    "\n",
    "|ID|Q1_human|Q1_machine|\n",
    "|--|--|--|\n",
    "|00002|5|90|\n",
    "|00001|4|90|\n",
    "\n",
    "are both proper orderings based on the values of `Q1_machine` and could occur if the list before sorting had different orderings. In all cases below, the difference between swaps needed for the pseudo-random and machine orderings exhibit a significant difference.\n",
    "\n",
    "Data from the cell above is cited in the paper so that comparisons can be made between the performance of machine, pseudo-random, and human orderings whereas the data in the cell below includes only comparisons between machine and pseudo-random orderings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>351e0d0e-9901-4878-9d31-decad062eb91</td>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>6</td>\n",
       "      <td>90</td>\n",
       "      <td>87</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0adab6a0-9008-4fdb-9fb7-a0be1edcda37</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>8</td>\n",
       "      <td>78</td>\n",
       "      <td>88</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>70d749d7-dd2e-4b83-be1a-1409b0dcb3e2</td>\n",
       "      <td>9</td>\n",
       "      <td>35</td>\n",
       "      <td>17</td>\n",
       "      <td>85</td>\n",
       "      <td>84</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7a25c1e8-4bd9-438f-ad5b-393ad679a0ab</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>11</td>\n",
       "      <td>82</td>\n",
       "      <td>81</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>031928ff-6248-4987-bb30-75af9273ea92</td>\n",
       "      <td>10</td>\n",
       "      <td>41</td>\n",
       "      <td>21</td>\n",
       "      <td>83</td>\n",
       "      <td>85</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x  Q1_x  Q2_x  SHORT_ANS_y  \\\n",
       "0  351e0d0e-9901-4878-9d31-decad062eb91            2    21     6           90   \n",
       "1  0adab6a0-9008-4fdb-9fb7-a0be1edcda37            8    16     8           78   \n",
       "2  70d749d7-dd2e-4b83-be1a-1409b0dcb3e2            9    35    17           85   \n",
       "3  7a25c1e8-4bd9-438f-ad5b-393ad679a0ab            3    13    11           82   \n",
       "4  031928ff-6248-4987-bb30-75af9273ea92           10    41    21           83   \n",
       "\n",
       "   Q1_y  Q2_y  \n",
       "0    87    81  \n",
       "1    88    84  \n",
       "2    84    86  \n",
       "3    81    83  \n",
       "4    85    85  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n",
      "Algo score, swaps needed: 989.50331\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1591, 1599, 1465, 1386, 1545, 1389, 1609, 1744, 1616, 1637]\n",
      "Pseudo-random, avergae swaps needed: 1499.72726\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n",
      "Algo score, swaps needed: 1021.46179\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1688, 1418, 1513, 1558, 1453, 1742, 1635, 1270, 1463, 1713]\n",
      "Pseudo-random, avergae swaps needed: 1566.17577\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n",
      "Algo score, swaps needed: 829.93601\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1560, 1477, 1644, 1828, 1799, 1743, 1766, 1683, 1562, 1394]\n",
      "Pseudo-random, avergae swaps needed: 1560.0728\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ecdebb90-4db9-4967-b6f9-ea3759679fcc</td>\n",
       "      <td>13.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>90</td>\n",
       "      <td>94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d57e3e5c-10f4-4610-9ea6-49852423ede4</td>\n",
       "      <td>18.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>88</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6f684adb-beb1-4e9c-8d7d-76e0637baf2b</td>\n",
       "      <td>14.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>85</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>d2a0868b-f1ef-43d6-a277-0b4b33bd78d0</td>\n",
       "      <td>11.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>82</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>da455e55-d2ac-4054-aae9-e62723a9dc6c</td>\n",
       "      <td>17.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>79</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  ecdebb90-4db9-4967-b6f9-ea3759679fcc  13.0  16.0    90    94\n",
       "1  d57e3e5c-10f4-4610-9ea6-49852423ede4  18.0  23.0    88    87\n",
       "2  6f684adb-beb1-4e9c-8d7d-76e0637baf2b  14.0  18.0    85    84\n",
       "3  d2a0868b-f1ef-43d6-a277-0b4b33bd78d0  11.5  21.0    82    79\n",
       "4  da455e55-d2ac-4054-aae9-e62723a9dc6c  17.0  17.0    79    84"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n",
      "Algo score, swaps needed: 1289.51587\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1774, 1743, 1938, 1834, 1669, 1834, 2189, 1843, 1991, 1652]\n",
      "Pseudo-random, avergae swaps needed: 1837.38272\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n",
      "Algo score, swaps needed: 1319.45138\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1776, 1771, 1959, 1737, 1595, 2008, 1706, 1698, 1872, 1868]\n",
      "Pseudo-random, avergae swaps needed: 1757.88976\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>eabdc089-b66a-4412-a8ca-7241ef5e59a3</td>\n",
       "      <td>31.0</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>981c3959-fe3b-4222-97de-742768dee8e9</td>\n",
       "      <td>23.0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f1a75c07-eda1-4552-907a-c976d9ae7c0c</td>\n",
       "      <td>42.0</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a7931454-2722-4600-ad92-baccd06a38a8</td>\n",
       "      <td>42.0</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8fee9d29-68ff-4b2d-b54d-58f8303249ff</td>\n",
       "      <td>29.5</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q2_x  Q2_y\n",
       "0  eabdc089-b66a-4412-a8ca-7241ef5e59a3  31.0    73\n",
       "1  981c3959-fe3b-4222-97de-742768dee8e9  23.0    80\n",
       "2  f1a75c07-eda1-4552-907a-c976d9ae7c0c  42.0    82\n",
       "3  a7931454-2722-4600-ad92-baccd06a38a8  42.0    96\n",
       "4  8fee9d29-68ff-4b2d-b54d-58f8303249ff  29.5    91"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n",
      "Algo score, swaps needed: 121.01466\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[195, 184, 209, 160, 178, 147, 170, 202, 153, 162]\n",
      "Pseudo-random, avergae swaps needed: 182.61788\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>66da3df6-08d8-4757-bbd0-2d5d0df848fa</td>\n",
       "      <td>9.5</td>\n",
       "      <td>5.5</td>\n",
       "      <td>74</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f7422fbb-df4a-4788-8d19-03ca5ab472f4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>74</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6c261e5d-2212-43a3-ab4f-28bd7dd90612</td>\n",
       "      <td>9.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>88</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>473b0dda-6847-467e-8589-32a882dcfde1</td>\n",
       "      <td>12.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>81</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>836dd271-a288-4467-b688-9d61bc2b99ff</td>\n",
       "      <td>12.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>85</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  66da3df6-08d8-4757-bbd0-2d5d0df848fa   9.5   5.5    74    85\n",
       "1  f7422fbb-df4a-4788-8d19-03ca5ab472f4   5.0   4.0    74    85\n",
       "2  6c261e5d-2212-43a3-ab4f-28bd7dd90612   9.0   3.0    88    85\n",
       "3  473b0dda-6847-467e-8589-32a882dcfde1  12.0   5.0    81    83\n",
       "4  836dd271-a288-4467-b688-9d61bc2b99ff  12.5   5.0    85    85"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n",
      "Algo score, swaps needed: 908.03549\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1502, 1318, 1217, 1319, 1269, 1358, 1401, 1272, 1337, 1195]\n",
      "Pseudo-random, avergae swaps needed: 1324.44786\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n",
      "Algo score, swaps needed: 814.05907\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1264, 1108, 1349, 1480, 1260, 963, 1205, 1043, 1220, 998]\n",
      "Pseudo-random, avergae swaps needed: 1219.65644\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>737af13d-d5ba-4cf8-beaa-63bdc403d760</td>\n",
       "      <td>38</td>\n",
       "      <td>2</td>\n",
       "      <td>17</td>\n",
       "      <td>85</td>\n",
       "      <td>79</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6c715ed9-9520-4c0d-8d4f-b56ceef0320a</td>\n",
       "      <td>53</td>\n",
       "      <td>43</td>\n",
       "      <td>15</td>\n",
       "      <td>81</td>\n",
       "      <td>86</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bb065bd5-206a-4c0f-ba0f-f193651f5ac6</td>\n",
       "      <td>62</td>\n",
       "      <td>23</td>\n",
       "      <td>40</td>\n",
       "      <td>83</td>\n",
       "      <td>84</td>\n",
       "      <td>88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>cbe7586a-af37-49e3-86ad-4e36380adad9</td>\n",
       "      <td>74</td>\n",
       "      <td>40</td>\n",
       "      <td>45</td>\n",
       "      <td>90</td>\n",
       "      <td>91</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>496aeba0-ea0b-4003-bd5b-7e4ac8bd9073</td>\n",
       "      <td>47</td>\n",
       "      <td>35</td>\n",
       "      <td>20</td>\n",
       "      <td>86</td>\n",
       "      <td>78</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q3_x  Q1_y  Q2_y  Q3_y\n",
       "0  737af13d-d5ba-4cf8-beaa-63bdc403d760    38     2    17    85    79    82\n",
       "1  6c715ed9-9520-4c0d-8d4f-b56ceef0320a    53    43    15    81    86    81\n",
       "2  bb065bd5-206a-4c0f-ba0f-f193651f5ac6    62    23    40    83    84    88\n",
       "3  cbe7586a-af37-49e3-86ad-4e36380adad9    74    40    45    90    91    90\n",
       "4  496aeba0-ea0b-4003-bd5b-7e4ac8bd9073    47    35    20    86    78    81"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n",
      "Algo score, swaps needed: 1176.01861\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1388, 1298, 1489, 1456, 1549, 1324, 1352, 1599, 1684, 1609]\n",
      "Pseudo-random, avergae swaps needed: 1471.9416\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n",
      "Algo score, swaps needed: 995.51218\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1358, 1507, 1473, 1510, 1686, 1271, 1371, 1605, 1644, 1360]\n",
      "Pseudo-random, avergae swaps needed: 1449.70805\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n",
      "Algo score, swaps needed: 769.55054\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1440, 1391, 1549, 1429, 1392, 1520, 1326, 1485, 1404, 1589]\n",
      "Pseudo-random, avergae swaps needed: 1457.44482\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>fc6ff039-d412-461c-a096-7039bac5bd73</td>\n",
       "      <td>25.25</td>\n",
       "      <td>20.5</td>\n",
       "      <td>76</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d18b3687-1b51-4f7b-b1e4-94b2918c2e1f</td>\n",
       "      <td>16.00</td>\n",
       "      <td>17.0</td>\n",
       "      <td>82</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>230325ce-77e5-46d5-adf5-e088088aee8f</td>\n",
       "      <td>21.00</td>\n",
       "      <td>21.0</td>\n",
       "      <td>87</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>da2f8241-c0df-4081-b6bc-347b35a3d470</td>\n",
       "      <td>21.00</td>\n",
       "      <td>21.0</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0c3b2e97-82af-40a0-81c9-0e19e9ddff09</td>\n",
       "      <td>19.00</td>\n",
       "      <td>20.0</td>\n",
       "      <td>90</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID   Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  fc6ff039-d412-461c-a096-7039bac5bd73  25.25  20.5    76    62\n",
       "1  d18b3687-1b51-4f7b-b1e4-94b2918c2e1f  16.00  17.0    82    89\n",
       "2  230325ce-77e5-46d5-adf5-e088088aee8f  21.00  21.0    87    81\n",
       "3  da2f8241-c0df-4081-b6bc-347b35a3d470  21.00  21.0    83    83\n",
       "4  0c3b2e97-82af-40a0-81c9-0e19e9ddff09  19.00  20.0    90    91"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n",
      "Algo score, swaps needed: 1777.99084\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2057, 1872, 1910, 1664, 2135, 2271, 2014, 1972, 2058, 1751]\n",
      "Pseudo-random, avergae swaps needed: 2045.42864\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n",
      "Algo score, swaps needed: 1635.99287\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2278, 1949, 1648, 2197, 1839, 1946, 1976, 2155, 2021, 2084]\n",
      "Pseudo-random, avergae swaps needed: 1994.22284\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[989.50331, 1021.46179, 829.93601, 1289.51587, 1319.45138, 121.01466, 908.03549, 814.05907, 1176.01861, 995.51218, 769.55054, 1777.99084, 1635.99287]\n",
      "N: 13 \tMean:  1049.8494323076923 \tVar:  160966.89750620702\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.72726, 1566.17577, 1560.0728, 1837.38272, 1757.88976, 182.61788, 1324.44786, 1219.65644, 1471.9416, 1449.70805, 1457.44482, 2045.42864, 1994.22284]\n",
      "N: 13 \tMean:  1489.7474184615385 \tVar:  198364.50812390135 \n",
      "\n",
      "1 ) 1499.72726 > 989.50331 => 40.994511006668205\n",
      "2 ) 1566.17577 > 1021.46179 => 42.101257797479185\n",
      "3 ) 1560.0728 > 829.93601 => 61.0990877477142\n",
      "4 ) 1837.38272 > 1289.51587 => 35.04218856039077\n",
      "5 ) 1757.88976 > 1319.45138 => 28.4946231213092\n",
      "6 ) 182.61788 > 121.01466 => 40.57748224218656\n",
      "7 ) 1324.44786 > 908.03549 => 37.304857839141334\n",
      "8 ) 1219.65644 > 814.05907 => 39.88732622686247\n",
      "9 ) 1471.9416 > 1176.01861 => 22.351014859094125\n",
      "10 ) 1449.70805 > 995.51218 => 37.14969019375404\n",
      "11 ) 1457.44482 > 769.55054 => 61.77779193935995\n",
      "12 ) 2045.42864 > 1777.99084 => 13.989456370086817\n",
      "13 ) 1994.22284 > 1635.99287 => 19.736015631974656\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "36.96194642584781 \n",
      "\n",
      "Ttest_relResult(statistic=-8.966066420769858, pvalue=1.1490042846605633e-06)\n",
      "Cohen's d:  0.9970991139971257\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp1',normv=0,score=0,goal=\"centroid\",runs=100000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cells present a variety of parameter selections, including a mix of vector normalization, grade translations, and \"best answer\" selections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a503bcd8-7709-45ef-b48f-d56131f99108</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "      <td>11</td>\n",
       "      <td>78</td>\n",
       "      <td>84</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>dcef7f09-f784-48cb-b65e-59d0293f4195</td>\n",
       "      <td>10</td>\n",
       "      <td>30</td>\n",
       "      <td>32</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ccd4c7f7-60a4-48c3-9dbf-96e923a98d93</td>\n",
       "      <td>6</td>\n",
       "      <td>23</td>\n",
       "      <td>28</td>\n",
       "      <td>86</td>\n",
       "      <td>79</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c74cd4a8-869a-4714-892e-be61183e7835</td>\n",
       "      <td>4</td>\n",
       "      <td>27</td>\n",
       "      <td>24</td>\n",
       "      <td>89</td>\n",
       "      <td>91</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>18</td>\n",
       "      <td>85</td>\n",
       "      <td>88</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x  Q1_x  Q2_x  SHORT_ANS_y  \\\n",
       "0  a503bcd8-7709-45ef-b48f-d56131f99108            6    15    11           78   \n",
       "1  dcef7f09-f784-48cb-b65e-59d0293f4195           10    30    32           87   \n",
       "2  ccd4c7f7-60a4-48c3-9dbf-96e923a98d93            6    23    28           86   \n",
       "3  c74cd4a8-869a-4714-892e-be61183e7835            4    27    24           89   \n",
       "4  7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e            5    18    18           85   \n",
       "\n",
       "   Q1_y  Q2_y  \n",
       "0    84    86  \n",
       "1    83    86  \n",
       "2    79    86  \n",
       "3    91    83  \n",
       "4    88    84  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n",
      "Algo score, swaps needed: 996.46505\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1310, 1488, 1320, 1562, 1612, 1426, 1638, 1551, 1372, 1488]\n",
      "Pseudo-random, avergae swaps needed: 1499.14618\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n",
      "Algo score, swaps needed: 1054.98323\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1658, 1752, 1703, 1683, 1707, 1709, 1600, 1737, 1774, 1408]\n",
      "Pseudo-random, avergae swaps needed: 1566.48594\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n",
      "Algo score, swaps needed: 959.91972\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1471, 1484, 1446, 1407, 1472, 1561, 1721, 1494, 1540, 1534]\n",
      "Pseudo-random, avergae swaps needed: 1560.00533\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8c9de0d1-eff5-4b5c-b1ff-7cecd2bb6c34</td>\n",
       "      <td>12.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>65</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1b15cca5-b292-4c29-8e83-384ed76e7b77</td>\n",
       "      <td>20.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>79</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d9863d9e-0473-482d-ae21-bcfb05f1753c</td>\n",
       "      <td>9.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>71</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a9738fe5-2df1-4661-b147-de435b7eeeb6</td>\n",
       "      <td>17.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>80</td>\n",
       "      <td>94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>529faf72-9186-4652-8fcc-58ba8b46a5cb</td>\n",
       "      <td>22.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>89</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  8c9de0d1-eff5-4b5c-b1ff-7cecd2bb6c34  12.0  12.0    65    80\n",
       "1  1b15cca5-b292-4c29-8e83-384ed76e7b77  20.0  18.0    79    83\n",
       "2  d9863d9e-0473-482d-ae21-bcfb05f1753c   9.0  17.0    71    82\n",
       "3  a9738fe5-2df1-4661-b147-de435b7eeeb6  17.0  20.0    80    94\n",
       "4  529faf72-9186-4652-8fcc-58ba8b46a5cb  22.0  20.0    89    86"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n",
      "Algo score, swaps needed: 1247.49193\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1813, 1981, 1568, 2121, 1575, 1863, 2028, 1707, 1575, 1886]\n",
      "Pseudo-random, avergae swaps needed: 1837.51867\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n",
      "Algo score, swaps needed: 1213.96858\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1938, 1632, 1899, 1795, 1660, 2098, 1546, 1900, 1834, 1816]\n",
      "Pseudo-random, avergae swaps needed: 1758.20714\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ab1fbbba-d6e0-4551-aed0-3fc164fa8570</td>\n",
       "      <td>27.0</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9ce4f501-176f-4180-ad8a-0a44c2c21a21</td>\n",
       "      <td>34.0</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a7931454-2722-4600-ad92-baccd06a38a8</td>\n",
       "      <td>42.0</td>\n",
       "      <td>95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>467e6ace-0fab-4f54-a773-979058184ac2</td>\n",
       "      <td>25.0</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3307e79a-3565-478c-8749-b095d12c7e56</td>\n",
       "      <td>20.5</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q2_x  Q2_y\n",
       "0  ab1fbbba-d6e0-4551-aed0-3fc164fa8570  27.0    86\n",
       "1  9ce4f501-176f-4180-ad8a-0a44c2c21a21  34.0    85\n",
       "2  a7931454-2722-4600-ad92-baccd06a38a8  42.0    95\n",
       "3  467e6ace-0fab-4f54-a773-979058184ac2  25.0    84\n",
       "4  3307e79a-3565-478c-8749-b095d12c7e56  20.5    82"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n",
      "Algo score, swaps needed: 114.0029\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[169, 222, 170, 208, 151, 148, 156, 168, 226, 157]\n",
      "Pseudo-random, avergae swaps needed: 182.47244\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>330e4993-30e4-4192-9e02-07e960fad404</td>\n",
       "      <td>16.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>89</td>\n",
       "      <td>86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bb846754-6c06-4082-a827-eee27975d766</td>\n",
       "      <td>10.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>80</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f835cc4c-68ff-4a67-a6a8-17c0d3997a2a</td>\n",
       "      <td>16.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>85</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>054dc8e6-dc85-4ebb-92f4-f09e5bc559c9</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>91</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>af7980d7-0f7d-4600-82c5-14f34f16f53a</td>\n",
       "      <td>5.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  330e4993-30e4-4192-9e02-07e960fad404  16.5   6.0    89    86\n",
       "1  bb846754-6c06-4082-a827-eee27975d766  10.0   4.5    80    85\n",
       "2  f835cc4c-68ff-4a67-a6a8-17c0d3997a2a  16.0   5.0    85    85\n",
       "3  054dc8e6-dc85-4ebb-92f4-f09e5bc559c9  10.0   3.0    91    59\n",
       "4  af7980d7-0f7d-4600-82c5-14f34f16f53a   5.5   3.0    77    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n",
      "Algo score, swaps needed: 908.98728\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1473, 1182, 1417, 1249, 1410, 1401, 1076, 1334, 1079, 1336]\n",
      "Pseudo-random, avergae swaps needed: 1324.01157\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n",
      "Algo score, swaps needed: 907.39679\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1154, 1370, 1052, 1319, 1287, 1226, 1149, 1108, 1222, 1072]\n",
      "Pseudo-random, avergae swaps needed: 1219.49946\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>496aeba0-ea0b-4003-bd5b-7e4ac8bd9073</td>\n",
       "      <td>47</td>\n",
       "      <td>35</td>\n",
       "      <td>20</td>\n",
       "      <td>83</td>\n",
       "      <td>74</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>49fdf82b-9794-42a5-9a7b-8fb177111688</td>\n",
       "      <td>28</td>\n",
       "      <td>25</td>\n",
       "      <td>17</td>\n",
       "      <td>80</td>\n",
       "      <td>82</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ed6cf722-94b9-4474-8bf7-318cb15713b8</td>\n",
       "      <td>67</td>\n",
       "      <td>25</td>\n",
       "      <td>28</td>\n",
       "      <td>85</td>\n",
       "      <td>89</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c1b34a08-528e-4824-85ac-1d2043c2bb32</td>\n",
       "      <td>15</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ee5b756f-84b7-4707-ab9a-a9064b65ebe2</td>\n",
       "      <td>43</td>\n",
       "      <td>25</td>\n",
       "      <td>16</td>\n",
       "      <td>87</td>\n",
       "      <td>85</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q3_x  Q1_y  Q2_y  Q3_y\n",
       "0  496aeba0-ea0b-4003-bd5b-7e4ac8bd9073    47    35    20    83    74    81\n",
       "1  49fdf82b-9794-42a5-9a7b-8fb177111688    28    25    17    80    82    80\n",
       "2  ed6cf722-94b9-4474-8bf7-318cb15713b8    67    25    28    85    89    87\n",
       "3  c1b34a08-528e-4824-85ac-1d2043c2bb32    15     5     2    80    81    78\n",
       "4  ee5b756f-84b7-4707-ab9a-a9064b65ebe2    43    25    16    87    85    80"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n",
      "Algo score, swaps needed: 1174.50544\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1373, 1263, 1463, 1603, 1533, 1465, 1509, 1349, 1563, 1521]\n",
      "Pseudo-random, avergae swaps needed: 1472.93425\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n",
      "Algo score, swaps needed: 1041.00029\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1430, 1487, 1436, 1619, 1536, 1363, 1301, 1294, 1438, 1435]\n",
      "Pseudo-random, avergae swaps needed: 1450.00772\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n",
      "Algo score, swaps needed: 803.94188\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1520, 1399, 1458, 1504, 1481, 1406, 1533, 1621, 1241, 1440]\n",
      "Pseudo-random, avergae swaps needed: 1458.14977\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0a1cbcb8-fb01-45be-bb70-9c97770f647c</td>\n",
       "      <td>18.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>83</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25b1719f-d684-490a-b15d-9596e54c29ee</td>\n",
       "      <td>14.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>76</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5b092aa8-350b-4919-8267-f3309dbcb8fc</td>\n",
       "      <td>19.5</td>\n",
       "      <td>19.0</td>\n",
       "      <td>81</td>\n",
       "      <td>81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>348bfa7e-1c58-4b5b-9b11-af9b507ee3cb</td>\n",
       "      <td>17.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>79</td>\n",
       "      <td>84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>858981e3-e43d-4777-ac97-16678c7f6ac0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>88</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  0a1cbcb8-fb01-45be-bb70-9c97770f647c  18.0  11.0    83    84\n",
       "1  25b1719f-d684-490a-b15d-9596e54c29ee  14.0   3.5    76    63\n",
       "2  5b092aa8-350b-4919-8267-f3309dbcb8fc  19.5  19.0    81    81\n",
       "3  348bfa7e-1c58-4b5b-9b11-af9b507ee3cb  17.0  15.0    79    84\n",
       "4  858981e3-e43d-4777-ac97-16678c7f6ac0  19.0  20.0    88    92"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n",
      "Algo score, swaps needed: 1688.05055\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2388, 1906, 2017, 2148, 2162, 2038, 2254, 1994, 2205, 1959]\n",
      "Pseudo-random, avergae swaps needed: 2045.85193\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n",
      "Algo score, swaps needed: 1578.9718\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2294, 1959, 1827, 1962, 2017, 2153, 2094, 1988, 1979, 1939]\n",
      "Pseudo-random, avergae swaps needed: 1993.83577\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[996.46505, 1054.98323, 959.91972, 1247.49193, 1213.96858, 114.0029, 908.98728, 907.39679, 1174.50544, 1041.00029, 803.94188, 1688.05055, 1578.9718]\n",
      "N: 13 \tMean:  1053.0527261538462 \tVar:  135086.9684930775\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.14618, 1566.48594, 1560.00533, 1837.51867, 1758.20714, 182.47244, 1324.01157, 1219.49946, 1472.93425, 1450.00772, 1458.14977, 2045.85193, 1993.83577]\n",
      "N: 13 \tMean:  1489.8558592307693 \tVar:  198432.03504219826 \n",
      "\n",
      "1 ) 1499.14618 > 996.46505 => 40.28521141091355\n",
      "2 ) 1566.48594 > 1054.98323 => 39.0241255440742\n",
      "3 ) 1560.00533 > 959.91972 => 47.62725859644119\n",
      "4 ) 1837.51867 > 1247.49193 => 38.25119693267829\n",
      "5 ) 1758.20714 > 1213.96858 => 36.62223308923336\n",
      "6 ) 182.47244 > 114.0029 => 46.18902874013063\n",
      "7 ) 1324.01157 > 908.98728 => 37.17192151711138\n",
      "8 ) 1219.49946 > 907.39679 => 29.34818000642956\n",
      "9 ) 1472.93425 > 1174.50544 => 22.544710735223592\n",
      "10 ) 1450.00772 > 1041.00029 => 32.83870853550568\n",
      "11 ) 1458.14977 > 803.94188 => 57.840971209101994\n",
      "12 ) 2045.85193 > 1688.05055 => 19.16500936575077\n",
      "13 ) 1993.83577 > 1578.9718 => 23.223415304172114\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "36.163997768212795 \n",
      "\n",
      "Ttest_relResult(statistic=-10.007411205172588, pvalue=3.5527684242044965e-07)\n",
      "Cohen's d:  1.0276835318353643\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp1',normv=1,score=0,goal=\"centroid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3563df1a-2ee1-4323-a204-fb3a13e13f57</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>80</td>\n",
       "      <td>70</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>70d749d7-dd2e-4b83-be1a-1409b0dcb3e2</td>\n",
       "      <td>9</td>\n",
       "      <td>35</td>\n",
       "      <td>17</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>354b8e2c-ce96-49fc-8a7b-80c7b8f04be3</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>80</td>\n",
       "      <td>63</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>e8c7236a-2cae-40ea-8d17-5405ce4a0e0f</td>\n",
       "      <td>12</td>\n",
       "      <td>17</td>\n",
       "      <td>14</td>\n",
       "      <td>87</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>07ac7940-e702-498a-a97c-333cb233a847</td>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x  Q1_x  Q2_x  SHORT_ANS_y  \\\n",
       "0  3563df1a-2ee1-4323-a204-fb3a13e13f57            1     1     4           80   \n",
       "1  70d749d7-dd2e-4b83-be1a-1409b0dcb3e2            9    35    17           83   \n",
       "2  354b8e2c-ce96-49fc-8a7b-80c7b8f04be3            3    13     2           80   \n",
       "3  e8c7236a-2cae-40ea-8d17-5405ce4a0e0f           12    17    14           87   \n",
       "4  07ac7940-e702-498a-a97c-333cb233a847            2    14     4           77   \n",
       "\n",
       "   Q1_y  Q2_y  \n",
       "0    70    77  \n",
       "1    83    83  \n",
       "2    63    80  \n",
       "3    80    80  \n",
       "4    83    77  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n",
      "Algo score, swaps needed: 991.10428\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1310, 1608, 1730, 1663, 1360, 1523, 1626, 1454, 1405, 1740]\n",
      "Pseudo-random, avergae swaps needed: 1500.25208\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n",
      "Algo score, swaps needed: 1023.11096\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1431, 1541, 1747, 1377, 1676, 1525, 1692, 1442, 1375, 1642]\n",
      "Pseudo-random, avergae swaps needed: 1566.50075\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n",
      "Algo score, swaps needed: 918.7735\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1702, 1558, 1459, 1443, 1767, 1453, 1580, 1413, 1650, 1367]\n",
      "Pseudo-random, avergae swaps needed: 1560.09365\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>05ab5f67-d7d3-441f-8d3c-e879ece151f9</td>\n",
       "      <td>19.0</td>\n",
       "      <td>22.0</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d3d89043-c41a-4cf9-aba0-17b47f4750c9</td>\n",
       "      <td>14.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6dfac62a-26c6-4c69-afe0-39aa1b1a3e44</td>\n",
       "      <td>15.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>d57e3e5c-10f4-4610-9ea6-49852423ede4</td>\n",
       "      <td>18.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6c430619-e918-45cc-8164-b092cb27ee8a</td>\n",
       "      <td>11.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  05ab5f67-d7d3-441f-8d3c-e879ece151f9  19.0  22.0    80    80\n",
       "1  d3d89043-c41a-4cf9-aba0-17b47f4750c9  14.0  12.0    87    87\n",
       "2  6dfac62a-26c6-4c69-afe0-39aa1b1a3e44  15.0  19.0    83    83\n",
       "3  d57e3e5c-10f4-4610-9ea6-49852423ede4  18.0  23.0    87    87\n",
       "4  6c430619-e918-45cc-8164-b092cb27ee8a  11.0  21.0    83    80"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n",
      "Algo score, swaps needed: 1331.50755\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1666, 1787, 1725, 1750, 1861, 1565, 2026, 1805, 1799, 2077]\n",
      "Pseudo-random, avergae swaps needed: 1837.78548\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n",
      "Algo score, swaps needed: 1371.00386\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1762, 1976, 1553, 1693, 1799, 1760, 1792, 1938, 1873, 1643]\n",
      "Pseudo-random, avergae swaps needed: 1757.7674\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a0b1247e-71db-499f-a8e9-5191a179ac6c</td>\n",
       "      <td>28.0</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>740ba105-1c40-49be-b74d-e78072fdbf34</td>\n",
       "      <td>25.0</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2872fa41-b643-452a-b6c6-87f4e140f336</td>\n",
       "      <td>43.0</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>467e6ace-0fab-4f54-a773-979058184ac2</td>\n",
       "      <td>25.0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a7931454-2722-4600-ad92-baccd06a38a8</td>\n",
       "      <td>42.0</td>\n",
       "      <td>93</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q2_x  Q2_y\n",
       "0  a0b1247e-71db-499f-a8e9-5191a179ac6c  28.0    77\n",
       "1  740ba105-1c40-49be-b74d-e78072fdbf34  25.0    73\n",
       "2  2872fa41-b643-452a-b6c6-87f4e140f336  43.0    87\n",
       "3  467e6ace-0fab-4f54-a773-979058184ac2  25.0    80\n",
       "4  a7931454-2722-4600-ad92-baccd06a38a8  42.0    93"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n",
      "Algo score, swaps needed: 122.00948\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[197, 212, 230, 219, 190, 212, 178, 175, 158, 144]\n",
      "Pseudo-random, avergae swaps needed: 182.51655\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>867eac98-6036-4439-ad09-d9c49c511b56</td>\n",
       "      <td>10.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5d7cfc35-9967-4ab0-bab2-011f73bd8856</td>\n",
       "      <td>12.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4e4a357a-bc9d-46b9-b0c5-01e68cb32ba0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>70</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>eb2a08b7-b5f4-4b84-b4bb-b3ae393af3e7</td>\n",
       "      <td>14.5</td>\n",
       "      <td>5.5</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>77b75df3-9aa3-4653-861d-e8dd9c5cf920</td>\n",
       "      <td>7.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  867eac98-6036-4439-ad09-d9c49c511b56  10.5   6.0    80    83\n",
       "1  5d7cfc35-9967-4ab0-bab2-011f73bd8856  12.0   4.0    87    83\n",
       "2  4e4a357a-bc9d-46b9-b0c5-01e68cb32ba0  10.0   3.0    70    83\n",
       "3  eb2a08b7-b5f4-4b84-b4bb-b3ae393af3e7  14.5   5.5    83    83\n",
       "4  77b75df3-9aa3-4653-861d-e8dd9c5cf920   7.0   4.0    77    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n",
      "Algo score, swaps needed: 903.94001\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1455, 1467, 1342, 1351, 1419, 1302, 1333, 1366, 1257, 1313]\n",
      "Pseudo-random, avergae swaps needed: 1324.79922\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n",
      "Algo score, swaps needed: 1074.27863\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1257, 1232, 1104, 1248, 1359, 1134, 1220, 1170, 1187, 1202]\n",
      "Pseudo-random, avergae swaps needed: 1218.77426\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>e4873808-277a-47f8-9235-2188343dc4ab</td>\n",
       "      <td>32</td>\n",
       "      <td>33</td>\n",
       "      <td>30</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>496aeba0-ea0b-4003-bd5b-7e4ac8bd9073</td>\n",
       "      <td>47</td>\n",
       "      <td>35</td>\n",
       "      <td>20</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>737af13d-d5ba-4cf8-beaa-63bdc403d760</td>\n",
       "      <td>38</td>\n",
       "      <td>2</td>\n",
       "      <td>17</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1dd97d65-c89a-4744-bfc1-c14311cc97b7</td>\n",
       "      <td>38</td>\n",
       "      <td>36</td>\n",
       "      <td>20</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7e5c12f6-95ca-485d-91ec-71a4031f22d5</td>\n",
       "      <td>20</td>\n",
       "      <td>5</td>\n",
       "      <td>20</td>\n",
       "      <td>80</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q3_x  Q1_y  Q2_y  Q3_y\n",
       "0  e4873808-277a-47f8-9235-2188343dc4ab    32    33    30    87    87    83\n",
       "1  496aeba0-ea0b-4003-bd5b-7e4ac8bd9073    47    35    20    83    77    80\n",
       "2  737af13d-d5ba-4cf8-beaa-63bdc403d760    38     2    17    83    77    80\n",
       "3  1dd97d65-c89a-4744-bfc1-c14311cc97b7    38    36    20    80    83    80\n",
       "4  7e5c12f6-95ca-485d-91ec-71a4031f22d5    20     5    20    80    77    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n",
      "Algo score, swaps needed: 1189.94081\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1534, 1358, 1507, 1549, 1670, 1694, 1405, 1519, 1365, 1353]\n",
      "Pseudo-random, avergae swaps needed: 1472.50809\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n",
      "Algo score, swaps needed: 1004.40795\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1523, 1338, 1593, 1306, 1328, 1397, 1350, 1304, 1455, 1503]\n",
      "Pseudo-random, avergae swaps needed: 1450.15038\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n",
      "Algo score, swaps needed: 773.30653\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1637, 1310, 1430, 1499, 1580, 1559, 1270, 1730, 1328, 1565]\n",
      "Pseudo-random, avergae swaps needed: 1458.04045\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>95eca05c-7d2d-4e74-80e7-f4dd862e71b1</td>\n",
       "      <td>24.5</td>\n",
       "      <td>24.0</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>c268db25-0c25-4dcb-a5dd-13e0b0b3b4e5</td>\n",
       "      <td>26.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d5556a5d-e8c0-4613-853e-e6448f546a32</td>\n",
       "      <td>20.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>d7cbd32b-0f8a-4ee7-a4a8-5679eab81ae5</td>\n",
       "      <td>11.5</td>\n",
       "      <td>13.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c2f90f17-5912-4155-a6d4-60da6b5733b4</td>\n",
       "      <td>17.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  95eca05c-7d2d-4e74-80e7-f4dd862e71b1  24.5  24.0    87    87\n",
       "1  c268db25-0c25-4dcb-a5dd-13e0b0b3b4e5  26.5  21.0    83    87\n",
       "2  d5556a5d-e8c0-4613-853e-e6448f546a32  20.5  18.0    87    87\n",
       "3  d7cbd32b-0f8a-4ee7-a4a8-5679eab81ae5  11.5  13.0    80    83\n",
       "4  c2f90f17-5912-4155-a6d4-60da6b5733b4  17.0  18.0    80    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n",
      "Algo score, swaps needed: 1767.51708\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1877, 2270, 1927, 2161, 2495, 1985, 2122, 2196, 1972, 1813]\n",
      "Pseudo-random, avergae swaps needed: 2045.88172\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n",
      "Algo score, swaps needed: 1633.86401\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2064, 1826, 1848, 1903, 1798, 2116, 1994, 1746, 1985, 2002]\n",
      "Pseudo-random, avergae swaps needed: 1993.96205\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[991.10428, 1023.11096, 918.7735, 1331.50755, 1371.00386, 122.00948, 903.94001, 1074.27863, 1189.94081, 1004.40795, 773.30653, 1767.51708, 1633.86401]\n",
      "N: 13 \tMean:  1084.981896153846 \tVar:  155756.32549119272\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1500.25208, 1566.50075, 1560.09365, 1837.78548, 1757.7674, 182.51655, 1324.79922, 1218.77426, 1472.50809, 1450.15038, 1458.04045, 2045.88172, 1993.96205]\n",
      "N: 13 \tMean:  1489.925544615385 \tVar:  198445.37818547385 \n",
      "\n",
      "1 ) 1500.25208 > 991.10428 => 40.87314108688972\n",
      "2 ) 1566.50075 > 1023.11096 => 41.966893175656814\n",
      "3 ) 1560.09365 > 918.7735 => 51.74300284708682\n",
      "4 ) 1837.78548 > 1331.50755 => 31.948950457257023\n",
      "5 ) 1757.7674 > 1371.00386 => 24.723030727404463\n",
      "6 ) 182.51655 > 122.00948 => 39.738520874553814\n",
      "7 ) 1324.79922 > 903.94001 => 37.76657262859774\n",
      "8 ) 1218.77426 > 1074.27863 => 12.602904244393587\n",
      "9 ) 1472.50809 > 1189.94081 => 21.226118555740165\n",
      "10 ) 1450.15038 > 1004.40795 => 36.31956303926988\n",
      "11 ) 1458.04045 > 773.30653 => 61.37404232845939\n",
      "12 ) 2045.88172 > 1767.51708 => 14.599293417724892\n",
      "13 ) 1993.96205 > 1633.86401 => 19.852001393914684\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "33.44107959822685 \n",
      "\n",
      "Ttest_relResult(statistic=-8.035474277283694, pvalue=3.5932817377187584e-06)\n",
      "Cohen's d:  0.9244921189473885\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp1',normv=0,score=1,goal=\"centroid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>68975dd8-703c-43ba-8247-1e6cde4fe067</td>\n",
       "      <td>0.120927</td>\n",
       "      <td>-0.147495</td>\n",
       "      <td>0.798654</td>\n",
       "      <td>0.503427</td>\n",
       "      <td>0.869034</td>\n",
       "      <td>0.537341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>72162df3-a93a-447b-9155-24b29b7c4a05</td>\n",
       "      <td>0.401122</td>\n",
       "      <td>0.367653</td>\n",
       "      <td>0.354091</td>\n",
       "      <td>0.530529</td>\n",
       "      <td>0.201332</td>\n",
       "      <td>0.047264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0</td>\n",
       "      <td>-0.999856</td>\n",
       "      <td>-1.486880</td>\n",
       "      <td>-0.535036</td>\n",
       "      <td>0.636763</td>\n",
       "      <td>0.584401</td>\n",
       "      <td>0.496988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b54398fb-14ce-4ad0-9a86-9813160d627f</td>\n",
       "      <td>2.362492</td>\n",
       "      <td>2.119157</td>\n",
       "      <td>1.391404</td>\n",
       "      <td>0.516124</td>\n",
       "      <td>0.262030</td>\n",
       "      <td>0.379471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>02694d63-a2bc-45b7-98fd-77c89066e667</td>\n",
       "      <td>-1.280052</td>\n",
       "      <td>-1.074762</td>\n",
       "      <td>-0.535036</td>\n",
       "      <td>0.007574</td>\n",
       "      <td>-1.101450</td>\n",
       "      <td>0.458851</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x      Q1_x      Q2_x  \\\n",
       "0  68975dd8-703c-43ba-8247-1e6cde4fe067     0.120927 -0.147495  0.798654   \n",
       "1  72162df3-a93a-447b-9155-24b29b7c4a05     0.401122  0.367653  0.354091   \n",
       "2  ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0    -0.999856 -1.486880 -0.535036   \n",
       "3  b54398fb-14ce-4ad0-9a86-9813160d627f     2.362492  2.119157  1.391404   \n",
       "4  02694d63-a2bc-45b7-98fd-77c89066e667    -1.280052 -1.074762 -0.535036   \n",
       "\n",
       "   SHORT_ANS_y      Q1_y      Q2_y  \n",
       "0     0.503427  0.869034  0.537341  \n",
       "1     0.530529  0.201332  0.047264  \n",
       "2     0.636763  0.584401  0.496988  \n",
       "3     0.516124  0.262030  0.379471  \n",
       "4     0.007574 -1.101450  0.458851  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>68975dd8-703c-43ba-8247-1e6cde4fe067</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.120927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>72162df3-a93a-447b-9155-24b29b7c4a05</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.401122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.999856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b54398fb-14ce-4ad0-9a86-9813160d627f</td>\n",
       "      <td>Human</td>\n",
       "      <td>2.362492</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>02694d63-a2bc-45b7-98fd-77c89066e667</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.280052</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  68975dd8-703c-43ba-8247-1e6cde4fe067  Human  0.120927\n",
       "1  72162df3-a93a-447b-9155-24b29b7c4a05  Human  0.401122\n",
       "2  ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0  Human -0.999856\n",
       "3  b54398fb-14ce-4ad0-9a86-9813160d627f  Human  2.362492\n",
       "4  02694d63-a2bc-45b7-98fd-77c89066e667  Human -1.280052"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.426259</td>\n",
       "      <td>2.485892</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000030</td>\n",
       "      <td>[0.23, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.424240</td>\n",
       "      <td>2.455869</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000041</td>\n",
       "      <td>[0.23, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.421274</td>\n",
       "      <td>2.455869</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000041</td>\n",
       "      <td>[0.22, 0.58]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.597730</td>\n",
       "      <td>2.485892</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000030</td>\n",
       "      <td>[0.38, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.595742</td>\n",
       "      <td>2.455869</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000041</td>\n",
       "      <td>[0.37, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.592812</td>\n",
       "      <td>2.455869</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000041</td>\n",
       "      <td>[0.37, 0.74]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.426259  2.485892   80   81  0.000030   \n",
       "ICC2      Single random raters  0.424240  2.455869   80   80  0.000041   \n",
       "ICC3       Single fixed raters  0.421274  2.455869   80   80  0.000041   \n",
       "ICC1k  Average raters absolute  0.597730  2.485892   80   81  0.000030   \n",
       "ICC2k    Average random raters  0.595742  2.455869   80   80  0.000041   \n",
       "ICC3k     Average fixed raters  0.592812  2.455869   80   80  0.000041   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.23, 0.59]  \n",
       "ICC2   [0.23, 0.59]  \n",
       "ICC3   [0.22, 0.58]  \n",
       "ICC1k  [0.38, 0.74]  \n",
       "ICC2k  [0.37, 0.74]  \n",
       "ICC3k  [0.37, 0.74]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.446784149383303\n",
      "Algo score, swaps needed: 998.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1501, 1258, 1382, 1416, 1286, 1546, 1583, 1355, 1486, 1469]\n",
      "Pseudo-random, avergae swaps needed: 1499.90568\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>68975dd8-703c-43ba-8247-1e6cde4fe067</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.147495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>72162df3-a93a-447b-9155-24b29b7c4a05</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.367653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.486880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b54398fb-14ce-4ad0-9a86-9813160d627f</td>\n",
       "      <td>Human</td>\n",
       "      <td>2.119157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>02694d63-a2bc-45b7-98fd-77c89066e667</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.074762</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  68975dd8-703c-43ba-8247-1e6cde4fe067  Human -0.147495\n",
       "1  72162df3-a93a-447b-9155-24b29b7c4a05  Human  0.367653\n",
       "2  ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0  Human -1.486880\n",
       "3  b54398fb-14ce-4ad0-9a86-9813160d627f  Human  2.119157\n",
       "4  02694d63-a2bc-45b7-98fd-77c89066e667  Human -1.074762"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.506941</td>\n",
       "      <td>3.056311</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>5.281393e-07</td>\n",
       "      <td>[0.33, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.505419</td>\n",
       "      <td>3.018623</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>7.604948e-07</td>\n",
       "      <td>[0.32, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.502317</td>\n",
       "      <td>3.018623</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>7.604948e-07</td>\n",
       "      <td>[0.32, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.672808</td>\n",
       "      <td>3.056311</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>5.281393e-07</td>\n",
       "      <td>[0.49, 0.79]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.671466</td>\n",
       "      <td>3.018623</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>7.604948e-07</td>\n",
       "      <td>[0.49, 0.79]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.668723</td>\n",
       "      <td>3.018623</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>7.604948e-07</td>\n",
       "      <td>[0.48, 0.79]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.506941  3.056311   80   81  5.281393e-07   \n",
       "ICC2      Single random raters  0.505419  3.018623   80   80  7.604948e-07   \n",
       "ICC3       Single fixed raters  0.502317  3.018623   80   80  7.604948e-07   \n",
       "ICC1k  Average raters absolute  0.672808  3.056311   80   81  5.281393e-07   \n",
       "ICC2k    Average random raters  0.671466  3.018623   80   80  7.604948e-07   \n",
       "ICC3k     Average fixed raters  0.668723  3.018623   80   80  7.604948e-07   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.33, 0.65]  \n",
       "ICC2   [0.32, 0.65]  \n",
       "ICC3   [0.32, 0.65]  \n",
       "ICC1k  [0.49, 0.79]  \n",
       "ICC2k  [0.49, 0.79]  \n",
       "ICC3k  [0.48, 0.79]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.48698613796947476\n",
      "Algo score, swaps needed: 1054.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1514, 1862, 1390, 1671, 1496, 1632, 1676, 1651, 1435, 1786]\n",
      "Pseudo-random, avergae swaps needed: 1566.23236\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>68975dd8-703c-43ba-8247-1e6cde4fe067</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.798654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>72162df3-a93a-447b-9155-24b29b7c4a05</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.354091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.535036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b54398fb-14ce-4ad0-9a86-9813160d627f</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.391404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>02694d63-a2bc-45b7-98fd-77c89066e667</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.535036</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  68975dd8-703c-43ba-8247-1e6cde4fe067  Human  0.798654\n",
       "1  72162df3-a93a-447b-9155-24b29b7c4a05  Human  0.354091\n",
       "2  ddee1d35-ea9d-4e90-b6dd-4a9c5ff90aa0  Human -0.535036\n",
       "3  b54398fb-14ce-4ad0-9a86-9813160d627f  Human  1.391404\n",
       "4  02694d63-a2bc-45b7-98fd-77c89066e667  Human -0.535036"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.418784</td>\n",
       "      <td>2.441062</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>[0.22, 0.58]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.416990</td>\n",
       "      <td>2.415478</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.22, 0.58]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.414430</td>\n",
       "      <td>2.415478</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.22, 0.58]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.590342</td>\n",
       "      <td>2.441062</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>[0.36, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.588557</td>\n",
       "      <td>2.415478</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.36, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.586003</td>\n",
       "      <td>2.415478</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.36, 0.73]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.418784  2.441062   80   81  0.000042   \n",
       "ICC2      Single random raters  0.416990  2.415478   80   80  0.000054   \n",
       "ICC3       Single fixed raters  0.414430  2.415478   80   80  0.000054   \n",
       "ICC1k  Average raters absolute  0.590342  2.441062   80   81  0.000042   \n",
       "ICC2k    Average random raters  0.588557  2.415478   80   80  0.000054   \n",
       "ICC3k     Average fixed raters  0.586003  2.415478   80   80  0.000054   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.22, 0.58]  \n",
       "ICC2   [0.22, 0.58]  \n",
       "ICC3   [0.22, 0.58]  \n",
       "ICC1k  [0.36, 0.74]  \n",
       "ICC2k  [0.36, 0.74]  \n",
       "ICC3k  [0.36, 0.73]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.457167253080513\n",
      "Algo score, swaps needed: 930.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1701, 1560, 1672, 1504, 1827, 1572, 1689, 1567, 1405, 1634]\n",
      "Pseudo-random, avergae swaps needed: 1560.29005\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>d2a0868b-f1ef-43d6-a277-0b4b33bd78d0</td>\n",
       "      <td>-1.220500</td>\n",
       "      <td>0.657180</td>\n",
       "      <td>-0.181182</td>\n",
       "      <td>-0.910573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d16fe8d2-bea5-4a56-bbd8-8f8bd80bfe97</td>\n",
       "      <td>-0.039371</td>\n",
       "      <td>0.957513</td>\n",
       "      <td>1.074028</td>\n",
       "      <td>0.502680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1a67bcd2-4be4-4221-ab02-0cea41c74809</td>\n",
       "      <td>0.196855</td>\n",
       "      <td>-0.243819</td>\n",
       "      <td>-0.390988</td>\n",
       "      <td>0.322597</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>fad9d7b4-6245-4e76-ae50-76d00c933b26</td>\n",
       "      <td>-1.811065</td>\n",
       "      <td>-1.144818</td>\n",
       "      <td>-0.706111</td>\n",
       "      <td>-0.550720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>140fa70f-38c7-45ce-86f4-f66ecc52218d</td>\n",
       "      <td>-0.393710</td>\n",
       "      <td>0.807347</td>\n",
       "      <td>-1.009999</td>\n",
       "      <td>-1.275871</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  d2a0868b-f1ef-43d6-a277-0b4b33bd78d0 -1.220500  0.657180 -0.181182   \n",
       "1  d16fe8d2-bea5-4a56-bbd8-8f8bd80bfe97 -0.039371  0.957513  1.074028   \n",
       "2  1a67bcd2-4be4-4221-ab02-0cea41c74809  0.196855 -0.243819 -0.390988   \n",
       "3  fad9d7b4-6245-4e76-ae50-76d00c933b26 -1.811065 -1.144818 -0.706111   \n",
       "4  140fa70f-38c7-45ce-86f4-f66ecc52218d -0.393710  0.807347 -1.009999   \n",
       "\n",
       "       Q2_y  \n",
       "0 -0.910573  \n",
       "1  0.502680  \n",
       "2  0.322597  \n",
       "3 -0.550720  \n",
       "4 -1.275871  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>d2a0868b-f1ef-43d6-a277-0b4b33bd78d0</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.220500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d16fe8d2-bea5-4a56-bbd8-8f8bd80bfe97</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.039371</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1a67bcd2-4be4-4221-ab02-0cea41c74809</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.196855</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>fad9d7b4-6245-4e76-ae50-76d00c933b26</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.811065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>140fa70f-38c7-45ce-86f4-f66ecc52218d</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.393710</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  d2a0868b-f1ef-43d6-a277-0b4b33bd78d0  Human -1.220500\n",
       "1  d16fe8d2-bea5-4a56-bbd8-8f8bd80bfe97  Human -0.039371\n",
       "2  1a67bcd2-4be4-4221-ab02-0cea41c74809  Human  0.196855\n",
       "3  fad9d7b4-6245-4e76-ae50-76d00c933b26  Human -1.811065\n",
       "4  140fa70f-38c7-45ce-86f4-f66ecc52218d  Human -0.393710"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.435186</td>\n",
       "      <td>2.540989</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000009</td>\n",
       "      <td>[0.25, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.433440</td>\n",
       "      <td>2.513555</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>[0.25, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.430776</td>\n",
       "      <td>2.513555</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>[0.24, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.606452</td>\n",
       "      <td>2.540989</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000009</td>\n",
       "      <td>[0.4, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.604755</td>\n",
       "      <td>2.513555</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>[0.4, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.602157</td>\n",
       "      <td>2.513555</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>[0.39, 0.74]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.435186  2.540989   87   88  0.000009   \n",
       "ICC2      Single random raters  0.433440  2.513555   87   87  0.000013   \n",
       "ICC3       Single fixed raters  0.430776  2.513555   87   87  0.000013   \n",
       "ICC1k  Average raters absolute  0.606452  2.540989   87   88  0.000009   \n",
       "ICC2k    Average random raters  0.604755  2.513555   87   87  0.000013   \n",
       "ICC3k     Average fixed raters  0.602157  2.513555   87   87  0.000013   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.25, 0.59]  \n",
       "ICC2   [0.25, 0.59]  \n",
       "ICC3   [0.24, 0.59]  \n",
       "ICC1k   [0.4, 0.74]  \n",
       "ICC2k   [0.4, 0.74]  \n",
       "ICC3k  [0.39, 0.74]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.4602594296957241\n",
      "Algo score, swaps needed: 1275.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1785, 2042, 1790, 1734, 1833, 1907, 1810, 1817, 1613, 1692]\n",
      "Pseudo-random, avergae swaps needed: 1837.85846\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>d2a0868b-f1ef-43d6-a277-0b4b33bd78d0</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.657180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d16fe8d2-bea5-4a56-bbd8-8f8bd80bfe97</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.957513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1a67bcd2-4be4-4221-ab02-0cea41c74809</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.243819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>fad9d7b4-6245-4e76-ae50-76d00c933b26</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.144818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>140fa70f-38c7-45ce-86f4-f66ecc52218d</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.807347</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  d2a0868b-f1ef-43d6-a277-0b4b33bd78d0  Human  0.657180\n",
       "1  d16fe8d2-bea5-4a56-bbd8-8f8bd80bfe97  Human  0.957513\n",
       "2  1a67bcd2-4be4-4221-ab02-0cea41c74809  Human -0.243819\n",
       "3  fad9d7b4-6245-4e76-ae50-76d00c933b26  Human -1.144818\n",
       "4  140fa70f-38c7-45ce-86f4-f66ecc52218d  Human  0.807347"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.403045</td>\n",
       "      <td>2.350336</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>[0.21, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.401305</td>\n",
       "      <td>2.327666</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.21, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.398978</td>\n",
       "      <td>2.327666</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.21, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.574529</td>\n",
       "      <td>2.350336</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000042</td>\n",
       "      <td>[0.35, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.572759</td>\n",
       "      <td>2.327666</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.35, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.570385</td>\n",
       "      <td>2.327666</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>[0.34, 0.72]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.403045  2.350336   87   88  0.000042   \n",
       "ICC2      Single random raters  0.401305  2.327666   87   87  0.000054   \n",
       "ICC3       Single fixed raters  0.398978  2.327666   87   87  0.000054   \n",
       "ICC1k  Average raters absolute  0.574529  2.350336   87   88  0.000042   \n",
       "ICC2k    Average random raters  0.572759  2.327666   87   87  0.000054   \n",
       "ICC3k     Average fixed raters  0.570385  2.327666   87   87  0.000054   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.21, 0.56]  \n",
       "ICC2   [0.21, 0.56]  \n",
       "ICC3   [0.21, 0.56]  \n",
       "ICC1k  [0.35, 0.72]  \n",
       "ICC2k  [0.35, 0.72]  \n",
       "ICC3k  [0.34, 0.72]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.4202472486054576\n",
      "Algo score, swaps needed: 1224.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1954, 2106, 1798, 1916, 1883, 1613, 1413, 1467, 1651, 1745]\n",
      "Pseudo-random, avergae swaps needed: 1757.2092\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c6f9b6cb-c5b8-4622-bdad-51aabad768cd</td>\n",
       "      <td>-0.540495</td>\n",
       "      <td>0.969560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>dda68a48-2475-4338-9956-67fc7efbf74b</td>\n",
       "      <td>1.010796</td>\n",
       "      <td>1.324034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f870a18d-a560-4a6a-90b8-e1660d8877d1</td>\n",
       "      <td>0.294815</td>\n",
       "      <td>-0.354441</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2872fa41-b643-452a-b6c6-87f4e140f336</td>\n",
       "      <td>1.726776</td>\n",
       "      <td>1.232272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>eabdc089-b66a-4412-a8ca-7241ef5e59a3</td>\n",
       "      <td>0.294815</td>\n",
       "      <td>-1.465595</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q2_x      Q2_y\n",
       "0  c6f9b6cb-c5b8-4622-bdad-51aabad768cd -0.540495  0.969560\n",
       "1  dda68a48-2475-4338-9956-67fc7efbf74b  1.010796  1.324034\n",
       "2  f870a18d-a560-4a6a-90b8-e1660d8877d1  0.294815 -0.354441\n",
       "3  2872fa41-b643-452a-b6c6-87f4e140f336  1.726776  1.232272\n",
       "4  eabdc089-b66a-4412-a8ca-7241ef5e59a3  0.294815 -1.465595"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c6f9b6cb-c5b8-4622-bdad-51aabad768cd</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.540495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>dda68a48-2475-4338-9956-67fc7efbf74b</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.010796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f870a18d-a560-4a6a-90b8-e1660d8877d1</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.294815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2872fa41-b643-452a-b6c6-87f4e140f336</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.726776</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3c3d7b28-148d-4465-a14c-14429181c310</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.063175</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  c6f9b6cb-c5b8-4622-bdad-51aabad768cd  Human -0.540495\n",
       "1  dda68a48-2475-4338-9956-67fc7efbf74b  Human  1.010796\n",
       "2  f870a18d-a560-4a6a-90b8-e1660d8877d1  Human  0.294815\n",
       "3  2872fa41-b643-452a-b6c6-87f4e140f336  Human  1.726776\n",
       "4  3c3d7b28-148d-4465-a14c-14429181c310  Human -0.063175"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.568949</td>\n",
       "      <td>3.639821</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.000549</td>\n",
       "      <td>[0.26, 0.77]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.566593</td>\n",
       "      <td>3.550264</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>[0.25, 0.77]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.560465</td>\n",
       "      <td>3.550264</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>[0.24, 0.77]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.725261</td>\n",
       "      <td>3.639821</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.000549</td>\n",
       "      <td>[0.41, 0.87]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.723344</td>\n",
       "      <td>3.550264</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>[0.4, 0.87]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.718331</td>\n",
       "      <td>3.550264</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000783</td>\n",
       "      <td>[0.39, 0.87]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.568949  3.639821   27   28  0.000549   \n",
       "ICC2      Single random raters  0.566593  3.550264   27   27  0.000783   \n",
       "ICC3       Single fixed raters  0.560465  3.550264   27   27  0.000783   \n",
       "ICC1k  Average raters absolute  0.725261  3.639821   27   28  0.000549   \n",
       "ICC2k    Average random raters  0.723344  3.550264   27   27  0.000783   \n",
       "ICC3k     Average fixed raters  0.718331  3.550264   27   27  0.000783   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.26, 0.77]  \n",
       "ICC2   [0.25, 0.77]  \n",
       "ICC3   [0.24, 0.77]  \n",
       "ICC1k  [0.41, 0.87]  \n",
       "ICC2k   [0.4, 0.87]  \n",
       "ICC3k  [0.39, 0.87]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.5166799574694312\n",
      "Algo score, swaps needed: 117.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[157, 209, 170, 190, 193, 193, 150, 215, 181, 198]\n",
      "Pseudo-random, avergae swaps needed: 182.56658\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7e891b5b-c0ab-461d-993c-86650e3db067</td>\n",
       "      <td>-0.136447</td>\n",
       "      <td>-0.957701</td>\n",
       "      <td>-0.670447</td>\n",
       "      <td>0.205259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1c8aeeb4-a1f3-492a-ae5c-87af4b3ad8e2</td>\n",
       "      <td>0.196644</td>\n",
       "      <td>-0.038751</td>\n",
       "      <td>-0.988681</td>\n",
       "      <td>0.217333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7e96b523-06ee-45f4-9c2e-6247c261f646</td>\n",
       "      <td>0.030099</td>\n",
       "      <td>-0.038751</td>\n",
       "      <td>0.761941</td>\n",
       "      <td>0.387743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5c11e5b2-953a-43ee-a21b-79f47664d5b2</td>\n",
       "      <td>0.529736</td>\n",
       "      <td>0.420724</td>\n",
       "      <td>-0.791942</td>\n",
       "      <td>0.066243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3c3821f0-5e8a-45a9-b9bb-06f850649a0a</td>\n",
       "      <td>0.196644</td>\n",
       "      <td>0.880199</td>\n",
       "      <td>0.186709</td>\n",
       "      <td>0.263660</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  7e891b5b-c0ab-461d-993c-86650e3db067 -0.136447 -0.957701 -0.670447   \n",
       "1  1c8aeeb4-a1f3-492a-ae5c-87af4b3ad8e2  0.196644 -0.038751 -0.988681   \n",
       "2  7e96b523-06ee-45f4-9c2e-6247c261f646  0.030099 -0.038751  0.761941   \n",
       "3  5c11e5b2-953a-43ee-a21b-79f47664d5b2  0.529736  0.420724 -0.791942   \n",
       "4  3c3821f0-5e8a-45a9-b9bb-06f850649a0a  0.196644  0.880199  0.186709   \n",
       "\n",
       "       Q2_y  \n",
       "0  0.205259  \n",
       "1  0.217333  \n",
       "2  0.387743  \n",
       "3  0.066243  \n",
       "4  0.263660  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7e891b5b-c0ab-461d-993c-86650e3db067</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.136447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1c8aeeb4-a1f3-492a-ae5c-87af4b3ad8e2</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.196644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7e96b523-06ee-45f4-9c2e-6247c261f646</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.030099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5c11e5b2-953a-43ee-a21b-79f47664d5b2</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.529736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3c3821f0-5e8a-45a9-b9bb-06f850649a0a</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.196644</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  7e891b5b-c0ab-461d-993c-86650e3db067  Human -0.136447\n",
       "1  1c8aeeb4-a1f3-492a-ae5c-87af4b3ad8e2  Human  0.196644\n",
       "2  7e96b523-06ee-45f4-9c2e-6247c261f646  Human  0.030099\n",
       "3  5c11e5b2-953a-43ee-a21b-79f47664d5b2  Human  0.529736\n",
       "4  3c3821f0-5e8a-45a9-b9bb-06f850649a0a  Human  0.196644"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.540244</td>\n",
       "      <td>3.350137</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>2.086765e-07</td>\n",
       "      <td>[0.36, 0.68]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.538824</td>\n",
       "      <td>3.305833</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>3.106074e-07</td>\n",
       "      <td>[0.36, 0.68]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.535514</td>\n",
       "      <td>3.305833</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>3.106074e-07</td>\n",
       "      <td>[0.35, 0.68]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.701505</td>\n",
       "      <td>3.350137</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>2.086765e-07</td>\n",
       "      <td>[0.53, 0.81]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.700306</td>\n",
       "      <td>3.305833</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>3.106074e-07</td>\n",
       "      <td>[0.52, 0.81]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.697504</td>\n",
       "      <td>3.305833</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>3.106074e-07</td>\n",
       "      <td>[0.52, 0.81]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.540244  3.350137   74   75  2.086765e-07   \n",
       "ICC2      Single random raters  0.538824  3.305833   74   74  3.106074e-07   \n",
       "ICC3       Single fixed raters  0.535514  3.305833   74   74  3.106074e-07   \n",
       "ICC1k  Average raters absolute  0.701505  3.350137   74   75  2.086765e-07   \n",
       "ICC2k    Average random raters  0.700306  3.305833   74   74  3.106074e-07   \n",
       "ICC3k     Average fixed raters  0.697504  3.305833   74   74  3.106074e-07   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.36, 0.68]  \n",
       "ICC2   [0.36, 0.68]  \n",
       "ICC3   [0.35, 0.68]  \n",
       "ICC1k  [0.53, 0.81]  \n",
       "ICC2k  [0.52, 0.81]  \n",
       "ICC3k  [0.52, 0.81]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.5102469877573461\n",
      "Algo score, swaps needed: 903.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1540, 1252, 1347, 1305, 1219, 1314, 1375, 1252, 1314, 1198]\n",
      "Pseudo-random, avergae swaps needed: 1323.97221\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7e891b5b-c0ab-461d-993c-86650e3db067</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.957701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1c8aeeb4-a1f3-492a-ae5c-87af4b3ad8e2</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.038751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7e96b523-06ee-45f4-9c2e-6247c261f646</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.038751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5c11e5b2-953a-43ee-a21b-79f47664d5b2</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.420724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3c3821f0-5e8a-45a9-b9bb-06f850649a0a</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.880199</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  7e891b5b-c0ab-461d-993c-86650e3db067  Human -0.957701\n",
       "1  1c8aeeb4-a1f3-492a-ae5c-87af4b3ad8e2  Human -0.038751\n",
       "2  7e96b523-06ee-45f4-9c2e-6247c261f646  Human -0.038751\n",
       "3  5c11e5b2-953a-43ee-a21b-79f47664d5b2  Human  0.420724\n",
       "4  3c3821f0-5e8a-45a9-b9bb-06f850649a0a  Human  0.880199"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.177572</td>\n",
       "      <td>1.431825</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.061668</td>\n",
       "      <td>[-0.05, 0.39]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.173036</td>\n",
       "      <td>1.412976</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>[-0.06, 0.39]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.171148</td>\n",
       "      <td>1.412976</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>[-0.06, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.301591</td>\n",
       "      <td>1.431825</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.061668</td>\n",
       "      <td>[-0.1, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.295023</td>\n",
       "      <td>1.412976</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>[-0.12, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.292274</td>\n",
       "      <td>1.412976</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.069672</td>\n",
       "      <td>[-0.12, 0.55]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.177572  1.431825   74   75  0.061668   \n",
       "ICC2      Single random raters  0.173036  1.412976   74   74  0.069672   \n",
       "ICC3       Single fixed raters  0.171148  1.412976   74   74  0.069672   \n",
       "ICC1k  Average raters absolute  0.301591  1.431825   74   75  0.061668   \n",
       "ICC2k    Average random raters  0.295023  1.412976   74   74  0.069672   \n",
       "ICC3k     Average fixed raters  0.292274  1.412976   74   74  0.069672   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.05, 0.39]  \n",
       "ICC2   [-0.06, 0.39]  \n",
       "ICC3   [-0.06, 0.38]  \n",
       "ICC1k   [-0.1, 0.56]  \n",
       "ICC2k  [-0.12, 0.56]  \n",
       "ICC3k  [-0.12, 0.55]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.23952975753122707\n",
      "Algo score, swaps needed: 831.00127\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1164, 1204, 1431, 1154, 1247, 1357, 1284, 1287, 1036, 1192]\n",
      "Pseudo-random, avergae swaps needed: 1219.08912\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39ed6bf0-e1fe-45eb-88ec-fdda59f87141</td>\n",
       "      <td>1.238243</td>\n",
       "      <td>1.374969</td>\n",
       "      <td>1.970159</td>\n",
       "      <td>1.333419</td>\n",
       "      <td>0.823378</td>\n",
       "      <td>0.792395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1460cb8a-dfcb-4279-97df-77be7213064a</td>\n",
       "      <td>-0.821550</td>\n",
       "      <td>0.349049</td>\n",
       "      <td>2.067547</td>\n",
       "      <td>-0.219469</td>\n",
       "      <td>0.768768</td>\n",
       "      <td>0.642212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4b2b17a2-b7f3-4386-b297-74b1f14afa6b</td>\n",
       "      <td>1.650202</td>\n",
       "      <td>0.451641</td>\n",
       "      <td>-0.367166</td>\n",
       "      <td>0.899784</td>\n",
       "      <td>-0.386050</td>\n",
       "      <td>0.069744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ed6cf722-94b9-4474-8bf7-318cb15713b8</td>\n",
       "      <td>1.375563</td>\n",
       "      <td>-0.163911</td>\n",
       "      <td>0.411942</td>\n",
       "      <td>0.227733</td>\n",
       "      <td>0.862354</td>\n",
       "      <td>0.521911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1dd97d65-c89a-4744-bfc1-c14311cc97b7</td>\n",
       "      <td>-0.615570</td>\n",
       "      <td>0.964601</td>\n",
       "      <td>-0.367166</td>\n",
       "      <td>-0.361493</td>\n",
       "      <td>-0.021564</td>\n",
       "      <td>-0.413383</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q3_x  \\\n",
       "0  39ed6bf0-e1fe-45eb-88ec-fdda59f87141  1.238243  1.374969  1.970159   \n",
       "1  1460cb8a-dfcb-4279-97df-77be7213064a -0.821550  0.349049  2.067547   \n",
       "2  4b2b17a2-b7f3-4386-b297-74b1f14afa6b  1.650202  0.451641 -0.367166   \n",
       "3  ed6cf722-94b9-4474-8bf7-318cb15713b8  1.375563 -0.163911  0.411942   \n",
       "4  1dd97d65-c89a-4744-bfc1-c14311cc97b7 -0.615570  0.964601 -0.367166   \n",
       "\n",
       "       Q1_y      Q2_y      Q3_y  \n",
       "0  1.333419  0.823378  0.792395  \n",
       "1 -0.219469  0.768768  0.642212  \n",
       "2  0.899784 -0.386050  0.069744  \n",
       "3  0.227733  0.862354  0.521911  \n",
       "4 -0.361493 -0.021564 -0.413383  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39ed6bf0-e1fe-45eb-88ec-fdda59f87141</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.238243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1460cb8a-dfcb-4279-97df-77be7213064a</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.821550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4b2b17a2-b7f3-4386-b297-74b1f14afa6b</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.650202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ed6cf722-94b9-4474-8bf7-318cb15713b8</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.375563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1dd97d65-c89a-4744-bfc1-c14311cc97b7</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.615570</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  39ed6bf0-e1fe-45eb-88ec-fdda59f87141  Human  1.238243\n",
       "1  1460cb8a-dfcb-4279-97df-77be7213064a  Human -0.821550\n",
       "2  4b2b17a2-b7f3-4386-b297-74b1f14afa6b  Human  1.650202\n",
       "3  ed6cf722-94b9-4474-8bf7-318cb15713b8  Human  1.375563\n",
       "4  1dd97d65-c89a-4744-bfc1-c14311cc97b7  Human -0.615570"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.277498</td>\n",
       "      <td>1.768157</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.006460</td>\n",
       "      <td>[0.06, 0.47]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.274216</td>\n",
       "      <td>1.746300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.007704</td>\n",
       "      <td>[0.05, 0.47]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.271748</td>\n",
       "      <td>1.746300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.007704</td>\n",
       "      <td>[0.05, 0.47]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.434439</td>\n",
       "      <td>1.768157</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.006460</td>\n",
       "      <td>[0.11, 0.64]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.430408</td>\n",
       "      <td>1.746300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.007704</td>\n",
       "      <td>[0.1, 0.64]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.427361</td>\n",
       "      <td>1.746300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.007704</td>\n",
       "      <td>[0.1, 0.63]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.277498  1.768157   77   78  0.006460   \n",
       "ICC2      Single random raters  0.274216  1.746300   77   77  0.007704   \n",
       "ICC3       Single fixed raters  0.271748  1.746300   77   77  0.007704   \n",
       "ICC1k  Average raters absolute  0.434439  1.768157   77   78  0.006460   \n",
       "ICC2k    Average random raters  0.430408  1.746300   77   77  0.007704   \n",
       "ICC3k     Average fixed raters  0.427361  1.746300   77   77  0.007704   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.06, 0.47]  \n",
       "ICC2   [0.05, 0.47]  \n",
       "ICC3   [0.05, 0.47]  \n",
       "ICC1k  [0.11, 0.64]  \n",
       "ICC2k   [0.1, 0.64]  \n",
       "ICC3k   [0.1, 0.63]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.2949785232758061\n",
      "Algo score, swaps needed: 1176.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1425, 1358, 1286, 1390, 1540, 1488, 1385, 1448, 1321, 1709]\n",
      "Pseudo-random, avergae swaps needed: 1472.53206\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39ed6bf0-e1fe-45eb-88ec-fdda59f87141</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.374969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1460cb8a-dfcb-4279-97df-77be7213064a</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.349049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4b2b17a2-b7f3-4386-b297-74b1f14afa6b</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.451641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ed6cf722-94b9-4474-8bf7-318cb15713b8</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.163911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1dd97d65-c89a-4744-bfc1-c14311cc97b7</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.964601</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  39ed6bf0-e1fe-45eb-88ec-fdda59f87141  Human  1.374969\n",
       "1  1460cb8a-dfcb-4279-97df-77be7213064a  Human  0.349049\n",
       "2  4b2b17a2-b7f3-4386-b297-74b1f14afa6b  Human  0.451641\n",
       "3  ed6cf722-94b9-4474-8bf7-318cb15713b8  Human -0.163911\n",
       "4  1dd97d65-c89a-4744-bfc1-c14311cc97b7  Human  0.964601"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.319052</td>\n",
       "      <td>1.937081</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.002003</td>\n",
       "      <td>[0.11, 0.5]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.316352</td>\n",
       "      <td>1.914873</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.002425</td>\n",
       "      <td>[0.1, 0.5]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.313864</td>\n",
       "      <td>1.914873</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.002425</td>\n",
       "      <td>[0.1, 0.5]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.483759</td>\n",
       "      <td>1.937081</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.002003</td>\n",
       "      <td>[0.19, 0.67]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.480650</td>\n",
       "      <td>1.914873</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.002425</td>\n",
       "      <td>[0.18, 0.67]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.477772</td>\n",
       "      <td>1.914873</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.002425</td>\n",
       "      <td>[0.18, 0.67]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.319052  1.937081   77   78  0.002003   \n",
       "ICC2      Single random raters  0.316352  1.914873   77   77  0.002425   \n",
       "ICC3       Single fixed raters  0.313864  1.914873   77   77  0.002425   \n",
       "ICC1k  Average raters absolute  0.483759  1.937081   77   78  0.002003   \n",
       "ICC2k    Average random raters  0.480650  1.914873   77   77  0.002425   \n",
       "ICC3k     Average fixed raters  0.477772  1.914873   77   77  0.002425   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1    [0.11, 0.5]  \n",
       "ICC2     [0.1, 0.5]  \n",
       "ICC3     [0.1, 0.5]  \n",
       "ICC1k  [0.19, 0.67]  \n",
       "ICC2k  [0.18, 0.67]  \n",
       "ICC3k  [0.18, 0.67]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.3509009258745911\n",
      "Algo score, swaps needed: 1043.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1781, 1446, 1468, 1600, 1529, 1356, 1529, 1433, 1557, 1438]\n",
      "Pseudo-random, avergae swaps needed: 1449.32702\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39ed6bf0-e1fe-45eb-88ec-fdda59f87141</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.970159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1460cb8a-dfcb-4279-97df-77be7213064a</td>\n",
       "      <td>Human</td>\n",
       "      <td>2.067547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4b2b17a2-b7f3-4386-b297-74b1f14afa6b</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.367166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ed6cf722-94b9-4474-8bf7-318cb15713b8</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.411942</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1dd97d65-c89a-4744-bfc1-c14311cc97b7</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.367166</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  39ed6bf0-e1fe-45eb-88ec-fdda59f87141  Human  1.970159\n",
       "1  1460cb8a-dfcb-4279-97df-77be7213064a  Human  2.067547\n",
       "2  4b2b17a2-b7f3-4386-b297-74b1f14afa6b  Human -0.367166\n",
       "3  ed6cf722-94b9-4474-8bf7-318cb15713b8  Human  0.411942\n",
       "4  1dd97d65-c89a-4744-bfc1-c14311cc97b7  Human -0.367166"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.557776</td>\n",
       "      <td>3.522595</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>4.057886e-08</td>\n",
       "      <td>[0.38, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.556504</td>\n",
       "      <td>3.477485</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>6.165675e-08</td>\n",
       "      <td>[0.38, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.553321</td>\n",
       "      <td>3.477485</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>6.165675e-08</td>\n",
       "      <td>[0.38, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.716118</td>\n",
       "      <td>3.522595</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>4.057886e-08</td>\n",
       "      <td>[0.56, 0.82]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.715069</td>\n",
       "      <td>3.477485</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>6.165675e-08</td>\n",
       "      <td>[0.55, 0.82]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.712436</td>\n",
       "      <td>3.477485</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>6.165675e-08</td>\n",
       "      <td>[0.55, 0.82]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.557776  3.522595   77   78  4.057886e-08   \n",
       "ICC2      Single random raters  0.556504  3.477485   77   77  6.165675e-08   \n",
       "ICC3       Single fixed raters  0.553321  3.477485   77   77  6.165675e-08   \n",
       "ICC1k  Average raters absolute  0.716118  3.522595   77   78  4.057886e-08   \n",
       "ICC2k    Average random raters  0.715069  3.477485   77   77  6.165675e-08   \n",
       "ICC3k     Average fixed raters  0.712436  3.477485   77   77  6.165675e-08   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.38, 0.69]  \n",
       "ICC2   [0.38, 0.69]  \n",
       "ICC3   [0.38, 0.69]  \n",
       "ICC1k  [0.56, 0.82]  \n",
       "ICC2k  [0.55, 0.82]  \n",
       "ICC3k  [0.55, 0.82]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.6055882812907261\n",
      "Algo score, swaps needed: 783.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1311, 1515, 1372, 1294, 1557, 1657, 1481, 1288, 1405, 1291]\n",
      "Pseudo-random, avergae swaps needed: 1457.98898\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1345e6e8-c15f-4a31-a12a-3642a8d2eb92</td>\n",
       "      <td>-0.047909</td>\n",
       "      <td>0.836545</td>\n",
       "      <td>1.295143</td>\n",
       "      <td>1.300717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b79bcda6-c021-453b-96e9-fd6d58c20469</td>\n",
       "      <td>-0.945323</td>\n",
       "      <td>-1.284442</td>\n",
       "      <td>-0.273292</td>\n",
       "      <td>-0.944506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>85ca12a0-f4e5-4045-8350-676e0e159221</td>\n",
       "      <td>0.849505</td>\n",
       "      <td>0.258094</td>\n",
       "      <td>-0.398175</td>\n",
       "      <td>-0.105680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>da2f8241-c0df-4081-b6bc-347b35a3d470</td>\n",
       "      <td>0.625152</td>\n",
       "      <td>1.029362</td>\n",
       "      <td>0.263096</td>\n",
       "      <td>-0.056564</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c2f90f17-5912-4155-a6d4-60da6b5733b4</td>\n",
       "      <td>-0.272262</td>\n",
       "      <td>0.450911</td>\n",
       "      <td>-0.509601</td>\n",
       "      <td>0.290321</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  1345e6e8-c15f-4a31-a12a-3642a8d2eb92 -0.047909  0.836545  1.295143   \n",
       "1  b79bcda6-c021-453b-96e9-fd6d58c20469 -0.945323 -1.284442 -0.273292   \n",
       "2  85ca12a0-f4e5-4045-8350-676e0e159221  0.849505  0.258094 -0.398175   \n",
       "3  da2f8241-c0df-4081-b6bc-347b35a3d470  0.625152  1.029362  0.263096   \n",
       "4  c2f90f17-5912-4155-a6d4-60da6b5733b4 -0.272262  0.450911 -0.509601   \n",
       "\n",
       "       Q2_y  \n",
       "0  1.300717  \n",
       "1 -0.944506  \n",
       "2 -0.105680  \n",
       "3 -0.056564  \n",
       "4  0.290321  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1345e6e8-c15f-4a31-a12a-3642a8d2eb92</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.047909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b79bcda6-c021-453b-96e9-fd6d58c20469</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.945323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>85ca12a0-f4e5-4045-8350-676e0e159221</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.849505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>da2f8241-c0df-4081-b6bc-347b35a3d470</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.625152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c2f90f17-5912-4155-a6d4-60da6b5733b4</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.272262</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  1345e6e8-c15f-4a31-a12a-3642a8d2eb92  Human -0.047909\n",
       "1  b79bcda6-c021-453b-96e9-fd6d58c20469  Human -0.945323\n",
       "2  85ca12a0-f4e5-4045-8350-676e0e159221  Human  0.849505\n",
       "3  da2f8241-c0df-4081-b6bc-347b35a3d470  Human  0.625152\n",
       "4  c2f90f17-5912-4155-a6d4-60da6b5733b4  Human -0.272262"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.090017</td>\n",
       "      <td>1.197843</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.194626</td>\n",
       "      <td>[-0.12, 0.29]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.085672</td>\n",
       "      <td>1.185463</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.209323</td>\n",
       "      <td>[-0.12, 0.29]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.084862</td>\n",
       "      <td>1.185463</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.209323</td>\n",
       "      <td>[-0.12, 0.28]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.165166</td>\n",
       "      <td>1.197843</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.194626</td>\n",
       "      <td>[-0.26, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.157823</td>\n",
       "      <td>1.185463</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.209323</td>\n",
       "      <td>[-0.28, 0.44]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.156447</td>\n",
       "      <td>1.185463</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.209323</td>\n",
       "      <td>[-0.28, 0.44]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.090017  1.197843   91   92  0.194626   \n",
       "ICC2      Single random raters  0.085672  1.185463   91   91  0.209323   \n",
       "ICC3       Single fixed raters  0.084862  1.185463   91   91  0.209323   \n",
       "ICC1k  Average raters absolute  0.165166  1.197843   91   92  0.194626   \n",
       "ICC2k    Average random raters  0.157823  1.185463   91   91  0.209323   \n",
       "ICC3k     Average fixed raters  0.156447  1.185463   91   91  0.209323   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.12, 0.29]  \n",
       "ICC2   [-0.12, 0.29]  \n",
       "ICC3   [-0.12, 0.28]  \n",
       "ICC1k  [-0.26, 0.45]  \n",
       "ICC2k  [-0.28, 0.44]  \n",
       "ICC3k  [-0.28, 0.44]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.15220449016453974\n",
      "Algo score, swaps needed: 1705.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2081, 2187, 2193, 2140, 2030, 1831, 2220, 1940, 2074, 1901]\n",
      "Pseudo-random, avergae swaps needed: 2045.00169\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1345e6e8-c15f-4a31-a12a-3642a8d2eb92</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.836545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b79bcda6-c021-453b-96e9-fd6d58c20469</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.284442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>85ca12a0-f4e5-4045-8350-676e0e159221</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.258094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>da2f8241-c0df-4081-b6bc-347b35a3d470</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.029362</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c2f90f17-5912-4155-a6d4-60da6b5733b4</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.450911</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  1345e6e8-c15f-4a31-a12a-3642a8d2eb92  Human  0.836545\n",
       "1  b79bcda6-c021-453b-96e9-fd6d58c20469  Human -1.284442\n",
       "2  85ca12a0-f4e5-4045-8350-676e0e159221  Human  0.258094\n",
       "3  da2f8241-c0df-4081-b6bc-347b35a3d470  Human  1.029362\n",
       "4  c2f90f17-5912-4155-a6d4-60da6b5733b4  Human  0.450911"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.267120</td>\n",
       "      <td>1.728959</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.004700</td>\n",
       "      <td>[0.07, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.264758</td>\n",
       "      <td>1.713932</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.005423</td>\n",
       "      <td>[0.06, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.263062</td>\n",
       "      <td>1.713932</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.005423</td>\n",
       "      <td>[0.06, 0.44]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.421617</td>\n",
       "      <td>1.728959</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.004700</td>\n",
       "      <td>[0.13, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.418669</td>\n",
       "      <td>1.713932</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.005423</td>\n",
       "      <td>[0.12, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.416546</td>\n",
       "      <td>1.713932</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.005423</td>\n",
       "      <td>[0.12, 0.61]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.267120  1.728959   91   92  0.004700   \n",
       "ICC2      Single random raters  0.264758  1.713932   91   91  0.005423   \n",
       "ICC3       Single fixed raters  0.263062  1.713932   91   91  0.005423   \n",
       "ICC1k  Average raters absolute  0.421617  1.728959   91   92  0.004700   \n",
       "ICC2k    Average random raters  0.418669  1.713932   91   91  0.005423   \n",
       "ICC3k     Average fixed raters  0.416546  1.713932   91   91  0.005423   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.07, 0.45]  \n",
       "ICC2   [0.06, 0.45]  \n",
       "ICC3   [0.06, 0.44]  \n",
       "ICC1k  [0.13, 0.62]  \n",
       "ICC2k  [0.12, 0.62]  \n",
       "ICC3k  [0.12, 0.61]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.25302929011688136\n",
      "Algo score, swaps needed: 1589.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2053, 1928, 1861, 2327, 2038, 1984, 1850, 1868, 1715, 2068]\n",
      "Pseudo-random, avergae swaps needed: 1994.42136\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[998.0, 1054.0, 930.0, 1275.0, 1224.0, 117.0, 903.0, 831.00127, 1176.0, 1043.0, 783.0, 1705.0, 1589.0]\n",
      "N: 13 \tMean:  1048.30779 \tVar:  141907.86286697842\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.90568, 1566.23236, 1560.29005, 1837.85846, 1757.2092, 182.56658, 1323.97221, 1219.08912, 1472.53206, 1449.32702, 1457.98898, 2045.00169, 1994.42136]\n",
      "N: 13 \tMean:  1489.722674615385 \tVar:  198388.26468606733 \n",
      "\n",
      "1 ) 1499.90568 > 998.0 => 40.186119437464114\n",
      "2 ) 1566.23236 > 1054.0 => 39.09823936377917\n",
      "3 ) 1560.29005 > 930.0 => 50.61981033092913\n",
      "4 ) 1837.85846 > 1275.0 => 36.16344702033127\n",
      "5 ) 1757.2092 > 1224.0 => 35.77133734861679\n",
      "6 ) 182.56658 > 117.0 => 43.77429551721022\n",
      "7 ) 1323.97221 > 903.0 => 37.806687313803515\n",
      "8 ) 1219.08912 > 831.00127 => 37.86055989462982\n",
      "9 ) 1472.53206 > 1176.0 => 22.392182030071403\n",
      "10 ) 1449.32702 > 1043.0 => 32.60623639990871\n",
      "11 ) 1457.98898 > 783.0 => 60.24027659430973\n",
      "12 ) 2045.00169 > 1705.0 => 18.133415294540846\n",
      "13 ) 1994.42136 > 1589.0 => 22.62761306976191\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "36.713863047335124 \n",
      "\n",
      "Ttest_relResult(statistic=-10.059009471465309, pvalue=3.36061487875277e-07)\n",
      "Cohen's d:  1.0281403763866397\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp1',normv=1,score=2,goal=\"centroid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>499fc744-a3e7-4927-afd5-b4df47ca7874</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>10</td>\n",
       "      <td>73</td>\n",
       "      <td>87</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ebd5efa9-f2f3-47f8-9959-1918046163c8</td>\n",
       "      <td>4</td>\n",
       "      <td>13</td>\n",
       "      <td>8</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>341a81d8-db2d-4017-95ca-f11d39c62331</td>\n",
       "      <td>5</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7b622388-4835-41b0-916e-bb4ca373d6c4</td>\n",
       "      <td>13</td>\n",
       "      <td>26</td>\n",
       "      <td>25</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6e472a30-2be4-4670-bff5-0a78a5fab822</td>\n",
       "      <td>9</td>\n",
       "      <td>33</td>\n",
       "      <td>21</td>\n",
       "      <td>70</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x  Q1_x  Q2_x  SHORT_ANS_y  \\\n",
       "0  499fc744-a3e7-4927-afd5-b4df47ca7874            0    25    10           73   \n",
       "1  ebd5efa9-f2f3-47f8-9959-1918046163c8            4    13     8           87   \n",
       "2  341a81d8-db2d-4017-95ca-f11d39c62331            5    10    10           87   \n",
       "3  7b622388-4835-41b0-916e-bb4ca373d6c4           13    26    25           83   \n",
       "4  6e472a30-2be4-4670-bff5-0a78a5fab822            9    33    21           70   \n",
       "\n",
       "   Q1_y  Q2_y  \n",
       "0    87    80  \n",
       "1    83    83  \n",
       "2    83    83  \n",
       "3    83    80  \n",
       "4    87    83  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n",
      "Algo score, swaps needed: 1441.96402\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1436, 1483, 1738, 1483, 1603, 1516, 1292, 1531, 1522, 1636]\n",
      "Pseudo-random, avergae swaps needed: 1499.1106\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n",
      "Algo score, swaps needed: 1165.3767\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1689, 1696, 1601, 1684, 1452, 1535, 1388, 1478, 1512, 1470]\n",
      "Pseudo-random, avergae swaps needed: 1566.27993\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n",
      "Algo score, swaps needed: 1284.64089\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1535, 1616, 1611, 1268, 1311, 1636, 1582, 1632, 1591, 1562]\n",
      "Pseudo-random, avergae swaps needed: 1560.16865\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8d5c28dd-7532-4b93-b654-7c254bd47fce</td>\n",
       "      <td>17.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>87</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>da455e55-d2ac-4054-aae9-e62723a9dc6c</td>\n",
       "      <td>17.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>73</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ebf4deeb-44b2-4c4d-992e-b4e44cda8cc9</td>\n",
       "      <td>17.5</td>\n",
       "      <td>20.0</td>\n",
       "      <td>90</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>af76b772-5886-4e53-a62d-e87b17f052c9</td>\n",
       "      <td>13.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>70</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>faca6a15-3d4c-4771-ad55-f8fc6f55b71a</td>\n",
       "      <td>10.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>73</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  8d5c28dd-7532-4b93-b654-7c254bd47fce  17.0  20.0    87    90\n",
       "1  da455e55-d2ac-4054-aae9-e62723a9dc6c  17.0  17.0    73    80\n",
       "2  ebf4deeb-44b2-4c4d-992e-b4e44cda8cc9  17.5  20.0    90    83\n",
       "3  af76b772-5886-4e53-a62d-e87b17f052c9  13.0  15.0    70    77\n",
       "4  faca6a15-3d4c-4771-ad55-f8fc6f55b71a  10.0  13.0    73    80"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n",
      "Algo score, swaps needed: 1217.07259\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1564, 1857, 1897, 2003, 1973, 1738, 1768, 1815, 1870, 1777]\n",
      "Pseudo-random, avergae swaps needed: 1837.17634\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n",
      "Algo score, swaps needed: 1703.51235\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1900, 1846, 1666, 1523, 1645, 1857, 1991, 1795, 1687, 1664]\n",
      "Pseudo-random, avergae swaps needed: 1757.39543\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>dbd130ac-e5e2-4938-9aad-d4a30c0406f0</td>\n",
       "      <td>45.0</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>47e96791-9e80-4638-b63a-a00903ce8508</td>\n",
       "      <td>30.0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a6db6331-93ae-48a4-ab5d-04cd34cf3989</td>\n",
       "      <td>25.0</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3307e79a-3565-478c-8749-b095d12c7e56</td>\n",
       "      <td>20.5</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>467e6ace-0fab-4f54-a773-979058184ac2</td>\n",
       "      <td>25.0</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q2_x  Q2_y\n",
       "0  dbd130ac-e5e2-4938-9aad-d4a30c0406f0  45.0    83\n",
       "1  47e96791-9e80-4638-b63a-a00903ce8508  30.0    80\n",
       "2  a6db6331-93ae-48a4-ab5d-04cd34cf3989  25.0    77\n",
       "3  3307e79a-3565-478c-8749-b095d12c7e56  20.5    80\n",
       "4  467e6ace-0fab-4f54-a773-979058184ac2  25.0    87"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n",
      "Algo score, swaps needed: 131.48975\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[170, 163, 183, 202, 168, 147, 195, 182, 159, 202]\n",
      "Pseudo-random, avergae swaps needed: 182.60821\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1ebf5189-e22f-4dfd-9f72-5ab06b37ca76</td>\n",
       "      <td>9.0</td>\n",
       "      <td>6.5</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>eecc8b68-5ac3-4146-8346-f775be3eceef</td>\n",
       "      <td>14.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>90</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>cbcfa92d-9d08-4487-84f4-9431b21fc8fa</td>\n",
       "      <td>13.5</td>\n",
       "      <td>4.5</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>25f7b700-3561-4102-ba38-4753c54f9eb0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5ff78e7f-5dbb-4fce-a97d-f2dc6c5a6fbe</td>\n",
       "      <td>12.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  1ebf5189-e22f-4dfd-9f72-5ab06b37ca76   9.0   6.5    80    83\n",
       "1  eecc8b68-5ac3-4146-8346-f775be3eceef  14.0   4.0    90    83\n",
       "2  cbcfa92d-9d08-4487-84f4-9431b21fc8fa  13.5   4.5    83    83\n",
       "3  25f7b700-3561-4102-ba38-4753c54f9eb0  11.0   5.0    83    83\n",
       "4  5ff78e7f-5dbb-4fce-a97d-f2dc6c5a6fbe  12.0   6.0    83    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n",
      "Algo score, swaps needed: 1024.01478\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1384, 1308, 1303, 1302, 1298, 1362, 1207, 1375, 1230, 1514]\n",
      "Pseudo-random, avergae swaps needed: 1324.45789\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n",
      "Algo score, swaps needed: 1129.23608\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1216, 1280, 1276, 1289, 1081, 1243, 1201, 1239, 1133, 1292]\n",
      "Pseudo-random, avergae swaps needed: 1219.24489\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>b21e248a-45c5-43ae-b06a-10261395fd1c</td>\n",
       "      <td>66</td>\n",
       "      <td>28</td>\n",
       "      <td>33</td>\n",
       "      <td>87</td>\n",
       "      <td>77</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1b273341-a255-40ef-9a90-17fbdf05af4b</td>\n",
       "      <td>48</td>\n",
       "      <td>39</td>\n",
       "      <td>31</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>a7bf4849-6187-42da-8439-600dce189d0f</td>\n",
       "      <td>37</td>\n",
       "      <td>28</td>\n",
       "      <td>19</td>\n",
       "      <td>73</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>30234d01-ef3f-41bf-ae4a-1efaba3522d8</td>\n",
       "      <td>72</td>\n",
       "      <td>41</td>\n",
       "      <td>23</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>fbc35066-d18b-49cc-9540-9ec417e0e160</td>\n",
       "      <td>38</td>\n",
       "      <td>25</td>\n",
       "      <td>31</td>\n",
       "      <td>87</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q3_x  Q1_y  Q2_y  Q3_y\n",
       "0  b21e248a-45c5-43ae-b06a-10261395fd1c    66    28    33    87    77    80\n",
       "1  1b273341-a255-40ef-9a90-17fbdf05af4b    48    39    31    80    83    87\n",
       "2  a7bf4849-6187-42da-8439-600dce189d0f    37    28    19    73    83    83\n",
       "3  30234d01-ef3f-41bf-ae4a-1efaba3522d8    72    41    23    83    87    87\n",
       "4  fbc35066-d18b-49cc-9540-9ec417e0e160    38    25    31    87    80    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n",
      "Algo score, swaps needed: 1279.13438\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1659, 1575, 1460, 1316, 1404, 1372, 1392, 1448, 1530, 1249]\n",
      "Pseudo-random, avergae swaps needed: 1472.41306\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n",
      "Algo score, swaps needed: 1111.52218\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1551, 1425, 1617, 1418, 1379, 1218, 1516, 1339, 1546, 1557]\n",
      "Pseudo-random, avergae swaps needed: 1449.40899\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n",
      "Algo score, swaps needed: 947.0725\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1365, 1473, 1398, 1606, 1517, 1234, 1233, 1528, 1500, 1492]\n",
      "Pseudo-random, avergae swaps needed: 1457.26523\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>f234936b-f804-47e5-b956-9f3c9feef83c</td>\n",
       "      <td>20.5</td>\n",
       "      <td>22.0</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>83c20b41-645b-4931-97bf-870d498e43de</td>\n",
       "      <td>14.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0091731f-bae9-4326-a66a-a744137584a2</td>\n",
       "      <td>13.5</td>\n",
       "      <td>8.0</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>858981e3-e43d-4777-ac97-16678c7f6ac0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>90</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>85ca12a0-f4e5-4045-8350-676e0e159221</td>\n",
       "      <td>22.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  f234936b-f804-47e5-b956-9f3c9feef83c  20.5  22.0    83    87\n",
       "1  83c20b41-645b-4931-97bf-870d498e43de  14.0   8.0    80    83\n",
       "2  0091731f-bae9-4326-a66a-a744137584a2  13.5   8.0    83    77\n",
       "3  858981e3-e43d-4777-ac97-16678c7f6ac0  19.0  20.0    90    90\n",
       "4  85ca12a0-f4e5-4045-8350-676e0e159221  22.0  17.0    80    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n",
      "Algo score, swaps needed: 2023.38838\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2152, 2110, 1868, 2089, 2156, 2139, 1961, 1961, 2235, 1950]\n",
      "Pseudo-random, avergae swaps needed: 2045.98698\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n",
      "Algo score, swaps needed: 1759.92724\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2193, 1875, 1769, 2061, 1820, 1979, 1942, 2009, 1967, 1817]\n",
      "Pseudo-random, avergae swaps needed: 1993.77345\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[1441.96402, 1165.3767, 1284.64089, 1217.07259, 1703.51235, 131.48975, 1024.01478, 1129.23608, 1279.13438, 1111.52218, 947.0725, 2023.38838, 1759.92724]\n",
      "N: 13 \tMean:  1247.5655261538461 \tVar:  195273.5144029216\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.1106, 1566.27993, 1560.16865, 1837.17634, 1757.39543, 182.60821, 1324.45789, 1219.24489, 1472.41306, 1449.40899, 1457.26523, 2045.98698, 1993.77345]\n",
      "N: 13 \tMean:  1489.6376653846155 \tVar:  198367.60072464152 \n",
      "\n",
      "1 ) 1499.1106 > 1441.96402 => 3.886102012603821\n",
      "2 ) 1566.27993 > 1165.3767 => 29.352388261184927\n",
      "3 ) 1560.16865 > 1284.64089 => 19.370559338042725\n",
      "4 ) 1837.17634 > 1217.07259 => 40.60597313526767\n",
      "5 ) 1757.39543 > 1703.51235 => 3.113811949071932\n",
      "6 ) 182.60821 > 131.48975 => 32.54937408698868\n",
      "7 ) 1324.45789 > 1024.01478 => 25.586255598196928\n",
      "8 ) 1219.24489 > 1129.23608 => 7.665279059084738\n",
      "9 ) 1472.41306 > 1279.13438 => 14.04872597799005\n",
      "10 ) 1449.40899 > 1111.52218 => 26.387808775040213\n",
      "11 ) 1457.26523 > 947.0725 => 42.43935647093971\n",
      "12 ) 2045.98698 > 2023.38838 => 1.1106667731924247\n",
      "13 ) 1993.77345 > 1759.92724 => 12.459502198615624\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "19.890446433555343 \n",
      "\n",
      "Ttest_relResult(statistic=-4.5918638949446615, pvalue=0.0006195845592729263)\n",
      "Cohen's d:  0.5242380338356752\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp1',normv=1,score=1,goal=\"medoid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>02694d63-a2bc-45b7-98fd-77c89066e667</td>\n",
       "      <td>-1.280052</td>\n",
       "      <td>-1.074762</td>\n",
       "      <td>-0.535036</td>\n",
       "      <td>1.229655</td>\n",
       "      <td>-1.120584</td>\n",
       "      <td>0.033619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8b2e671a-8164-4467-9f19-a6dc7cbf313c</td>\n",
       "      <td>-0.719661</td>\n",
       "      <td>0.367653</td>\n",
       "      <td>0.205903</td>\n",
       "      <td>0.555071</td>\n",
       "      <td>0.557392</td>\n",
       "      <td>0.432039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>07ac7940-e702-498a-a97c-333cb233a847</td>\n",
       "      <td>-1.280052</td>\n",
       "      <td>-0.971732</td>\n",
       "      <td>-1.572349</td>\n",
       "      <td>-0.494572</td>\n",
       "      <td>0.849569</td>\n",
       "      <td>-0.376901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>499fc744-a3e7-4927-afd5-b4df47ca7874</td>\n",
       "      <td>-1.840443</td>\n",
       "      <td>0.161594</td>\n",
       "      <td>-0.683223</td>\n",
       "      <td>-1.526789</td>\n",
       "      <td>1.024490</td>\n",
       "      <td>0.133701</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0adab6a0-9008-4fdb-9fb7-a0be1edcda37</td>\n",
       "      <td>0.401122</td>\n",
       "      <td>-0.765673</td>\n",
       "      <td>-0.979599</td>\n",
       "      <td>-1.051691</td>\n",
       "      <td>0.474231</td>\n",
       "      <td>0.012859</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x      Q1_x      Q2_x  \\\n",
       "0  02694d63-a2bc-45b7-98fd-77c89066e667    -1.280052 -1.074762 -0.535036   \n",
       "1  8b2e671a-8164-4467-9f19-a6dc7cbf313c    -0.719661  0.367653  0.205903   \n",
       "2  07ac7940-e702-498a-a97c-333cb233a847    -1.280052 -0.971732 -1.572349   \n",
       "3  499fc744-a3e7-4927-afd5-b4df47ca7874    -1.840443  0.161594 -0.683223   \n",
       "4  0adab6a0-9008-4fdb-9fb7-a0be1edcda37     0.401122 -0.765673 -0.979599   \n",
       "\n",
       "   SHORT_ANS_y      Q1_y      Q2_y  \n",
       "0     1.229655 -1.120584  0.033619  \n",
       "1     0.555071  0.557392  0.432039  \n",
       "2    -0.494572  0.849569 -0.376901  \n",
       "3    -1.526789  1.024490  0.133701  \n",
       "4    -1.051691  0.474231  0.012859  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>02694d63-a2bc-45b7-98fd-77c89066e667</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.280052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8b2e671a-8164-4467-9f19-a6dc7cbf313c</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.719661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7ab0ea89-c22f-4ca6-bd74-c3aec57452f8</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.999856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>07ac7940-e702-498a-a97c-333cb233a847</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.280052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>499fc744-a3e7-4927-afd5-b4df47ca7874</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.840443</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  02694d63-a2bc-45b7-98fd-77c89066e667  Human -1.280052\n",
       "1  8b2e671a-8164-4467-9f19-a6dc7cbf313c  Human -0.719661\n",
       "2  7ab0ea89-c22f-4ca6-bd74-c3aec57452f8  Human -0.999856\n",
       "3  07ac7940-e702-498a-a97c-333cb233a847  Human -1.280052\n",
       "4  499fc744-a3e7-4927-afd5-b4df47ca7874  Human -1.840443"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.137978</td>\n",
       "      <td>1.320126</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.107329</td>\n",
       "      <td>[-0.08, 0.34]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.133377</td>\n",
       "      <td>1.304064</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.118631</td>\n",
       "      <td>[-0.09, 0.34]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.131969</td>\n",
       "      <td>1.304064</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.118631</td>\n",
       "      <td>[-0.09, 0.34]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.242496</td>\n",
       "      <td>1.320126</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.107329</td>\n",
       "      <td>[-0.18, 0.51]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.235362</td>\n",
       "      <td>1.304064</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.118631</td>\n",
       "      <td>[-0.2, 0.51]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.233167</td>\n",
       "      <td>1.304064</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.118631</td>\n",
       "      <td>[-0.19, 0.51]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.137978  1.320126   80   81  0.107329   \n",
       "ICC2      Single random raters  0.133377  1.304064   80   80  0.118631   \n",
       "ICC3       Single fixed raters  0.131969  1.304064   80   80  0.118631   \n",
       "ICC1k  Average raters absolute  0.242496  1.320126   80   81  0.107329   \n",
       "ICC2k    Average random raters  0.235362  1.304064   80   80  0.118631   \n",
       "ICC3k     Average fixed raters  0.233167  1.304064   80   80  0.118631   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.08, 0.34]  \n",
       "ICC2   [-0.09, 0.34]  \n",
       "ICC3   [-0.09, 0.34]  \n",
       "ICC1k  [-0.18, 0.51]  \n",
       "ICC2k   [-0.2, 0.51]  \n",
       "ICC3k  [-0.19, 0.51]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.16994789109397024\n",
      "Algo score, swaps needed: 1320.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1474, 1561, 1577, 1650, 1573, 1428, 1492, 1199, 1508, 1441]\n",
      "Pseudo-random, avergae swaps needed: 1498.90842\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>02694d63-a2bc-45b7-98fd-77c89066e667</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.074762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8b2e671a-8164-4467-9f19-a6dc7cbf313c</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.367653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7ab0ea89-c22f-4ca6-bd74-c3aec57452f8</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.765673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>07ac7940-e702-498a-a97c-333cb233a847</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.971732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>499fc744-a3e7-4927-afd5-b4df47ca7874</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.161594</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  02694d63-a2bc-45b7-98fd-77c89066e667  Human -1.074762\n",
       "1  8b2e671a-8164-4467-9f19-a6dc7cbf313c  Human  0.367653\n",
       "2  7ab0ea89-c22f-4ca6-bd74-c3aec57452f8  Human -0.765673\n",
       "3  07ac7940-e702-498a-a97c-333cb233a847  Human -0.971732\n",
       "4  499fc744-a3e7-4927-afd5-b4df47ca7874  Human  0.161594"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.348950</td>\n",
       "      <td>2.071960</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000628</td>\n",
       "      <td>[0.14, 0.53]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.346292</td>\n",
       "      <td>2.046403</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000791</td>\n",
       "      <td>[0.14, 0.52]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.343488</td>\n",
       "      <td>2.046403</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000791</td>\n",
       "      <td>[0.14, 0.52]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.517365</td>\n",
       "      <td>2.071960</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000628</td>\n",
       "      <td>[0.25, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.514438</td>\n",
       "      <td>2.046403</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000791</td>\n",
       "      <td>[0.24, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.511338</td>\n",
       "      <td>2.046403</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000791</td>\n",
       "      <td>[0.24, 0.69]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.348950  2.071960   80   81  0.000628   \n",
       "ICC2      Single random raters  0.346292  2.046403   80   80  0.000791   \n",
       "ICC3       Single fixed raters  0.343488  2.046403   80   80  0.000791   \n",
       "ICC1k  Average raters absolute  0.517365  2.071960   80   81  0.000628   \n",
       "ICC2k    Average random raters  0.514438  2.046403   80   80  0.000791   \n",
       "ICC3k     Average fixed raters  0.511338  2.046403   80   80  0.000791   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.14, 0.53]  \n",
       "ICC2   [0.14, 0.52]  \n",
       "ICC3   [0.14, 0.52]  \n",
       "ICC1k  [0.25, 0.69]  \n",
       "ICC2k  [0.24, 0.69]  \n",
       "ICC3k  [0.24, 0.69]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.3545040820172465\n",
      "Algo score, swaps needed: 1231.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1566, 1671, 1587, 1454, 1224, 1567, 1473, 1583, 1492, 1568]\n",
      "Pseudo-random, avergae swaps needed: 1566.7739\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>02694d63-a2bc-45b7-98fd-77c89066e667</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.535036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8b2e671a-8164-4467-9f19-a6dc7cbf313c</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.205903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7ab0ea89-c22f-4ca6-bd74-c3aec57452f8</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.386848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>07ac7940-e702-498a-a97c-333cb233a847</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.572349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>499fc744-a3e7-4927-afd5-b4df47ca7874</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.683223</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  02694d63-a2bc-45b7-98fd-77c89066e667  Human -0.535036\n",
       "1  8b2e671a-8164-4467-9f19-a6dc7cbf313c  Human  0.205903\n",
       "2  7ab0ea89-c22f-4ca6-bd74-c3aec57452f8  Human -0.386848\n",
       "3  07ac7940-e702-498a-a97c-333cb233a847  Human -1.572349\n",
       "4  499fc744-a3e7-4927-afd5-b4df47ca7874  Human -0.683223"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.429470</td>\n",
       "      <td>2.505512</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000026</td>\n",
       "      <td>[0.23, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.427747</td>\n",
       "      <td>2.479341</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000034</td>\n",
       "      <td>[0.23, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.425178</td>\n",
       "      <td>2.479341</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000034</td>\n",
       "      <td>[0.23, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.600880</td>\n",
       "      <td>2.505512</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000026</td>\n",
       "      <td>[0.38, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.599191</td>\n",
       "      <td>2.479341</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000034</td>\n",
       "      <td>[0.38, 0.74]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.596667</td>\n",
       "      <td>2.479341</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000034</td>\n",
       "      <td>[0.37, 0.74]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.429470  2.505512   80   81  0.000026   \n",
       "ICC2      Single random raters  0.427747  2.479341   80   80  0.000034   \n",
       "ICC3       Single fixed raters  0.425178  2.479341   80   80  0.000034   \n",
       "ICC1k  Average raters absolute  0.600880  2.505512   80   81  0.000026   \n",
       "ICC2k    Average random raters  0.599191  2.479341   80   80  0.000034   \n",
       "ICC3k     Average fixed raters  0.596667  2.479341   80   80  0.000034   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.23, 0.59]  \n",
       "ICC2   [0.23, 0.59]  \n",
       "ICC3   [0.23, 0.59]  \n",
       "ICC1k  [0.38, 0.74]  \n",
       "ICC2k  [0.38, 0.74]  \n",
       "ICC3k  [0.37, 0.74]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.4491160108350567\n",
      "Algo score, swaps needed: 976.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1571, 1506, 1777, 1670, 1781, 1536, 1645, 1401, 1514, 1447]\n",
      "Pseudo-random, avergae swaps needed: 1559.69847\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51373c65-23cf-4027-bf63-ab9c1ce537b4</td>\n",
       "      <td>-0.866162</td>\n",
       "      <td>0.657180</td>\n",
       "      <td>0.467738</td>\n",
       "      <td>0.504613</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>83af18c1-132e-4728-b685-77436f376b67</td>\n",
       "      <td>0.787420</td>\n",
       "      <td>0.957513</td>\n",
       "      <td>0.460779</td>\n",
       "      <td>1.178211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0b4a8297-98e1-4153-bddb-d6562cd5041e</td>\n",
       "      <td>-0.748049</td>\n",
       "      <td>0.957513</td>\n",
       "      <td>-0.008358</td>\n",
       "      <td>-0.030989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>05baba87-2dab-4b20-b95e-1150bd449ba4</td>\n",
       "      <td>0.078742</td>\n",
       "      <td>-0.544152</td>\n",
       "      <td>0.056882</td>\n",
       "      <td>0.289218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f1a855a4-dfef-48b6-99bc-b5e2324ce2fc</td>\n",
       "      <td>1.496097</td>\n",
       "      <td>-0.243819</td>\n",
       "      <td>0.308445</td>\n",
       "      <td>-1.200206</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  51373c65-23cf-4027-bf63-ab9c1ce537b4 -0.866162  0.657180  0.467738   \n",
       "1  83af18c1-132e-4728-b685-77436f376b67  0.787420  0.957513  0.460779   \n",
       "2  0b4a8297-98e1-4153-bddb-d6562cd5041e -0.748049  0.957513 -0.008358   \n",
       "3  05baba87-2dab-4b20-b95e-1150bd449ba4  0.078742 -0.544152  0.056882   \n",
       "4  f1a855a4-dfef-48b6-99bc-b5e2324ce2fc  1.496097 -0.243819  0.308445   \n",
       "\n",
       "       Q2_y  \n",
       "0  0.504613  \n",
       "1  1.178211  \n",
       "2 -0.030989  \n",
       "3  0.289218  \n",
       "4 -1.200206  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51373c65-23cf-4027-bf63-ab9c1ce537b4</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.866162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>83af18c1-132e-4728-b685-77436f376b67</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.787420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0b4a8297-98e1-4153-bddb-d6562cd5041e</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.748049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>05baba87-2dab-4b20-b95e-1150bd449ba4</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.078742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f1a855a4-dfef-48b6-99bc-b5e2324ce2fc</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.496097</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  51373c65-23cf-4027-bf63-ab9c1ce537b4  Human -0.866162\n",
       "1  83af18c1-132e-4728-b685-77436f376b67  Human  0.787420\n",
       "2  0b4a8297-98e1-4153-bddb-d6562cd5041e  Human -0.748049\n",
       "3  05baba87-2dab-4b20-b95e-1150bd449ba4  Human  0.078742\n",
       "4  f1a855a4-dfef-48b6-99bc-b5e2324ce2fc  Human  1.496097"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.407178</td>\n",
       "      <td>2.373694</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000035</td>\n",
       "      <td>[0.22, 0.57]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.405249</td>\n",
       "      <td>2.348002</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000046</td>\n",
       "      <td>[0.21, 0.57]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.402629</td>\n",
       "      <td>2.348002</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000046</td>\n",
       "      <td>[0.21, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.578716</td>\n",
       "      <td>2.373694</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000035</td>\n",
       "      <td>[0.36, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.576765</td>\n",
       "      <td>2.348002</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000046</td>\n",
       "      <td>[0.35, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.574106</td>\n",
       "      <td>2.348002</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000046</td>\n",
       "      <td>[0.35, 0.72]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.407178  2.373694   87   88  0.000035   \n",
       "ICC2      Single random raters  0.405249  2.348002   87   87  0.000046   \n",
       "ICC3       Single fixed raters  0.402629  2.348002   87   87  0.000046   \n",
       "ICC1k  Average raters absolute  0.578716  2.373694   87   88  0.000035   \n",
       "ICC2k    Average random raters  0.576765  2.348002   87   87  0.000046   \n",
       "ICC3k     Average fixed raters  0.574106  2.348002   87   87  0.000046   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.22, 0.57]  \n",
       "ICC2   [0.21, 0.57]  \n",
       "ICC3   [0.21, 0.56]  \n",
       "ICC1k  [0.36, 0.72]  \n",
       "ICC2k  [0.35, 0.72]  \n",
       "ICC3k  [0.35, 0.72]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.47507119479746673\n",
      "Algo score, swaps needed: 1205.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1643, 1502, 1963, 1926, 2102, 1804, 2026, 2149, 1937, 1971]\n",
      "Pseudo-random, avergae swaps needed: 1838.54332\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>51373c65-23cf-4027-bf63-ab9c1ce537b4</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.657180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>83af18c1-132e-4728-b685-77436f376b67</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.957513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0b4a8297-98e1-4153-bddb-d6562cd5041e</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.957513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>05baba87-2dab-4b20-b95e-1150bd449ba4</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.544152</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f1a855a4-dfef-48b6-99bc-b5e2324ce2fc</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.243819</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  51373c65-23cf-4027-bf63-ab9c1ce537b4  Human  0.657180\n",
       "1  83af18c1-132e-4728-b685-77436f376b67  Human  0.957513\n",
       "2  0b4a8297-98e1-4153-bddb-d6562cd5041e  Human  0.957513\n",
       "3  05baba87-2dab-4b20-b95e-1150bd449ba4  Human -0.544152\n",
       "4  f1a855a4-dfef-48b6-99bc-b5e2324ce2fc  Human -0.243819"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.285825</td>\n",
       "      <td>1.800433</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.003216</td>\n",
       "      <td>[0.08, 0.47]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.283259</td>\n",
       "      <td>1.782560</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.003795</td>\n",
       "      <td>[0.08, 0.46]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.281237</td>\n",
       "      <td>1.782560</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.003795</td>\n",
       "      <td>[0.08, 0.46]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.444578</td>\n",
       "      <td>1.800433</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.003216</td>\n",
       "      <td>[0.15, 0.64]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.441468</td>\n",
       "      <td>1.782560</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.003795</td>\n",
       "      <td>[0.14, 0.63]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.439009</td>\n",
       "      <td>1.782560</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.003795</td>\n",
       "      <td>[0.14, 0.63]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.285825  1.800433   87   88  0.003216   \n",
       "ICC2      Single random raters  0.283259  1.782560   87   87  0.003795   \n",
       "ICC3       Single fixed raters  0.281237  1.782560   87   87  0.003795   \n",
       "ICC1k  Average raters absolute  0.444578  1.800433   87   88  0.003216   \n",
       "ICC2k    Average random raters  0.441468  1.782560   87   87  0.003795   \n",
       "ICC3k     Average fixed raters  0.439009  1.782560   87   87  0.003795   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.08, 0.47]  \n",
       "ICC2   [0.08, 0.46]  \n",
       "ICC3   [0.08, 0.46]  \n",
       "ICC1k  [0.15, 0.64]  \n",
       "ICC2k  [0.14, 0.63]  \n",
       "ICC3k  [0.14, 0.63]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.3682931944811738\n",
      "Algo score, swaps needed: 1235.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1635, 1698, 1366, 1886, 1916, 1924, 1624, 1526, 1797, 1838]\n",
      "Pseudo-random, avergae swaps needed: 1756.92053\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>799aa117-453f-459f-8d42-dfc84e23f308</td>\n",
       "      <td>-0.182505</td>\n",
       "      <td>0.202074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f1a75c07-eda1-4552-907a-c976d9ae7c0c</td>\n",
       "      <td>1.607446</td>\n",
       "      <td>-0.942145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3307e79a-3565-478c-8749-b095d12c7e56</td>\n",
       "      <td>-0.958150</td>\n",
       "      <td>-0.540016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ab1fbbba-d6e0-4551-aed0-3fc164fa8570</td>\n",
       "      <td>-0.182505</td>\n",
       "      <td>-0.305673</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c1dcfb9e-4f1e-4c8b-9ab6-7290cd1c035b</td>\n",
       "      <td>0.294815</td>\n",
       "      <td>0.126680</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q2_x      Q2_y\n",
       "0  799aa117-453f-459f-8d42-dfc84e23f308 -0.182505  0.202074\n",
       "1  f1a75c07-eda1-4552-907a-c976d9ae7c0c  1.607446 -0.942145\n",
       "2  3307e79a-3565-478c-8749-b095d12c7e56 -0.958150 -0.540016\n",
       "3  ab1fbbba-d6e0-4551-aed0-3fc164fa8570 -0.182505 -0.305673\n",
       "4  c1dcfb9e-4f1e-4c8b-9ab6-7290cd1c035b  0.294815  0.126680"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>799aa117-453f-459f-8d42-dfc84e23f308</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.182505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f1a75c07-eda1-4552-907a-c976d9ae7c0c</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.607446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4852efe1-c79c-4b78-94b2-de3958cdf5aa</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.779155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3307e79a-3565-478c-8749-b095d12c7e56</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.958150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ab1fbbba-d6e0-4551-aed0-3fc164fa8570</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.182505</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  799aa117-453f-459f-8d42-dfc84e23f308  Human -0.182505\n",
       "1  f1a75c07-eda1-4552-907a-c976d9ae7c0c  Human  1.607446\n",
       "2  4852efe1-c79c-4b78-94b2-de3958cdf5aa  Human -0.779155\n",
       "3  3307e79a-3565-478c-8749-b095d12c7e56  Human -0.958150\n",
       "4  ab1fbbba-d6e0-4551-aed0-3fc164fa8570  Human -0.182505"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>-0.103230</td>\n",
       "      <td>0.812859</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.703723</td>\n",
       "      <td>[-0.45, 0.27]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>-0.123284</td>\n",
       "      <td>0.787375</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.730599</td>\n",
       "      <td>[-0.49, 0.27]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>-0.118959</td>\n",
       "      <td>0.787375</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.730599</td>\n",
       "      <td>[-0.47, 0.26]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>-0.230226</td>\n",
       "      <td>0.812859</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.703723</td>\n",
       "      <td>[-1.63, 0.43]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>-0.281241</td>\n",
       "      <td>0.787375</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.730599</td>\n",
       "      <td>[-1.91, 0.42]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>-0.270043</td>\n",
       "      <td>0.787375</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.730599</td>\n",
       "      <td>[-1.74, 0.41]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute -0.103230  0.812859   27   28  0.703723   \n",
       "ICC2      Single random raters -0.123284  0.787375   27   27  0.730599   \n",
       "ICC3       Single fixed raters -0.118959  0.787375   27   27  0.730599   \n",
       "ICC1k  Average raters absolute -0.230226  0.812859   27   28  0.703723   \n",
       "ICC2k    Average random raters -0.281241  0.787375   27   27  0.730599   \n",
       "ICC3k     Average fixed raters -0.270043  0.787375   27   27  0.730599   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.45, 0.27]  \n",
       "ICC2   [-0.49, 0.27]  \n",
       "ICC3   [-0.47, 0.26]  \n",
       "ICC1k  [-1.63, 0.43]  \n",
       "ICC2k  [-1.91, 0.42]  \n",
       "ICC3k  [-1.74, 0.41]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): -0.0984864410342654\n",
      "Algo score, swaps needed: 191.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[177, 154, 157, 214, 116, 208, 193, 162, 137, 185]\n",
      "Pseudo-random, avergae swaps needed: 182.4611\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>f7422fbb-df4a-4788-8d19-03ca5ab472f4</td>\n",
       "      <td>-2.134996</td>\n",
       "      <td>-0.498226</td>\n",
       "      <td>-1.009460</td>\n",
       "      <td>0.428823</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f835cc4c-68ff-4a67-a6a8-17c0d3997a2a</td>\n",
       "      <td>1.529010</td>\n",
       "      <td>0.420724</td>\n",
       "      <td>-0.427363</td>\n",
       "      <td>0.366663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>330e4993-30e4-4192-9e02-07e960fad404</td>\n",
       "      <td>1.695556</td>\n",
       "      <td>1.339674</td>\n",
       "      <td>-0.151769</td>\n",
       "      <td>0.181630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>22e8b506-0aa5-400a-aca0-ad1457dc1f19</td>\n",
       "      <td>0.529736</td>\n",
       "      <td>-1.417176</td>\n",
       "      <td>-0.107893</td>\n",
       "      <td>0.056022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>473b0dda-6847-467e-8589-32a882dcfde1</td>\n",
       "      <td>0.196644</td>\n",
       "      <td>0.420724</td>\n",
       "      <td>-1.033410</td>\n",
       "      <td>-0.244192</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  f7422fbb-df4a-4788-8d19-03ca5ab472f4 -2.134996 -0.498226 -1.009460   \n",
       "1  f835cc4c-68ff-4a67-a6a8-17c0d3997a2a  1.529010  0.420724 -0.427363   \n",
       "2  330e4993-30e4-4192-9e02-07e960fad404  1.695556  1.339674 -0.151769   \n",
       "3  22e8b506-0aa5-400a-aca0-ad1457dc1f19  0.529736 -1.417176 -0.107893   \n",
       "4  473b0dda-6847-467e-8589-32a882dcfde1  0.196644  0.420724 -1.033410   \n",
       "\n",
       "       Q2_y  \n",
       "0  0.428823  \n",
       "1  0.366663  \n",
       "2  0.181630  \n",
       "3  0.056022  \n",
       "4 -0.244192  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>f7422fbb-df4a-4788-8d19-03ca5ab472f4</td>\n",
       "      <td>Human</td>\n",
       "      <td>-2.134996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f835cc4c-68ff-4a67-a6a8-17c0d3997a2a</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.529010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>330e4993-30e4-4192-9e02-07e960fad404</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.695556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>457c5af7-4834-422d-927e-fe3c52bcc0f6</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.196644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>22e8b506-0aa5-400a-aca0-ad1457dc1f19</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.529736</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  f7422fbb-df4a-4788-8d19-03ca5ab472f4  Human -2.134996\n",
       "1  f835cc4c-68ff-4a67-a6a8-17c0d3997a2a  Human  1.529010\n",
       "2  330e4993-30e4-4192-9e02-07e960fad404  Human  1.695556\n",
       "3  457c5af7-4834-422d-927e-fe3c52bcc0f6  Human  0.196644\n",
       "4  22e8b506-0aa5-400a-aca0-ad1457dc1f19  Human  0.529736"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.304870</td>\n",
       "      <td>1.877158</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.003581</td>\n",
       "      <td>[0.09, 0.5]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.301607</td>\n",
       "      <td>1.852265</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.004372</td>\n",
       "      <td>[0.08, 0.49]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.298803</td>\n",
       "      <td>1.852265</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.004372</td>\n",
       "      <td>[0.08, 0.49]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.467280</td>\n",
       "      <td>1.877158</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.003581</td>\n",
       "      <td>[0.16, 0.66]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.463438</td>\n",
       "      <td>1.852265</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.004372</td>\n",
       "      <td>[0.15, 0.66]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.460120</td>\n",
       "      <td>1.852265</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.004372</td>\n",
       "      <td>[0.15, 0.66]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.304870  1.877158   74   75  0.003581   \n",
       "ICC2      Single random raters  0.301607  1.852265   74   74  0.004372   \n",
       "ICC3       Single fixed raters  0.298803  1.852265   74   74  0.004372   \n",
       "ICC1k  Average raters absolute  0.467280  1.877158   74   75  0.003581   \n",
       "ICC2k    Average random raters  0.463438  1.852265   74   74  0.004372   \n",
       "ICC3k     Average fixed raters  0.460120  1.852265   74   74  0.004372   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1    [0.09, 0.5]  \n",
       "ICC2   [0.08, 0.49]  \n",
       "ICC3   [0.08, 0.49]  \n",
       "ICC1k  [0.16, 0.66]  \n",
       "ICC2k  [0.15, 0.66]  \n",
       "ICC3k  [0.15, 0.66]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.25088830064062484\n",
      "Algo score, swaps needed: 1107.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1293, 1303, 1224, 1403, 1239, 1260, 1380, 1331, 1408, 1393]\n",
      "Pseudo-random, avergae swaps needed: 1323.96503\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>f7422fbb-df4a-4788-8d19-03ca5ab472f4</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.498226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f835cc4c-68ff-4a67-a6a8-17c0d3997a2a</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.420724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>330e4993-30e4-4192-9e02-07e960fad404</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.339674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>457c5af7-4834-422d-927e-fe3c52bcc0f6</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.339674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>22e8b506-0aa5-400a-aca0-ad1457dc1f19</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.417176</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  f7422fbb-df4a-4788-8d19-03ca5ab472f4  Human -0.498226\n",
       "1  f835cc4c-68ff-4a67-a6a8-17c0d3997a2a  Human  0.420724\n",
       "2  330e4993-30e4-4192-9e02-07e960fad404  Human  1.339674\n",
       "3  457c5af7-4834-422d-927e-fe3c52bcc0f6  Human  1.339674\n",
       "4  22e8b506-0aa5-400a-aca0-ad1457dc1f19  Human -1.417176"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.161149</td>\n",
       "      <td>1.384214</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.081296</td>\n",
       "      <td>[-0.07, 0.37]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.156428</td>\n",
       "      <td>1.365987</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.091065</td>\n",
       "      <td>[-0.07, 0.37]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.154687</td>\n",
       "      <td>1.365987</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.091065</td>\n",
       "      <td>[-0.07, 0.37]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.277568</td>\n",
       "      <td>1.384214</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.081296</td>\n",
       "      <td>[-0.14, 0.54]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.270536</td>\n",
       "      <td>1.365987</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.091065</td>\n",
       "      <td>[-0.16, 0.54]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.267929</td>\n",
       "      <td>1.365987</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.091065</td>\n",
       "      <td>[-0.16, 0.54]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.161149  1.384214   74   75  0.081296   \n",
       "ICC2      Single random raters  0.156428  1.365987   74   74  0.091065   \n",
       "ICC3       Single fixed raters  0.154687  1.365987   74   74  0.091065   \n",
       "ICC1k  Average raters absolute  0.277568  1.384214   74   75  0.081296   \n",
       "ICC2k    Average random raters  0.270536  1.365987   74   74  0.091065   \n",
       "ICC3k     Average fixed raters  0.267929  1.365987   74   74  0.091065   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.07, 0.37]  \n",
       "ICC2   [-0.07, 0.37]  \n",
       "ICC3   [-0.07, 0.37]  \n",
       "ICC1k  [-0.14, 0.54]  \n",
       "ICC2k  [-0.16, 0.54]  \n",
       "ICC3k  [-0.16, 0.54]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.13987914505579846\n",
      "Algo score, swaps needed: 1026.9973\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1227, 1247, 1206, 1162, 1225, 1022, 1158, 1218, 1310, 1174]\n",
      "Pseudo-random, avergae swaps needed: 1219.80319\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>aa684706-1bbf-47a8-8828-62f8fde0f3c7</td>\n",
       "      <td>0.414326</td>\n",
       "      <td>0.041273</td>\n",
       "      <td>-0.075000</td>\n",
       "      <td>-0.841008</td>\n",
       "      <td>-1.606080</td>\n",
       "      <td>-0.522871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>bb065bd5-206a-4c0f-ba0f-f193651f5ac6</td>\n",
       "      <td>1.032264</td>\n",
       "      <td>-0.369096</td>\n",
       "      <td>1.580605</td>\n",
       "      <td>-0.876663</td>\n",
       "      <td>-0.508243</td>\n",
       "      <td>0.291306</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>e039c490-f389-4a42-a88e-15fc431b7865</td>\n",
       "      <td>0.414326</td>\n",
       "      <td>-0.266504</td>\n",
       "      <td>-0.464554</td>\n",
       "      <td>0.368989</td>\n",
       "      <td>-0.914040</td>\n",
       "      <td>0.246203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>55530149-172a-4627-ba6a-99a4f2ae48c9</td>\n",
       "      <td>-0.958869</td>\n",
       "      <td>0.041273</td>\n",
       "      <td>0.509331</td>\n",
       "      <td>0.265634</td>\n",
       "      <td>0.151630</td>\n",
       "      <td>-0.018894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0193ec10-5235-470a-8037-73db98e4f036</td>\n",
       "      <td>0.345666</td>\n",
       "      <td>-0.266504</td>\n",
       "      <td>0.119777</td>\n",
       "      <td>0.255167</td>\n",
       "      <td>-0.762980</td>\n",
       "      <td>-0.607294</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q3_x  \\\n",
       "0  aa684706-1bbf-47a8-8828-62f8fde0f3c7  0.414326  0.041273 -0.075000   \n",
       "1  bb065bd5-206a-4c0f-ba0f-f193651f5ac6  1.032264 -0.369096  1.580605   \n",
       "2  e039c490-f389-4a42-a88e-15fc431b7865  0.414326 -0.266504 -0.464554   \n",
       "3  55530149-172a-4627-ba6a-99a4f2ae48c9 -0.958869  0.041273  0.509331   \n",
       "4  0193ec10-5235-470a-8037-73db98e4f036  0.345666 -0.266504  0.119777   \n",
       "\n",
       "       Q1_y      Q2_y      Q3_y  \n",
       "0 -0.841008 -1.606080 -0.522871  \n",
       "1 -0.876663 -0.508243  0.291306  \n",
       "2  0.368989 -0.914040  0.246203  \n",
       "3  0.265634  0.151630 -0.018894  \n",
       "4  0.255167 -0.762980 -0.607294  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>aa684706-1bbf-47a8-8828-62f8fde0f3c7</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.414326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cf60d4fc-75ab-45fb-97ab-775ce4333299</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.164848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bb065bd5-206a-4c0f-ba0f-f193651f5ac6</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.032264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>e039c490-f389-4a42-a88e-15fc431b7865</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.414326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55530149-172a-4627-ba6a-99a4f2ae48c9</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.958869</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  aa684706-1bbf-47a8-8828-62f8fde0f3c7  Human  0.414326\n",
       "1  cf60d4fc-75ab-45fb-97ab-775ce4333299  Human -1.164848\n",
       "2  bb065bd5-206a-4c0f-ba0f-f193651f5ac6  Human  1.032264\n",
       "3  e039c490-f389-4a42-a88e-15fc431b7865  Human  0.414326\n",
       "4  55530149-172a-4627-ba6a-99a4f2ae48c9  Human -0.958869"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.071258</td>\n",
       "      <td>1.153451</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.265440</td>\n",
       "      <td>[-0.15, 0.29]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.065783</td>\n",
       "      <td>1.139076</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.284585</td>\n",
       "      <td>[-0.16, 0.28]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.065017</td>\n",
       "      <td>1.139076</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.284585</td>\n",
       "      <td>[-0.16, 0.28]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.133037</td>\n",
       "      <td>1.153451</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.265440</td>\n",
       "      <td>[-0.36, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.123446</td>\n",
       "      <td>1.139076</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.284585</td>\n",
       "      <td>[-0.38, 0.44]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.122095</td>\n",
       "      <td>1.139076</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.284585</td>\n",
       "      <td>[-0.38, 0.44]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.071258  1.153451   77   78  0.265440   \n",
       "ICC2      Single random raters  0.065783  1.139076   77   77  0.284585   \n",
       "ICC3       Single fixed raters  0.065017  1.139076   77   77  0.284585   \n",
       "ICC1k  Average raters absolute  0.133037  1.153451   77   78  0.265440   \n",
       "ICC2k    Average random raters  0.123446  1.139076   77   77  0.284585   \n",
       "ICC3k     Average fixed raters  0.122095  1.139076   77   77  0.284585   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.15, 0.29]  \n",
       "ICC2   [-0.16, 0.28]  \n",
       "ICC3   [-0.16, 0.28]  \n",
       "ICC1k  [-0.36, 0.45]  \n",
       "ICC2k  [-0.38, 0.44]  \n",
       "ICC3k  [-0.38, 0.44]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.06282814994898334\n",
      "Algo score, swaps needed: 1395.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1420, 1594, 1488, 1392, 1551, 1455, 1469, 1517, 1471, 1520]\n",
      "Pseudo-random, avergae swaps needed: 1472.55787\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>aa684706-1bbf-47a8-8828-62f8fde0f3c7</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.041273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cf60d4fc-75ab-45fb-97ab-775ce4333299</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.369096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bb065bd5-206a-4c0f-ba0f-f193651f5ac6</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.369096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>e039c490-f389-4a42-a88e-15fc431b7865</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.266504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55530149-172a-4627-ba6a-99a4f2ae48c9</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.041273</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  aa684706-1bbf-47a8-8828-62f8fde0f3c7  Human  0.041273\n",
       "1  cf60d4fc-75ab-45fb-97ab-775ce4333299  Human -0.369096\n",
       "2  bb065bd5-206a-4c0f-ba0f-f193651f5ac6  Human -0.369096\n",
       "3  e039c490-f389-4a42-a88e-15fc431b7865  Human -0.266504\n",
       "4  55530149-172a-4627-ba6a-99a4f2ae48c9  Human  0.041273"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.237117</td>\n",
       "      <td>1.621635</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.017324</td>\n",
       "      <td>[0.02, 0.44]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.233684</td>\n",
       "      <td>1.602808</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.020002</td>\n",
       "      <td>[0.01, 0.43]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.231599</td>\n",
       "      <td>1.602808</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.020002</td>\n",
       "      <td>[0.01, 0.43]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.383338</td>\n",
       "      <td>1.621635</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.017324</td>\n",
       "      <td>[0.03, 0.61]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.378839</td>\n",
       "      <td>1.602808</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.020002</td>\n",
       "      <td>[0.02, 0.6]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.376095</td>\n",
       "      <td>1.602808</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.020002</td>\n",
       "      <td>[0.02, 0.6]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.237117  1.621635   77   78  0.017324   \n",
       "ICC2      Single random raters  0.233684  1.602808   77   77  0.020002   \n",
       "ICC3       Single fixed raters  0.231599  1.602808   77   77  0.020002   \n",
       "ICC1k  Average raters absolute  0.383338  1.621635   77   78  0.017324   \n",
       "ICC2k    Average random raters  0.378839  1.602808   77   77  0.020002   \n",
       "ICC3k     Average fixed raters  0.376095  1.602808   77   77  0.020002   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.02, 0.44]  \n",
       "ICC2   [0.01, 0.43]  \n",
       "ICC3   [0.01, 0.43]  \n",
       "ICC1k  [0.03, 0.61]  \n",
       "ICC2k   [0.02, 0.6]  \n",
       "ICC3k   [0.02, 0.6]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.2186304510280832\n",
      "Algo score, swaps needed: 1176.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1557, 1458, 1541, 1435, 1422, 1508, 1454, 1342, 1443, 1460]\n",
      "Pseudo-random, avergae swaps needed: 1450.1969\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>aa684706-1bbf-47a8-8828-62f8fde0f3c7</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.075000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cf60d4fc-75ab-45fb-97ab-775ce4333299</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.367166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>bb065bd5-206a-4c0f-ba0f-f193651f5ac6</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.580605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>e039c490-f389-4a42-a88e-15fc431b7865</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.464554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55530149-172a-4627-ba6a-99a4f2ae48c9</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.509331</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  aa684706-1bbf-47a8-8828-62f8fde0f3c7  Human -0.075000\n",
       "1  cf60d4fc-75ab-45fb-97ab-775ce4333299  Human -0.367166\n",
       "2  bb065bd5-206a-4c0f-ba0f-f193651f5ac6  Human  1.580605\n",
       "3  e039c490-f389-4a42-a88e-15fc431b7865  Human -0.464554\n",
       "4  55530149-172a-4627-ba6a-99a4f2ae48c9  Human  0.509331"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.505779</td>\n",
       "      <td>3.046773</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>8.944293e-07</td>\n",
       "      <td>[0.32, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.504190</td>\n",
       "      <td>3.007752</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>1.287034e-06</td>\n",
       "      <td>[0.32, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.500967</td>\n",
       "      <td>3.007752</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>1.287034e-06</td>\n",
       "      <td>[0.31, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.671784</td>\n",
       "      <td>3.046773</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>8.944293e-07</td>\n",
       "      <td>[0.49, 0.79]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.670380</td>\n",
       "      <td>3.007752</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>1.287034e-06</td>\n",
       "      <td>[0.48, 0.79]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.667526</td>\n",
       "      <td>3.007752</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>1.287034e-06</td>\n",
       "      <td>[0.48, 0.79]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.505779  3.046773   77   78  8.944293e-07   \n",
       "ICC2      Single random raters  0.504190  3.007752   77   77  1.287034e-06   \n",
       "ICC3       Single fixed raters  0.500967  3.007752   77   77  1.287034e-06   \n",
       "ICC1k  Average raters absolute  0.671784  3.046773   77   78  8.944293e-07   \n",
       "ICC2k    Average random raters  0.670380  3.007752   77   77  1.287034e-06   \n",
       "ICC3k     Average fixed raters  0.667526  3.007752   77   77  1.287034e-06   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.32, 0.65]  \n",
       "ICC2   [0.32, 0.65]  \n",
       "ICC3   [0.31, 0.65]  \n",
       "ICC1k  [0.49, 0.79]  \n",
       "ICC2k  [0.48, 0.79]  \n",
       "ICC3k  [0.48, 0.79]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.49625066373222937\n",
      "Algo score, swaps needed: 942.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1647, 1209, 1235, 1466, 1296, 1677, 1583, 1400, 1502, 1450]\n",
      "Pseudo-random, avergae swaps needed: 1458.46293\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59b5388b-5b2b-430c-a246-0d8b83249966</td>\n",
       "      <td>0.176445</td>\n",
       "      <td>0.836545</td>\n",
       "      <td>0.990707</td>\n",
       "      <td>0.671642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e2459d8b-2f3f-469e-b9f9-f5acfe633b0d</td>\n",
       "      <td>1.298213</td>\n",
       "      <td>1.800630</td>\n",
       "      <td>-0.421015</td>\n",
       "      <td>-0.399676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0ab76566-bd98-4796-b32d-f23794084401</td>\n",
       "      <td>1.129947</td>\n",
       "      <td>-0.127540</td>\n",
       "      <td>0.669283</td>\n",
       "      <td>0.465763</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>af5f9a64-909b-4e1b-8100-e82f9813b148</td>\n",
       "      <td>1.186036</td>\n",
       "      <td>1.414996</td>\n",
       "      <td>0.801939</td>\n",
       "      <td>1.342865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f0166970-fdaf-4f6b-9ae6-e8e25ef8b198</td>\n",
       "      <td>0.008180</td>\n",
       "      <td>-0.513174</td>\n",
       "      <td>-0.564429</td>\n",
       "      <td>-0.736895</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  59b5388b-5b2b-430c-a246-0d8b83249966  0.176445  0.836545  0.990707   \n",
       "1  e2459d8b-2f3f-469e-b9f9-f5acfe633b0d  1.298213  1.800630 -0.421015   \n",
       "2  0ab76566-bd98-4796-b32d-f23794084401  1.129947 -0.127540  0.669283   \n",
       "3  af5f9a64-909b-4e1b-8100-e82f9813b148  1.186036  1.414996  0.801939   \n",
       "4  f0166970-fdaf-4f6b-9ae6-e8e25ef8b198  0.008180 -0.513174 -0.564429   \n",
       "\n",
       "       Q2_y  \n",
       "0  0.671642  \n",
       "1 -0.399676  \n",
       "2  0.465763  \n",
       "3  1.342865  \n",
       "4 -0.736895  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59b5388b-5b2b-430c-a246-0d8b83249966</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.176445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e2459d8b-2f3f-469e-b9f9-f5acfe633b0d</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.298213</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0ab76566-bd98-4796-b32d-f23794084401</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.129947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>af5f9a64-909b-4e1b-8100-e82f9813b148</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.186036</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f0166970-fdaf-4f6b-9ae6-e8e25ef8b198</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.008180</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  59b5388b-5b2b-430c-a246-0d8b83249966  Human  0.176445\n",
       "1  e2459d8b-2f3f-469e-b9f9-f5acfe633b0d  Human  1.298213\n",
       "2  0ab76566-bd98-4796-b32d-f23794084401  Human  1.129947\n",
       "3  af5f9a64-909b-4e1b-8100-e82f9813b148  Human  1.186036\n",
       "4  f0166970-fdaf-4f6b-9ae6-e8e25ef8b198  Human  0.008180"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.038908</td>\n",
       "      <td>1.080967</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.355056</td>\n",
       "      <td>[-0.17, 0.24]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.034047</td>\n",
       "      <td>1.069763</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.374203</td>\n",
       "      <td>[-0.17, 0.24]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.033706</td>\n",
       "      <td>1.069763</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.374203</td>\n",
       "      <td>[-0.17, 0.24]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.074902</td>\n",
       "      <td>1.080967</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.355056</td>\n",
       "      <td>[-0.4, 0.39]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.065852</td>\n",
       "      <td>1.069763</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.374203</td>\n",
       "      <td>[-0.42, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.065214</td>\n",
       "      <td>1.069763</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.374203</td>\n",
       "      <td>[-0.41, 0.38]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.038908  1.080967   91   92  0.355056   \n",
       "ICC2      Single random raters  0.034047  1.069763   91   91  0.374203   \n",
       "ICC3       Single fixed raters  0.033706  1.069763   91   91  0.374203   \n",
       "ICC1k  Average raters absolute  0.074902  1.080967   91   92  0.355056   \n",
       "ICC2k    Average random raters  0.065852  1.069763   91   91  0.374203   \n",
       "ICC3k     Average fixed raters  0.065214  1.069763   91   91  0.374203   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.17, 0.24]  \n",
       "ICC2   [-0.17, 0.24]  \n",
       "ICC3   [-0.17, 0.24]  \n",
       "ICC1k   [-0.4, 0.39]  \n",
       "ICC2k  [-0.42, 0.38]  \n",
       "ICC3k  [-0.41, 0.38]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.1003820015488901\n",
      "Algo score, swaps needed: 1757.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2043, 2059, 1988, 2385, 2113, 2180, 1834, 2101, 1961, 2254]\n",
      "Pseudo-random, avergae swaps needed: 2045.23351\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>59b5388b-5b2b-430c-a246-0d8b83249966</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.836545</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e2459d8b-2f3f-469e-b9f9-f5acfe633b0d</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.800630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0ab76566-bd98-4796-b32d-f23794084401</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.127540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>af5f9a64-909b-4e1b-8100-e82f9813b148</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.414996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f0166970-fdaf-4f6b-9ae6-e8e25ef8b198</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.513174</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  59b5388b-5b2b-430c-a246-0d8b83249966  Human  0.836545\n",
       "1  e2459d8b-2f3f-469e-b9f9-f5acfe633b0d  Human  1.800630\n",
       "2  0ab76566-bd98-4796-b32d-f23794084401  Human -0.127540\n",
       "3  af5f9a64-909b-4e1b-8100-e82f9813b148  Human  1.414996\n",
       "4  f0166970-fdaf-4f6b-9ae6-e8e25ef8b198  Human -0.513174"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.095113</td>\n",
       "      <td>1.210220</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.181434</td>\n",
       "      <td>[-0.11, 0.29]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.091335</td>\n",
       "      <td>1.199202</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.193938</td>\n",
       "      <td>[-0.12, 0.29]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.090579</td>\n",
       "      <td>1.199202</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.193938</td>\n",
       "      <td>[-0.12, 0.29]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.173704</td>\n",
       "      <td>1.210220</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.181434</td>\n",
       "      <td>[-0.25, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.167383</td>\n",
       "      <td>1.199202</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.193938</td>\n",
       "      <td>[-0.26, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.166112</td>\n",
       "      <td>1.199202</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.193938</td>\n",
       "      <td>[-0.26, 0.45]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.095113  1.210220   91   92  0.181434   \n",
       "ICC2      Single random raters  0.091335  1.199202   91   91  0.193938   \n",
       "ICC3       Single fixed raters  0.090579  1.199202   91   91  0.193938   \n",
       "ICC1k  Average raters absolute  0.173704  1.210220   91   92  0.181434   \n",
       "ICC2k    Average random raters  0.167383  1.199202   91   91  0.193938   \n",
       "ICC3k     Average fixed raters  0.166112  1.199202   91   91  0.193938   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.11, 0.29]  \n",
       "ICC2   [-0.12, 0.29]  \n",
       "ICC3   [-0.12, 0.29]  \n",
       "ICC1k  [-0.25, 0.45]  \n",
       "ICC2k  [-0.26, 0.45]  \n",
       "ICC3k  [-0.26, 0.45]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.14589349325298806\n",
      "Algo score, swaps needed: 1715.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2004, 1996, 1802, 2049, 2284, 1970, 1738, 2122, 2075, 1794]\n",
      "Pseudo-random, avergae swaps needed: 1995.11491\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[1320.0, 1231.0, 976.0, 1205.0, 1235.0, 191.0, 1107.0, 1026.9973, 1395.0, 1176.0, 942.0, 1757.0, 1715.0]\n",
      "N: 13 \tMean:  1175.1536384615383 \tVar:  138158.49941116854\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1498.90842, 1566.7739, 1559.69847, 1838.54332, 1756.92053, 182.4611, 1323.96503, 1219.80319, 1472.55787, 1450.1969, 1458.46293, 2045.23351, 1995.11491]\n",
      "N: 13 \tMean:  1489.8953907692307 \tVar:  198469.2821672779 \n",
      "\n",
      "1 ) 1498.90842 > 1320.0 => 12.693453872474509\n",
      "2 ) 1566.7739 > 1231.0 => 24.002933189132968\n",
      "3 ) 1559.69847 > 976.0 => 46.03847633350506\n",
      "4 ) 1838.54332 > 1205.0 => 41.63195679435902\n",
      "5 ) 1756.92053 > 1235.0 => 34.88866263436483\n",
      "6 ) 182.4611 > 191.0 => -4.57284573948934\n",
      "7 ) 1323.96503 > 1107.0 => 17.850115268832152\n",
      "8 ) 1219.80319 > 1026.9973 => 17.162706778651277\n",
      "9 ) 1472.55787 > 1395.0 => 5.409332506339277\n",
      "10 ) 1450.1969 > 1176.0 => 20.88167113440732\n",
      "11 ) 1458.46293 > 942.0 => 43.030277497349225\n",
      "12 ) 2045.23351 > 1757.0 => 15.161273458978064\n",
      "13 ) 1995.11491 > 1715.0 => 15.10006653675317\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "22.252160020435195 \n",
      "\n",
      "Ttest_relResult(statistic=-5.76351253347693, pvalue=8.971217078133437e-05)\n",
      "Cohen's d:  0.7370777909631482\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp1',normv=1,score=2,goal=\"medoid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>e021f764-a5d2-432a-89f5-00ba2b760f0f</td>\n",
       "      <td>9</td>\n",
       "      <td>31</td>\n",
       "      <td>23</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4eb794d7-21bd-4bcd-96e7-6807a2197049</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>77</td>\n",
       "      <td>73</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7a25c1e8-4bd9-438f-ad5b-393ad679a0ab</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>11</td>\n",
       "      <td>70</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b8efdf86-6377-4fc4-b0b3-64d1bf7df43a</td>\n",
       "      <td>5</td>\n",
       "      <td>27</td>\n",
       "      <td>18</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>351e0d0e-9901-4878-9d31-decad062eb91</td>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>6</td>\n",
       "      <td>87</td>\n",
       "      <td>90</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x  Q1_x  Q2_x  SHORT_ANS_y  \\\n",
       "0  e021f764-a5d2-432a-89f5-00ba2b760f0f            9    31    23           87   \n",
       "1  4eb794d7-21bd-4bcd-96e7-6807a2197049            5     8     5           77   \n",
       "2  7a25c1e8-4bd9-438f-ad5b-393ad679a0ab            3    13    11           70   \n",
       "3  b8efdf86-6377-4fc4-b0b3-64d1bf7df43a            5    27    18           80   \n",
       "4  351e0d0e-9901-4878-9d31-decad062eb91            2    21     6           87   \n",
       "\n",
       "   Q1_y  Q2_y  \n",
       "0    87    87  \n",
       "1    73    80  \n",
       "2    83    80  \n",
       "3    80    87  \n",
       "4    90    70  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n",
      "Algo score, swaps needed: 1002.03083\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1519, 1547, 1536, 1539, 1672, 1510, 1401, 1408, 1372, 1525]\n",
      "Pseudo-random, avergae swaps needed: 1498.81278\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n",
      "Algo score, swaps needed: 1193.03933\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1555, 1660, 1662, 1749, 1595, 1720, 1584, 1280, 1739, 1650]\n",
      "Pseudo-random, avergae swaps needed: 1566.86461\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n",
      "Algo score, swaps needed: 970.57382\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1492, 1512, 1656, 1665, 1685, 1666, 1411, 1295, 1426, 1706]\n",
      "Pseudo-random, avergae swaps needed: 1560.04081\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>68202246-13a0-4c1f-a238-2590685176fa</td>\n",
       "      <td>20.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>90</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d6db68eb-a97f-49b6-b974-8146d81c7417</td>\n",
       "      <td>13.5</td>\n",
       "      <td>13.0</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d14fe6dc-ad38-4146-815b-82d3e66692cf</td>\n",
       "      <td>19.5</td>\n",
       "      <td>23.0</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>165f436c-ce5e-4cf6-8005-5c96b0801ab8</td>\n",
       "      <td>18.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a6f29834-7a4e-4947-a2f0-3daeb1b23284</td>\n",
       "      <td>18.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  68202246-13a0-4c1f-a238-2590685176fa  20.0  21.0    90    87\n",
       "1  d6db68eb-a97f-49b6-b974-8146d81c7417  13.5  13.0    80    80\n",
       "2  d14fe6dc-ad38-4146-815b-82d3e66692cf  19.5  23.0    87    83\n",
       "3  165f436c-ce5e-4cf6-8005-5c96b0801ab8  18.0  21.0    83    83\n",
       "4  a6f29834-7a4e-4947-a2f0-3daeb1b23284  18.0  20.0    80    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n",
      "Algo score, swaps needed: 1283.03442\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1884, 1924, 1945, 1884, 1731, 1831, 1816, 1727, 1834, 1888]\n",
      "Pseudo-random, avergae swaps needed: 1837.85914\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n",
      "Algo score, swaps needed: 1230.06142\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2108, 1767, 1512, 1821, 1741, 1660, 1653, 1618, 1630, 1764]\n",
      "Pseudo-random, avergae swaps needed: 1757.28153\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c6f9b6cb-c5b8-4622-bdad-51aabad768cd</td>\n",
       "      <td>24.0</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>981c3959-fe3b-4222-97de-742768dee8e9</td>\n",
       "      <td>23.0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2872fa41-b643-452a-b6c6-87f4e140f336</td>\n",
       "      <td>43.0</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dda68a48-2475-4338-9956-67fc7efbf74b</td>\n",
       "      <td>37.0</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8fee9d29-68ff-4b2d-b54d-58f8303249ff</td>\n",
       "      <td>29.5</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q2_x  Q2_y\n",
       "0  c6f9b6cb-c5b8-4622-bdad-51aabad768cd  24.0    90\n",
       "1  981c3959-fe3b-4222-97de-742768dee8e9  23.0    80\n",
       "2  2872fa41-b643-452a-b6c6-87f4e140f336  43.0    87\n",
       "3  dda68a48-2475-4338-9956-67fc7efbf74b  37.0    90\n",
       "4  8fee9d29-68ff-4b2d-b54d-58f8303249ff  29.5    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n",
      "Algo score, swaps needed: 91.00509\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[168, 193, 175, 192, 212, 191, 228, 212, 191, 175]\n",
      "Pseudo-random, avergae swaps needed: 182.40981\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a4a3f431-59a4-4fd1-b7fa-41bb5effd731</td>\n",
       "      <td>10.5</td>\n",
       "      <td>3.5</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>73c7a609-ce12-4e96-b1f4-58059cb60d4a</td>\n",
       "      <td>13.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>77</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1ebf5189-e22f-4dfd-9f72-5ab06b37ca76</td>\n",
       "      <td>9.0</td>\n",
       "      <td>6.5</td>\n",
       "      <td>80</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>64ec3c75-b047-4c85-8b3e-99c63a1953aa</td>\n",
       "      <td>5.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>70</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>098adb0c-c0ab-4a2d-a590-8d917e3604f2</td>\n",
       "      <td>12.5</td>\n",
       "      <td>4.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  a4a3f431-59a4-4fd1-b7fa-41bb5effd731  10.5   3.5    77    83\n",
       "1  73c7a609-ce12-4e96-b1f4-58059cb60d4a  13.5   5.0    77    73\n",
       "2  1ebf5189-e22f-4dfd-9f72-5ab06b37ca76   9.0   6.5    80    90\n",
       "3  64ec3c75-b047-4c85-8b3e-99c63a1953aa   5.5   3.0    70    77\n",
       "4  098adb0c-c0ab-4a2d-a590-8d917e3604f2  12.5   4.0    83    80"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n",
      "Algo score, swaps needed: 919.00724\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1498, 1277, 1260, 1280, 1377, 1230, 1555, 1374, 1264, 1274]\n",
      "Pseudo-random, avergae swaps needed: 1324.88911\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n",
      "Algo score, swaps needed: 834.03755\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1213, 1156, 1373, 1219, 1103, 1335, 1077, 1069, 1339, 1345]\n",
      "Pseudo-random, avergae swaps needed: 1219.81396\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ba674cb4-50fb-4023-8bb7-5ae68452d116</td>\n",
       "      <td>55</td>\n",
       "      <td>40</td>\n",
       "      <td>41</td>\n",
       "      <td>83</td>\n",
       "      <td>93</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ab397bfe-7c68-4f02-baee-46407830b974</td>\n",
       "      <td>36</td>\n",
       "      <td>28</td>\n",
       "      <td>22</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11db58c4-af5b-4ffb-a2be-a807e6d3b0f6</td>\n",
       "      <td>34</td>\n",
       "      <td>26</td>\n",
       "      <td>22</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>09e22550-57f9-46d9-b5e2-63a3b0f30890</td>\n",
       "      <td>43</td>\n",
       "      <td>31</td>\n",
       "      <td>23</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c013d719-f4b0-4788-8955-41e80cd92edf</td>\n",
       "      <td>41</td>\n",
       "      <td>32</td>\n",
       "      <td>41</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q3_x  Q1_y  Q2_y  Q3_y\n",
       "0  ba674cb4-50fb-4023-8bb7-5ae68452d116    55    40    41    83    93    87\n",
       "1  ab397bfe-7c68-4f02-baee-46407830b974    36    28    22    87    87    83\n",
       "2  11db58c4-af5b-4ffb-a2be-a807e6d3b0f6    34    26    22    83    87    80\n",
       "3  09e22550-57f9-46d9-b5e2-63a3b0f30890    43    31    23    83    83    83\n",
       "4  c013d719-f4b0-4788-8955-41e80cd92edf    41    32    41    83    87    87"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n",
      "Algo score, swaps needed: 1139.95249\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1473, 1248, 1600, 1508, 1557, 1409, 1458, 1547, 1574, 1450]\n",
      "Pseudo-random, avergae swaps needed: 1471.74111\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n",
      "Algo score, swaps needed: 908.52944\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1438, 1543, 1439, 1637, 1741, 1580, 1491, 1531, 1358, 1271]\n",
      "Pseudo-random, avergae swaps needed: 1450.04051\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n",
      "Algo score, swaps needed: 857.42491\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1573, 1620, 1499, 1287, 1396, 1525, 1296, 1598, 1455, 1386]\n",
      "Pseudo-random, avergae swaps needed: 1458.2128\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5588c2b6-6cba-4955-a41c-b54a085d6751</td>\n",
       "      <td>15.50</td>\n",
       "      <td>15.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>922969bb-2109-4d45-8d53-919f843faa07</td>\n",
       "      <td>23.00</td>\n",
       "      <td>16.0</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>19504cf2-0e8e-47e7-8cd8-37ccc42cc860</td>\n",
       "      <td>20.50</td>\n",
       "      <td>15.0</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>59b5388b-5b2b-430c-a246-0d8b83249966</td>\n",
       "      <td>19.00</td>\n",
       "      <td>20.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f9b1bd52-ca9b-4c23-8758-7dcb7006a1b1</td>\n",
       "      <td>6.25</td>\n",
       "      <td>12.0</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID   Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  5588c2b6-6cba-4955-a41c-b54a085d6751  15.50  15.0    83    80\n",
       "1  922969bb-2109-4d45-8d53-919f843faa07  23.00  16.0    77    83\n",
       "2  19504cf2-0e8e-47e7-8cd8-37ccc42cc860  20.50  15.0    83    83\n",
       "3  59b5388b-5b2b-430c-a246-0d8b83249966  19.00  20.0    83    80\n",
       "4  f9b1bd52-ca9b-4c23-8758-7dcb7006a1b1   6.25  12.0    77    77"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n",
      "Algo score, swaps needed: 1745.88055\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2349, 1807, 2094, 2042, 2084, 1777, 1822, 2271, 1952, 2124]\n",
      "Pseudo-random, avergae swaps needed: 2045.07331\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n",
      "Algo score, swaps needed: 1731.0502\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1862, 1824, 1812, 1953, 1979, 1899, 1893, 2309, 1853, 1980]\n",
      "Pseudo-random, avergae swaps needed: 1993.76677\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[1002.03083, 1193.03933, 970.57382, 1283.03442, 1230.06142, 91.00509, 919.00724, 834.03755, 1139.95249, 908.52944, 857.42491, 1745.88055, 1731.0502]\n",
      "N: 13 \tMean:  1069.6636376923077 \tVar:  162116.16715370378\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1498.81278, 1566.86461, 1560.04081, 1837.85914, 1757.28153, 182.40981, 1324.88911, 1219.81396, 1471.74111, 1450.04051, 1458.2128, 2045.07331, 1993.76677]\n",
      "N: 13 \tMean:  1489.7543269230769 \tVar:  198324.48312159677 \n",
      "\n",
      "1 ) 1498.81278 > 1002.03083 => 39.729149636829945\n",
      "2 ) 1566.86461 > 1193.03933 => 27.089731246225913\n",
      "3 ) 1560.04081 > 970.57382 => 46.58686336607482\n",
      "4 ) 1837.85914 > 1283.03442 => 35.55550417425964\n",
      "5 ) 1757.28153 > 1230.06142 => 35.29692565093673\n",
      "6 ) 182.40981 > 91.00509 => 66.86154997405043\n",
      "7 ) 1324.88911 > 919.00724 => 36.17652571162657\n",
      "8 ) 1219.81396 > 834.03755 => 37.56614420484565\n",
      "9 ) 1471.74111 > 1139.95249 => 25.407928403239957\n",
      "10 ) 1450.04051 > 908.52944 => 45.91859317125617\n",
      "11 ) 1458.2128 > 857.42491 => 51.88962741498972\n",
      "12 ) 2045.07331 > 1745.88055 => 15.784563518797345\n",
      "13 ) 1993.76677 > 1731.0502 => 14.106280771159614\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "36.766875941868655 \n",
      "\n",
      "Ttest_relResult(statistic=-10.091581943427244, pvalue=3.2450841337734005e-07)\n",
      "Cohen's d:  0.9507363930529238\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp2',normv=1,score=1,goal=\"centroid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e</td>\n",
       "      <td>-0.439465</td>\n",
       "      <td>-0.559614</td>\n",
       "      <td>0.502278</td>\n",
       "      <td>0.807046</td>\n",
       "      <td>1.343111</td>\n",
       "      <td>-0.026743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d2054499-4dd2-4ea1-9dae-7751c37f58e5</td>\n",
       "      <td>0.120927</td>\n",
       "      <td>-1.074762</td>\n",
       "      <td>0.205903</td>\n",
       "      <td>1.037645</td>\n",
       "      <td>0.220430</td>\n",
       "      <td>0.220944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f09ccb39-f576-42af-9803-e5d3c10fdfba</td>\n",
       "      <td>-0.159269</td>\n",
       "      <td>1.191890</td>\n",
       "      <td>0.798654</td>\n",
       "      <td>-1.409790</td>\n",
       "      <td>0.657383</td>\n",
       "      <td>0.681871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>341a81d8-db2d-4017-95ca-f11d39c62331</td>\n",
       "      <td>-0.439465</td>\n",
       "      <td>-1.383851</td>\n",
       "      <td>-0.683223</td>\n",
       "      <td>-1.035278</td>\n",
       "      <td>0.005974</td>\n",
       "      <td>0.183160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68975dd8-703c-43ba-8247-1e6cde4fe067</td>\n",
       "      <td>0.120927</td>\n",
       "      <td>-0.147495</td>\n",
       "      <td>0.798654</td>\n",
       "      <td>0.288590</td>\n",
       "      <td>0.551271</td>\n",
       "      <td>0.357628</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x      Q1_x      Q2_x  \\\n",
       "0  7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e    -0.439465 -0.559614  0.502278   \n",
       "1  d2054499-4dd2-4ea1-9dae-7751c37f58e5     0.120927 -1.074762  0.205903   \n",
       "2  f09ccb39-f576-42af-9803-e5d3c10fdfba    -0.159269  1.191890  0.798654   \n",
       "3  341a81d8-db2d-4017-95ca-f11d39c62331    -0.439465 -1.383851 -0.683223   \n",
       "4  68975dd8-703c-43ba-8247-1e6cde4fe067     0.120927 -0.147495  0.798654   \n",
       "\n",
       "   SHORT_ANS_y      Q1_y      Q2_y  \n",
       "0     0.807046  1.343111 -0.026743  \n",
       "1     1.037645  0.220430  0.220944  \n",
       "2    -1.409790  0.657383  0.681871  \n",
       "3    -1.035278  0.005974  0.183160  \n",
       "4     0.288590  0.551271  0.357628  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.439465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d2054499-4dd2-4ea1-9dae-7751c37f58e5</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.120927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f09ccb39-f576-42af-9803-e5d3c10fdfba</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.159269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>341a81d8-db2d-4017-95ca-f11d39c62331</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.439465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68975dd8-703c-43ba-8247-1e6cde4fe067</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.120927</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e  Human -0.439465\n",
       "1  d2054499-4dd2-4ea1-9dae-7751c37f58e5  Human  0.120927\n",
       "2  f09ccb39-f576-42af-9803-e5d3c10fdfba  Human -0.159269\n",
       "3  341a81d8-db2d-4017-95ca-f11d39c62331  Human -0.439465\n",
       "4  68975dd8-703c-43ba-8247-1e6cde4fe067  Human  0.120927"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.469717</td>\n",
       "      <td>2.771568</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000004</td>\n",
       "      <td>[0.28, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.467995</td>\n",
       "      <td>2.738156</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>[0.28, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.464977</td>\n",
       "      <td>2.738156</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>[0.28, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.639193</td>\n",
       "      <td>2.771568</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000004</td>\n",
       "      <td>[0.44, 0.77]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.637598</td>\n",
       "      <td>2.738156</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>[0.44, 0.77]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.634791</td>\n",
       "      <td>2.738156</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>[0.43, 0.77]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.469717  2.771568   80   81  0.000004   \n",
       "ICC2      Single random raters  0.467995  2.738156   80   80  0.000005   \n",
       "ICC3       Single fixed raters  0.464977  2.738156   80   80  0.000005   \n",
       "ICC1k  Average raters absolute  0.639193  2.771568   80   81  0.000004   \n",
       "ICC2k    Average random raters  0.637598  2.738156   80   80  0.000005   \n",
       "ICC3k     Average fixed raters  0.634791  2.738156   80   80  0.000005   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.28, 0.62]  \n",
       "ICC2   [0.28, 0.62]  \n",
       "ICC3   [0.28, 0.62]  \n",
       "ICC1k  [0.44, 0.77]  \n",
       "ICC2k  [0.44, 0.77]  \n",
       "ICC3k  [0.43, 0.77]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.44649725553974373\n",
      "Algo score, swaps needed: 1004.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1407, 1646, 1709, 1520, 1446, 1424, 1600, 1612, 1681, 1355]\n",
      "Pseudo-random, avergae swaps needed: 1499.66689\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.559614</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d2054499-4dd2-4ea1-9dae-7751c37f58e5</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.074762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f09ccb39-f576-42af-9803-e5d3c10fdfba</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.191890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>341a81d8-db2d-4017-95ca-f11d39c62331</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.383851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68975dd8-703c-43ba-8247-1e6cde4fe067</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.147495</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e  Human -0.559614\n",
       "1  d2054499-4dd2-4ea1-9dae-7751c37f58e5  Human -1.074762\n",
       "2  f09ccb39-f576-42af-9803-e5d3c10fdfba  Human  1.191890\n",
       "3  341a81d8-db2d-4017-95ca-f11d39c62331  Human -1.383851\n",
       "4  68975dd8-703c-43ba-8247-1e6cde4fe067  Human -0.147495"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.457011</td>\n",
       "      <td>2.683315</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>[0.27, 0.61]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.455164</td>\n",
       "      <td>2.650223</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000010</td>\n",
       "      <td>[0.26, 0.61]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.452088</td>\n",
       "      <td>2.650223</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000010</td>\n",
       "      <td>[0.26, 0.61]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.627327</td>\n",
       "      <td>2.683315</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>[0.42, 0.76]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.625584</td>\n",
       "      <td>2.650223</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000010</td>\n",
       "      <td>[0.42, 0.76]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.622673</td>\n",
       "      <td>2.650223</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000010</td>\n",
       "      <td>[0.41, 0.76]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.457011  2.683315   80   81  0.000007   \n",
       "ICC2      Single random raters  0.455164  2.650223   80   80  0.000010   \n",
       "ICC3       Single fixed raters  0.452088  2.650223   80   80  0.000010   \n",
       "ICC1k  Average raters absolute  0.627327  2.683315   80   81  0.000007   \n",
       "ICC2k    Average random raters  0.625584  2.650223   80   80  0.000010   \n",
       "ICC3k     Average fixed raters  0.622673  2.650223   80   80  0.000010   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.27, 0.61]  \n",
       "ICC2   [0.26, 0.61]  \n",
       "ICC3   [0.26, 0.61]  \n",
       "ICC1k  [0.42, 0.76]  \n",
       "ICC2k  [0.42, 0.76]  \n",
       "ICC3k  [0.41, 0.76]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.43873165193414976\n",
      "Algo score, swaps needed: 1176.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1615, 1477, 1725, 1751, 1514, 1830, 1620, 1521, 1481, 1526]\n",
      "Pseudo-random, avergae swaps needed: 1566.66396\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.502278</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d2054499-4dd2-4ea1-9dae-7751c37f58e5</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.205903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>f09ccb39-f576-42af-9803-e5d3c10fdfba</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.798654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>341a81d8-db2d-4017-95ca-f11d39c62331</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.683223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>68975dd8-703c-43ba-8247-1e6cde4fe067</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.798654</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  7a4dfc2e-553b-4bc6-bc18-3b0626c17a1e  Human  0.502278\n",
       "1  d2054499-4dd2-4ea1-9dae-7751c37f58e5  Human  0.205903\n",
       "2  f09ccb39-f576-42af-9803-e5d3c10fdfba  Human  0.798654\n",
       "3  341a81d8-db2d-4017-95ca-f11d39c62331  Human -0.683223\n",
       "4  68975dd8-703c-43ba-8247-1e6cde4fe067  Human  0.798654"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.533812</td>\n",
       "      <td>3.290118</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>1.069540e-07</td>\n",
       "      <td>[0.36, 0.67]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.532710</td>\n",
       "      <td>3.257148</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>1.504265e-07</td>\n",
       "      <td>[0.36, 0.67]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.530202</td>\n",
       "      <td>3.257148</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>1.504265e-07</td>\n",
       "      <td>[0.35, 0.67]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.696059</td>\n",
       "      <td>3.290118</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>1.069540e-07</td>\n",
       "      <td>[0.53, 0.8]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.695122</td>\n",
       "      <td>3.257148</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>1.504265e-07</td>\n",
       "      <td>[0.53, 0.8]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.692983</td>\n",
       "      <td>3.257148</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>1.504265e-07</td>\n",
       "      <td>[0.52, 0.8]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.533812  3.290118   80   81  1.069540e-07   \n",
       "ICC2      Single random raters  0.532710  3.257148   80   80  1.504265e-07   \n",
       "ICC3       Single fixed raters  0.530202  3.257148   80   80  1.504265e-07   \n",
       "ICC1k  Average raters absolute  0.696059  3.290118   80   81  1.069540e-07   \n",
       "ICC2k    Average random raters  0.695122  3.257148   80   80  1.504265e-07   \n",
       "ICC3k     Average fixed raters  0.692983  3.257148   80   80  1.504265e-07   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.36, 0.67]  \n",
       "ICC2   [0.36, 0.67]  \n",
       "ICC3   [0.35, 0.67]  \n",
       "ICC1k   [0.53, 0.8]  \n",
       "ICC2k   [0.53, 0.8]  \n",
       "ICC3k   [0.52, 0.8]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.5623944153304041\n",
      "Algo score, swaps needed: 898.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1656, 1621, 1500, 1456, 1641, 1532, 1604, 1468, 1524, 1539]\n",
      "Pseudo-random, avergae swaps needed: 1560.97148\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8a2c52d8-2f79-4f23-bbca-9dacca67bf27</td>\n",
       "      <td>-1.811065</td>\n",
       "      <td>-1.144818</td>\n",
       "      <td>-2.912223</td>\n",
       "      <td>0.193750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>af76b772-5886-4e53-a62d-e87b17f052c9</td>\n",
       "      <td>-0.866162</td>\n",
       "      <td>-1.144818</td>\n",
       "      <td>-0.974972</td>\n",
       "      <td>-1.062013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>381e6532-4d26-4cb3-bd80-79b5628f6ba8</td>\n",
       "      <td>-0.866162</td>\n",
       "      <td>0.356847</td>\n",
       "      <td>-0.010128</td>\n",
       "      <td>0.501987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>68202246-13a0-4c1f-a238-2590685176fa</td>\n",
       "      <td>0.787420</td>\n",
       "      <td>0.657180</td>\n",
       "      <td>1.060997</td>\n",
       "      <td>0.638339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>05baba87-2dab-4b20-b95e-1150bd449ba4</td>\n",
       "      <td>0.078742</td>\n",
       "      <td>-0.544152</td>\n",
       "      <td>-0.257584</td>\n",
       "      <td>-0.270717</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  8a2c52d8-2f79-4f23-bbca-9dacca67bf27 -1.811065 -1.144818 -2.912223   \n",
       "1  af76b772-5886-4e53-a62d-e87b17f052c9 -0.866162 -1.144818 -0.974972   \n",
       "2  381e6532-4d26-4cb3-bd80-79b5628f6ba8 -0.866162  0.356847 -0.010128   \n",
       "3  68202246-13a0-4c1f-a238-2590685176fa  0.787420  0.657180  1.060997   \n",
       "4  05baba87-2dab-4b20-b95e-1150bd449ba4  0.078742 -0.544152 -0.257584   \n",
       "\n",
       "       Q2_y  \n",
       "0  0.193750  \n",
       "1 -1.062013  \n",
       "2  0.501987  \n",
       "3  0.638339  \n",
       "4 -0.270717  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8a2c52d8-2f79-4f23-bbca-9dacca67bf27</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.811065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>af76b772-5886-4e53-a62d-e87b17f052c9</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.866162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>381e6532-4d26-4cb3-bd80-79b5628f6ba8</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.866162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>68202246-13a0-4c1f-a238-2590685176fa</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.787420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>05baba87-2dab-4b20-b95e-1150bd449ba4</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.078742</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  8a2c52d8-2f79-4f23-bbca-9dacca67bf27  Human -1.811065\n",
       "1  af76b772-5886-4e53-a62d-e87b17f052c9  Human -0.866162\n",
       "2  381e6532-4d26-4cb3-bd80-79b5628f6ba8  Human -0.866162\n",
       "3  68202246-13a0-4c1f-a238-2590685176fa  Human  0.787420\n",
       "4  05baba87-2dab-4b20-b95e-1150bd449ba4  Human  0.078742"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.443737</td>\n",
       "      <td>2.595423</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000006</td>\n",
       "      <td>[0.26, 0.6]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.442045</td>\n",
       "      <td>2.567424</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>[0.26, 0.6]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.439371</td>\n",
       "      <td>2.567424</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>[0.25, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.614706</td>\n",
       "      <td>2.595423</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000006</td>\n",
       "      <td>[0.41, 0.75]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.613081</td>\n",
       "      <td>2.567424</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>[0.41, 0.75]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.610505</td>\n",
       "      <td>2.567424</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>[0.41, 0.74]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.443737  2.595423   87   88  0.000006   \n",
       "ICC2      Single random raters  0.442045  2.567424   87   87  0.000008   \n",
       "ICC3       Single fixed raters  0.439371  2.567424   87   87  0.000008   \n",
       "ICC1k  Average raters absolute  0.614706  2.595423   87   88  0.000006   \n",
       "ICC2k    Average random raters  0.613081  2.567424   87   87  0.000008   \n",
       "ICC3k     Average fixed raters  0.610505  2.567424   87   87  0.000008   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1    [0.26, 0.6]  \n",
       "ICC2    [0.26, 0.6]  \n",
       "ICC3   [0.25, 0.59]  \n",
       "ICC1k  [0.41, 0.75]  \n",
       "ICC2k  [0.41, 0.75]  \n",
       "ICC3k  [0.41, 0.74]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.43825737722516567\n",
      "Algo score, swaps needed: 1271.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1887, 1918, 1749, 1747, 1875, 1862, 1663, 1777, 1825, 1948]\n",
      "Pseudo-random, avergae swaps needed: 1837.65831\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8a2c52d8-2f79-4f23-bbca-9dacca67bf27</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.144818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>af76b772-5886-4e53-a62d-e87b17f052c9</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.144818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>381e6532-4d26-4cb3-bd80-79b5628f6ba8</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.356847</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>68202246-13a0-4c1f-a238-2590685176fa</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.657180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>05baba87-2dab-4b20-b95e-1150bd449ba4</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.544152</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  8a2c52d8-2f79-4f23-bbca-9dacca67bf27  Human -1.144818\n",
       "1  af76b772-5886-4e53-a62d-e87b17f052c9  Human -1.144818\n",
       "2  381e6532-4d26-4cb3-bd80-79b5628f6ba8  Human  0.356847\n",
       "3  68202246-13a0-4c1f-a238-2590685176fa  Human  0.657180\n",
       "4  05baba87-2dab-4b20-b95e-1150bd449ba4  Human -0.544152"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.411264</td>\n",
       "      <td>2.397110</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000029</td>\n",
       "      <td>[0.22, 0.57]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.409576</td>\n",
       "      <td>2.374046</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000037</td>\n",
       "      <td>[0.22, 0.57]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.407240</td>\n",
       "      <td>2.374046</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000037</td>\n",
       "      <td>[0.22, 0.57]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.582831</td>\n",
       "      <td>2.397110</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.000029</td>\n",
       "      <td>[0.36, 0.73]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.581133</td>\n",
       "      <td>2.374046</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000037</td>\n",
       "      <td>[0.36, 0.73]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.578778</td>\n",
       "      <td>2.374046</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.000037</td>\n",
       "      <td>[0.36, 0.72]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.411264  2.397110   87   88  0.000029   \n",
       "ICC2      Single random raters  0.409576  2.374046   87   87  0.000037   \n",
       "ICC3       Single fixed raters  0.407240  2.374046   87   87  0.000037   \n",
       "ICC1k  Average raters absolute  0.582831  2.397110   87   88  0.000029   \n",
       "ICC2k    Average random raters  0.581133  2.374046   87   87  0.000037   \n",
       "ICC3k     Average fixed raters  0.578778  2.374046   87   87  0.000037   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.22, 0.57]  \n",
       "ICC2   [0.22, 0.57]  \n",
       "ICC3   [0.22, 0.57]  \n",
       "ICC1k  [0.36, 0.73]  \n",
       "ICC2k  [0.36, 0.73]  \n",
       "ICC3k  [0.36, 0.72]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.4449625465040623\n",
      "Algo score, swaps needed: 1196.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1957, 1785, 1632, 1665, 1572, 1925, 1867, 1909, 1699, 1764]\n",
      "Pseudo-random, avergae swaps needed: 1757.52432\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4e406595-0ea8-477c-af86-86048cfcb9b8</td>\n",
       "      <td>-0.600160</td>\n",
       "      <td>0.391077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a7931454-2722-4600-ad92-baccd06a38a8</td>\n",
       "      <td>1.607446</td>\n",
       "      <td>0.803181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8fee9d29-68ff-4b2d-b54d-58f8303249ff</td>\n",
       "      <td>0.115820</td>\n",
       "      <td>0.064055</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f1a75c07-eda1-4552-907a-c976d9ae7c0c</td>\n",
       "      <td>1.607446</td>\n",
       "      <td>1.073256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a6db6331-93ae-48a4-ab5d-04cd34cf3989</td>\n",
       "      <td>-0.421165</td>\n",
       "      <td>-0.033444</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q2_x      Q2_y\n",
       "0  4e406595-0ea8-477c-af86-86048cfcb9b8 -0.600160  0.391077\n",
       "1  a7931454-2722-4600-ad92-baccd06a38a8  1.607446  0.803181\n",
       "2  8fee9d29-68ff-4b2d-b54d-58f8303249ff  0.115820  0.064055\n",
       "3  f1a75c07-eda1-4552-907a-c976d9ae7c0c  1.607446  1.073256\n",
       "4  a6db6331-93ae-48a4-ab5d-04cd34cf3989 -0.421165 -0.033444"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4e406595-0ea8-477c-af86-86048cfcb9b8</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.600160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a7931454-2722-4600-ad92-baccd06a38a8</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.607446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8fee9d29-68ff-4b2d-b54d-58f8303249ff</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.115820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f1a75c07-eda1-4552-907a-c976d9ae7c0c</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.607446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1d16424e-886f-4420-9479-d806d0fd5262</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.182505</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  4e406595-0ea8-477c-af86-86048cfcb9b8  Human -0.600160\n",
       "1  a7931454-2722-4600-ad92-baccd06a38a8  Human  1.607446\n",
       "2  8fee9d29-68ff-4b2d-b54d-58f8303249ff  Human  0.115820\n",
       "3  f1a75c07-eda1-4552-907a-c976d9ae7c0c  Human  1.607446\n",
       "4  1d16424e-886f-4420-9479-d806d0fd5262  Human -0.182505"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.613381</td>\n",
       "      <td>4.173052</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.000167</td>\n",
       "      <td>[0.32, 0.8]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.611588</td>\n",
       "      <td>4.075730</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000247</td>\n",
       "      <td>[0.31, 0.8]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.605968</td>\n",
       "      <td>4.075730</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000247</td>\n",
       "      <td>[0.31, 0.8]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.760367</td>\n",
       "      <td>4.173052</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.000167</td>\n",
       "      <td>[0.49, 0.89]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.758988</td>\n",
       "      <td>4.075730</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000247</td>\n",
       "      <td>[0.48, 0.89]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.754645</td>\n",
       "      <td>4.075730</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.000247</td>\n",
       "      <td>[0.47, 0.89]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.613381  4.173052   27   28  0.000167   \n",
       "ICC2      Single random raters  0.611588  4.075730   27   27  0.000247   \n",
       "ICC3       Single fixed raters  0.605968  4.075730   27   27  0.000247   \n",
       "ICC1k  Average raters absolute  0.760367  4.173052   27   28  0.000167   \n",
       "ICC2k    Average random raters  0.758988  4.075730   27   27  0.000247   \n",
       "ICC3k     Average fixed raters  0.754645  4.075730   27   27  0.000247   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1    [0.32, 0.8]  \n",
       "ICC2    [0.31, 0.8]  \n",
       "ICC3    [0.31, 0.8]  \n",
       "ICC1k  [0.49, 0.89]  \n",
       "ICC2k  [0.48, 0.89]  \n",
       "ICC3k  [0.47, 0.89]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.6450260023485992\n",
      "Algo score, swaps needed: 92.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[179, 237, 214, 164, 199, 154, 172, 223, 166, 194]\n",
      "Pseudo-random, avergae swaps needed: 182.57292\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3d29b5ad-e82f-4972-89a4-adb948591eb1</td>\n",
       "      <td>1.195919</td>\n",
       "      <td>0.880199</td>\n",
       "      <td>0.369450</td>\n",
       "      <td>-2.099819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6c261e5d-2212-43a3-ab4f-28bd7dd90612</td>\n",
       "      <td>-0.802630</td>\n",
       "      <td>-1.417176</td>\n",
       "      <td>0.676527</td>\n",
       "      <td>0.349529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>75fe19f4-4652-4851-afe6-affd379502de</td>\n",
       "      <td>-1.468813</td>\n",
       "      <td>-0.498226</td>\n",
       "      <td>-0.256541</td>\n",
       "      <td>-0.019297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a868c5d1-d01c-44b6-bbd8-1f9eb5857baa</td>\n",
       "      <td>0.030099</td>\n",
       "      <td>0.880199</td>\n",
       "      <td>-0.298173</td>\n",
       "      <td>-0.002833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1ebf5189-e22f-4dfd-9f72-5ab06b37ca76</td>\n",
       "      <td>-0.802630</td>\n",
       "      <td>1.799149</td>\n",
       "      <td>-0.392862</td>\n",
       "      <td>1.111576</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  3d29b5ad-e82f-4972-89a4-adb948591eb1  1.195919  0.880199  0.369450   \n",
       "1  6c261e5d-2212-43a3-ab4f-28bd7dd90612 -0.802630 -1.417176  0.676527   \n",
       "2  75fe19f4-4652-4851-afe6-affd379502de -1.468813 -0.498226 -0.256541   \n",
       "3  a868c5d1-d01c-44b6-bbd8-1f9eb5857baa  0.030099  0.880199 -0.298173   \n",
       "4  1ebf5189-e22f-4dfd-9f72-5ab06b37ca76 -0.802630  1.799149 -0.392862   \n",
       "\n",
       "       Q2_y  \n",
       "0 -2.099819  \n",
       "1  0.349529  \n",
       "2 -0.019297  \n",
       "3 -0.002833  \n",
       "4  1.111576  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>80abd277-bd34-4f34-b425-30190a9eaa98</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.196644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3d29b5ad-e82f-4972-89a4-adb948591eb1</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.195919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6c261e5d-2212-43a3-ab4f-28bd7dd90612</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.802630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>75fe19f4-4652-4851-afe6-affd379502de</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.468813</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a868c5d1-d01c-44b6-bbd8-1f9eb5857baa</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.030099</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  80abd277-bd34-4f34-b425-30190a9eaa98  Human  0.196644\n",
       "1  3d29b5ad-e82f-4972-89a4-adb948591eb1  Human  1.195919\n",
       "2  6c261e5d-2212-43a3-ab4f-28bd7dd90612  Human -0.802630\n",
       "3  75fe19f4-4652-4851-afe6-affd379502de  Human -1.468813\n",
       "4  a868c5d1-d01c-44b6-bbd8-1f9eb5857baa  Human  0.030099"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.505370</td>\n",
       "      <td>3.043424</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>[0.32, 0.66]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.503724</td>\n",
       "      <td>3.003152</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>[0.31, 0.66]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.500394</td>\n",
       "      <td>3.003152</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>[0.31, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.671423</td>\n",
       "      <td>3.043424</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>[0.48, 0.79]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.669969</td>\n",
       "      <td>3.003152</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>[0.48, 0.79]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.667017</td>\n",
       "      <td>3.003152</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>[0.47, 0.79]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.505370  3.043424   74   75  0.000001   \n",
       "ICC2      Single random raters  0.503724  3.003152   74   74  0.000002   \n",
       "ICC3       Single fixed raters  0.500394  3.003152   74   74  0.000002   \n",
       "ICC1k  Average raters absolute  0.671423  3.043424   74   75  0.000001   \n",
       "ICC2k    Average random raters  0.669969  3.003152   74   74  0.000002   \n",
       "ICC3k     Average fixed raters  0.667017  3.003152   74   74  0.000002   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.32, 0.66]  \n",
       "ICC2   [0.31, 0.66]  \n",
       "ICC3   [0.31, 0.65]  \n",
       "ICC1k  [0.48, 0.79]  \n",
       "ICC2k  [0.48, 0.79]  \n",
       "ICC3k  [0.47, 0.79]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.48119882696588434\n",
      "Algo score, swaps needed: 919.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1241, 1148, 1364, 1282, 1272, 1410, 1209, 1216, 1255, 1265]\n",
      "Pseudo-random, avergae swaps needed: 1324.95001\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>80abd277-bd34-4f34-b425-30190a9eaa98</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.420724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3d29b5ad-e82f-4972-89a4-adb948591eb1</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.880199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6c261e5d-2212-43a3-ab4f-28bd7dd90612</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.417176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>75fe19f4-4652-4851-afe6-affd379502de</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.498226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a868c5d1-d01c-44b6-bbd8-1f9eb5857baa</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.880199</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  80abd277-bd34-4f34-b425-30190a9eaa98  Human  0.420724\n",
       "1  3d29b5ad-e82f-4972-89a4-adb948591eb1  Human  0.880199\n",
       "2  6c261e5d-2212-43a3-ab4f-28bd7dd90612  Human -1.417176\n",
       "3  75fe19f4-4652-4851-afe6-affd379502de  Human -0.498226\n",
       "4  a868c5d1-d01c-44b6-bbd8-1f9eb5857baa  Human  0.880199"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.375496</td>\n",
       "      <td>2.202543</td>\n",
       "      <td>70</td>\n",
       "      <td>71</td>\n",
       "      <td>0.000547</td>\n",
       "      <td>[0.16, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.373043</td>\n",
       "      <td>2.175286</td>\n",
       "      <td>70</td>\n",
       "      <td>70</td>\n",
       "      <td>0.000687</td>\n",
       "      <td>[0.15, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.370136</td>\n",
       "      <td>2.175286</td>\n",
       "      <td>70</td>\n",
       "      <td>70</td>\n",
       "      <td>0.000687</td>\n",
       "      <td>[0.15, 0.55]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.545979</td>\n",
       "      <td>2.202543</td>\n",
       "      <td>70</td>\n",
       "      <td>71</td>\n",
       "      <td>0.000547</td>\n",
       "      <td>[0.27, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.543382</td>\n",
       "      <td>2.175286</td>\n",
       "      <td>70</td>\n",
       "      <td>70</td>\n",
       "      <td>0.000687</td>\n",
       "      <td>[0.27, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.540290</td>\n",
       "      <td>2.175286</td>\n",
       "      <td>70</td>\n",
       "      <td>70</td>\n",
       "      <td>0.000687</td>\n",
       "      <td>[0.26, 0.71]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.375496  2.202543   70   71  0.000547   \n",
       "ICC2      Single random raters  0.373043  2.175286   70   70  0.000687   \n",
       "ICC3       Single fixed raters  0.370136  2.175286   70   70  0.000687   \n",
       "ICC1k  Average raters absolute  0.545979  2.202543   70   71  0.000547   \n",
       "ICC2k    Average random raters  0.543382  2.175286   70   70  0.000687   \n",
       "ICC3k     Average fixed raters  0.540290  2.175286   70   70  0.000687   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.16, 0.56]  \n",
       "ICC2   [0.15, 0.56]  \n",
       "ICC3   [0.15, 0.55]  \n",
       "ICC1k  [0.27, 0.72]  \n",
       "ICC2k  [0.27, 0.72]  \n",
       "ICC3k  [0.26, 0.71]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.3848289799263249\n",
      "Algo score, swaps needed: 750.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1017, 1013, 1238, 1015, 924, 1062, 1042, 1183, 845, 1043]\n",
      "Pseudo-random, avergae swaps needed: 1091.64396\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ba674cb4-50fb-4023-8bb7-5ae68452d116</td>\n",
       "      <td>0.551646</td>\n",
       "      <td>1.374969</td>\n",
       "      <td>1.677993</td>\n",
       "      <td>0.405869</td>\n",
       "      <td>1.715751</td>\n",
       "      <td>0.950219</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41120ba2-92db-431f-935f-f7904ad647e8</td>\n",
       "      <td>0.414326</td>\n",
       "      <td>0.041273</td>\n",
       "      <td>0.314554</td>\n",
       "      <td>0.017621</td>\n",
       "      <td>0.589728</td>\n",
       "      <td>0.868936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fc5a292f-eeb9-413b-82b7-feae67846c21</td>\n",
       "      <td>-0.752890</td>\n",
       "      <td>-0.266504</td>\n",
       "      <td>-0.269777</td>\n",
       "      <td>-1.010606</td>\n",
       "      <td>0.337776</td>\n",
       "      <td>0.176556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4c68352f-27a5-49c2-aecd-887e788ab441</td>\n",
       "      <td>0.551646</td>\n",
       "      <td>-1.189832</td>\n",
       "      <td>-0.075000</td>\n",
       "      <td>-0.163684</td>\n",
       "      <td>0.090107</td>\n",
       "      <td>0.386832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30234d01-ef3f-41bf-ae4a-1efaba3522d8</td>\n",
       "      <td>1.718862</td>\n",
       "      <td>1.477562</td>\n",
       "      <td>-0.075000</td>\n",
       "      <td>1.154458</td>\n",
       "      <td>1.391002</td>\n",
       "      <td>0.837809</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q3_x  \\\n",
       "0  ba674cb4-50fb-4023-8bb7-5ae68452d116  0.551646  1.374969  1.677993   \n",
       "1  41120ba2-92db-431f-935f-f7904ad647e8  0.414326  0.041273  0.314554   \n",
       "2  fc5a292f-eeb9-413b-82b7-feae67846c21 -0.752890 -0.266504 -0.269777   \n",
       "3  4c68352f-27a5-49c2-aecd-887e788ab441  0.551646 -1.189832 -0.075000   \n",
       "4  30234d01-ef3f-41bf-ae4a-1efaba3522d8  1.718862  1.477562 -0.075000   \n",
       "\n",
       "       Q1_y      Q2_y      Q3_y  \n",
       "0  0.405869  1.715751  0.950219  \n",
       "1  0.017621  0.589728  0.868936  \n",
       "2 -1.010606  0.337776  0.176556  \n",
       "3 -0.163684  0.090107  0.386832  \n",
       "4  1.154458  1.391002  0.837809  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ba674cb4-50fb-4023-8bb7-5ae68452d116</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.551646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41120ba2-92db-431f-935f-f7904ad647e8</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.414326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fc5a292f-eeb9-413b-82b7-feae67846c21</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.752890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f65fe2ab-5f3e-4b04-9a45-8eb8a63e0c19</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.409591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>417062c9-d62a-4452-8878-4c9c32b7a455</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.345666</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  ba674cb4-50fb-4023-8bb7-5ae68452d116  Human  0.551646\n",
       "1  41120ba2-92db-431f-935f-f7904ad647e8  Human  0.414326\n",
       "2  fc5a292f-eeb9-413b-82b7-feae67846c21  Human -0.752890\n",
       "3  f65fe2ab-5f3e-4b04-9a45-8eb8a63e0c19  Human -0.409591\n",
       "4  417062c9-d62a-4452-8878-4c9c32b7a455  Human  0.345666"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.348297</td>\n",
       "      <td>2.068883</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.000793</td>\n",
       "      <td>[0.14, 0.53]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.345639</td>\n",
       "      <td>2.043412</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000991</td>\n",
       "      <td>[0.13, 0.53]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.342843</td>\n",
       "      <td>2.043412</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000991</td>\n",
       "      <td>[0.13, 0.52]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.516647</td>\n",
       "      <td>2.068883</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.000793</td>\n",
       "      <td>[0.24, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.513718</td>\n",
       "      <td>2.043412</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000991</td>\n",
       "      <td>[0.23, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.510622</td>\n",
       "      <td>2.043412</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000991</td>\n",
       "      <td>[0.23, 0.69]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.348297  2.068883   77   78  0.000793   \n",
       "ICC2      Single random raters  0.345639  2.043412   77   77  0.000991   \n",
       "ICC3       Single fixed raters  0.342843  2.043412   77   77  0.000991   \n",
       "ICC1k  Average raters absolute  0.516647  2.068883   77   78  0.000793   \n",
       "ICC2k    Average random raters  0.513718  2.043412   77   77  0.000991   \n",
       "ICC3k     Average fixed raters  0.510622  2.043412   77   77  0.000991   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.14, 0.53]  \n",
       "ICC2   [0.13, 0.53]  \n",
       "ICC3   [0.13, 0.52]  \n",
       "ICC1k  [0.24, 0.69]  \n",
       "ICC2k  [0.23, 0.69]  \n",
       "ICC3k  [0.23, 0.69]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.3314799343147705\n",
      "Algo score, swaps needed: 1184.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1463, 1557, 1576, 1463, 1453, 1446, 1514, 1426, 1520, 1489]\n",
      "Pseudo-random, avergae swaps needed: 1472.83579\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ba674cb4-50fb-4023-8bb7-5ae68452d116</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.374969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41120ba2-92db-431f-935f-f7904ad647e8</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.041273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fc5a292f-eeb9-413b-82b7-feae67846c21</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.266504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f65fe2ab-5f3e-4b04-9a45-8eb8a63e0c19</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.554233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>417062c9-d62a-4452-8878-4c9c32b7a455</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.061319</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  ba674cb4-50fb-4023-8bb7-5ae68452d116  Human  1.374969\n",
       "1  41120ba2-92db-431f-935f-f7904ad647e8  Human  0.041273\n",
       "2  fc5a292f-eeb9-413b-82b7-feae67846c21  Human -0.266504\n",
       "3  f65fe2ab-5f3e-4b04-9a45-8eb8a63e0c19  Human  0.554233\n",
       "4  417062c9-d62a-4452-8878-4c9c32b7a455  Human -0.061319"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.449742</td>\n",
       "      <td>2.634659</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>[0.25, 0.61]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.448031</td>\n",
       "      <td>2.605300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>[0.25, 0.61]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.445261</td>\n",
       "      <td>2.605300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>[0.25, 0.61]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.620444</td>\n",
       "      <td>2.634659</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>[0.41, 0.76]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.618814</td>\n",
       "      <td>2.605300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>[0.4, 0.76]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.616167</td>\n",
       "      <td>2.605300</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>[0.4, 0.76]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.449742  2.634659   77   78  0.000015   \n",
       "ICC2      Single random raters  0.448031  2.605300   77   77  0.000020   \n",
       "ICC3       Single fixed raters  0.445261  2.605300   77   77  0.000020   \n",
       "ICC1k  Average raters absolute  0.620444  2.634659   77   78  0.000015   \n",
       "ICC2k    Average random raters  0.618814  2.605300   77   77  0.000020   \n",
       "ICC3k     Average fixed raters  0.616167  2.605300   77   77  0.000020   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.25, 0.61]  \n",
       "ICC2   [0.25, 0.61]  \n",
       "ICC3   [0.25, 0.61]  \n",
       "ICC1k  [0.41, 0.76]  \n",
       "ICC2k   [0.4, 0.76]  \n",
       "ICC3k   [0.4, 0.76]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.4694298872478958\n",
      "Algo score, swaps needed: 905.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1586, 1400, 1382, 1464, 1372, 1383, 1560, 1658, 1394, 1431]\n",
      "Pseudo-random, avergae swaps needed: 1450.35153\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ba674cb4-50fb-4023-8bb7-5ae68452d116</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.677993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41120ba2-92db-431f-935f-f7904ad647e8</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.314554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fc5a292f-eeb9-413b-82b7-feae67846c21</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.269777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f65fe2ab-5f3e-4b04-9a45-8eb8a63e0c19</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.464554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>417062c9-d62a-4452-8878-4c9c32b7a455</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.483216</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  ba674cb4-50fb-4023-8bb7-5ae68452d116  Human  1.677993\n",
       "1  41120ba2-92db-431f-935f-f7904ad647e8  Human  0.314554\n",
       "2  fc5a292f-eeb9-413b-82b7-feae67846c21  Human -0.269777\n",
       "3  f65fe2ab-5f3e-4b04-9a45-8eb8a63e0c19  Human -0.464554\n",
       "4  417062c9-d62a-4452-8878-4c9c32b7a455  Human  1.483216"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.561804</td>\n",
       "      <td>3.564167</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>3.122207e-08</td>\n",
       "      <td>[0.39, 0.7]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.560555</td>\n",
       "      <td>3.518526</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>4.765327e-08</td>\n",
       "      <td>[0.39, 0.7]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.557378</td>\n",
       "      <td>3.518526</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>4.765327e-08</td>\n",
       "      <td>[0.38, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.719430</td>\n",
       "      <td>3.564167</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>3.122207e-08</td>\n",
       "      <td>[0.56, 0.82]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.718405</td>\n",
       "      <td>3.518526</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>4.765327e-08</td>\n",
       "      <td>[0.56, 0.82]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.715790</td>\n",
       "      <td>3.518526</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>4.765327e-08</td>\n",
       "      <td>[0.55, 0.82]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.561804  3.564167   77   78  3.122207e-08   \n",
       "ICC2      Single random raters  0.560555  3.518526   77   77  4.765327e-08   \n",
       "ICC3       Single fixed raters  0.557378  3.518526   77   77  4.765327e-08   \n",
       "ICC1k  Average raters absolute  0.719430  3.564167   77   78  3.122207e-08   \n",
       "ICC2k    Average random raters  0.718405  3.518526   77   77  4.765327e-08   \n",
       "ICC3k     Average fixed raters  0.715790  3.518526   77   77  4.765327e-08   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1    [0.39, 0.7]  \n",
       "ICC2    [0.39, 0.7]  \n",
       "ICC3   [0.38, 0.69]  \n",
       "ICC1k  [0.56, 0.82]  \n",
       "ICC2k  [0.56, 0.82]  \n",
       "ICC3k  [0.55, 0.82]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.5560003359180361\n",
      "Algo score, swaps needed: 831.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1474, 1434, 1411, 1304, 1532, 1387, 1481, 1600, 1345, 1469]\n",
      "Pseudo-random, avergae swaps needed: 1457.60248\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0091731f-bae9-4326-a66a-a744137584a2</td>\n",
       "      <td>-1.057500</td>\n",
       "      <td>-1.477259</td>\n",
       "      <td>0.367094</td>\n",
       "      <td>0.180184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>83cb5f25-30c8-4aff-b6e7-1b9e323e657c</td>\n",
       "      <td>-1.169677</td>\n",
       "      <td>0.065277</td>\n",
       "      <td>0.590992</td>\n",
       "      <td>0.701665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>546ad478-6cd2-4ad5-92c6-2f32ef76d808</td>\n",
       "      <td>-1.169677</td>\n",
       "      <td>-0.705991</td>\n",
       "      <td>-1.265803</td>\n",
       "      <td>-0.338207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f9b1bd52-ca9b-4c23-8758-7dcb7006a1b1</td>\n",
       "      <td>-2.684063</td>\n",
       "      <td>-0.705991</td>\n",
       "      <td>-0.672031</td>\n",
       "      <td>-0.991709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>90104063-f02a-4ab0-b693-12b727a108e5</td>\n",
       "      <td>0.625152</td>\n",
       "      <td>-0.127540</td>\n",
       "      <td>-2.015043</td>\n",
       "      <td>-1.561243</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  0091731f-bae9-4326-a66a-a744137584a2 -1.057500 -1.477259  0.367094   \n",
       "1  83cb5f25-30c8-4aff-b6e7-1b9e323e657c -1.169677  0.065277  0.590992   \n",
       "2  546ad478-6cd2-4ad5-92c6-2f32ef76d808 -1.169677 -0.705991 -1.265803   \n",
       "3  f9b1bd52-ca9b-4c23-8758-7dcb7006a1b1 -2.684063 -0.705991 -0.672031   \n",
       "4  90104063-f02a-4ab0-b693-12b727a108e5  0.625152 -0.127540 -2.015043   \n",
       "\n",
       "       Q2_y  \n",
       "0  0.180184  \n",
       "1  0.701665  \n",
       "2 -0.338207  \n",
       "3 -0.991709  \n",
       "4 -1.561243  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0568651d-0e40-4bf5-8e24-d8f16403f49e</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.496616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0091731f-bae9-4326-a66a-a744137584a2</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.057500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>83cb5f25-30c8-4aff-b6e7-1b9e323e657c</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.169677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>546ad478-6cd2-4ad5-92c6-2f32ef76d808</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.169677</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f9b1bd52-ca9b-4c23-8758-7dcb7006a1b1</td>\n",
       "      <td>Human</td>\n",
       "      <td>-2.684063</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  0568651d-0e40-4bf5-8e24-d8f16403f49e  Human -0.496616\n",
       "1  0091731f-bae9-4326-a66a-a744137584a2  Human -1.057500\n",
       "2  83cb5f25-30c8-4aff-b6e7-1b9e323e657c  Human -1.169677\n",
       "3  546ad478-6cd2-4ad5-92c6-2f32ef76d808  Human -1.169677\n",
       "4  f9b1bd52-ca9b-4c23-8758-7dcb7006a1b1  Human -2.684063"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.194317</td>\n",
       "      <td>1.482365</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.030563</td>\n",
       "      <td>[-0.01, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.190936</td>\n",
       "      <td>1.467145</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.034549</td>\n",
       "      <td>[-0.02, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.189346</td>\n",
       "      <td>1.467145</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.034549</td>\n",
       "      <td>[-0.02, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.325402</td>\n",
       "      <td>1.482365</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.030563</td>\n",
       "      <td>[-0.02, 0.55]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.320648</td>\n",
       "      <td>1.467145</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.034549</td>\n",
       "      <td>[-0.03, 0.55]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.318404</td>\n",
       "      <td>1.467145</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.034549</td>\n",
       "      <td>[-0.03, 0.55]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.194317  1.482365   91   92  0.030563   \n",
       "ICC2      Single random raters  0.190936  1.467145   91   91  0.034549   \n",
       "ICC3       Single fixed raters  0.189346  1.467145   91   91  0.034549   \n",
       "ICC1k  Average raters absolute  0.325402  1.482365   91   92  0.030563   \n",
       "ICC2k    Average random raters  0.320648  1.467145   91   91  0.034549   \n",
       "ICC3k     Average fixed raters  0.318404  1.467145   91   91  0.034549   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.01, 0.38]  \n",
       "ICC2   [-0.02, 0.38]  \n",
       "ICC3   [-0.02, 0.38]  \n",
       "ICC1k  [-0.02, 0.55]  \n",
       "ICC2k  [-0.03, 0.55]  \n",
       "ICC3k  [-0.03, 0.55]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.1986696787148594\n",
      "Algo score, swaps needed: 1729.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2098, 2279, 2321, 2313, 1974, 1858, 2197, 2276, 1821, 2032]\n",
      "Pseudo-random, avergae swaps needed: 2045.02919\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0568651d-0e40-4bf5-8e24-d8f16403f49e</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.127540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0091731f-bae9-4326-a66a-a744137584a2</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.477259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>83cb5f25-30c8-4aff-b6e7-1b9e323e657c</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.065277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>546ad478-6cd2-4ad5-92c6-2f32ef76d808</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.705991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f9b1bd52-ca9b-4c23-8758-7dcb7006a1b1</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.705991</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  0568651d-0e40-4bf5-8e24-d8f16403f49e  Human -0.127540\n",
       "1  0091731f-bae9-4326-a66a-a744137584a2  Human -1.477259\n",
       "2  83cb5f25-30c8-4aff-b6e7-1b9e323e657c  Human  0.065277\n",
       "3  546ad478-6cd2-4ad5-92c6-2f32ef76d808  Human -0.705991\n",
       "4  f9b1bd52-ca9b-4c23-8758-7dcb7006a1b1  Human -0.705991"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.196336</td>\n",
       "      <td>1.488603</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.029217</td>\n",
       "      <td>[-0.01, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.193431</td>\n",
       "      <td>1.475380</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.032580</td>\n",
       "      <td>[-0.01, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.192043</td>\n",
       "      <td>1.475380</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.032580</td>\n",
       "      <td>[-0.01, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.328229</td>\n",
       "      <td>1.488603</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.029217</td>\n",
       "      <td>[-0.01, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.324160</td>\n",
       "      <td>1.475380</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.032580</td>\n",
       "      <td>[-0.02, 0.55]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.322208</td>\n",
       "      <td>1.475380</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.032580</td>\n",
       "      <td>[-0.02, 0.55]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.196336  1.488603   91   92  0.029217   \n",
       "ICC2      Single random raters  0.193431  1.475380   91   91  0.032580   \n",
       "ICC3       Single fixed raters  0.192043  1.475380   91   91  0.032580   \n",
       "ICC1k  Average raters absolute  0.328229  1.488603   91   92  0.029217   \n",
       "ICC2k    Average random raters  0.324160  1.475380   91   91  0.032580   \n",
       "ICC3k     Average fixed raters  0.322208  1.475380   91   91  0.032580   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.01, 0.38]  \n",
       "ICC2   [-0.01, 0.38]  \n",
       "ICC3   [-0.01, 0.38]  \n",
       "ICC1k  [-0.01, 0.56]  \n",
       "ICC2k  [-0.02, 0.55]  \n",
       "ICC3k  [-0.02, 0.55]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.2047859977228783\n",
      "Algo score, swaps needed: 1753.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1804, 1946, 2214, 2029, 2130, 2049, 2043, 1926, 1742, 2155]\n",
      "Pseudo-random, avergae swaps needed: 1993.26688\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[1004.0, 1176.0, 898.0, 1271.0, 1196.0, 92.0, 919.0, 750.0, 1184.0, 905.0, 831.0, 1729.0, 1753.0]\n",
      "N: 13 \tMean:  1054.4615384615386 \tVar:  167547.01775147932\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.66689, 1566.66396, 1560.97148, 1837.65831, 1757.52432, 182.57292, 1324.95001, 1091.64396, 1472.83579, 1450.35153, 1457.60248, 2045.02919, 1993.26688]\n",
      "N: 13 \tMean:  1480.0567476923075 \tVar:  204746.44657664374 \n",
      "\n",
      "1 ) 1499.66689 > 1004.0 => 39.59527459341845\n",
      "2 ) 1566.66396 > 1176.0 => 28.487920189828877\n",
      "3 ) 1560.97148 > 898.0 => 53.92266526002976\n",
      "4 ) 1837.65831 > 1271.0 => 36.45677675009577\n",
      "5 ) 1757.52432 > 1196.0 => 38.02401870860505\n",
      "6 ) 182.57292 > 92.0 => 65.9736728589258\n",
      "7 ) 1324.95001 > 919.0 => 36.18173383461426\n",
      "8 ) 1091.64396 > 750.0 => 37.102063962461024\n",
      "9 ) 1472.83579 > 1184.0 => 21.742840945393926\n",
      "10 ) 1450.35153 > 905.0 => 46.30744269412727\n",
      "11 ) 1457.60248 > 831.0 => 54.758524949252\n",
      "12 ) 2045.02919 > 1729.0 => 16.747575288361773\n",
      "13 ) 1993.26688 > 1753.0 => 12.827002864248684\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "37.54827022302789 \n",
      "\n",
      "Ttest_relResult(statistic=-9.111941248957306, pvalue=9.687893536286064e-07)\n",
      "Cohen's d:  0.9477372908142065\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp2',normv=1,score=2,goal=\"centroid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>e8c7236a-2cae-40ea-8d17-5405ce4a0e0f</td>\n",
       "      <td>12</td>\n",
       "      <td>17</td>\n",
       "      <td>14</td>\n",
       "      <td>90</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cfa09378-0303-42fb-9ff1-066c8e2f7e88</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>12</td>\n",
       "      <td>70</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4fda7a79-36eb-4380-923e-3878eaadddde</td>\n",
       "      <td>7</td>\n",
       "      <td>22</td>\n",
       "      <td>12</td>\n",
       "      <td>70</td>\n",
       "      <td>70</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8b2e671a-8164-4467-9f19-a6dc7cbf313c</td>\n",
       "      <td>4</td>\n",
       "      <td>27</td>\n",
       "      <td>16</td>\n",
       "      <td>90</td>\n",
       "      <td>83</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c6b6baea-3f79-4d14-bc22-83d685e04cfb</td>\n",
       "      <td>9</td>\n",
       "      <td>30</td>\n",
       "      <td>22</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x  Q1_x  Q2_x  SHORT_ANS_y  \\\n",
       "0  e8c7236a-2cae-40ea-8d17-5405ce4a0e0f           12    17    14           90   \n",
       "1  cfa09378-0303-42fb-9ff1-066c8e2f7e88            4    14    12           70   \n",
       "2  4fda7a79-36eb-4380-923e-3878eaadddde            7    22    12           70   \n",
       "3  8b2e671a-8164-4467-9f19-a6dc7cbf313c            4    27    16           90   \n",
       "4  c6b6baea-3f79-4d14-bc22-83d685e04cfb            9    30    22           83   \n",
       "\n",
       "   Q1_y  Q2_y  \n",
       "0    80    83  \n",
       "1    83    80  \n",
       "2    70    80  \n",
       "3    83    90  \n",
       "4    80    80  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n",
      "Algo score, swaps needed: 1063.58667\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1416, 1695, 1575, 1607, 1510, 1478, 1474, 1643, 1455, 1365]\n",
      "Pseudo-random, avergae swaps needed: 1499.33637\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n",
      "Algo score, swaps needed: 1364.57761\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1670, 1440, 1581, 1430, 1473, 1577, 1719, 1602, 1378, 1392]\n",
      "Pseudo-random, avergae swaps needed: 1566.50795\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n",
      "Algo score, swaps needed: 1112.97992\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1358, 1589, 1523, 1634, 1553, 1586, 1582, 1622, 1535, 1484]\n",
      "Pseudo-random, avergae swaps needed: 1559.56629\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1b15cca5-b292-4c29-8e83-384ed76e7b77</td>\n",
       "      <td>20.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>da455e55-d2ac-4054-aae9-e62723a9dc6c</td>\n",
       "      <td>17.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>73</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d43eda57-827c-4f18-af5f-dcd5b9660eec</td>\n",
       "      <td>24.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9db4494e-5ddd-4a7d-b3e9-031a33003a02</td>\n",
       "      <td>22.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>90</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>346c7a84-2b02-4aaa-81f4-4ae8633b9b69</td>\n",
       "      <td>24.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>87</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  1b15cca5-b292-4c29-8e83-384ed76e7b77  20.0  18.0    77    77\n",
       "1  da455e55-d2ac-4054-aae9-e62723a9dc6c  17.0  17.0    73    83\n",
       "2  d43eda57-827c-4f18-af5f-dcd5b9660eec  24.0  21.0    80    83\n",
       "3  9db4494e-5ddd-4a7d-b3e9-031a33003a02  22.0  25.0    90    87\n",
       "4  346c7a84-2b02-4aaa-81f4-4ae8633b9b69  24.0  25.0    87    80"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n",
      "Algo score, swaps needed: 1395.29161\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1914, 1909, 1743, 1821, 1602, 1776, 2085, 1889, 2085, 1876]\n",
      "Pseudo-random, avergae swaps needed: 1837.29656\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n",
      "Algo score, swaps needed: 1554.92878\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1740, 1819, 1511, 1745, 1815, 1743, 1948, 1765, 1956, 1831]\n",
      "Pseudo-random, avergae swaps needed: 1756.82817\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ab1fbbba-d6e0-4551-aed0-3fc164fa8570</td>\n",
       "      <td>27.0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f1a75c07-eda1-4552-907a-c976d9ae7c0c</td>\n",
       "      <td>42.0</td>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3307e79a-3565-478c-8749-b095d12c7e56</td>\n",
       "      <td>20.5</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>daa06ff3-2614-4aa8-8fad-914ddc4b1cb3</td>\n",
       "      <td>28.0</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>981c3959-fe3b-4222-97de-742768dee8e9</td>\n",
       "      <td>23.0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q2_x  Q2_y\n",
       "0  ab1fbbba-d6e0-4551-aed0-3fc164fa8570  27.0    80\n",
       "1  f1a75c07-eda1-4552-907a-c976d9ae7c0c  42.0    97\n",
       "2  3307e79a-3565-478c-8749-b095d12c7e56  20.5    83\n",
       "3  daa06ff3-2614-4aa8-8fad-914ddc4b1cb3  28.0    83\n",
       "4  981c3959-fe3b-4222-97de-742768dee8e9  23.0    80"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n",
      "Algo score, swaps needed: 107.99537\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[148, 182, 159, 239, 188, 185, 224, 188, 201, 155]\n",
      "Pseudo-random, avergae swaps needed: 182.46159\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4e4a357a-bc9d-46b9-b0c5-01e68cb32ba0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>473b0dda-6847-467e-8589-32a882dcfde1</td>\n",
       "      <td>12.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>80</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>836dd271-a288-4467-b688-9d61bc2b99ff</td>\n",
       "      <td>12.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>47285a02-f301-408b-b50a-6a25530cadae</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>872764dd-e402-419b-b819-b4e2337263c3</td>\n",
       "      <td>9.5</td>\n",
       "      <td>3.5</td>\n",
       "      <td>77</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  4e4a357a-bc9d-46b9-b0c5-01e68cb32ba0  10.0   3.0    83    77\n",
       "1  473b0dda-6847-467e-8589-32a882dcfde1  12.0   5.0    80    73\n",
       "2  836dd271-a288-4467-b688-9d61bc2b99ff  12.5   5.0    80    80\n",
       "3  47285a02-f301-408b-b50a-6a25530cadae   7.0   3.5    83    77\n",
       "4  872764dd-e402-419b-b819-b4e2337263c3   9.5   3.5    77    80"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n",
      "Algo score, swaps needed: 959.03298\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1200, 1360, 1517, 1144, 1154, 1326, 1191, 1400, 1227, 1367]\n",
      "Pseudo-random, avergae swaps needed: 1324.14649\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n",
      "Algo score, swaps needed: 1046.08702\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1441, 1139, 1282, 1129, 1251, 1229, 1027, 1248, 1286, 1377]\n",
      "Pseudo-random, avergae swaps needed: 1219.73759\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39268066-b798-49dd-8159-eda5299389ef</td>\n",
       "      <td>61</td>\n",
       "      <td>16</td>\n",
       "      <td>24</td>\n",
       "      <td>77</td>\n",
       "      <td>80</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cee3d697-38c2-403f-a5e8-2bf351cced67</td>\n",
       "      <td>60</td>\n",
       "      <td>11</td>\n",
       "      <td>18</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5a6f914f-609c-4043-b2ba-6ad26b7fb36c</td>\n",
       "      <td>41</td>\n",
       "      <td>4</td>\n",
       "      <td>25</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ab397bfe-7c68-4f02-baee-46407830b974</td>\n",
       "      <td>36</td>\n",
       "      <td>28</td>\n",
       "      <td>22</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ee5b756f-84b7-4707-ab9a-a9064b65ebe2</td>\n",
       "      <td>43</td>\n",
       "      <td>25</td>\n",
       "      <td>16</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q3_x  Q1_y  Q2_y  Q3_y\n",
       "0  39268066-b798-49dd-8159-eda5299389ef    61    16    24    77    80    77\n",
       "1  cee3d697-38c2-403f-a5e8-2bf351cced67    60    11    18    87    87    83\n",
       "2  5a6f914f-609c-4043-b2ba-6ad26b7fb36c    41     4    25    87    87    83\n",
       "3  ab397bfe-7c68-4f02-baee-46407830b974    36    28    22    83    83    83\n",
       "4  ee5b756f-84b7-4707-ab9a-a9064b65ebe2    43    25    16    77    77    80"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n",
      "Algo score, swaps needed: 1073.8407\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1327, 1577, 1514, 1633, 1585, 1558, 1714, 1623, 1338, 1434]\n",
      "Pseudo-random, avergae swaps needed: 1472.20717\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n",
      "Algo score, swaps needed: 1225.50466\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1508, 1353, 1572, 1447, 1486, 1367, 1369, 1511, 1357, 1354]\n",
      "Pseudo-random, avergae swaps needed: 1450.233\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n",
      "Algo score, swaps needed: 1030.00018\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1336, 1551, 1475, 1520, 1395, 1334, 1650, 1456, 1199, 1303]\n",
      "Pseudo-random, avergae swaps needed: 1457.32272\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1f8baf28-0b57-47fe-bddb-0a27b92be278</td>\n",
       "      <td>14.5</td>\n",
       "      <td>19.0</td>\n",
       "      <td>73</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3608d390-675a-4e40-afc1-1f17fdebe8e8</td>\n",
       "      <td>23.5</td>\n",
       "      <td>22.0</td>\n",
       "      <td>83</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fcfd3df1-72e1-4e8a-b798-68fb0162b8b8</td>\n",
       "      <td>18.5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>90104063-f02a-4ab0-b693-12b727a108e5</td>\n",
       "      <td>21.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>70</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>a577fd48-7230-4784-a7eb-6e268b042b9c</td>\n",
       "      <td>13.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  1f8baf28-0b57-47fe-bddb-0a27b92be278  14.5  19.0    73    80\n",
       "1  3608d390-675a-4e40-afc1-1f17fdebe8e8  23.5  22.0    83    90\n",
       "2  fcfd3df1-72e1-4e8a-b798-68fb0162b8b8  18.5  21.0    87    87\n",
       "3  90104063-f02a-4ab0-b693-12b727a108e5  21.0  15.0    70    73\n",
       "4  a577fd48-7230-4784-a7eb-6e268b042b9c  13.0  16.0    80    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n",
      "Algo score, swaps needed: 1977.50891\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2138, 1921, 2000, 2165, 2030, 2339, 2295, 2168, 2084, 1891]\n",
      "Pseudo-random, avergae swaps needed: 2045.7049\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n",
      "Algo score, swaps needed: 1852.88044\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1987, 2261, 1935, 1799, 1967, 2180, 1988, 1884, 1907, 2168]\n",
      "Pseudo-random, avergae swaps needed: 1993.2798\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[1063.58667, 1364.57761, 1112.97992, 1395.29161, 1554.92878, 107.99537, 959.03298, 1046.08702, 1073.8407, 1225.50466, 1030.00018, 1977.50891, 1852.88044]\n",
      "N: 13 \tMean:  1212.6319115384617 \tVar:  197367.21358401564\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.33637, 1566.50795, 1559.56629, 1837.29656, 1756.82817, 182.46159, 1324.14649, 1219.73759, 1472.20717, 1450.233, 1457.32272, 2045.7049, 1993.2798]\n",
      "N: 13 \tMean:  1489.5868153846154 \tVar:  198296.9387529382 \n",
      "\n",
      "1 ) 1499.33637 > 1063.58667 => 34.00411898439214\n",
      "2 ) 1566.50795 > 1364.57761 => 13.778536031544565\n",
      "3 ) 1559.56629 > 1112.97992 => 33.4202917299604\n",
      "4 ) 1837.29656 > 1395.29161 => 27.3468147970114\n",
      "5 ) 1756.82817 > 1554.92878 => 12.19288692064193\n",
      "6 ) 182.46159 > 107.99537 => 51.2752181941173\n",
      "7 ) 1324.14649 > 959.03298 => 31.982900582055436\n",
      "8 ) 1219.73759 > 1046.08702 => 15.327803328961108\n",
      "9 ) 1472.20717 > 1073.8407 => 31.29292851826859\n",
      "10 ) 1450.233 > 1225.50466 => 16.797486790988312\n",
      "11 ) 1457.32272 > 1030.00018 => 34.360037452314685\n",
      "12 ) 2045.7049 > 1977.50891 => 3.3901250701861123\n",
      "13 ) 1993.2798 > 1852.88044 => 7.300754583225587\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "24.036146383359043 \n",
      "\n",
      "Ttest_relResult(statistic=-6.86031798230282, pvalue=1.7464596895455474e-05)\n",
      "Cohen's d:  0.5982457909985546\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp2',normv=1,score=1,goal=\"medoid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cfa09378-0303-42fb-9ff1-066c8e2f7e88</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>12</td>\n",
       "      <td>77</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b4b58e7e-2a41-4e45-aca3-7b69511f3866</td>\n",
       "      <td>12</td>\n",
       "      <td>39</td>\n",
       "      <td>24</td>\n",
       "      <td>87</td>\n",
       "      <td>67</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>45942e14-a556-414d-b4d7-9287409db104</td>\n",
       "      <td>13</td>\n",
       "      <td>43</td>\n",
       "      <td>19</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>dcef7f09-f784-48cb-b65e-59d0293f4195</td>\n",
       "      <td>10</td>\n",
       "      <td>30</td>\n",
       "      <td>32</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c74cd4a8-869a-4714-892e-be61183e7835</td>\n",
       "      <td>4</td>\n",
       "      <td>27</td>\n",
       "      <td>24</td>\n",
       "      <td>90</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x  Q1_x  Q2_x  SHORT_ANS_y  \\\n",
       "0  cfa09378-0303-42fb-9ff1-066c8e2f7e88            4    14    12           77   \n",
       "1  b4b58e7e-2a41-4e45-aca3-7b69511f3866           12    39    24           87   \n",
       "2  45942e14-a556-414d-b4d7-9287409db104           13    43    19           80   \n",
       "3  dcef7f09-f784-48cb-b65e-59d0293f4195           10    30    32           87   \n",
       "4  c74cd4a8-869a-4714-892e-be61183e7835            4    27    24           90   \n",
       "\n",
       "   Q1_y  Q2_y  \n",
       "0    87    87  \n",
       "1    67    80  \n",
       "2    83    83  \n",
       "3    83    87  \n",
       "4    87    83  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n",
      "Algo score, swaps needed: 1327.50167\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1528, 1180, 1568, 1544, 1479, 1658, 1465, 1672, 1642, 1495]\n",
      "Pseudo-random, avergae swaps needed: 1499.65638\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n",
      "Algo score, swaps needed: 1283.97738\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1542, 1534, 1714, 1654, 1706, 1472, 1491, 1555, 1663, 1650]\n",
      "Pseudo-random, avergae swaps needed: 1566.25132\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n",
      "Algo score, swaps needed: 1217.39444\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1452, 1579, 1374, 1611, 1422, 1657, 1536, 1609, 1729, 1509]\n",
      "Pseudo-random, avergae swaps needed: 1559.69437\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ecdebb90-4db9-4967-b6f9-ea3759679fcc</td>\n",
       "      <td>13.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0cfd3781-8b12-4e17-8d9d-16c15ce0f246</td>\n",
       "      <td>12.5</td>\n",
       "      <td>18.0</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6726ea7e-8d3e-46f5-bb70-5aff7c5b6402</td>\n",
       "      <td>9.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>80</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a9738fe5-2df1-4661-b147-de435b7eeeb6</td>\n",
       "      <td>17.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>77</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>79fc7532-8d8e-4995-adbc-23e3aa4c75af</td>\n",
       "      <td>11.5</td>\n",
       "      <td>11.0</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  ecdebb90-4db9-4967-b6f9-ea3759679fcc  13.0  16.0    83    80\n",
       "1  0cfd3781-8b12-4e17-8d9d-16c15ce0f246  12.5  18.0    87    87\n",
       "2  6726ea7e-8d3e-46f5-bb70-5aff7c5b6402   9.0  15.0    80    87\n",
       "3  a9738fe5-2df1-4661-b147-de435b7eeeb6  17.0  20.0    77    87\n",
       "4  79fc7532-8d8e-4995-adbc-23e3aa4c75af  11.5  11.0    83    77"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n",
      "Algo score, swaps needed: 1578.45767\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2188, 1949, 1709, 1739, 1654, 1537, 1849, 1894, 1901, 1999]\n",
      "Pseudo-random, avergae swaps needed: 1838.07482\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n",
      "Algo score, swaps needed: 1572.3984\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1861, 1911, 1747, 1763, 1721, 2084, 1601, 1552, 1730, 1722]\n",
      "Pseudo-random, avergae swaps needed: 1757.55939\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4e406595-0ea8-477c-af86-86048cfcb9b8</td>\n",
       "      <td>23.5</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>c1609ab5-b028-447c-8f26-60c711841a33</td>\n",
       "      <td>20.0</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3ad4aed5-023b-47c7-9ddf-3f2284a403b1</td>\n",
       "      <td>13.0</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>467e6ace-0fab-4f54-a773-979058184ac2</td>\n",
       "      <td>25.0</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f870a18d-a560-4a6a-90b8-e1660d8877d1</td>\n",
       "      <td>31.0</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q2_x  Q2_y\n",
       "0  4e406595-0ea8-477c-af86-86048cfcb9b8  23.5    83\n",
       "1  c1609ab5-b028-447c-8f26-60c711841a33  20.0    60\n",
       "2  3ad4aed5-023b-47c7-9ddf-3f2284a403b1  13.0    77\n",
       "3  467e6ace-0fab-4f54-a773-979058184ac2  25.0    67\n",
       "4  f870a18d-a560-4a6a-90b8-e1660d8877d1  31.0    87"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n",
      "Algo score, swaps needed: 108.49357\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[217, 217, 183, 166, 208, 177, 189, 202, 152, 188]\n",
      "Pseudo-random, avergae swaps needed: 182.49323\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bb846754-6c06-4082-a827-eee27975d766</td>\n",
       "      <td>10.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>70</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6c1573ff-01fa-4d9a-afce-2592baa321ef</td>\n",
       "      <td>12.5</td>\n",
       "      <td>6.0</td>\n",
       "      <td>87</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>098adb0c-c0ab-4a2d-a590-8d917e3604f2</td>\n",
       "      <td>12.5</td>\n",
       "      <td>4.0</td>\n",
       "      <td>87</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1c8aeeb4-a1f3-492a-ae5c-87af4b3ad8e2</td>\n",
       "      <td>12.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>83</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>25f7b700-3561-4102-ba38-4753c54f9eb0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  bb846754-6c06-4082-a827-eee27975d766  10.0   4.5    70    77\n",
       "1  6c1573ff-01fa-4d9a-afce-2592baa321ef  12.5   6.0    87    90\n",
       "2  098adb0c-c0ab-4a2d-a590-8d917e3604f2  12.5   4.0    87    80\n",
       "3  1c8aeeb4-a1f3-492a-ae5c-87af4b3ad8e2  12.0   4.5    83    73\n",
       "4  25f7b700-3561-4102-ba38-4753c54f9eb0  11.0   5.0    83    80"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n",
      "Algo score, swaps needed: 926.46827\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1257, 1476, 1332, 1302, 1213, 1242, 1292, 1587, 1375, 1339]\n",
      "Pseudo-random, avergae swaps needed: 1324.23719\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n",
      "Algo score, swaps needed: 1010.98591\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1294, 1186, 1123, 1187, 1364, 1105, 1404, 1091, 1111, 1172]\n",
      "Pseudo-random, avergae swaps needed: 1219.24378\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bb065bd5-206a-4c0f-ba0f-f193651f5ac6</td>\n",
       "      <td>62</td>\n",
       "      <td>23</td>\n",
       "      <td>40</td>\n",
       "      <td>77</td>\n",
       "      <td>73</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7fc61220-6c30-428e-9cbe-ba29c1d2e8fb</td>\n",
       "      <td>62</td>\n",
       "      <td>40</td>\n",
       "      <td>32</td>\n",
       "      <td>90</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>aa684706-1bbf-47a8-8828-62f8fde0f3c7</td>\n",
       "      <td>53</td>\n",
       "      <td>27</td>\n",
       "      <td>23</td>\n",
       "      <td>77</td>\n",
       "      <td>70</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6c715ed9-9520-4c0d-8d4f-b56ceef0320a</td>\n",
       "      <td>53</td>\n",
       "      <td>43</td>\n",
       "      <td>15</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4556f34c-b15d-4f3e-b818-09cff2ec3400</td>\n",
       "      <td>54</td>\n",
       "      <td>28</td>\n",
       "      <td>6</td>\n",
       "      <td>67</td>\n",
       "      <td>77</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q3_x  Q1_y  Q2_y  Q3_y\n",
       "0  bb065bd5-206a-4c0f-ba0f-f193651f5ac6    62    23    40    77    73    90\n",
       "1  7fc61220-6c30-428e-9cbe-ba29c1d2e8fb    62    40    32    90    83    80\n",
       "2  aa684706-1bbf-47a8-8828-62f8fde0f3c7    53    27    23    77    70    80\n",
       "3  6c715ed9-9520-4c0d-8d4f-b56ceef0320a    53    43    15    83    83    80\n",
       "4  4556f34c-b15d-4f3e-b818-09cff2ec3400    54    28     6    67    77    63"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n",
      "Algo score, swaps needed: 1355.55177\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1262, 1592, 1482, 1730, 1432, 1467, 1540, 1663, 1254, 1458]\n",
      "Pseudo-random, avergae swaps needed: 1472.71635\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n",
      "Algo score, swaps needed: 1163.31542\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1685, 1454, 1395, 1511, 1317, 1490, 1587, 1411, 1326, 1382]\n",
      "Pseudo-random, avergae swaps needed: 1449.21663\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n",
      "Algo score, swaps needed: 1059.66915\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1593, 1364, 1310, 1344, 1345, 1578, 1490, 1285, 1628, 1401]\n",
      "Pseudo-random, avergae swaps needed: 1458.5898\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>509aec91-34c1-4d28-b94d-f7845bfe5226</td>\n",
       "      <td>14.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>67</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a577fd48-7230-4784-a7eb-6e268b042b9c</td>\n",
       "      <td>13.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1f8baf28-0b57-47fe-bddb-0a27b92be278</td>\n",
       "      <td>14.5</td>\n",
       "      <td>19.0</td>\n",
       "      <td>63</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>83c20b41-645b-4931-97bf-870d498e43de</td>\n",
       "      <td>14.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>87</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ba2dd55c-5abd-4f7f-8560-18efd2679188</td>\n",
       "      <td>23.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  509aec91-34c1-4d28-b94d-f7845bfe5226  14.0  19.0    67    70\n",
       "1  a577fd48-7230-4784-a7eb-6e268b042b9c  13.0  16.0    80    80\n",
       "2  1f8baf28-0b57-47fe-bddb-0a27b92be278  14.5  19.0    63    80\n",
       "3  83c20b41-645b-4931-97bf-870d498e43de  14.0   8.0    87    80\n",
       "4  ba2dd55c-5abd-4f7f-8560-18efd2679188  23.0  27.0    80    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n",
      "Algo score, swaps needed: 1718.97797\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1936, 2171, 2226, 2158, 1791, 1995, 2136, 2242, 2061, 2156]\n",
      "Pseudo-random, avergae swaps needed: 2044.98433\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n",
      "Algo score, swaps needed: 1802.09925\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2217, 2013, 1930, 2002, 2257, 2055, 1930, 1918, 1862, 1917]\n",
      "Pseudo-random, avergae swaps needed: 1993.58644\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[1327.50167, 1283.97738, 1217.39444, 1578.45767, 1572.3984, 108.49357, 926.46827, 1010.98591, 1355.55177, 1163.31542, 1059.66915, 1718.97797, 1802.09925]\n",
      "N: 13 \tMean:  1240.40699 \tVar:  174101.9627388273\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.65638, 1566.25132, 1559.69437, 1838.07482, 1757.55939, 182.49323, 1324.23719, 1219.24378, 1472.71635, 1449.21663, 1458.5898, 2044.98433, 1993.58644]\n",
      "N: 13 \tMean:  1489.7156946153846 \tVar:  198340.275571395 \n",
      "\n",
      "1 ) 1499.65638 > 1327.50167 => 12.178640667082606\n",
      "2 ) 1566.25132 > 1283.97738 => 19.80710811030708\n",
      "3 ) 1559.69437 > 1217.39444 => 24.651709284010966\n",
      "4 ) 1838.07482 > 1578.45767 => 15.197698295560482\n",
      "5 ) 1757.55939 > 1572.3984 => 11.120921145369826\n",
      "6 ) 182.49323 > 108.49357 => 50.861179957303904\n",
      "7 ) 1324.23719 > 926.46827 => 35.3461549784484\n",
      "8 ) 1219.24378 > 1010.98591 => 18.67591225547715\n",
      "9 ) 1472.71635 > 1355.55177 => 8.285252672578999\n",
      "10 ) 1449.21663 > 1163.31542 => 21.8869054639923\n",
      "11 ) 1458.5898 > 1059.66915 => 31.682258093433962\n",
      "12 ) 2044.98433 > 1718.97797 => 17.322509314187343\n",
      "13 ) 1993.58644 > 1802.09925 => 10.089728478018422\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "21.3158445165978 \n",
      "\n",
      "Ttest_relResult(statistic=-8.815186765052696, pvalue=1.3738105274588818e-06)\n",
      "Cohen's d:  0.555062545312017\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp3',normv=1,score=1,goal=\"centroid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>afaca025-5d82-46e8-bfba-a1ce2167adca</td>\n",
       "      <td>-0.159269</td>\n",
       "      <td>0.470683</td>\n",
       "      <td>0.502278</td>\n",
       "      <td>0.144799</td>\n",
       "      <td>1.420513</td>\n",
       "      <td>-0.126103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>d2054499-4dd2-4ea1-9dae-7751c37f58e5</td>\n",
       "      <td>0.120927</td>\n",
       "      <td>-1.074762</td>\n",
       "      <td>0.205903</td>\n",
       "      <td>0.994781</td>\n",
       "      <td>-0.083617</td>\n",
       "      <td>0.661264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5b58226f-9ae5-4ad2-8261-d659f95da503</td>\n",
       "      <td>0.681318</td>\n",
       "      <td>0.161594</td>\n",
       "      <td>0.354091</td>\n",
       "      <td>-0.413266</td>\n",
       "      <td>-1.325318</td>\n",
       "      <td>-0.985834</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>e021f764-a5d2-432a-89f5-00ba2b760f0f</td>\n",
       "      <td>0.681318</td>\n",
       "      <td>0.779772</td>\n",
       "      <td>1.243217</td>\n",
       "      <td>1.063094</td>\n",
       "      <td>0.701413</td>\n",
       "      <td>0.702417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>07ac7940-e702-498a-a97c-333cb233a847</td>\n",
       "      <td>-1.280052</td>\n",
       "      <td>-0.971732</td>\n",
       "      <td>-1.572349</td>\n",
       "      <td>-2.286492</td>\n",
       "      <td>-3.295831</td>\n",
       "      <td>-1.693152</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x      Q1_x      Q2_x  \\\n",
       "0  afaca025-5d82-46e8-bfba-a1ce2167adca    -0.159269  0.470683  0.502278   \n",
       "1  d2054499-4dd2-4ea1-9dae-7751c37f58e5     0.120927 -1.074762  0.205903   \n",
       "2  5b58226f-9ae5-4ad2-8261-d659f95da503     0.681318  0.161594  0.354091   \n",
       "3  e021f764-a5d2-432a-89f5-00ba2b760f0f     0.681318  0.779772  1.243217   \n",
       "4  07ac7940-e702-498a-a97c-333cb233a847    -1.280052 -0.971732 -1.572349   \n",
       "\n",
       "   SHORT_ANS_y      Q1_y      Q2_y  \n",
       "0     0.144799  1.420513 -0.126103  \n",
       "1     0.994781 -0.083617  0.661264  \n",
       "2    -0.413266 -1.325318 -0.985834  \n",
       "3     1.063094  0.701413  0.702417  \n",
       "4    -2.286492 -3.295831 -1.693152  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>afaca025-5d82-46e8-bfba-a1ce2167adca</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.159269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>065476cf-e25b-4039-a5b0-69bfd27e61dd</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.999856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d2054499-4dd2-4ea1-9dae-7751c37f58e5</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.120927</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5b58226f-9ae5-4ad2-8261-d659f95da503</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.681318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>e021f764-a5d2-432a-89f5-00ba2b760f0f</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.681318</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  afaca025-5d82-46e8-bfba-a1ce2167adca  Human -0.159269\n",
       "1  065476cf-e25b-4039-a5b0-69bfd27e61dd  Human -0.999856\n",
       "2  d2054499-4dd2-4ea1-9dae-7751c37f58e5  Human  0.120927\n",
       "3  5b58226f-9ae5-4ad2-8261-d659f95da503  Human  0.681318\n",
       "4  e021f764-a5d2-432a-89f5-00ba2b760f0f  Human  0.681318"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.260254</td>\n",
       "      <td>1.703632</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.008882</td>\n",
       "      <td>[0.05, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.256877</td>\n",
       "      <td>1.682955</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.010504</td>\n",
       "      <td>[0.04, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.254553</td>\n",
       "      <td>1.682955</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.010504</td>\n",
       "      <td>[0.04, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.413019</td>\n",
       "      <td>1.703632</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.008882</td>\n",
       "      <td>[0.09, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.408755</td>\n",
       "      <td>1.682955</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.010504</td>\n",
       "      <td>[0.08, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.405807</td>\n",
       "      <td>1.682955</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.010504</td>\n",
       "      <td>[0.08, 0.62]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.260254  1.703632   80   81  0.008882   \n",
       "ICC2      Single random raters  0.256877  1.682955   80   80  0.010504   \n",
       "ICC3       Single fixed raters  0.254553  1.682955   80   80  0.010504   \n",
       "ICC1k  Average raters absolute  0.413019  1.703632   80   81  0.008882   \n",
       "ICC2k    Average random raters  0.408755  1.682955   80   80  0.010504   \n",
       "ICC3k     Average fixed raters  0.405807  1.682955   80   80  0.010504   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.05, 0.45]  \n",
       "ICC2   [0.04, 0.45]  \n",
       "ICC3   [0.04, 0.45]  \n",
       "ICC1k  [0.09, 0.62]  \n",
       "ICC2k  [0.08, 0.62]  \n",
       "ICC3k  [0.08, 0.62]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.2481144672391804\n",
      "Algo score, swaps needed: 1261.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1657, 1563, 1476, 1497, 1484, 1431, 1528, 1640, 1694, 1440]\n",
      "Pseudo-random, avergae swaps needed: 1499.90796\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>afaca025-5d82-46e8-bfba-a1ce2167adca</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.470683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>065476cf-e25b-4039-a5b0-69bfd27e61dd</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.250525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d2054499-4dd2-4ea1-9dae-7751c37f58e5</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.074762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5b58226f-9ae5-4ad2-8261-d659f95da503</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.161594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>e021f764-a5d2-432a-89f5-00ba2b760f0f</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.779772</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  afaca025-5d82-46e8-bfba-a1ce2167adca  Human  0.470683\n",
       "1  065476cf-e25b-4039-a5b0-69bfd27e61dd  Human -0.250525\n",
       "2  d2054499-4dd2-4ea1-9dae-7751c37f58e5  Human -1.074762\n",
       "3  5b58226f-9ae5-4ad2-8261-d659f95da503  Human  0.161594\n",
       "4  e021f764-a5d2-432a-89f5-00ba2b760f0f  Human  0.779772"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.260084</td>\n",
       "      <td>1.703010</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.008921</td>\n",
       "      <td>[0.05, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.256649</td>\n",
       "      <td>1.682002</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.010573</td>\n",
       "      <td>[0.04, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.254288</td>\n",
       "      <td>1.682002</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.010573</td>\n",
       "      <td>[0.04, 0.45]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.412804</td>\n",
       "      <td>1.703010</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.008921</td>\n",
       "      <td>[0.09, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.408466</td>\n",
       "      <td>1.682002</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.010573</td>\n",
       "      <td>[0.08, 0.62]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.405470</td>\n",
       "      <td>1.682002</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.010573</td>\n",
       "      <td>[0.08, 0.62]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.260084  1.703010   80   81  0.008921   \n",
       "ICC2      Single random raters  0.256649  1.682002   80   80  0.010573   \n",
       "ICC3       Single fixed raters  0.254288  1.682002   80   80  0.010573   \n",
       "ICC1k  Average raters absolute  0.412804  1.703010   80   81  0.008921   \n",
       "ICC2k    Average random raters  0.408466  1.682002   80   80  0.010573   \n",
       "ICC3k     Average fixed raters  0.405470  1.682002   80   80  0.010573   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.05, 0.45]  \n",
       "ICC2   [0.04, 0.45]  \n",
       "ICC3   [0.04, 0.45]  \n",
       "ICC1k  [0.09, 0.62]  \n",
       "ICC2k  [0.08, 0.62]  \n",
       "ICC3k  [0.08, 0.62]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.24637465479630627\n",
      "Algo score, swaps needed: 1261.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1386, 1516, 1642, 1454, 1477, 1422, 1454, 1516, 1556, 1599]\n",
      "Pseudo-random, avergae swaps needed: 1566.48321\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>afaca025-5d82-46e8-bfba-a1ce2167adca</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.502278</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>065476cf-e25b-4039-a5b0-69bfd27e61dd</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.386848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>d2054499-4dd2-4ea1-9dae-7751c37f58e5</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.205903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5b58226f-9ae5-4ad2-8261-d659f95da503</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.354091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>e021f764-a5d2-432a-89f5-00ba2b760f0f</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.243217</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  afaca025-5d82-46e8-bfba-a1ce2167adca  Human  0.502278\n",
       "1  065476cf-e25b-4039-a5b0-69bfd27e61dd  Human -0.386848\n",
       "2  d2054499-4dd2-4ea1-9dae-7751c37f58e5  Human  0.205903\n",
       "3  5b58226f-9ae5-4ad2-8261-d659f95da503  Human  0.354091\n",
       "4  e021f764-a5d2-432a-89f5-00ba2b760f0f  Human  1.243217"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.388763</td>\n",
       "      <td>2.272052</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000145</td>\n",
       "      <td>[0.19, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.386760</td>\n",
       "      <td>2.248032</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000183</td>\n",
       "      <td>[0.18, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.384243</td>\n",
       "      <td>2.248032</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000183</td>\n",
       "      <td>[0.18, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.559869</td>\n",
       "      <td>2.272052</td>\n",
       "      <td>80</td>\n",
       "      <td>81</td>\n",
       "      <td>0.000145</td>\n",
       "      <td>[0.32, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.557790</td>\n",
       "      <td>2.248032</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000183</td>\n",
       "      <td>[0.31, 0.72]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.555166</td>\n",
       "      <td>2.248032</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>0.000183</td>\n",
       "      <td>[0.31, 0.71]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.388763  2.272052   80   81  0.000145   \n",
       "ICC2      Single random raters  0.386760  2.248032   80   80  0.000183   \n",
       "ICC3       Single fixed raters  0.384243  2.248032   80   80  0.000183   \n",
       "ICC1k  Average raters absolute  0.559869  2.272052   80   81  0.000145   \n",
       "ICC2k    Average random raters  0.557790  2.248032   80   80  0.000183   \n",
       "ICC3k     Average fixed raters  0.555166  2.248032   80   80  0.000183   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.19, 0.56]  \n",
       "ICC2   [0.18, 0.56]  \n",
       "ICC3   [0.18, 0.56]  \n",
       "ICC1k  [0.32, 0.72]  \n",
       "ICC2k  [0.31, 0.72]  \n",
       "ICC3k  [0.31, 0.71]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.3588961567372144\n",
      "Algo score, swaps needed: 1165.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1596, 1573, 1416, 1584, 1633, 1674, 1734, 1627, 1799, 1432]\n",
      "Pseudo-random, avergae swaps needed: 1560.01438\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ef5e07b6-ab79-4558-880e-90df1e36afd2</td>\n",
       "      <td>0.787420</td>\n",
       "      <td>-0.243819</td>\n",
       "      <td>1.043109</td>\n",
       "      <td>-0.423503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8c9de0d1-eff5-4b5c-b1ff-7cecd2bb6c34</td>\n",
       "      <td>-1.102387</td>\n",
       "      <td>-2.045816</td>\n",
       "      <td>-2.175645</td>\n",
       "      <td>0.714271</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6726ea7e-8d3e-46f5-bb70-5aff7c5b6402</td>\n",
       "      <td>-1.811065</td>\n",
       "      <td>-1.144818</td>\n",
       "      <td>-0.268016</td>\n",
       "      <td>0.661540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f1a855a4-dfef-48b6-99bc-b5e2324ce2fc</td>\n",
       "      <td>1.496097</td>\n",
       "      <td>-0.243819</td>\n",
       "      <td>0.085860</td>\n",
       "      <td>-0.344353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b2a62af1-d571-48c7-bece-f369e4005ea1</td>\n",
       "      <td>1.377984</td>\n",
       "      <td>0.356847</td>\n",
       "      <td>-0.103461</td>\n",
       "      <td>-0.202596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  ef5e07b6-ab79-4558-880e-90df1e36afd2  0.787420 -0.243819  1.043109   \n",
       "1  8c9de0d1-eff5-4b5c-b1ff-7cecd2bb6c34 -1.102387 -2.045816 -2.175645   \n",
       "2  6726ea7e-8d3e-46f5-bb70-5aff7c5b6402 -1.811065 -1.144818 -0.268016   \n",
       "3  f1a855a4-dfef-48b6-99bc-b5e2324ce2fc  1.496097 -0.243819  0.085860   \n",
       "4  b2a62af1-d571-48c7-bece-f369e4005ea1  1.377984  0.356847 -0.103461   \n",
       "\n",
       "       Q2_y  \n",
       "0 -0.423503  \n",
       "1  0.714271  \n",
       "2  0.661540  \n",
       "3 -0.344353  \n",
       "4 -0.202596  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ef5e07b6-ab79-4558-880e-90df1e36afd2</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.787420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8c9de0d1-eff5-4b5c-b1ff-7cecd2bb6c34</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.102387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6726ea7e-8d3e-46f5-bb70-5aff7c5b6402</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.811065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>65949493-9050-46a2-82a6-eaf632b66ce1</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.102387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f1a855a4-dfef-48b6-99bc-b5e2324ce2fc</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.496097</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  ef5e07b6-ab79-4558-880e-90df1e36afd2  Human  0.787420\n",
       "1  8c9de0d1-eff5-4b5c-b1ff-7cecd2bb6c34  Human -1.102387\n",
       "2  6726ea7e-8d3e-46f5-bb70-5aff7c5b6402  Human -1.811065\n",
       "3  65949493-9050-46a2-82a6-eaf632b66ce1  Human -1.102387\n",
       "4  f1a855a4-dfef-48b6-99bc-b5e2324ce2fc  Human  1.496097"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.236197</td>\n",
       "      <td>1.618475</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.012669</td>\n",
       "      <td>[0.03, 0.42]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.232955</td>\n",
       "      <td>1.600762</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.014705</td>\n",
       "      <td>[0.02, 0.42]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.230995</td>\n",
       "      <td>1.600762</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.014705</td>\n",
       "      <td>[0.02, 0.42]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.382134</td>\n",
       "      <td>1.618475</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.012669</td>\n",
       "      <td>[0.06, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.377881</td>\n",
       "      <td>1.600762</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.014705</td>\n",
       "      <td>[0.05, 0.59]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.375298</td>\n",
       "      <td>1.600762</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.014705</td>\n",
       "      <td>[0.05, 0.59]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.236197  1.618475   87   88  0.012669   \n",
       "ICC2      Single random raters  0.232955  1.600762   87   87  0.014705   \n",
       "ICC3       Single fixed raters  0.230995  1.600762   87   87  0.014705   \n",
       "ICC1k  Average raters absolute  0.382134  1.618475   87   88  0.012669   \n",
       "ICC2k    Average random raters  0.377881  1.600762   87   87  0.014705   \n",
       "ICC3k     Average fixed raters  0.375298  1.600762   87   87  0.014705   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.03, 0.42]  \n",
       "ICC2   [0.02, 0.42]  \n",
       "ICC3   [0.02, 0.42]  \n",
       "ICC1k  [0.06, 0.59]  \n",
       "ICC2k  [0.05, 0.59]  \n",
       "ICC3k  [0.05, 0.59]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.23349407418070156\n",
      "Algo score, swaps needed: 1571.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1687, 2044, 1821, 1907, 1937, 1958, 1896, 1776, 1888, 1933]\n",
      "Pseudo-random, avergae swaps needed: 1837.17309\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ef5e07b6-ab79-4558-880e-90df1e36afd2</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.243819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8c9de0d1-eff5-4b5c-b1ff-7cecd2bb6c34</td>\n",
       "      <td>Human</td>\n",
       "      <td>-2.045816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6726ea7e-8d3e-46f5-bb70-5aff7c5b6402</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.144818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>65949493-9050-46a2-82a6-eaf632b66ce1</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.056514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f1a855a4-dfef-48b6-99bc-b5e2324ce2fc</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.243819</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  ef5e07b6-ab79-4558-880e-90df1e36afd2  Human -0.243819\n",
       "1  8c9de0d1-eff5-4b5c-b1ff-7cecd2bb6c34  Human -2.045816\n",
       "2  6726ea7e-8d3e-46f5-bb70-5aff7c5b6402  Human -1.144818\n",
       "3  65949493-9050-46a2-82a6-eaf632b66ce1  Human  0.056514\n",
       "4  f1a855a4-dfef-48b6-99bc-b5e2324ce2fc  Human -0.243819"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.141915</td>\n",
       "      <td>1.330772</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.091562</td>\n",
       "      <td>[-0.07, 0.34]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.138117</td>\n",
       "      <td>1.317242</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.100390</td>\n",
       "      <td>[-0.07, 0.34]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.136905</td>\n",
       "      <td>1.317242</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.100390</td>\n",
       "      <td>[-0.07, 0.34]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.248557</td>\n",
       "      <td>1.330772</td>\n",
       "      <td>87</td>\n",
       "      <td>88</td>\n",
       "      <td>0.091562</td>\n",
       "      <td>[-0.15, 0.51]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.242711</td>\n",
       "      <td>1.317242</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.100390</td>\n",
       "      <td>[-0.16, 0.51]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.240838</td>\n",
       "      <td>1.317242</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>0.100390</td>\n",
       "      <td>[-0.16, 0.5]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.141915  1.330772   87   88  0.091562   \n",
       "ICC2      Single random raters  0.138117  1.317242   87   87  0.100390   \n",
       "ICC3       Single fixed raters  0.136905  1.317242   87   87  0.100390   \n",
       "ICC1k  Average raters absolute  0.248557  1.330772   87   88  0.091562   \n",
       "ICC2k    Average random raters  0.242711  1.317242   87   87  0.100390   \n",
       "ICC3k     Average fixed raters  0.240838  1.317242   87   87  0.100390   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.07, 0.34]  \n",
       "ICC2   [-0.07, 0.34]  \n",
       "ICC3   [-0.07, 0.34]  \n",
       "ICC1k  [-0.15, 0.51]  \n",
       "ICC2k  [-0.16, 0.51]  \n",
       "ICC3k   [-0.16, 0.5]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.14707776126795447\n",
      "Algo score, swaps needed: 1500.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1797, 1585, 1739, 1934, 1827, 1921, 1911, 1744, 1878, 1636]\n",
      "Pseudo-random, avergae swaps needed: 1756.61942\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>467e6ace-0fab-4f54-a773-979058184ac2</td>\n",
       "      <td>-0.421165</td>\n",
       "      <td>-2.100272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>c1609ab5-b028-447c-8f26-60c711841a33</td>\n",
       "      <td>-1.017815</td>\n",
       "      <td>-3.333730</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>740ba105-1c40-49be-b74d-e78072fdbf34</td>\n",
       "      <td>-0.421165</td>\n",
       "      <td>-0.653895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>f870a18d-a560-4a6a-90b8-e1660d8877d1</td>\n",
       "      <td>0.294815</td>\n",
       "      <td>0.648157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8fee9d29-68ff-4b2d-b54d-58f8303249ff</td>\n",
       "      <td>0.115820</td>\n",
       "      <td>0.580038</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q2_x      Q2_y\n",
       "0  467e6ace-0fab-4f54-a773-979058184ac2 -0.421165 -2.100272\n",
       "1  c1609ab5-b028-447c-8f26-60c711841a33 -1.017815 -3.333730\n",
       "2  740ba105-1c40-49be-b74d-e78072fdbf34 -0.421165 -0.653895\n",
       "3  f870a18d-a560-4a6a-90b8-e1660d8877d1  0.294815  0.648157\n",
       "4  8fee9d29-68ff-4b2d-b54d-58f8303249ff  0.115820  0.580038"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1d16424e-886f-4420-9479-d806d0fd5262</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.182505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>467e6ace-0fab-4f54-a773-979058184ac2</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.421165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c1609ab5-b028-447c-8f26-60c711841a33</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.017815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>86ec23af-daa0-4987-b123-8b1aa763ea43</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.428451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>740ba105-1c40-49be-b74d-e78072fdbf34</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.421165</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  1d16424e-886f-4420-9479-d806d0fd5262  Human -0.182505\n",
       "1  467e6ace-0fab-4f54-a773-979058184ac2  Human -0.421165\n",
       "2  c1609ab5-b028-447c-8f26-60c711841a33  Human -1.017815\n",
       "3  86ec23af-daa0-4987-b123-8b1aa763ea43  Human  1.428451\n",
       "4  740ba105-1c40-49be-b74d-e78072fdbf34  Human -0.421165"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.549521</td>\n",
       "      <td>3.439716</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.000876</td>\n",
       "      <td>[0.23, 0.76]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.546895</td>\n",
       "      <td>3.353430</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.001233</td>\n",
       "      <td>[0.22, 0.76]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.540592</td>\n",
       "      <td>3.353430</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.001233</td>\n",
       "      <td>[0.22, 0.76]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.709278</td>\n",
       "      <td>3.439716</td>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>0.000876</td>\n",
       "      <td>[0.38, 0.86]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.707087</td>\n",
       "      <td>3.353430</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.001233</td>\n",
       "      <td>[0.36, 0.86]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.701798</td>\n",
       "      <td>3.353430</td>\n",
       "      <td>27</td>\n",
       "      <td>27</td>\n",
       "      <td>0.001233</td>\n",
       "      <td>[0.36, 0.86]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.549521  3.439716   27   28  0.000876   \n",
       "ICC2      Single random raters  0.546895  3.353430   27   27  0.001233   \n",
       "ICC3       Single fixed raters  0.540592  3.353430   27   27  0.001233   \n",
       "ICC1k  Average raters absolute  0.709278  3.439716   27   28  0.000876   \n",
       "ICC2k    Average random raters  0.707087  3.353430   27   27  0.001233   \n",
       "ICC3k     Average fixed raters  0.701798  3.353430   27   27  0.001233   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.23, 0.76]  \n",
       "ICC2   [0.22, 0.76]  \n",
       "ICC3   [0.22, 0.76]  \n",
       "ICC1k  [0.38, 0.86]  \n",
       "ICC2k  [0.36, 0.86]  \n",
       "ICC3k  [0.36, 0.86]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.6192638954277327\n",
      "Algo score, swaps needed: 101.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[192, 207, 208, 172, 239, 181, 156, 200, 232, 185]\n",
      "Pseudo-random, avergae swaps needed: 182.54523\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>25f7b700-3561-4102-ba38-4753c54f9eb0</td>\n",
       "      <td>-0.136447</td>\n",
       "      <td>0.420724</td>\n",
       "      <td>0.309788</td>\n",
       "      <td>-0.168120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7ab9f35c-8fee-43f8-9dbb-3f0a89dcd43d</td>\n",
       "      <td>1.029373</td>\n",
       "      <td>0.420724</td>\n",
       "      <td>1.065766</td>\n",
       "      <td>1.126083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7e96b523-06ee-45f4-9c2e-6247c261f646</td>\n",
       "      <td>0.030099</td>\n",
       "      <td>-0.038751</td>\n",
       "      <td>0.750147</td>\n",
       "      <td>0.107727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>eecc8b68-5ac3-4146-8346-f775be3eceef</td>\n",
       "      <td>0.862827</td>\n",
       "      <td>-0.498226</td>\n",
       "      <td>0.984727</td>\n",
       "      <td>-1.088773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>735884fe-4a63-4515-a686-0086c998fa04</td>\n",
       "      <td>0.529736</td>\n",
       "      <td>0.880199</td>\n",
       "      <td>0.556233</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  25f7b700-3561-4102-ba38-4753c54f9eb0 -0.136447  0.420724  0.309788   \n",
       "1  7ab9f35c-8fee-43f8-9dbb-3f0a89dcd43d  1.029373  0.420724  1.065766   \n",
       "2  7e96b523-06ee-45f4-9c2e-6247c261f646  0.030099 -0.038751  0.750147   \n",
       "3  eecc8b68-5ac3-4146-8346-f775be3eceef  0.862827 -0.498226  0.984727   \n",
       "4  735884fe-4a63-4515-a686-0086c998fa04  0.529736  0.880199  0.556233   \n",
       "\n",
       "       Q2_y  \n",
       "0 -0.168120  \n",
       "1  1.126083  \n",
       "2  0.107727  \n",
       "3 -1.088773  \n",
       "4       NaN  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>457c5af7-4834-422d-927e-fe3c52bcc0f6</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.196644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25f7b700-3561-4102-ba38-4753c54f9eb0</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.136447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7ab9f35c-8fee-43f8-9dbb-3f0a89dcd43d</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.029373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7e96b523-06ee-45f4-9c2e-6247c261f646</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.030099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>eecc8b68-5ac3-4146-8346-f775be3eceef</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.862827</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  457c5af7-4834-422d-927e-fe3c52bcc0f6  Human  0.196644\n",
       "1  25f7b700-3561-4102-ba38-4753c54f9eb0  Human -0.136447\n",
       "2  7ab9f35c-8fee-43f8-9dbb-3f0a89dcd43d  Human  1.029373\n",
       "3  7e96b523-06ee-45f4-9c2e-6247c261f646  Human  0.030099\n",
       "4  eecc8b68-5ac3-4146-8346-f775be3eceef  Human  0.862827"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.547111</td>\n",
       "      <td>3.416093</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>1.387694e-07</td>\n",
       "      <td>[0.37, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.545733</td>\n",
       "      <td>3.370922</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>2.080702e-07</td>\n",
       "      <td>[0.36, 0.69]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.542431</td>\n",
       "      <td>3.370922</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>2.080702e-07</td>\n",
       "      <td>[0.36, 0.68]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.707268</td>\n",
       "      <td>3.416093</td>\n",
       "      <td>74</td>\n",
       "      <td>75</td>\n",
       "      <td>1.387694e-07</td>\n",
       "      <td>[0.54, 0.81]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.706115</td>\n",
       "      <td>3.370922</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>2.080702e-07</td>\n",
       "      <td>[0.53, 0.81]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.703345</td>\n",
       "      <td>3.370922</td>\n",
       "      <td>74</td>\n",
       "      <td>74</td>\n",
       "      <td>2.080702e-07</td>\n",
       "      <td>[0.53, 0.81]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2          pval  \\\n",
       "Type                                                                         \n",
       "ICC1    Single raters absolute  0.547111  3.416093   74   75  1.387694e-07   \n",
       "ICC2      Single random raters  0.545733  3.370922   74   74  2.080702e-07   \n",
       "ICC3       Single fixed raters  0.542431  3.370922   74   74  2.080702e-07   \n",
       "ICC1k  Average raters absolute  0.707268  3.416093   74   75  1.387694e-07   \n",
       "ICC2k    Average random raters  0.706115  3.370922   74   74  2.080702e-07   \n",
       "ICC3k     Average fixed raters  0.703345  3.370922   74   74  2.080702e-07   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.37, 0.69]  \n",
       "ICC2   [0.36, 0.69]  \n",
       "ICC3   [0.36, 0.68]  \n",
       "ICC1k  [0.54, 0.81]  \n",
       "ICC2k  [0.53, 0.81]  \n",
       "ICC3k  [0.53, 0.81]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.5176223234540123\n",
      "Algo score, swaps needed: 891.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1337, 1390, 1485, 1361, 1347, 1121, 1224, 1365, 1413, 1325]\n",
      "Pseudo-random, avergae swaps needed: 1324.22571\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>457c5af7-4834-422d-927e-fe3c52bcc0f6</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.339674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25f7b700-3561-4102-ba38-4753c54f9eb0</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.420724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7ab9f35c-8fee-43f8-9dbb-3f0a89dcd43d</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.420724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7e96b523-06ee-45f4-9c2e-6247c261f646</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.038751</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>eecc8b68-5ac3-4146-8346-f775be3eceef</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.498226</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  457c5af7-4834-422d-927e-fe3c52bcc0f6  Human  1.339674\n",
       "1  25f7b700-3561-4102-ba38-4753c54f9eb0  Human  0.420724\n",
       "2  7ab9f35c-8fee-43f8-9dbb-3f0a89dcd43d  Human  0.420724\n",
       "3  7e96b523-06ee-45f4-9c2e-6247c261f646  Human -0.038751\n",
       "4  eecc8b68-5ac3-4146-8346-f775be3eceef  Human -0.498226"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.180677</td>\n",
       "      <td>1.441040</td>\n",
       "      <td>70</td>\n",
       "      <td>71</td>\n",
       "      <td>0.063576</td>\n",
       "      <td>[-0.05, 0.4]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.176309</td>\n",
       "      <td>1.422623</td>\n",
       "      <td>70</td>\n",
       "      <td>70</td>\n",
       "      <td>0.071378</td>\n",
       "      <td>[-0.06, 0.39]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.174449</td>\n",
       "      <td>1.422623</td>\n",
       "      <td>70</td>\n",
       "      <td>70</td>\n",
       "      <td>0.071378</td>\n",
       "      <td>[-0.06, 0.39]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.306057</td>\n",
       "      <td>1.441040</td>\n",
       "      <td>70</td>\n",
       "      <td>71</td>\n",
       "      <td>0.063576</td>\n",
       "      <td>[-0.11, 0.57]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.299766</td>\n",
       "      <td>1.422623</td>\n",
       "      <td>70</td>\n",
       "      <td>70</td>\n",
       "      <td>0.071378</td>\n",
       "      <td>[-0.13, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.297073</td>\n",
       "      <td>1.422623</td>\n",
       "      <td>70</td>\n",
       "      <td>70</td>\n",
       "      <td>0.071378</td>\n",
       "      <td>[-0.13, 0.56]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.180677  1.441040   70   71  0.063576   \n",
       "ICC2      Single random raters  0.176309  1.422623   70   70  0.071378   \n",
       "ICC3       Single fixed raters  0.174449  1.422623   70   70  0.071378   \n",
       "ICC1k  Average raters absolute  0.306057  1.441040   70   71  0.063576   \n",
       "ICC2k    Average random raters  0.299766  1.422623   70   70  0.071378   \n",
       "ICC3k     Average fixed raters  0.297073  1.422623   70   70  0.071378   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1    [-0.05, 0.4]  \n",
       "ICC2   [-0.06, 0.39]  \n",
       "ICC3   [-0.06, 0.39]  \n",
       "ICC1k  [-0.11, 0.57]  \n",
       "ICC2k  [-0.13, 0.56]  \n",
       "ICC3k  [-0.13, 0.56]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.19914633009511273\n",
      "Algo score, swaps needed: 927.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1061, 1163, 1159, 1110, 1160, 1128, 1084, 860, 1016, 971]\n",
      "Pseudo-random, avergae swaps needed: 1091.79753\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>f5c0bbaa-1a35-4f2f-8c89-2267b0420452</td>\n",
       "      <td>1.444223</td>\n",
       "      <td>-1.189832</td>\n",
       "      <td>2.067547</td>\n",
       "      <td>-0.242691</td>\n",
       "      <td>-0.554376</td>\n",
       "      <td>-0.746189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>fc5a292f-eeb9-413b-82b7-feae67846c21</td>\n",
       "      <td>-0.752890</td>\n",
       "      <td>-0.266504</td>\n",
       "      <td>-0.269777</td>\n",
       "      <td>-0.500509</td>\n",
       "      <td>0.385384</td>\n",
       "      <td>0.252360</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1b694dbb-dcfa-4dfe-a12a-7bbc836ca5e3</td>\n",
       "      <td>-0.409591</td>\n",
       "      <td>-0.676872</td>\n",
       "      <td>-1.146274</td>\n",
       "      <td>1.034031</td>\n",
       "      <td>0.424481</td>\n",
       "      <td>0.736469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>242b03f3-ec32-4e62-8bc1-26283ca38604</td>\n",
       "      <td>-2.538044</td>\n",
       "      <td>-1.087240</td>\n",
       "      <td>-1.438440</td>\n",
       "      <td>-2.210997</td>\n",
       "      <td>-2.543719</td>\n",
       "      <td>-2.074636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>27169089-71cf-44da-906b-6690fbcd8a43</td>\n",
       "      <td>1.238243</td>\n",
       "      <td>1.272377</td>\n",
       "      <td>-0.075000</td>\n",
       "      <td>-0.907106</td>\n",
       "      <td>-1.194529</td>\n",
       "      <td>-0.328051</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q3_x  \\\n",
       "0  f5c0bbaa-1a35-4f2f-8c89-2267b0420452  1.444223 -1.189832  2.067547   \n",
       "1  fc5a292f-eeb9-413b-82b7-feae67846c21 -0.752890 -0.266504 -0.269777   \n",
       "2  1b694dbb-dcfa-4dfe-a12a-7bbc836ca5e3 -0.409591 -0.676872 -1.146274   \n",
       "3  242b03f3-ec32-4e62-8bc1-26283ca38604 -2.538044 -1.087240 -1.438440   \n",
       "4  27169089-71cf-44da-906b-6690fbcd8a43  1.238243  1.272377 -0.075000   \n",
       "\n",
       "       Q1_y      Q2_y      Q3_y  \n",
       "0 -0.242691 -0.554376 -0.746189  \n",
       "1 -0.500509  0.385384  0.252360  \n",
       "2  1.034031  0.424481  0.736469  \n",
       "3 -2.210997 -2.543719 -2.074636  \n",
       "4 -0.907106 -1.194529 -0.328051  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1016b5f2-7f8a-496a-bb0b-42be41dcdfdc</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.375563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f5c0bbaa-1a35-4f2f-8c89-2267b0420452</td>\n",
       "      <td>Human</td>\n",
       "      <td>1.444223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fc5a292f-eeb9-413b-82b7-feae67846c21</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.752890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1b694dbb-dcfa-4dfe-a12a-7bbc836ca5e3</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.409591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>242b03f3-ec32-4e62-8bc1-26283ca38604</td>\n",
       "      <td>Human</td>\n",
       "      <td>-2.538044</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  1016b5f2-7f8a-496a-bb0b-42be41dcdfdc  Human  1.375563\n",
       "1  f5c0bbaa-1a35-4f2f-8c89-2267b0420452  Human  1.444223\n",
       "2  fc5a292f-eeb9-413b-82b7-feae67846c21  Human -0.752890\n",
       "3  1b694dbb-dcfa-4dfe-a12a-7bbc836ca5e3  Human -0.409591\n",
       "4  242b03f3-ec32-4e62-8bc1-26283ca38604  Human -2.538044"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.181365</td>\n",
       "      <td>1.443091</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.054147</td>\n",
       "      <td>[-0.04, 0.39]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.177131</td>\n",
       "      <td>1.425175</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.061123</td>\n",
       "      <td>[-0.05, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.175317</td>\n",
       "      <td>1.425175</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.061123</td>\n",
       "      <td>[-0.05, 0.38]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.307043</td>\n",
       "      <td>1.443091</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.054147</td>\n",
       "      <td>[-0.08, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.300954</td>\n",
       "      <td>1.425175</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.061123</td>\n",
       "      <td>[-0.1, 0.56]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.298332</td>\n",
       "      <td>1.425175</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.061123</td>\n",
       "      <td>[-0.1, 0.55]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.181365  1.443091   77   78  0.054147   \n",
       "ICC2      Single random raters  0.177131  1.425175   77   77  0.061123   \n",
       "ICC3       Single fixed raters  0.175317  1.425175   77   77  0.061123   \n",
       "ICC1k  Average raters absolute  0.307043  1.443091   77   78  0.054147   \n",
       "ICC2k    Average random raters  0.300954  1.425175   77   77  0.061123   \n",
       "ICC3k     Average fixed raters  0.298332  1.425175   77   77  0.061123   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.04, 0.39]  \n",
       "ICC2   [-0.05, 0.38]  \n",
       "ICC3   [-0.05, 0.38]  \n",
       "ICC1k  [-0.08, 0.56]  \n",
       "ICC2k   [-0.1, 0.56]  \n",
       "ICC3k   [-0.1, 0.55]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.15788305287927895\n",
      "Algo score, swaps needed: 1337.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1448, 1495, 1623, 1408, 1477, 1524, 1428, 1524, 1650, 1322]\n",
      "Pseudo-random, avergae swaps needed: 1472.17265\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1016b5f2-7f8a-496a-bb0b-42be41dcdfdc</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.779464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f5c0bbaa-1a35-4f2f-8c89-2267b0420452</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.189832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fc5a292f-eeb9-413b-82b7-feae67846c21</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.266504</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1b694dbb-dcfa-4dfe-a12a-7bbc836ca5e3</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.676872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>242b03f3-ec32-4e62-8bc1-26283ca38604</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.087240</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  1016b5f2-7f8a-496a-bb0b-42be41dcdfdc  Human -0.779464\n",
       "1  f5c0bbaa-1a35-4f2f-8c89-2267b0420452  Human -1.189832\n",
       "2  fc5a292f-eeb9-413b-82b7-feae67846c21  Human -0.266504\n",
       "3  1b694dbb-dcfa-4dfe-a12a-7bbc836ca5e3  Human -0.676872\n",
       "4  242b03f3-ec32-4e62-8bc1-26283ca38604  Human -1.087240"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.289093</td>\n",
       "      <td>1.813309</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.004737</td>\n",
       "      <td>[0.07, 0.48]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.286136</td>\n",
       "      <td>1.792416</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.005632</td>\n",
       "      <td>[0.07, 0.48]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.283774</td>\n",
       "      <td>1.792416</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.005632</td>\n",
       "      <td>[0.07, 0.48]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.448522</td>\n",
       "      <td>1.813309</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.004737</td>\n",
       "      <td>[0.14, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.444954</td>\n",
       "      <td>1.792416</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.005632</td>\n",
       "      <td>[0.13, 0.65]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.442094</td>\n",
       "      <td>1.792416</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.005632</td>\n",
       "      <td>[0.12, 0.64]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.289093  1.813309   77   78  0.004737   \n",
       "ICC2      Single random raters  0.286136  1.792416   77   77  0.005632   \n",
       "ICC3       Single fixed raters  0.283774  1.792416   77   77  0.005632   \n",
       "ICC1k  Average raters absolute  0.448522  1.813309   77   78  0.004737   \n",
       "ICC2k    Average random raters  0.444954  1.792416   77   77  0.005632   \n",
       "ICC3k     Average fixed raters  0.442094  1.792416   77   77  0.005632   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1   [0.07, 0.48]  \n",
       "ICC2   [0.07, 0.48]  \n",
       "ICC3   [0.07, 0.48]  \n",
       "ICC1k  [0.14, 0.65]  \n",
       "ICC2k  [0.13, 0.65]  \n",
       "ICC3k  [0.12, 0.64]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.30327190428767214\n",
      "Algo score, swaps needed: 1112.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1382, 1456, 1489, 1456, 1502, 1497, 1362, 1474, 1492, 1417]\n",
      "Pseudo-random, avergae swaps needed: 1449.68919\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1016b5f2-7f8a-496a-bb0b-42be41dcdfdc</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.217165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>f5c0bbaa-1a35-4f2f-8c89-2267b0420452</td>\n",
       "      <td>Human</td>\n",
       "      <td>2.067547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fc5a292f-eeb9-413b-82b7-feae67846c21</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.269777</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1b694dbb-dcfa-4dfe-a12a-7bbc836ca5e3</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.146274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>242b03f3-ec32-4e62-8bc1-26283ca38604</td>\n",
       "      <td>Human</td>\n",
       "      <td>-1.438440</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  1016b5f2-7f8a-496a-bb0b-42be41dcdfdc  Human  0.217165\n",
       "1  f5c0bbaa-1a35-4f2f-8c89-2267b0420452  Human  2.067547\n",
       "2  fc5a292f-eeb9-413b-82b7-feae67846c21  Human -0.269777\n",
       "3  1b694dbb-dcfa-4dfe-a12a-7bbc836ca5e3  Human -1.146274\n",
       "4  242b03f3-ec32-4e62-8bc1-26283ca38604  Human -1.438440"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.442772</td>\n",
       "      <td>2.589198</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>[0.25, 0.6]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.440751</td>\n",
       "      <td>2.556033</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000028</td>\n",
       "      <td>[0.24, 0.6]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.437576</td>\n",
       "      <td>2.556033</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000028</td>\n",
       "      <td>[0.24, 0.6]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.613780</td>\n",
       "      <td>2.589198</td>\n",
       "      <td>77</td>\n",
       "      <td>78</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>[0.4, 0.75]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.611835</td>\n",
       "      <td>2.556033</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000028</td>\n",
       "      <td>[0.39, 0.75]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.608769</td>\n",
       "      <td>2.556033</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>0.000028</td>\n",
       "      <td>[0.39, 0.75]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.442772  2.589198   77   78  0.000020   \n",
       "ICC2      Single random raters  0.440751  2.556033   77   77  0.000028   \n",
       "ICC3       Single fixed raters  0.437576  2.556033   77   77  0.000028   \n",
       "ICC1k  Average raters absolute  0.613780  2.589198   77   78  0.000020   \n",
       "ICC2k    Average random raters  0.611835  2.556033   77   77  0.000028   \n",
       "ICC3k     Average fixed raters  0.608769  2.556033   77   77  0.000028   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1    [0.25, 0.6]  \n",
       "ICC2    [0.24, 0.6]  \n",
       "ICC3    [0.24, 0.6]  \n",
       "ICC1k   [0.4, 0.75]  \n",
       "ICC2k  [0.39, 0.75]  \n",
       "ICC3k  [0.39, 0.75]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.4348797198997916\n",
      "Algo score, swaps needed: 1040.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1274, 1442, 1494, 1658, 1769, 1608, 1452, 1387, 1498, 1384]\n",
      "Pseudo-random, avergae swaps needed: 1457.75468\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n",
      "Leaving z scores in place\n",
      "Leaving z scores in place\n",
      "Computing z scores for human scoring\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>e1308431-d29b-47a7-8201-12fd04f5cac9</td>\n",
       "      <td>0.849505</td>\n",
       "      <td>0.258094</td>\n",
       "      <td>0.246790</td>\n",
       "      <td>1.207579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7b74720b-c2e8-4eb3-9274-ea3fb17441e2</td>\n",
       "      <td>0.400798</td>\n",
       "      <td>-0.127540</td>\n",
       "      <td>-0.626280</td>\n",
       "      <td>0.388980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>348bfa7e-1c58-4b5b-9b11-af9b507ee3cb</td>\n",
       "      <td>-0.272262</td>\n",
       "      <td>-0.127540</td>\n",
       "      <td>0.278983</td>\n",
       "      <td>0.527461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>e63e6f2b-24aa-45e2-8cf4-ceba745a1010</td>\n",
       "      <td>-0.889235</td>\n",
       "      <td>-0.898808</td>\n",
       "      <td>-0.844614</td>\n",
       "      <td>-0.351062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f0166970-fdaf-4f6b-9ae6-e8e25ef8b198</td>\n",
       "      <td>0.008180</td>\n",
       "      <td>-0.513174</td>\n",
       "      <td>-2.355852</td>\n",
       "      <td>-0.574387</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID      Q1_x      Q2_x      Q1_y  \\\n",
       "0  e1308431-d29b-47a7-8201-12fd04f5cac9  0.849505  0.258094  0.246790   \n",
       "1  7b74720b-c2e8-4eb3-9274-ea3fb17441e2  0.400798 -0.127540 -0.626280   \n",
       "2  348bfa7e-1c58-4b5b-9b11-af9b507ee3cb -0.272262 -0.127540  0.278983   \n",
       "3  e63e6f2b-24aa-45e2-8cf4-ceba745a1010 -0.889235 -0.898808 -0.844614   \n",
       "4  f0166970-fdaf-4f6b-9ae6-e8e25ef8b198  0.008180 -0.513174 -2.355852   \n",
       "\n",
       "       Q2_y  \n",
       "0  1.207579  \n",
       "1  0.388980  \n",
       "2  0.527461  \n",
       "3 -0.351062  \n",
       "4 -0.574387  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>e1308431-d29b-47a7-8201-12fd04f5cac9</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.849505</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7b74720b-c2e8-4eb3-9274-ea3fb17441e2</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.400798</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>348bfa7e-1c58-4b5b-9b11-af9b507ee3cb</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.272262</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>e63e6f2b-24aa-45e2-8cf4-ceba745a1010</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.889235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f0166970-fdaf-4f6b-9ae6-e8e25ef8b198</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.008180</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  e1308431-d29b-47a7-8201-12fd04f5cac9  Human  0.849505\n",
       "1  7b74720b-c2e8-4eb3-9274-ea3fb17441e2  Human  0.400798\n",
       "2  348bfa7e-1c58-4b5b-9b11-af9b507ee3cb  Human -0.272262\n",
       "3  e63e6f2b-24aa-45e2-8cf4-ceba745a1010  Human -0.889235\n",
       "4  f0166970-fdaf-4f6b-9ae6-e8e25ef8b198  Human  0.008180"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.182515</td>\n",
       "      <td>1.446528</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.039475</td>\n",
       "      <td>[-0.02, 0.37]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.179031</td>\n",
       "      <td>1.431663</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.044348</td>\n",
       "      <td>[-0.03, 0.37]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.177518</td>\n",
       "      <td>1.431663</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.044348</td>\n",
       "      <td>[-0.03, 0.37]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.308689</td>\n",
       "      <td>1.446528</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.039475</td>\n",
       "      <td>[-0.04, 0.54]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.303692</td>\n",
       "      <td>1.431663</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.044348</td>\n",
       "      <td>[-0.06, 0.54]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.301512</td>\n",
       "      <td>1.431663</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.044348</td>\n",
       "      <td>[-0.06, 0.54]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2      pval  \\\n",
       "Type                                                                     \n",
       "ICC1    Single raters absolute  0.182515  1.446528   91   92  0.039475   \n",
       "ICC2      Single random raters  0.179031  1.431663   91   91  0.044348   \n",
       "ICC3       Single fixed raters  0.177518  1.431663   91   91  0.044348   \n",
       "ICC1k  Average raters absolute  0.308689  1.446528   91   92  0.039475   \n",
       "ICC2k    Average random raters  0.303692  1.431663   91   91  0.044348   \n",
       "ICC3k     Average fixed raters  0.301512  1.431663   91   91  0.044348   \n",
       "\n",
       "               CI95%  \n",
       "Type                  \n",
       "ICC1   [-0.02, 0.37]  \n",
       "ICC2   [-0.03, 0.37]  \n",
       "ICC3   [-0.03, 0.37]  \n",
       "ICC1k  [-0.04, 0.54]  \n",
       "ICC2k  [-0.06, 0.54]  \n",
       "ICC3k  [-0.06, 0.54]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.17165245801214968\n",
      "Algo score, swaps needed: 1755.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2298, 2055, 2016, 2089, 2077, 2159, 1799, 1980, 1914, 1988]\n",
      "Pseudo-random, avergae swaps needed: 2045.44676\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>judge</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>e1308431-d29b-47a7-8201-12fd04f5cac9</td>\n",
       "      <td>Human</td>\n",
       "      <td>0.258094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7b74720b-c2e8-4eb3-9274-ea3fb17441e2</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.127540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>348bfa7e-1c58-4b5b-9b11-af9b507ee3cb</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.127540</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>e63e6f2b-24aa-45e2-8cf4-ceba745a1010</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.898808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>f0166970-fdaf-4f6b-9ae6-e8e25ef8b198</td>\n",
       "      <td>Human</td>\n",
       "      <td>-0.513174</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  judge     score\n",
       "0  e1308431-d29b-47a7-8201-12fd04f5cac9  Human  0.258094\n",
       "1  7b74720b-c2e8-4eb3-9274-ea3fb17441e2  Human -0.127540\n",
       "2  348bfa7e-1c58-4b5b-9b11-af9b507ee3cb  Human -0.127540\n",
       "3  e63e6f2b-24aa-45e2-8cf4-ceba745a1010  Human -0.898808\n",
       "4  f0166970-fdaf-4f6b-9ae6-e8e25ef8b198  Human -0.513174"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Description</th>\n",
       "      <th>ICC</th>\n",
       "      <th>F</th>\n",
       "      <th>df1</th>\n",
       "      <th>df2</th>\n",
       "      <th>pval</th>\n",
       "      <th>CI95%</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ICC1</th>\n",
       "      <td>Single raters absolute</td>\n",
       "      <td>0.217202</td>\n",
       "      <td>1.554939</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.01794</td>\n",
       "      <td>[0.01, 0.4]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2</th>\n",
       "      <td>Single random raters</td>\n",
       "      <td>0.214463</td>\n",
       "      <td>1.541209</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.02020</td>\n",
       "      <td>[0.01, 0.4]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3</th>\n",
       "      <td>Single fixed raters</td>\n",
       "      <td>0.212973</td>\n",
       "      <td>1.541209</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.02020</td>\n",
       "      <td>[0.01, 0.4]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC1k</th>\n",
       "      <td>Average raters absolute</td>\n",
       "      <td>0.356888</td>\n",
       "      <td>1.554939</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>0.01794</td>\n",
       "      <td>[0.03, 0.57]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC2k</th>\n",
       "      <td>Average random raters</td>\n",
       "      <td>0.353182</td>\n",
       "      <td>1.541209</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.02020</td>\n",
       "      <td>[0.02, 0.57]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ICC3k</th>\n",
       "      <td>Average fixed raters</td>\n",
       "      <td>0.351159</td>\n",
       "      <td>1.541209</td>\n",
       "      <td>91</td>\n",
       "      <td>91</td>\n",
       "      <td>0.02020</td>\n",
       "      <td>[0.02, 0.57]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Description       ICC         F  df1  df2     pval  \\\n",
       "Type                                                                    \n",
       "ICC1    Single raters absolute  0.217202  1.554939   91   92  0.01794   \n",
       "ICC2      Single random raters  0.214463  1.541209   91   91  0.02020   \n",
       "ICC3       Single fixed raters  0.212973  1.541209   91   91  0.02020   \n",
       "ICC1k  Average raters absolute  0.356888  1.554939   91   92  0.01794   \n",
       "ICC2k    Average random raters  0.353182  1.541209   91   91  0.02020   \n",
       "ICC3k     Average fixed raters  0.351159  1.541209   91   91  0.02020   \n",
       "\n",
       "              CI95%  \n",
       "Type                 \n",
       "ICC1    [0.01, 0.4]  \n",
       "ICC2    [0.01, 0.4]  \n",
       "ICC3    [0.01, 0.4]  \n",
       "ICC1k  [0.03, 0.57]  \n",
       "ICC2k  [0.02, 0.57]  \n",
       "ICC3k  [0.02, 0.57]  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohen's kappa for rounded(z scores x 10): 0.16671429748632016\n",
      "Algo score, swaps needed: 1796.0\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1891, 1725, 2297, 1936, 1873, 2096, 2122, 2058, 2143, 1717]\n",
      "Pseudo-random, avergae swaps needed: 1994.1412\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[1261.0, 1261.0, 1165.0, 1571.0, 1500.0, 101.0, 891.0, 927.0, 1337.0, 1112.0, 1040.0, 1755.0, 1796.0]\n",
      "N: 13 \tMean:  1209.0 \tVar:  179109.23076923078\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.90796, 1566.48321, 1560.01438, 1837.17309, 1756.61942, 182.54523, 1324.22571, 1091.79753, 1472.17265, 1449.68919, 1457.75468, 2045.44676, 1994.1412]\n",
      "N: 13 \tMean:  1479.843923846154 \tVar:  204790.024822612 \n",
      "\n",
      "1 ) 1499.90796 > 1261.0 => 17.306477684971433\n",
      "2 ) 1566.48321 > 1261.0 => 21.608136092167992\n",
      "3 ) 1560.01438 > 1165.0 => 28.991728109706347\n",
      "4 ) 1837.17309 > 1571.0 => 15.619693188763485\n",
      "5 ) 1756.61942 > 1500.0 => 15.759865486523445\n",
      "6 ) 182.54523 > 101.0 => 57.51832256180081\n",
      "7 ) 1324.22571 > 891.0 => 39.11345991014162\n",
      "8 ) 1091.79753 > 927.0 => 16.326305887643922\n",
      "9 ) 1472.17265 > 1337.0 => 9.62366268232036\n",
      "10 ) 1449.68919 > 1112.0 => 26.364571573962106\n",
      "11 ) 1457.75468 > 1040.0 => 33.45041715625972\n",
      "12 ) 2045.44676 > 1755.0 => 15.28487456037932\n",
      "13 ) 1994.1412 > 1796.0 => 10.45561046643856\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "23.647932720083006 \n",
      "\n",
      "Ttest_relResult(statistic=-8.994953240253562, pvalue=1.1106490832088464e-06)\n",
      "Cohen's d:  0.5939425055466511\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp3',normv=1,score=2,goal=\"centroid\",runs=100000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=========================================================\n",
      "../data/property_instructor_A\n",
      "=========================================================\n",
      "\n",
      "['ID', 'SHORT_ANS', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>SHORT_ANS_x</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>SHORT_ANS_y</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>c74cd4a8-869a-4714-892e-be61183e7835</td>\n",
       "      <td>4</td>\n",
       "      <td>27</td>\n",
       "      <td>24</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>88b8c7fa-b807-4ab3-b690-602e4b0787aa</td>\n",
       "      <td>6</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>afaca025-5d82-46e8-bfba-a1ce2167adca</td>\n",
       "      <td>6</td>\n",
       "      <td>28</td>\n",
       "      <td>18</td>\n",
       "      <td>90</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>de219fed-6a0e-4796-875c-300be31e1347</td>\n",
       "      <td>11</td>\n",
       "      <td>23</td>\n",
       "      <td>8</td>\n",
       "      <td>73</td>\n",
       "      <td>80</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>01759358-64dd-4801-b923-5a9914b62f19</td>\n",
       "      <td>6</td>\n",
       "      <td>33</td>\n",
       "      <td>19</td>\n",
       "      <td>87</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  SHORT_ANS_x  Q1_x  Q2_x  SHORT_ANS_y  \\\n",
       "0  c74cd4a8-869a-4714-892e-be61183e7835            4    27    24           87   \n",
       "1  88b8c7fa-b807-4ab3-b690-602e4b0787aa            6    18    20           83   \n",
       "2  afaca025-5d82-46e8-bfba-a1ce2167adca            6    28    18           90   \n",
       "3  de219fed-6a0e-4796-875c-300be31e1347           11    23     8           73   \n",
       "4  01759358-64dd-4801-b923-5a9914b62f19            6    33    19           87   \n",
       "\n",
       "   Q1_y  Q2_y  \n",
       "0    87    80  \n",
       "1    87    83  \n",
       "2    87    83  \n",
       "3    80    77  \n",
       "4    83    83  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 81\n",
      "\n",
      "SHORT_ANS\n",
      "Mean Words: 436\n",
      "Algo score, swaps needed: 1362.98474\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1516, 1345, 1513, 1286, 1490, 1488, 1463, 1562, 1705, 1500]\n",
      "Pseudo-random, avergae swaps needed: 1499.459\n",
      "\n",
      "Q1\n",
      "Mean Words: 2048\n",
      "Algo score, swaps needed: 1259.61929\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1469, 1559, 1586, 1635, 1725, 1580, 1492, 1626, 1477, 1518]\n",
      "Pseudo-random, avergae swaps needed: 1566.4181\n",
      "\n",
      "Q2\n",
      "Mean Words: 947\n",
      "Algo score, swaps needed: 1312.33292\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1542, 1433, 1483, 1537, 1357, 1666, 1590, 1522, 1510, 1596]\n",
      "Pseudo-random, avergae swaps needed: 1560.06736\n",
      "\n",
      "=========================================================\n",
      "../data/property_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4ef2af3a-05d9-4992-9746-a24945431d7b</td>\n",
       "      <td>16.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>77</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0b4a8297-98e1-4153-bddb-d6562cd5041e</td>\n",
       "      <td>13.5</td>\n",
       "      <td>22.0</td>\n",
       "      <td>80</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5b2a6aa4-fdf6-40fc-880c-09a15c2fe77b</td>\n",
       "      <td>12.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>732a8471-dca7-4b81-be0c-606cc6e86d0a</td>\n",
       "      <td>22.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6dc8ed5f-bc91-4ed5-bc1e-2de723a3aae9</td>\n",
       "      <td>11.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  4ef2af3a-05d9-4992-9746-a24945431d7b  16.0  23.0    77    60\n",
       "1  0b4a8297-98e1-4153-bddb-d6562cd5041e  13.5  22.0    80    77\n",
       "2  5b2a6aa4-fdf6-40fc-880c-09a15c2fe77b  12.0  21.0    83    80\n",
       "3  732a8471-dca7-4b81-be0c-606cc6e86d0a  22.0  16.0    77    83\n",
       "4  6dc8ed5f-bc91-4ed5-bc1e-2de723a3aae9  11.0  13.0    83    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 88\n",
      "\n",
      "Q1\n",
      "Mean Words: 1431\n",
      "Algo score, swaps needed: 1426.94488\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1883, 1719, 1765, 1931, 2222, 1610, 2100, 1847, 2133, 1695]\n",
      "Pseudo-random, avergae swaps needed: 1837.57631\n",
      "\n",
      "Q2\n",
      "Mean Words: 1546\n",
      "Algo score, swaps needed: 1692.05737\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1819, 1696, 2002, 2010, 1823, 1655, 1400, 1700, 1796, 1394]\n",
      "Pseudo-random, avergae swaps needed: 1758.10921\n",
      "\n",
      "=========================================================\n",
      "../data/environ_instructor_B\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>47e96791-9e80-4638-b63a-a00903ce8508</td>\n",
       "      <td>30.0</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>a6db6331-93ae-48a4-ab5d-04cd34cf3989</td>\n",
       "      <td>25.0</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8fee9d29-68ff-4b2d-b54d-58f8303249ff</td>\n",
       "      <td>29.5</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>467e6ace-0fab-4f54-a773-979058184ac2</td>\n",
       "      <td>25.0</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>dda68a48-2475-4338-9956-67fc7efbf74b</td>\n",
       "      <td>37.0</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q2_x  Q2_y\n",
       "0  47e96791-9e80-4638-b63a-a00903ce8508  30.0    87\n",
       "1  a6db6331-93ae-48a4-ab5d-04cd34cf3989  25.0    80\n",
       "2  8fee9d29-68ff-4b2d-b54d-58f8303249ff  29.5    80\n",
       "3  467e6ace-0fab-4f54-a773-979058184ac2  25.0    77\n",
       "4  dda68a48-2475-4338-9956-67fc7efbf74b  37.0    87"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 28\n",
      "\n",
      "Q2\n",
      "Mean Words: 2094\n",
      "Algo score, swaps needed: 134.47636\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[192, 172, 172, 162, 202, 198, 219, 165, 156, 158]\n",
      "Pseudo-random, avergae swaps needed: 182.52418\n",
      "\n",
      "=========================================================\n",
      "../data/PR_instructor_C\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6c261e5d-2212-43a3-ab4f-28bd7dd90612</td>\n",
       "      <td>9.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3d29b5ad-e82f-4972-89a4-adb948591eb1</td>\n",
       "      <td>15.0</td>\n",
       "      <td>5.5</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>12190cda-1a67-461b-8ffd-f080188457c5</td>\n",
       "      <td>15.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>47285a02-f301-408b-b50a-6a25530cadae</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7525acd7-87ff-42d3-b5f0-700d3234cb05</td>\n",
       "      <td>7.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>59</td>\n",
       "      <td>59</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  6c261e5d-2212-43a3-ab4f-28bd7dd90612   9.0   3.0    77    83\n",
       "1  3d29b5ad-e82f-4972-89a4-adb948591eb1  15.0   5.5    83    77\n",
       "2  12190cda-1a67-461b-8ffd-f080188457c5  15.5   5.0    80    83\n",
       "3  47285a02-f301-408b-b50a-6a25530cadae   7.0   3.5    83    83\n",
       "4  7525acd7-87ff-42d3-b5f0-700d3234cb05   7.0   4.0    59    59"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 75\n",
      "\n",
      "Q1\n",
      "Mean Words: 936\n",
      "Algo score, swaps needed: 1075.06208\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1296, 1434, 1373, 1328, 1232, 1426, 1332, 1390, 1110, 1508]\n",
      "Pseudo-random, avergae swaps needed: 1324.45835\n",
      "\n",
      "Q2\n",
      "Mean Words: 570\n",
      "Algo score, swaps needed: 1107.89282\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1227, 1052, 1203, 1293, 1183, 1170, 1071, 1094, 1185, 1243]\n",
      "Pseudo-random, avergae swaps needed: 1219.53493\n",
      "\n",
      "=========================================================\n",
      "../data/contracts_instructor_D\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2', 'Q3']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q3_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "      <th>Q3_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1b273341-a255-40ef-9a90-17fbdf05af4b</td>\n",
       "      <td>48</td>\n",
       "      <td>39</td>\n",
       "      <td>31</td>\n",
       "      <td>83</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ed6cf722-94b9-4474-8bf7-318cb15713b8</td>\n",
       "      <td>67</td>\n",
       "      <td>25</td>\n",
       "      <td>28</td>\n",
       "      <td>70</td>\n",
       "      <td>83</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4c68352f-27a5-49c2-aecd-887e788ab441</td>\n",
       "      <td>55</td>\n",
       "      <td>15</td>\n",
       "      <td>23</td>\n",
       "      <td>77</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6266e45d-2172-43e6-98f5-f9ed6d82b414</td>\n",
       "      <td>62</td>\n",
       "      <td>31</td>\n",
       "      <td>29</td>\n",
       "      <td>87</td>\n",
       "      <td>90</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>c04e80c3-8a55-408c-afd7-c6d5197b5fd2</td>\n",
       "      <td>50</td>\n",
       "      <td>40</td>\n",
       "      <td>18</td>\n",
       "      <td>83</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q3_x  Q1_y  Q2_y  Q3_y\n",
       "0  1b273341-a255-40ef-9a90-17fbdf05af4b    48    39    31    83    83    87\n",
       "1  ed6cf722-94b9-4474-8bf7-318cb15713b8    67    25    28    70    83    87\n",
       "2  4c68352f-27a5-49c2-aecd-887e788ab441    55    15    23    77    77    83\n",
       "3  6266e45d-2172-43e6-98f5-f9ed6d82b414    62    31    29    87    90    87\n",
       "4  c04e80c3-8a55-408c-afd7-c6d5197b5fd2    50    40    18    83    77    83"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 78\n",
      "\n",
      "Q1\n",
      "Mean Words: 1823\n",
      "Algo score, swaps needed: 1334.62252\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1595, 1470, 1469, 1542, 1520, 1501, 1542, 1452, 1287, 1395]\n",
      "Pseudo-random, avergae swaps needed: 1471.70444\n",
      "\n",
      "Q2\n",
      "Mean Words: 1050\n",
      "Algo score, swaps needed: 1083.63437\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1337, 1567, 1517, 1591, 1493, 1389, 1308, 1467, 1237, 1405]\n",
      "Pseudo-random, avergae swaps needed: 1449.61455\n",
      "\n",
      "Q3\n",
      "Mean Words: 839\n",
      "Algo score, swaps needed: 1002.36012\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1443, 1478, 1428, 1579, 1330, 1318, 1407, 1477, 1443, 1369]\n",
      "Pseudo-random, avergae swaps needed: 1458.0963\n",
      "\n",
      "=========================================================\n",
      "../data/crim_instructor_E\n",
      "=========================================================\n",
      "\n",
      "['ID', 'Q1', 'Q2']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>Q1_x</th>\n",
       "      <th>Q2_x</th>\n",
       "      <th>Q1_y</th>\n",
       "      <th>Q2_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19504cf2-0e8e-47e7-8cd8-37ccc42cc860</td>\n",
       "      <td>20.5</td>\n",
       "      <td>15.0</td>\n",
       "      <td>73</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e1308431-d29b-47a7-8201-12fd04f5cac9</td>\n",
       "      <td>22.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>77</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9db89209-0865-45ae-a102-45967994336f</td>\n",
       "      <td>21.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>90</td>\n",
       "      <td>83</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>85ca12a0-f4e5-4045-8350-676e0e159221</td>\n",
       "      <td>22.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>97</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0a1cbcb8-fb01-45be-bb70-9c97770f647c</td>\n",
       "      <td>18.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>87</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     ID  Q1_x  Q2_x  Q1_y  Q2_y\n",
       "0  19504cf2-0e8e-47e7-8cd8-37ccc42cc860  20.5  15.0    73    80\n",
       "1  e1308431-d29b-47a7-8201-12fd04f5cac9  22.0  17.0    77    83\n",
       "2  9db89209-0865-45ae-a102-45967994336f  21.0  14.0    90    83\n",
       "3  85ca12a0-f4e5-4045-8350-676e0e159221  22.0  17.0    97    73\n",
       "4  0a1cbcb8-fb01-45be-bb70-9c97770f647c  18.0  11.0    87    87"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 92\n",
      "\n",
      "Q1\n",
      "Mean Words: 3353\n",
      "Algo score, swaps needed: 1803.05968\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[1878, 2178, 1916, 2063, 1978, 2051, 2153, 2108, 2209, 2108]\n",
      "Pseudo-random, avergae swaps needed: 2045.09657\n",
      "\n",
      "Q2\n",
      "Mean Words: 2137\n",
      "Algo score, swaps needed: 1807.38171\n",
      "Swaps needed for first 10 out of 100000 pseudo-random runs:\n",
      "[2117, 2052, 1817, 2172, 2064, 2062, 2176, 1942, 2130, 2155]\n",
      "Pseudo-random, avergae swaps needed: 1993.39235\n",
      "\n",
      "=========================================================\n",
      "Swaps (Machine):\n",
      "[1362.98474, 1259.61929, 1312.33292, 1426.94488, 1692.05737, 134.47636, 1075.06208, 1107.89282, 1334.62252, 1083.63437, 1002.36012, 1803.05968, 1807.38171]\n",
      "N: 13 \tMean:  1261.725296923077 \tVar:  173044.9624971154\n",
      "Avergae swaps (Pseudo-random):\n",
      "[1499.459, 1566.4181, 1560.06736, 1837.57631, 1758.10921, 182.52418, 1324.45835, 1219.53493, 1471.70444, 1449.61455, 1458.0963, 2045.09657, 1993.39235]\n",
      "N: 13 \tMean:  1489.696280769231 \tVar:  198315.17725239138 \n",
      "\n",
      "1 ) 1499.459 > 1362.98474 => 9.535506888250664\n",
      "2 ) 1566.4181 > 1259.61929 => 21.712296594915184\n",
      "3 ) 1560.06736 > 1312.33292 => 17.24929785900174\n",
      "4 ) 1837.57631 > 1426.94488 => 25.157222520586544\n",
      "5 ) 1758.10921 > 1692.05737 => 3.8289072987310706\n",
      "6 ) 182.52418 > 134.47636 => 30.314030379885157\n",
      "7 ) 1324.45835 > 1075.06208 => 20.78717621087312\n",
      "8 ) 1219.53493 > 1107.89282 => 9.59360478536874\n",
      "9 ) 1471.70444 > 1334.62252 => 9.769490294887099\n",
      "10 ) 1449.61455 > 1083.63437 => 28.894134888252516\n",
      "11 ) 1458.0963 > 1002.36012 => 37.04484877647212\n",
      "12 ) 2045.09657 > 1803.05968 => 12.579369145938388\n",
      "13 ) 1993.39235 > 1807.38171 => 9.788039860490944\n",
      "\n",
      "Avergae percent difference between swaps needed for pseudo-random and machine sorting:\n",
      "18.173378884896408 \n",
      "\n",
      "Ttest_relResult(statistic=-6.334260481104443, pvalue=3.751124090260203e-05)\n",
      "Cohen's d:  0.5082950436327329\n"
     ]
    }
   ],
   "source": [
    "score_exams(exams,model='nlp3',normv=1,score=1,goal=\"medoid\",runs=100000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to contents](#Contents)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}