3-Large DF of Unique Words.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"aet9TbfqOtc0"},"source":["# Large DataFrame of Unique Words\n","\n","Notebook by Melinee Her\n","\n","The goal for this notebook is to create a large dataframe of all unqiue words (via their \"id_word\" tag) from the word-dfs of all ORACC Projects."]},{"cell_type":"markdown","metadata":{"id":"8o9WOYlYRbH5"},"source":["# Mount Google Drive folder + imports + steps"]},{"cell_type":"markdown","source":["The code snippet below is to mount Google Drive files so that we can interact with our Google Drive files using the file browser or command line. Running it will give a permissions prompt."],"metadata":{"id":"gOeRV8mNQy1j"}},{"cell_type":"code","execution_count":1,"metadata":{"id":"oi1BummVRaZy","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1694466818222,"user_tz":420,"elapsed":1236,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"}},"outputId":"6b7e9760-8524-4ab2-d09f-ca28ffcec66d"},"outputs":[{"output_type":"stream","name":"stdout","text":["Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"]}],"source":["from google.colab import drive\n","drive.mount('/content/drive')"]},{"cell_type":"code","execution_count":2,"metadata":{"id":"oCIRD6koRoGc","executionInfo":{"status":"ok","timestamp":1694466818432,"user_tz":420,"elapsed":214,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"}}},"outputs":[],"source":["#any necessary imports\n","import pandas as pd\n","import zipfile\n","from zipfile import ZipFile\n","import json\n","import requests\n","from tqdm import tqdm\n","import os\n","import errno\n","import re\n","import random\n","import numpy as np\n","import sys\n","import copy\n","import networkx as nx\n","from pathlib import Path\n","\n","#Set folder for remote drive\n","#folder = '/content/drive/My Drive/FactGrid Cuneiform (AWCA)/people/Melinee'\n","folder = '/content/drive/MyDrive/Melinee/'\n","\n","#importing utils for the method which downloads the current text json files\n","os.chdir(folder + 'network/utils/')\n","from utils import oracc_download\n","\n","# This is a user defined module that searches through the texts to find the entities in the text that\n","# are people and places, to be imported as nodes into the network\n","os.chdir(folder + 'network/')\n","import rank_parser4 as rp"]},{"cell_type":"markdown","metadata":{"id":"CM6Psv6rox2U"},"source":["The steps:\n","1. Use the directory of the dataframes created called ORACC_DFS/PROJECT_DFS to access the words_df for all projects\n","2. Move id_text to be the index and merge all dataframes into a large dataframe\n","3. Sort by unique \"id_text\"\n","\n","For reference, there is a smaller example at the bottom of the notebook to watch the process of cleaning and merging two dataframes at a closer level."]},{"cell_type":"markdown","source":["List of the projects - split and final"],"metadata":{"id":"UhLqQgg9ErJv"}},{"cell_type":"code","execution_count":3,"metadata":{"executionInfo":{"elapsed":189,"status":"ok","timestamp":1694466818618,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"RRG5JWtfQnir"},"outputs":[],"source":["#projects alphabetical, starts with 'a'\n","p1 = ['adsd','adsd/adart1','adsd/adart2','adsd/adart3','adsd/adart5','adsd/adart6','aemw/alalakh/idrimi','aemw/amarna','akklove', 'ario',\n","      'armep','asbp','asbp/ninmed','asbp/rlasb','atae','atae/assur','atae/burmarina','atae/durkatlimmu',\n","      'atae/guzana','atae/huzirina','atae/imgurenlil','atae/kalhu','atae/mallanate','atae/marqasu',\n","      'atae/nineveh','atae/samal','atae/szibaniba','atae/tilbarsip','atae/tuszhan']\n","\n","\n","#projects alphabetical, starts with 'b' through 'e'\n","p2 = ['babcity','blms','borsippa','btmao','btto','cams','cams/akno','cams/anzu','cams/barutu','cams/etana','cams/ludlul',\n","      'cams/selbi','cams/tlab','ckst','cmawro','cmawro/cmawr1','cmawro/cmawr2','cmawro/cmawr3', 'cmawro/maqlu','contrib/amarna', 'ctij',\n","      'dcclt','dcclt/ebla','dcclt/jena','dcclt/nineveh','dcclt/signlists','dccmt','dsst', 'ecut', 'eisl','epsd2','etcsri',]\n","\n","\n","#projects alphabetical, 'g' through 'r'\n","p3 = ['glass','hbtin','lacost','nere','obel','obmc','obta','oimea','qcat','riao',\n","      'ribo','ribo/bab7scores','ribo/babylon10','ribo/babylon2','ribo/babylon3','ribo/babylon4','ribo/babylon5','ribo/babylon6',\n","      'ribo/babylon7','ribo/babylon8','ribo/sources','rimanum','rinap','rinap/rinap1','rinap/rinap2','rinap/rinap3',\n","      'rinap/rinap4','rinap/rinap5','rinap/rinap5p1','rinap/scores','rinap/sources',]\n","\n","\n","#projects alphabetical, 's' through 'x'\n","p4 = ['saao','saao/aebp','saao/knpp','saao/saa01','saao/saa02','saao/saa03','saao/saa04','saao/saa05','saao/saa06', 'saao/saa07','saao/saa08',\n","      'saao/saa09','saao/saa10','saao/saa11','saao/saa12','saao/saa13','saao/saa14','saao/saa15','saao/saa16','saao/saa17',\n","      'saao/saa18','saao/saa19','saao/saa20','saao/saa21','saao/saas2','suhu','tcma','tsae','xcat']\n","\n","#the buried projects\n","p5 = ['epsd2/earlylit', 'epsd2/literary', 'epsd2/praxis', 'epsd2/praxis/liturgy','epsd2/admin/ed12', 'epsd2/admin/ed3b', 'epsd2/admin/lagash2',\n","      'epsd2/admin/oakk', 'epsd2/admin/oldbab', 'epsd2/admin/ur3']\n","\n","p6 =  [\"tcma/ali1\",\"tcma/amarna\",\"tcma/assur\",\"tcma/barri\",\"tcma/bazmusian\",\"tcma/billa\", \"tcma/brak\",\"tcma/chuera\",\"tcma/emar\",\n","      \"tcma/fekheriye\",\"tcma/giricano\",\"tcma/hana\",\"tcma/haradum\",\"tcma/hatti\",\"tcma/kalhu\",\"tcma/kartn\",\"tcma/kulishinas\",\n","      \"tcma/miscellaneous\",\"tcma/nineveh\",\"tcma/nippur\",\"tcma/nuzi\",\"tcma/qitar\",\"tcma/rimah\",\"tcma/suri\",\n","      \"tcma/taban\",\"tcma/tsa1\",\"tcma/tsh1\",\"tcma/ugarit\"]\n","\n","pfinal = p1+p2+p3+p4+p5+p6"]},{"cell_type":"markdown","metadata":{"id":"xuHJHg53jOjM"},"source":["# Final DataFrame\n","This section of the notebook creates a final dataframe by concatenating the words-df of all projects."]},{"cell_type":"markdown","source":[],"metadata":{"id":"niWNqGE5FQ7R"}},{"cell_type":"code","execution_count":4,"metadata":{"id":"9gIN63dljwz-","executionInfo":{"status":"ok","timestamp":1694466819163,"user_tz":420,"elapsed":3,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"}}},"outputs":[],"source":["general_path = '/content/drive/MyDrive/Melinee/ORACC_DFS/PROJECT_DFS/'\n","pluscsv = '-words-df.csv'\n","\n","def createlargedf(lst):\n","  dash_proj = [word.replace(\"/\", \"-\") for word in lst]\n","  temp = pd.DataFrame()\n","  for i in range(len(lst)):\n","    print(lst[i])                   #uncomment this line to see progress as runs\n","    path = general_path + dash_proj[i] + pluscsv\n","    if i == 0:\n","      temp = pd.read_csv(path, low_memory=False, index_col=False)\n","    else:\n","      df = pd.read_csv(path, low_memory=False, index_col=False)\n","      temp = pd.concat([temp, df], ignore_index = True).drop(columns=['number']).drop_duplicates(subset=[\"id_word\"])\n","  return temp"]},{"cell_type":"markdown","source":["## Creating the finaldf using the smaller sections\n","\n","I found that separating the projects into smaller data frames first allows me to run and create the final df in a shorter amount of time and without going over the RAM limit."],"metadata":{"id":"0bwIZpkgGLfM"}},{"cell_type":"code","source":["#~17 minute runtime\n","p1_p4= createlargedf(p1+p2+p3+p4)"],"metadata":{"id":"dddfFthUB3ko"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["#~13 minute runtime\n","p5_p6 = createlargedf(p5+p6)"],"metadata":{"id":"ezTKzhdJHSfp"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["finaldf = pd.concat([p1_p4, p5_p6], ignore_index=True).drop_duplicates(subset = ['id_word'])"],"metadata":{"id":"8Q_Kb88rJ38h"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["##Creating the finaldf using the full path list\n","Using the createlargedf function, we can pass the entire list of the project path to get the finaldf.\n"],"metadata":{"id":"GBIaIGi-JIj9"}},{"cell_type":"code","source":["finaldf = createlargedf(pfinal)"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"wAdV_zswFrCx","outputId":"d7a1ab4f-9739-4b34-fa6b-f6974672af0a"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["adsd\n","adsd/adart1\n","adsd/adart2\n","adsd/adart3\n","adsd/adart5\n","adsd/adart6\n","aemw/alalakh/idrimi\n","aemw/amarna\n","akklove\n","ario\n","armep\n","asbp\n","asbp/ninmed\n","asbp/rlasb\n","atae\n","atae/assur\n","atae/burmarina\n","atae/durkatlimmu\n","atae/guzana\n","atae/huzirina\n","atae/imgurenlil\n","atae/kalhu\n","atae/mallanate\n","atae/marqasu\n","atae/nineveh\n","atae/samal\n","atae/szibaniba\n","atae/tilbarsip\n","atae/tuszhan\n","babcity\n","blms\n","borsippa\n","btmao\n","btto\n","cams\n","cams/akno\n","cams/anzu\n","cams/barutu\n","cams/etana\n","cams/ludlul\n","cams/selbi\n","cams/tlab\n","ckst\n","cmawro\n","cmawro/cmawr1\n","cmawro/cmawr2\n","cmawro/cmawr3\n","cmawro/maqlu\n","contrib/amarna\n","ctij\n","dcclt\n","dcclt/ebla\n","dcclt/jena\n","dcclt/nineveh\n","dcclt/signlists\n","dccmt\n","dsst\n","ecut\n","eisl\n","epsd2\n","etcsri\n","glass\n","hbtin\n","lacost\n","nere\n","obel\n","obmc\n","obta\n","oimea\n","qcat\n","riao\n","ribo\n","ribo/bab7scores\n","ribo/babylon10\n","ribo/babylon2\n","ribo/babylon3\n","ribo/babylon4\n","ribo/babylon5\n","ribo/babylon6\n","ribo/babylon7\n","ribo/babylon8\n","ribo/sources\n","rimanum\n","rinap\n","rinap/rinap1\n","rinap/rinap2\n","rinap/rinap3\n","rinap/rinap4\n","rinap/rinap5\n","rinap/rinap5p1\n","rinap/scores\n","rinap/sources\n","saao\n","saao/aebp\n","saao/knpp\n","saao/saa01\n","saao/saa02\n","saao/saa03\n","saao/saa04\n","saao/saa05\n","saao/saa06\n","saao/saa07\n","saao/saa08\n","saao/saa09\n","saao/saa10\n","saao/saa11\n","saao/saa12\n","saao/saa13\n","saao/saa14\n","saao/saa15\n","saao/saa16\n","saao/saa17\n","saao/saa18\n","saao/saa19\n","saao/saa20\n","saao/saa21\n","saao/saas2\n","suhu\n","tcma\n","tsae\n","xcat\n","epsd2/earlylit\n","epsd2/literary\n","epsd2/praxis\n"]}]},{"cell_type":"markdown","source":["## Analyzing the finaldf DataFrame"],"metadata":{"id":"D1Bb6XllKKI2"}},{"cell_type":"markdown","source":["To preview the size and unique values for each column in the finaldf"],"metadata":{"id":"hRhnSbhnHl90"}},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":22841,"status":"ok","timestamp":1688680229549,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"wBtOwyH6yx34","outputId":"7a4ea6ad-621b-4235-d34b-4dc6118aa36c"},"outputs":[{"output_type":"stream","name":"stdout","text":["lang (46): ['arc' 'akk-x-neoass' 'akk-x-mbperi']\n","form (372095): ['mmxx' 't' 'rmyt']\n","id_word (7271149): ['P522613.2.1' 'P522613.2.2' 'P522613.2.3']\n","label (63625): ['o 1' 'o 2' 'o 3']\n","id_text (117729): ['P522613' 'P522597' 'P522600']\n","delim (1): [nan]\n","gdl (7214709): [nan\n"," \"[{'n': 'n', 'sexified': '1/2(disz)', 'form': '1/2', 'id': 'P522597.2.1.0', 'seq': [{'r': '1/2'}]}]\"\n"," \"[{'gg': 'logo', 'gdl_type': 'logo', 'group': [{'s': 'MA', 'id': 'P522597.2.2.0', 'role': 'logo', 'logolang': 'sux', 'delim': '.'}, {'s': 'NA', 'id': 'P522597.2.2.1', 'role': 'logo', 'logolang': 'sux'}]}]\"]\n","pos (55): [nan 'n' 'N']\n","cf (80585): [nan 'manû' 'ṣarpu']\n","gw (14917): [nan 'unit' 'silver']\n","sense (26224): [nan 'a unit of weight' 'silver']\n","norm (94863): [nan 'manā' 'ṣarpu']\n","epos (47): [nan 'N' 'DET']\n","headform (455): [nan 'ib-ba-lak' 'x-š']\n","contrefs (1079): [nan 'P522604.16.3 P522604.17.1' 'Q007155.1.4 Q007155.1.1001']\n","norm0 (101809): [nan 'še' 'an']\n","base (68084): [nan 'ši' 'an']\n","morph (13592): [nan '~' '~;a']\n","stem (20): [nan 'G' 'Š']\n","cont (173): [nan '-ra=r.a' '-na=n.a']\n","syntax_ub-after (2): [nan '-.']\n","morph2 (1409): [nan 'N1=NAME.N5=DAT-H' 'N1=STEM.N5=ABS']\n","aform (5194): [nan 'uzud-zu' 'kug-zu']\n"]}],"source":["for col in finaldf.columns:\n","  uniq = finaldf[col].unique()\n","  print(col + ' (' + str(len(uniq)) + '): ' + str(uniq[:3]))"]},{"cell_type":"markdown","source":["The count of unique forms and dialects per language"],"metadata":{"id":"YQ9_XGpsIOYV"}},{"cell_type":"code","execution_count":1,"metadata":{"id":"Qgeh0oMyV4es","colab":{"base_uri":"https://localhost:8080/","height":194},"executionInfo":{"status":"error","timestamp":1694468043299,"user_tz":420,"elapsed":153,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"}},"outputId":"a1596ff7-f55e-448d-8b18-ac8f2ee281c6"},"outputs":[{"output_type":"error","ename":"NameError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)","\u001b[0;32m<ipython-input-1-b0816b18f067>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mlang_summary\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfinaldf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroupby\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'lang'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcount\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'form'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mlang_summary\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'unique forms'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfinaldf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgroupby\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'lang'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnunique\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'form'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mNameError\u001b[0m: name 'finaldf' is not defined"]}],"source":["lang_summary = finaldf.groupby('lang').count()[['form']]\n","lang_summary['unique forms'] = finaldf.groupby('lang').nunique()[['form']]"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":176},"executionInfo":{"elapsed":156,"status":"error","timestamp":1694468043882,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"lA8dpT1sNjBj","outputId":"3ab13361-d1f5-4f53-f709-b7bc63e2b726"},"outputs":[{"output_type":"error","ename":"NameError","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)","\u001b[0;32m<ipython-input-2-b3c3eefc0bc8>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mlang_summary\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m","\u001b[0;31mNameError\u001b[0m: name 'lang_summary' is not defined"]}],"source":["lang_summary"]},{"cell_type":"markdown","source":["Looking at the unique amount of words and texts in finaldf"],"metadata":{"id":"DwMO8bOBK0uC"}},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":6355,"status":"ok","timestamp":1688680362070,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"z1nEo0l9wJcy","outputId":"10c19929-d4a4-415a-ba6f-95c0e703496d"},"outputs":[{"output_type":"stream","name":"stdout","text":["No.of.unique words : 7271149\n","No.of.unique texts : 117729\n"]}],"source":["id_words = len(pd.unique(finaldf['id_word']))\n","id_texts = len(pd.unique(finaldf['id_text']))\n","print(\"No.of.unique words :\", id_words)\n","print(\"No.of.unique texts :\", id_texts)"]},{"cell_type":"markdown","source":["Exporting the finaldf to the ORACC_DFS directory"],"metadata":{"id":"QDPqJ7hKLnam"}},{"cell_type":"code","source":["finaldf.to_csv(folder + 'ORACC_DFS/finaldf.csv')"],"metadata":{"id":"af9V1o5o1RoT"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"cD301zjkz3Ju"},"source":["##This workflow accomplished:\n","The creation of a large dataframe of total words with their metadata from words_df.\n","\n","No.of.unique words : 7271149\n","\n","No.of.unique texts : 117729"]},{"cell_type":"markdown","metadata":{"id":"4PvQhvgwQ3GS"},"source":["# EXAMPLE: Merging Two DataFrames"]},{"cell_type":"markdown","source":["In this example, both the adsd and adsd-adart1 projects are to be merged.\n","In order to do this, we need to get and open each words_df csv file and look at the dataframes."],"metadata":{"id":"I8LHaP8iNgZb"}},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":381},"executionInfo":{"elapsed":2483,"status":"error","timestamp":1688494498143,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"XWZyvCVERby8","outputId":"94d16c29-610f-45e4-ffd7-88610223cd45"},"outputs":[{"output_type":"error","ename":"KeyboardInterrupt","evalue":"ignored","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)","\u001b[0;32m<ipython-input-5-1cebbb02b17d>\u001b[0m in \u001b[0;36m<cell line: 12>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0mpath1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgeneral_path\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mlst\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'/'\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mdash_lst\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mpluscsv\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0mpath2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgeneral_path\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mlst\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'/'\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mdash_lst\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mpluscsv\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 12\u001b[0;31m \u001b[0mdf1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlow_memory\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex_col\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     13\u001b[0m \u001b[0mdf2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlow_memory\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mindex_col\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0mdf1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhead\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/util/_decorators.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    209\u001b[0m                 \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    210\u001b[0m                     \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mnew_arg_name\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_arg_value\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 211\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    212\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    213\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mF\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/util/_decorators.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    329\u001b[0m                     \u001b[0mstacklevel\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfind_stack_level\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    330\u001b[0m                 )\n\u001b[0;32m--> 331\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    332\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    333\u001b[0m         \u001b[0;31m# error: \"Callable[[VarArg(Any), KwArg(Any)], Any]\" has no\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/io/parsers/readers.py\u001b[0m in \u001b[0;36mread_csv\u001b[0;34m(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, error_bad_lines, warn_bad_lines, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options)\u001b[0m\n\u001b[1;32m    948\u001b[0m     \u001b[0mkwds\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkwds_defaults\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    949\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 950\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0m_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilepath_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    951\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    952\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/io/parsers/readers.py\u001b[0m in \u001b[0;36m_read\u001b[0;34m(filepath_or_buffer, kwds)\u001b[0m\n\u001b[1;32m    609\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    610\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0mparser\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 611\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mparser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    612\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    613\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/io/parsers/readers.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, nrows)\u001b[0m\n\u001b[1;32m   1776\u001b[0m                     \u001b[0mcolumns\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1777\u001b[0m                     \u001b[0mcol_dict\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1778\u001b[0;31m                 \u001b[0;34m)\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m  \u001b[0;31m# type: ignore[attr-defined]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1779\u001b[0m                     \u001b[0mnrows\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1780\u001b[0m                 )\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/io/parsers/c_parser_wrapper.py\u001b[0m in \u001b[0;36mread\u001b[0;34m(self, nrows)\u001b[0m\n\u001b[1;32m    233\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    234\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 235\u001b[0;31m                 \u001b[0mdata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_reader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnrows\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    236\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    237\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_first_chunk\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader.read\u001b[0;34m()\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._read_rows\u001b[0;34m()\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_column_data\u001b[0;34m()\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_tokens\u001b[0;34m()\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/_libs/parsers.pyx\u001b[0m in \u001b[0;36mpandas._libs.parsers.TextReader._convert_with_dtype\u001b[0;34m()\u001b[0m\n","\u001b[0;32m/usr/local/lib/python3.10/dist-packages/pandas/core/dtypes/common.py\u001b[0m in \u001b[0;36mis_extension_array_dtype\u001b[0;34m(arr_or_dtype)\u001b[0m\n\u001b[1;32m   1431\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1432\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1433\u001b[0;31m \u001b[0;32mdef\u001b[0m \u001b[0mis_extension_array_dtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marr_or_dtype\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1434\u001b[0m     \"\"\"\n\u001b[1;32m   1435\u001b[0m     \u001b[0mCheck\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0man\u001b[0m \u001b[0mobject\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0ma\u001b[0m \u001b[0mpandas\u001b[0m \u001b[0mextension\u001b[0m \u001b[0marray\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n","\u001b[0;31mKeyboardInterrupt\u001b[0m: "]}],"source":["\n","\n","#gets the path in the right form\n","general_path = '/content/drive/MyDrive/Melinee/ORACC_DFS/'\n","pluscsv = '-words-df.csv'\n","lst = ['adsd', 'adsd/adart1']\n","dash_lst = [word.replace(\"/\", \"-\") for word in lst]\n","\n","path1 = general_path + lst[0] + '/' + dash_lst[0] + pluscsv\n","path2 = general_path + lst[1] + '/' + dash_lst[1] + pluscsv\n","df1 = pd.read_csv(path1, low_memory=False, index_col=False)\n","df2 = pd.read_csv(path2, low_memory=False, index_col=False)\n","df1.head(5)"]},{"cell_type":"markdown","source":["With the dataframes imported, we can concatenate both dataframes and drop any duplicates."],"metadata":{"id":"8THeXU8KN7Zp"}},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":693},"executionInfo":{"elapsed":272,"status":"ok","timestamp":1687193186561,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"N8Lq2g04S34S","outputId":"821aa915-b9a4-41d5-88cc-ea74523c1735"},"outputs":[{"data":{"text/html":["\n","  <div id=\"df-c6e205d8-3f10-49f8-b71c-47efc54c905a\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>lang</th>\n","      <th>form</th>\n","      <th>delim</th>\n","      <th>gdl</th>\n","      <th>pos</th>\n","      <th>id_word</th>\n","      <th>label</th>\n","      <th>id_text</th>\n","      <th>cf</th>\n","      <th>gw</th>\n","      <th>sense</th>\n","      <th>norm</th>\n","      <th>epos</th>\n","      <th>headform</th>\n","      <th>contrefs</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'x': 'ellipsis', 'id': 'X500020.2.1.0', 'bre...</td>\n","      <td>u</td>\n","      <td>X500020.2.1</td>\n","      <td>o 1'</td>\n","      <td>X500020</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'v': 'x', 'id': 'X500020.2.2.0', 'break': 'd...</td>\n","      <td>u</td>\n","      <td>X500020.2.2</td>\n","      <td>o 1'</td>\n","      <td>X500020</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>akk</td>\n","      <td>LUGAL</td>\n","      <td>NaN</td>\n","      <td>[{'gg': 'logo', 'gdl_type': 'logo', 'group': [...</td>\n","      <td>N</td>\n","      <td>X500020.2.3</td>\n","      <td>o 1'</td>\n","      <td>X500020</td>\n","      <td>šarru</td>\n","      <td>king</td>\n","      <td>king</td>\n","      <td>šarru</td>\n","      <td>N</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'v': 'x', 'id': 'X500020.3.1.0', 'breakStart...</td>\n","      <td>u</td>\n","      <td>X500020.3.1</td>\n","      <td>o 2'</td>\n","      <td>X500020</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'v': 'x', 'id': 'X500020.3.2.0', 'statusStar...</td>\n","      <td>u</td>\n","      <td>X500020.3.2</td>\n","      <td>o 2'</td>\n","      <td>X500020</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>613690</th>\n","      <td>akk</td>\n","      <td>TIL</td>\n","      <td>NaN</td>\n","      <td>[{'gg': 'logo', 'gdl_type': 'logo', 'group': [...</td>\n","      <td>N</td>\n","      <td>X102910.110.4</td>\n","      <td>l.e. B1</td>\n","      <td>X102910</td>\n","      <td>qītu</td>\n","      <td>end</td>\n","      <td>end</td>\n","      <td>qītu</td>\n","      <td>N</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>613691</th>\n","      <td>akk</td>\n","      <td>KIN</td>\n","      <td>NaN</td>\n","      <td>[{'gg': 'logo', 'gdl_type': 'logo', 'group': [...</td>\n","      <td>MN</td>\n","      <td>X102910.110.5</td>\n","      <td>l.e. B1</td>\n","      <td>X102910</td>\n","      <td>Ulūlu</td>\n","      <td>Month VI</td>\n","      <td>Month VI</td>\n","      <td>Ulūlu</td>\n","      <td>MN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>613692</th>\n","      <td>akk</td>\n","      <td>MU.20.KAM₂</td>\n","      <td>NaN</td>\n","      <td>[{'gg': 'logo', 'gdl_type': 'logo', 'group': [...</td>\n","      <td>n</td>\n","      <td>X102910.110.6</td>\n","      <td>l.e. B1</td>\n","      <td>X102910</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>613693</th>\n","      <td>akk</td>\n","      <td>{m}se-lu-ku</td>\n","      <td>NaN</td>\n","      <td>[{'det': 'semantic', 'pos': 'pre', 'seq': [{'v...</td>\n","      <td>RN</td>\n","      <td>X102910.110.7</td>\n","      <td>l.e. B1</td>\n","      <td>X102910</td>\n","      <td>Seleucus</td>\n","      <td>Seleucus</td>\n","      <td>Seleucus</td>\n","      <td>Seleucus</td>\n","      <td>RN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>613694</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'x': 'ellipsis', 'id': 'X102910.110.8.0', 'b...</td>\n","      <td>u</td>\n","      <td>X102910.110.8</td>\n","      <td>l.e. B1</td>\n","      <td>X102910</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>613695 rows × 15 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c6e205d8-3f10-49f8-b71c-47efc54c905a')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-c6e205d8-3f10-49f8-b71c-47efc54c905a button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-c6e205d8-3f10-49f8-b71c-47efc54c905a');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "],"text/plain":["       lang         form  delim  \\\n","0       akk            x    NaN   \n","1       akk            x    NaN   \n","2       akk        LUGAL    NaN   \n","3       akk            x    NaN   \n","4       akk            x    NaN   \n","...     ...          ...    ...   \n","613690  akk          TIL    NaN   \n","613691  akk          KIN    NaN   \n","613692  akk   MU.20.KAM₂    NaN   \n","613693  akk  {m}se-lu-ku    NaN   \n","613694  akk            x    NaN   \n","\n","                                                      gdl pos        id_word  \\\n","0       [{'x': 'ellipsis', 'id': 'X500020.2.1.0', 'bre...   u    X500020.2.1   \n","1       [{'v': 'x', 'id': 'X500020.2.2.0', 'break': 'd...   u    X500020.2.2   \n","2       [{'gg': 'logo', 'gdl_type': 'logo', 'group': [...   N    X500020.2.3   \n","3       [{'v': 'x', 'id': 'X500020.3.1.0', 'breakStart...   u    X500020.3.1   \n","4       [{'v': 'x', 'id': 'X500020.3.2.0', 'statusStar...   u    X500020.3.2   \n","...                                                   ...  ..            ...   \n","613690  [{'gg': 'logo', 'gdl_type': 'logo', 'group': [...   N  X102910.110.4   \n","613691  [{'gg': 'logo', 'gdl_type': 'logo', 'group': [...  MN  X102910.110.5   \n","613692  [{'gg': 'logo', 'gdl_type': 'logo', 'group': [...   n  X102910.110.6   \n","613693  [{'det': 'semantic', 'pos': 'pre', 'seq': [{'v...  RN  X102910.110.7   \n","613694  [{'x': 'ellipsis', 'id': 'X102910.110.8.0', 'b...   u  X102910.110.8   \n","\n","          label  id_text        cf        gw     sense      norm epos  \\\n","0          o 1'  X500020       NaN       NaN       NaN       NaN  NaN   \n","1          o 1'  X500020       NaN       NaN       NaN       NaN  NaN   \n","2          o 1'  X500020     šarru      king      king     šarru    N   \n","3          o 2'  X500020       NaN       NaN       NaN       NaN  NaN   \n","4          o 2'  X500020       NaN       NaN       NaN       NaN  NaN   \n","...         ...      ...       ...       ...       ...       ...  ...   \n","613690  l.e. B1  X102910      qītu       end       end      qītu    N   \n","613691  l.e. B1  X102910     Ulūlu  Month VI  Month VI     Ulūlu   MN   \n","613692  l.e. B1  X102910       NaN       NaN       NaN       NaN  NaN   \n","613693  l.e. B1  X102910  Seleucus  Seleucus  Seleucus  Seleucus   RN   \n","613694  l.e. B1  X102910       NaN       NaN       NaN       NaN  NaN   \n","\n","       headform contrefs  \n","0           NaN      NaN  \n","1           NaN      NaN  \n","2           NaN      NaN  \n","3           NaN      NaN  \n","4           NaN      NaN  \n","...         ...      ...  \n","613690      NaN      NaN  \n","613691      NaN      NaN  \n","613692      NaN      NaN  \n","613693      NaN      NaN  \n","613694      NaN      NaN  \n","\n","[613695 rows x 15 columns]"]},"execution_count":118,"metadata":{},"output_type":"execute_result"}],"source":["frames = [df1, df2]\n","result = pd.concat(frames, ignore_index = True).drop(columns = 'number')\n","result"]},{"cell_type":"markdown","source":["This cell focuses on a dataframe for all texts that have the id 'X500020'.\n","\n","In the initial concatenation, there are 216 recorded texts. After dropping all duplicate rows, there are 108 texts with this id. there is unique information for each 108 inputs, however the id of the text is the same."],"metadata":{"id":"X9ELX_P2OJge"}},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":368,"status":"ok","timestamp":1687193186926,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"YFnGJth-evfa","outputId":"c4c6b7e3-a549-434f-c654-fcbc19c5a5b9"},"outputs":[{"data":{"text/plain":["((216, 15), (108, 15))"]},"execution_count":119,"metadata":{},"output_type":"execute_result"}],"source":["before = result.loc[result['id_text']=='X500020']\n","after = result.loc[result['id_text']=='X500020'].drop_duplicates()\n","\n","before.shape, after.shape"]},{"cell_type":"markdown","source":["Here is how to get a list of all unique id_texts and the number of unique texts there are in the merged \"result\" dataframe."],"metadata":{"id":"-ZWNpwNBOWaE"}},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":6,"status":"ok","timestamp":1687193186927,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"ejWhGAbNfvXn","outputId":"16ad9844-d3b2-40e1-cd40-acb5c26df975"},"outputs":[{"data":{"text/plain":["1609"]},"execution_count":120,"metadata":{},"output_type":"execute_result"}],"source":["unique_ids_adart = result['id_text'].unique()\n","print(unique_ids_adart)\n","len(unique_ids_adart)"]},{"cell_type":"markdown","source":["In this cell, we want to drop duplicates from the entire result dataframe. notice that result df has 613695 rows × 15 columns.\n","\n","Once we dropped all duplicate rows and renamed the dataframe \"clean\", we see that it has 364669 rows × 15 columns."],"metadata":{"id":"h2Jsn3vjOj2t"}},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":423},"executionInfo":{"elapsed":1435,"status":"ok","timestamp":1687193188359,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"eHzUc3ycVGWa","outputId":"fa4be8ff-109c-4eb9-f804-92581d203100"},"outputs":[{"data":{"text/html":["\n","  <div id=\"df-71668a9c-12d2-43e3-95dd-c43982a916a2\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>lang</th>\n","      <th>form</th>\n","      <th>delim</th>\n","      <th>gdl</th>\n","      <th>pos</th>\n","      <th>id_word</th>\n","      <th>label</th>\n","      <th>id_text</th>\n","      <th>cf</th>\n","      <th>gw</th>\n","      <th>sense</th>\n","      <th>norm</th>\n","      <th>epos</th>\n","      <th>headform</th>\n","      <th>contrefs</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'x': 'ellipsis', 'id': 'X500020.2.1.0', 'bre...</td>\n","      <td>u</td>\n","      <td>X500020.2.1</td>\n","      <td>o 1'</td>\n","      <td>X500020</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'v': 'x', 'id': 'X500020.2.2.0', 'break': 'd...</td>\n","      <td>u</td>\n","      <td>X500020.2.2</td>\n","      <td>o 1'</td>\n","      <td>X500020</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>akk</td>\n","      <td>LUGAL</td>\n","      <td>NaN</td>\n","      <td>[{'gg': 'logo', 'gdl_type': 'logo', 'group': [...</td>\n","      <td>N</td>\n","      <td>X500020.2.3</td>\n","      <td>o 1'</td>\n","      <td>X500020</td>\n","      <td>šarru</td>\n","      <td>king</td>\n","      <td>king</td>\n","      <td>šarru</td>\n","      <td>N</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'v': 'x', 'id': 'X500020.3.1.0', 'breakStart...</td>\n","      <td>u</td>\n","      <td>X500020.3.1</td>\n","      <td>o 2'</td>\n","      <td>X500020</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'v': 'x', 'id': 'X500020.3.2.0', 'statusStar...</td>\n","      <td>u</td>\n","      <td>X500020.3.2</td>\n","      <td>o 2'</td>\n","      <td>X500020</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>...</th>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","      <td>...</td>\n","    </tr>\n","    <tr>\n","      <th>364778</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'x': 'ellipsis', 'id': 'X600043.8.6.0', 'bre...</td>\n","      <td>u</td>\n","      <td>X600043.8.6</td>\n","      <td>o 6'</td>\n","      <td>X600043</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>364779</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'v': 'x', 'id': 'X600043.9.1.0', 'break': 'd...</td>\n","      <td>u</td>\n","      <td>X600043.9.1</td>\n","      <td>o 7'</td>\n","      <td>X600043</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>364780</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'v': 'x', 'id': 'X600043.9.2.0', 'break': 'd...</td>\n","      <td>u</td>\n","      <td>X600043.9.2</td>\n","      <td>o 7'</td>\n","      <td>X600043</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>364781</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'v': 'x', 'id': 'X600043.9.3.0', 'break': 'd...</td>\n","      <td>u</td>\n","      <td>X600043.9.3</td>\n","      <td>o 7'</td>\n","      <td>X600043</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","    <tr>\n","      <th>364782</th>\n","      <td>akk</td>\n","      <td>x</td>\n","      <td>NaN</td>\n","      <td>[{'x': 'ellipsis', 'id': 'X600043.9.4.0', 'bre...</td>\n","      <td>u</td>\n","      <td>X600043.9.4</td>\n","      <td>o 7'</td>\n","      <td>X600043</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","      <td>NaN</td>\n","    </tr>\n","  </tbody>\n","</table>\n","<p>364669 rows × 15 columns</p>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-71668a9c-12d2-43e3-95dd-c43982a916a2')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-71668a9c-12d2-43e3-95dd-c43982a916a2 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-71668a9c-12d2-43e3-95dd-c43982a916a2');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "],"text/plain":["       lang   form  delim                                                gdl  \\\n","0       akk      x    NaN  [{'x': 'ellipsis', 'id': 'X500020.2.1.0', 'bre...   \n","1       akk      x    NaN  [{'v': 'x', 'id': 'X500020.2.2.0', 'break': 'd...   \n","2       akk  LUGAL    NaN  [{'gg': 'logo', 'gdl_type': 'logo', 'group': [...   \n","3       akk      x    NaN  [{'v': 'x', 'id': 'X500020.3.1.0', 'breakStart...   \n","4       akk      x    NaN  [{'v': 'x', 'id': 'X500020.3.2.0', 'statusStar...   \n","...     ...    ...    ...                                                ...   \n","364778  akk      x    NaN  [{'x': 'ellipsis', 'id': 'X600043.8.6.0', 'bre...   \n","364779  akk      x    NaN  [{'v': 'x', 'id': 'X600043.9.1.0', 'break': 'd...   \n","364780  akk      x    NaN  [{'v': 'x', 'id': 'X600043.9.2.0', 'break': 'd...   \n","364781  akk      x    NaN  [{'v': 'x', 'id': 'X600043.9.3.0', 'break': 'd...   \n","364782  akk      x    NaN  [{'x': 'ellipsis', 'id': 'X600043.9.4.0', 'bre...   \n","\n","       pos      id_word label  id_text     cf    gw sense   norm epos  \\\n","0        u  X500020.2.1  o 1'  X500020    NaN   NaN   NaN    NaN  NaN   \n","1        u  X500020.2.2  o 1'  X500020    NaN   NaN   NaN    NaN  NaN   \n","2        N  X500020.2.3  o 1'  X500020  šarru  king  king  šarru    N   \n","3        u  X500020.3.1  o 2'  X500020    NaN   NaN   NaN    NaN  NaN   \n","4        u  X500020.3.2  o 2'  X500020    NaN   NaN   NaN    NaN  NaN   \n","...     ..          ...   ...      ...    ...   ...   ...    ...  ...   \n","364778   u  X600043.8.6  o 6'  X600043    NaN   NaN   NaN    NaN  NaN   \n","364779   u  X600043.9.1  o 7'  X600043    NaN   NaN   NaN    NaN  NaN   \n","364780   u  X600043.9.2  o 7'  X600043    NaN   NaN   NaN    NaN  NaN   \n","364781   u  X600043.9.3  o 7'  X600043    NaN   NaN   NaN    NaN  NaN   \n","364782   u  X600043.9.4  o 7'  X600043    NaN   NaN   NaN    NaN  NaN   \n","\n","       headform contrefs  \n","0           NaN      NaN  \n","1           NaN      NaN  \n","2           NaN      NaN  \n","3           NaN      NaN  \n","4           NaN      NaN  \n","...         ...      ...  \n","364778      NaN      NaN  \n","364779      NaN      NaN  \n","364780      NaN      NaN  \n","364781      NaN      NaN  \n","364782      NaN      NaN  \n","\n","[364669 rows x 15 columns]"]},"execution_count":121,"metadata":{},"output_type":"execute_result"}],"source":["clean = result.drop_duplicates()\n","clean"]},{"cell_type":"markdown","source":["If we run this similar cell from before but with the clean df, we should get the same value of unique id_texts."],"metadata":{"id":"DN2-wBhOPOue"}},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":3,"status":"ok","timestamp":1687193188359,"user":{"displayName":"Melinee Her","userId":"06864672402504422512"},"user_tz":420},"id":"nRowoOdKgkzD","outputId":"4b74ca59-ea06-47bf-da4a-519b85165685"},"outputs":[{"data":{"text/plain":["1609"]},"execution_count":122,"metadata":{},"output_type":"execute_result"}],"source":["unique_clean = clean['id_text'].unique()\n","len(unique_clean)"]}],"metadata":{"colab":{"collapsed_sections":["4PvQhvgwQ3GS"],"provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"}},"nbformat":4,"nbformat_minor":0}