hindi_data_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Generation\n",
    "### Contributed by Praveen Yadav, [GitHub Repo](https://github.com/fnc11/nball4treehindi)\n",
    "In this part, we already had two input files, the tree structure or paths for all nodes to the topmost parent and a file which had the set to word mapping which we printed from the modernSyn2Words dictionary in the pre-processing section.\n",
    "So the first thing we did, we made two dictionaries out of the set to word mapping, one from set to word and another is the reverse, so by doing this we kind of ignored all other words in a set, kept only the first one in the list, because that was the easiest way to handle this data.\n",
    "We formed a Tree class with standard attributes and methods which you’ll understand once you go through the implementation of that and some extra methods which we needed for later experiments also.\n",
    "Now before we start building the tree from the paths we once again passed all the sets through our newly formed dictionary set to word to make sure there should not be any set number in the tree nodes for which we don’t have a corresponding set. After this filtering, we started building the tree, now we made *root* as the ultimate parent of each node. While making the tree we tried to store the set numbers in the nodes, further details can be found in the implementation.\n",
    "Now the stage was set to just print out the files we needed for experiment 1, used some simple code to print out the data in the required format, wordSenseChildren was simple tree traversal while for printing category codes we needed to write some complicated code because we need to print the category codes for the intermediate nodes as well.\n",
    "\n",
    "## Difficulties/Choices:\n",
    "1. Taking only first word from the modernSyn2Words dictionary, ignoring all other words in the set.\n",
    "2. While making the tree we stored the reference to all the node objects in a dictionary for faster access, to attach the child nodes directly and this dictionary was also used in later experiments.\n",
    "\n",
    "## Run the below code\n",
    "### To generate the tree out of the paths and also generate the wordSenseChildren.txt and catCodes.txt file which we ultimately need.\n",
    "\n",
    "### Tree Class\n",
    "* a utility class that we need for building the tree from our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Tree(object):\n",
    "    # \"Generic tree node.\"\n",
    "    def __init__(self, name='root', children=None):\n",
    "        self.name = name\n",
    "        self.children = []\n",
    "        if children is not None:\n",
    "            for child in children:\n",
    "                self.add_child(child)\n",
    "\n",
    "    def add_child(self, node):\n",
    "        assert isinstance(node, Tree)\n",
    "        self.children.append(node)\n",
    "\n",
    "    def printTree(self):\n",
    "        assert isinstance(self, Tree)\n",
    "        stck = [self]\n",
    "        while(True):\n",
    "            if len(stck) != 0:\n",
    "                temp = stck.pop(0)\n",
    "                print(temp.name)\n",
    "                if len(temp.children) > 0:\n",
    "                    for child in temp.children:\n",
    "                        stck.insert(0, child)\n",
    "            else:\n",
    "                break\n",
    "\n",
    "    def printLevelOrder(self):\n",
    "        assert isinstance(self, Tree)\n",
    "        que = [self]\n",
    "        nums_at_level = 1\n",
    "        levelNum = 0\n",
    "        # maxLevel = -1\n",
    "        levelDict = {0:[]}\n",
    "        for i in range(1,15):\n",
    "            levelDict[i]=[]\n",
    "        count = 0\n",
    "        while(True):\n",
    "            if nums_at_level == 0:\n",
    "                # print(\"level: \"+str(levelNum))\n",
    "                # if levelNum > maxLevel:\n",
    "                #     maxLevel = levelNum\n",
    "                levelNum += 1\n",
    "                nums_at_level = count\n",
    "                count = 0\n",
    "\n",
    "            if len(que) != 0:\n",
    "                nums_at_level -= 1\n",
    "                temp = que.pop(0)\n",
    "                # print(temp.name)\n",
    "                levelDict[levelNum].append(temp.name)\n",
    "                if len(temp.children) > 0:\n",
    "                    for child in temp.children:\n",
    "                        que.append(child)\n",
    "                        count += 1\n",
    "            else:\n",
    "                break\n",
    "        return levelDict\n",
    "        # print(maxLevel)\n",
    "\n",
    "    def child_size(self, sz):\n",
    "        assert isinstance(self, Tree)\n",
    "        clist = len(self.children)\n",
    "        count = 0\n",
    "        for i in range(0, clist):\n",
    "            if count > sz:\n",
    "                return count\n",
    "            count += self.children[i].size(sz)\n",
    "        return count+1\n",
    "\n",
    "    def node_and_children(self):\n",
    "        assert isinstance(self, Tree)\n",
    "        stck = [self]\n",
    "        child_list = []\n",
    "        while len(stck) != 0:\n",
    "            node = stck.pop()\n",
    "            child_list.append(node.name)\n",
    "            for child in node.children:\n",
    "                stck.append(child)\n",
    "        return child_list\n",
    "\n",
    "    def print_all_paths(self, path, filest):\n",
    "        assert isinstance(self, Tree)\n",
    "        new_path = self.name+\" \"+path\n",
    "        # print(new_path)\n",
    "        filest.write(new_path+\"\\n\")\n",
    "        if len(self.children)!= 0:\n",
    "            for child in self.children:\n",
    "                child.print_all_paths(new_path, filest)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### set2Word and word2Set Dictionaries\n",
    "* The below code makes two dictionaries which help us interchange synsets to words and vice versa whenever we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set2Word = {}\n",
    "word2Set = {}\n",
    "\n",
    "with open(\"data/set2WordV.txt\",\"r\") as s2w:\n",
    "    cont = s2w.read()\n",
    "    sets = cont.split(\"$\")\n",
    "    for set in sets:\n",
    "        pairs = set.split(\":\")\n",
    "        num = pairs[0]\n",
    "        word = pairs[1]\n",
    "        set2Word[num] = word\n",
    "        word2Set[word] = num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building the tree \n",
    "* Building the tree out of paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code to generate tree and printing wordsense and catcodes starts from here\n",
    "root = Tree(\"root\")\n",
    "allNodes = {\"root\": root}\n",
    "\n",
    "# final cleansing of the paths\n",
    "paths = []\n",
    "clean_paths = []\n",
    "with open(\"data/tree_struct.txt\", 'r') as tree_struct:\n",
    "    struct_cont = tree_struct.read()\n",
    "    paths = struct_cont.split(\"$\")\n",
    "    # count = 0\n",
    "    for path in paths:\n",
    "        tokens = path.split(\"<-\")\n",
    "        num_of_tokens = len(tokens)\n",
    "        new_path = tokens[0]\n",
    "        for i in range(1, num_of_tokens):\n",
    "            if tokens[i] in set2Word.keys():\n",
    "                new_path += \"<-\" + tokens[i]\n",
    "        clean_paths.append(new_path)\n",
    "\n",
    "    for path in clean_paths:\n",
    "        tokens = path.split(\"<-\")\n",
    "        # print(tokens)\n",
    "        num_tokens = len(tokens)\n",
    "        # saving prev_token to save it's child if they don't exist already\n",
    "        prev_token = tokens[num_tokens - 1]\n",
    "        # remember to leave last set, need to replace it with actual word\n",
    "        for i in range(0, num_tokens - 2):\n",
    "            name = str(tokens[num_tokens - 1 - i])\n",
    "            # to avoid index out of bound error\n",
    "            if i > 0:\n",
    "                prev_token = tokens[num_tokens - i]\n",
    "            if name in allNodes.keys():\n",
    "                continue\n",
    "            else:\n",
    "                # create new node and save it's mapping in the allNodes dict\n",
    "                temp = Tree(name)\n",
    "                allNodes[name] = temp\n",
    "                if i == 0:\n",
    "                    # if it's the node that goes below root\n",
    "                    allNodes[\"root\"].add_child(temp)\n",
    "                else:\n",
    "                    # if it is some other node's child, other than root(prev_token's)\n",
    "                    allNodes[prev_token].add_child(temp)\n",
    "        # now for the last node, as word\n",
    "        name = str(tokens[0])\n",
    "        # if the word is not directly attached to the root\n",
    "        if num_tokens >= 3:\n",
    "            if name in word2Set.keys():\n",
    "                name = word2Set[name]\n",
    "            if name not in allNodes.keys():\n",
    "                temp = Tree(name)\n",
    "                allNodes[name] = temp\n",
    "                allNodes[tokens[2]].add_child(temp)\n",
    "        else:\n",
    "            if name in word2Set.keys():\n",
    "                name = word2Set[name]\n",
    "            if name not in allNodes.keys():\n",
    "                temp = Tree(name)\n",
    "                allNodes[name] = temp\n",
    "                allNodes[\"root\"].add_child(temp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  setOrderNum Dictionary\n",
    "* This code makes a dictionary which keeps track of the number of a certain set under certain parent which will be used while printing catCodes of the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a dictionary to hold the order of the set in certain level under certain parent\n",
    "setOrderNum = {'root': 1 }\n",
    "with open(\"data/sameLevelWords.txt\", \"w\") as slw:\n",
    "    stck =[root]\n",
    "    while(True):\n",
    "        if len(stck) != 0:\n",
    "            temp = stck.pop(0)\n",
    "            if len(temp.children) > 0:\n",
    "                o_list = temp.children\n",
    "                c_list  = []\n",
    "                for child in o_list:\n",
    "                    word_name = child.name\n",
    "                    if child.name in set2Word.keys():    \n",
    "                        word_name = set2Word[child.name]\n",
    "                    c_list.append(word_name)\n",
    "                    stck.append(child)\n",
    "                c_list.sort()\n",
    "                k=0\n",
    "                for i in c_list:\n",
    "                    k=k+1\n",
    "                    if i in word2Set.keys():\n",
    "                        setOrderNum[word2Set[i]] = k\n",
    "                    else:\n",
    "                        setOrderNum[i] = k\n",
    "        else:\n",
    "            break\n",
    "        slw.write(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Printing CatCodes \n",
    "* Printing CatCodes for all the words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# printing cat codes of all the words in a file\n",
    "def print_catcodes():\n",
    "    with open(\"data/catCodes.txt\", \"w\") as ctcd:\n",
    "        # count = 0\n",
    "        # A dictionary to hold all the words whose cat_codes are already generated\n",
    "        cat_printed = {}\n",
    "        for path in clean_paths:\n",
    "            tokens = path.split(\"<-\")\n",
    "            num_of_tokens = len(tokens)\n",
    "\n",
    "            for j in range(0, num_of_tokens):\n",
    "                leaf = tokens[j]\n",
    "                if leaf in set2Word.keys():\n",
    "                    leaf = set2Word[leaf]\n",
    "                if leaf not in cat_printed:\n",
    "                    cat_printed[leaf]=True\n",
    "                    sen = \"\"\n",
    "                    slen = 0\n",
    "                    for i in range(j, num_of_tokens):\n",
    "                        if j==0 and i==1:\n",
    "                            slen = 1\n",
    "                            continue\n",
    "                        word = tokens[i]\n",
    "                        if word in set2Word.keys():\n",
    "                            word = set2Word[word]\n",
    "                        if word in word2Set.keys():\n",
    "                            sen = str(setOrderNum[word2Set[word]])+\" \"+sen\n",
    "                        # discussion need to be done here\n",
    "                        elif word in setOrderNum.keys():\n",
    "                            # count += 1\n",
    "                            sen = str(setOrderNum[word])+\" \"+sen\n",
    "                        else:\n",
    "                            print(\"Evil case\")\n",
    "                            continue\n",
    "                    # root order for every word\n",
    "                    # need to add special case for root*\n",
    "                    sen = \"1 \"+sen.strip()\n",
    "                    for k in range(0, 13-(num_of_tokens-j-slen)):\n",
    "                        sen += \" 0\"\n",
    "                    ctcd.write(leaf+\" \"+sen+\"\\n\")\n",
    "        # print(count)\n",
    "\n",
    "print_catcodes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Printing WordSenses\n",
    "* Printing WordSenses for all the words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# root.printTree()\n",
    "# printing word sense children file as English\n",
    "def print_word_senses():\n",
    "    with open(\"data/wordSenseChildren.txt\",\"w\") as wsc:\n",
    "        stck = [root]\n",
    "        while (True):\n",
    "            if len(stck) != 0:\n",
    "                temp = stck.pop(0)\n",
    "\n",
    "                if temp.name in set2Word.keys():\n",
    "                    wsc.write(set2Word[temp.name])\n",
    "                else:\n",
    "                    wsc.write(temp.name)\n",
    "\n",
    "                if len(temp.children) > 0:\n",
    "                    for child in temp.children:\n",
    "                        stck.insert(0, child)\n",
    "                        if child.name in set2Word.keys():\n",
    "                            wsc.write(\" \"+set2Word[child.name])\n",
    "                        else:\n",
    "                            wsc.write(\" \"+child.name)\n",
    "\n",
    "                wsc.write(\"\\n\")\n",
    "            else:\n",
    "                break\n",
    "\n",
    "print_word_senses()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some modification has to be done in both the files\n",
    "    * in wordSenseChildren.txt file change root to \\*root\\*\n",
    "    * in catCodes.txt also make root to \\*root\\* and the make the second 1 as 0 in it's code.\n",
    "\n",
    "## Now you are ready to use these files for experiments"
   ]
  },
   {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "  <tr>\n",
    "      <td colspan=\"1\" style=\"text-align:left;background-color:#0071BD;color:white\">\n",
    "        <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\">\n",
    "            <img alt=\"Creative Commons License\" style=\"border-width:0;float:left;padding-right:10pt\"\n",
    "                 src=\"https://i.creativecommons.org/l/by-nc/4.0/88x31.png\" />\n",
    "        </a>\n",
    "        &copy; T. Dong, C. Bauckhage<br/>\n",
    "        Licensed under a \n",
    "        <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\" style=\"color:white\">\n",
    "            CC BY-NC 4.0\n",
    "        </a>.\n",
    "      </td>\n",
    "      <td colspan=\"2\" style=\"text-align:left;background-color:#66A5D1\">\n",
    "          <b>Acknowledgments:</b>\n",
    "          This material was prepared within the project\n",
    "          <a href=\"http://www.b-it-center.de/b-it-programmes/teaching-material/p3ml/\" style=\"color:black\">\n",
    "              P3ML\n",
    "          </a> \n",
    "          which is funded by the Ministry of Education and Research of Germany (BMBF)\n",
    "          under grant number 01/S17064. The authors gratefully acknowledge this support.\n",
    "      </td>\n",
    "  </tr>\n",
    "</table>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}