tutorials/tutorial-07-graph-node_classification.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\";\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using Keras version: 2.2.4\n"
     ]
    }
   ],
   "source": [
    "import ktrain\n",
    "from ktrain import graph as gr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Node Classification in Graphs\n",
    "\n",
    "\n",
    "In this notebook, we will use *ktrain* to perform node classificaiton on the PubMed Diabetes citation graph.  In the PubMed graph, each node represents a paper pertaining to one of three topics:  *Diabetes Mellitus - Experimental*, *Diabetes Mellitus - Type 1*, and *Diabetes Mellitus - Type 2*.  Links represent citations between papers.  The attributes or features assigned to each node are in the form of a vector of words in each paper and their corresponding TF-IDF scores.  The dataset is available [here](https://linqs-data.soe.ucsc.edu/public/Pubmed-Diabetes.tgz).\n",
    "\n",
    "*ktrain* expects two files for node classification problems.  The first is comma or tab delimited file listing the edges in the graph, where each row contains the node IDs forming the edge.  The second is a comma or tab delimted file listing the features or attributes associated with each node in the graph.  The first column in this file is the User ID and the last column should be string representing the target or label of the node.  All other nodes should be numerical features assumed to be standardized appropriately and non-null.  \n",
    "\n",
    "We must prepare the raw data to conform to the above before we begin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the Data\n",
    "The code below will create two files that can be processed directly by *ktrain*:\n",
    "- `/tmp/pubmed-nodes.tab`\n",
    "- `/tmp/pubmed-edges.tab`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set this to the location of the downloaded Pubmed-Diabetes data\n",
    "DATADIR = 'data/pubmed/Pubmed-Diabetes/data'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path\n",
    "import pandas as pd\n",
    "import itertools\n",
    "\n",
    "# process links\n",
    "edgelist = pd.read_csv(os.path.join(DATADIR, 'Pubmed-Diabetes.DIRECTED.cites.tab'), \n",
    "                      skiprows=2, header=None,delimiter='\\t')\n",
    "edgelist.drop(columns=[0,2], inplace=True)\n",
    "edgelist.columns = ['source', 'target']\n",
    "edgelist['source'] = edgelist['source'].map(lambda x: x.lstrip('paper:')) \n",
    "edgelist['target'] = edgelist['target'].map(lambda x: x.lstrip('paper:'))\n",
    "edgelist.head()\n",
    "edgelist.to_csv('/tmp/pubmed-edges.tab', sep='\\t', header=None, index=False )\n",
    "\n",
    "# process nodes and their attributes\n",
    "nodes_as_dict = []\n",
    "with open(os.path.join(os.path.expanduser(DATADIR), \"Pubmed-Diabetes.NODE.paper.tab\")) as fp:\n",
    "    for line in itertools.islice(fp, 2, None):\n",
    "        line_res = line.split(\"\\t\")\n",
    "        pid = line_res[0]\n",
    "        feat_name = ['pid'] + [l.split(\"=\")[0] for l in line_res[1:]][:-1] # delete summary\n",
    "        feat_value = [l.split(\"=\")[1] for l in line_res[1:]][:-1] # delete summary\n",
    "        feat_value = [pid] + [ float(x) for x in feat_value ] # change to numeric from str\n",
    "        row = dict(zip(feat_name, feat_value))\n",
    "        nodes_as_dict.append(row)\n",
    "colnames = set()\n",
    "for row in nodes_as_dict:\n",
    "    colnames.update(list(row.keys()))\n",
    "colnames = list(colnames)\n",
    "colnames.sort()\n",
    "colnames.remove('label')\n",
    "colnames.append('label')\n",
    "target_dict = {1:'Diabetes_Mellitus-Experimental', 2: 'Diabetes_Mellitus-Type_1', 3:'Diabetes_Mellitus-Type_2', }\n",
    "with open('/tmp/pubmed-nodes.tab', 'w') as fp:\n",
    "    #fp.write(\"\\t\".join(colnames)+'\\n')\n",
    "    for row in nodes_as_dict:\n",
    "        feats = []\n",
    "        for col in colnames:\n",
    "            feats.append(row.get(col, 0.0))\n",
    "        feats = [str(feat) for feat in feats]\n",
    "        feats[-1] = round(float(feats[-1]))\n",
    "        feats[-1] = target_dict[feats[-1]]\n",
    "        fp.write(\"\\t\".join(feats) + '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 1: Load and Preprocess Data\n",
    "\n",
    "We will hold out 20% of the nodes as test nodes by setting `holdout_pct=0.2`.  Since we specified `holdout_for_inductive=True`, these heldout nodes are removed from the graph in order to later simulate making predicitions on new nodes added to the graph later (or *inductive inference*). If `holdout_for_inductive=False`, the features (not labels) of these nodes are accessible to the model during training.  Of the remaining nodes, 5% will be used for training and the remaining nodes will be used for validation (or *transductive inference*).  More information on transductive and inductive inference and the return values `df_holdout` and `df_complete` are provided below.\n",
    "\n",
    "Note that if there are any unlabeled nodes in the graph, these will be automatically used as heldout nodes for which predictions can be made once the model is trained.  See the [twitter example notebook](https://github.com/amaiya/ktrain/blob/master/examples/graphs/hateful_twitter_users-GraphSAGE.ipynb) for an example of this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Largest subgraph statistics: 19717 nodes, 44327 edges\n",
      "Size of training graph: 15774 nodes\n",
      "Training nodes: 788\n",
      "Validation nodes: 14986\n",
      "Nodes treated as unlabeled for testing/inference: 3943\n",
      "Size of graph with added holdout nodes: 19717\n",
      "Holdout node features are not visible during training (inductive_inference)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(train_data, val_data, preproc, \n",
    " df_holdout, G_complete)        = gr.graph_nodes_from_csv('/tmp/pubmed-nodes.tab',\n",
    "                                           '/tmp/pubmed-edges.tab',\n",
    "                                           sample_size=10, holdout_pct=0.2, holdout_for_inductive=True,\n",
    "                                           train_pct=0.05, sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `preproc` object includes a reference to the training graph and a dataframe showing the features and target for each node in the graph (both training and validation nodes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Diabetes_Mellitus-Type_1          6255\n",
       "Diabetes_Mellitus-Type_2          6242\n",
       "Diabetes_Mellitus-Experimental    3277\n",
       "Name: target, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preproc.df.target.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 2: Build a Model and Wrap in Learner Object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "graphsage: GraphSAGE:  https://arxiv.org/pdf/1706.02216.pdf\n"
     ]
    }
   ],
   "source": [
    "gr.print_node_classifiers()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is Multi-Label? False\n",
      "done\n"
     ]
    }
   ],
   "source": [
    "learner = ktrain.get_learner(model=gr.graph_node_classifier('graphsage', train_data), \n",
    "                             train_data=train_data, \n",
    "                             val_data=val_data, \n",
    "                             batch_size=64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 3:  Estimate LR \n",
    "Given the small number of batches per epoch, a larger number of epochs is required to estimate the learning rate. We will cap it at 100 here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "simulating training for different learning rates... this may take a few moments...\n",
      "Epoch 1/100\n",
      "12/12 [==============================] - 1s 85ms/step - loss: 1.1021 - acc: 0.3638\n",
      "Epoch 2/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 1.0971 - acc: 0.3743\n",
      "Epoch 3/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 1.1027 - acc: 0.3324\n",
      "Epoch 4/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 1.1024 - acc: 0.3402\n",
      "Epoch 5/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 1.1034 - acc: 0.3294\n",
      "Epoch 6/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 1.0960 - acc: 0.3763\n",
      "Epoch 7/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 1.0965 - acc: 0.3534\n",
      "Epoch 8/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 1.1091 - acc: 0.3430\n",
      "Epoch 9/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 1.1116 - acc: 0.3320\n",
      "Epoch 10/100\n",
      "12/12 [==============================] - 0s 31ms/step - loss: 1.0876 - acc: 0.3856\n",
      "Epoch 11/100\n",
      "12/12 [==============================] - 0s 31ms/step - loss: 1.0977 - acc: 0.3698\n",
      "Epoch 12/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 1.1057 - acc: 0.3329\n",
      "Epoch 13/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 1.1049 - acc: 0.3359\n",
      "Epoch 14/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 1.1036 - acc: 0.3468\n",
      "Epoch 15/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 1.0954 - acc: 0.3613\n",
      "Epoch 16/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 1.1098 - acc: 0.3744\n",
      "Epoch 17/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 1.0955 - acc: 0.3547\n",
      "Epoch 18/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 1.1050 - acc: 0.3443\n",
      "Epoch 19/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 1.0893 - acc: 0.3744\n",
      "Epoch 20/100\n",
      "12/12 [==============================] - 0s 40ms/step - loss: 1.0921 - acc: 0.3665\n",
      "Epoch 21/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 1.0996 - acc: 0.3600\n",
      "Epoch 22/100\n",
      "12/12 [==============================] - 0s 29ms/step - loss: 1.1018 - acc: 0.3390\n",
      "Epoch 23/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 1.0971 - acc: 0.3555\n",
      "Epoch 24/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 1.0936 - acc: 0.3613\n",
      "Epoch 25/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 1.0910 - acc: 0.3795\n",
      "Epoch 26/100\n",
      "12/12 [==============================] - 0s 26ms/step - loss: 1.0853 - acc: 0.3953\n",
      "Epoch 27/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 1.0881 - acc: 0.3613\n",
      "Epoch 28/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 1.0881 - acc: 0.3756\n",
      "Epoch 29/100\n",
      "12/12 [==============================] - 0s 38ms/step - loss: 1.0866 - acc: 0.3808\n",
      "Epoch 30/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 1.0757 - acc: 0.4281\n",
      "Epoch 31/100\n",
      "12/12 [==============================] - 0s 31ms/step - loss: 1.0709 - acc: 0.4267\n",
      "Epoch 32/100\n",
      "12/12 [==============================] - 0s 31ms/step - loss: 1.0838 - acc: 0.3919\n",
      "Epoch 33/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 1.0801 - acc: 0.3847\n",
      "Epoch 34/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 1.0669 - acc: 0.4266\n",
      "Epoch 35/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 1.0600 - acc: 0.4369\n",
      "Epoch 36/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 1.0477 - acc: 0.4766\n",
      "Epoch 37/100\n",
      "12/12 [==============================] - 0s 41ms/step - loss: 1.0432 - acc: 0.4712\n",
      "Epoch 38/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 1.0141 - acc: 0.5523\n",
      "Epoch 39/100\n",
      "12/12 [==============================] - 0s 28ms/step - loss: 1.0296 - acc: 0.4856\n",
      "Epoch 40/100\n",
      "12/12 [==============================] - 0s 31ms/step - loss: 1.0022 - acc: 0.5169\n",
      "Epoch 41/100\n",
      "12/12 [==============================] - 0s 37ms/step - loss: 0.9882 - acc: 0.5443\n",
      "Epoch 42/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.9765 - acc: 0.5565\n",
      "Epoch 43/100\n",
      "12/12 [==============================] - 0s 36ms/step - loss: 0.9715 - acc: 0.5536\n",
      "Epoch 44/100\n",
      "12/12 [==============================] - 0s 39ms/step - loss: 0.9235 - acc: 0.6191\n",
      "Epoch 45/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.9103 - acc: 0.6361\n",
      "Epoch 46/100\n",
      "12/12 [==============================] - 0s 40ms/step - loss: 0.8570 - acc: 0.6875\n",
      "Epoch 47/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 0.8375 - acc: 0.7134\n",
      "Epoch 48/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 0.7789 - acc: 0.7579\n",
      "Epoch 49/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 0.7346 - acc: 0.7943\n",
      "Epoch 50/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 0.6784 - acc: 0.8115\n",
      "Epoch 51/100\n",
      "12/12 [==============================] - 0s 41ms/step - loss: 0.6411 - acc: 0.8181\n",
      "Epoch 52/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 0.5853 - acc: 0.8338\n",
      "Epoch 53/100\n",
      "12/12 [==============================] - 0s 29ms/step - loss: 0.5497 - acc: 0.8469\n",
      "Epoch 54/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 0.4958 - acc: 0.8587\n",
      "Epoch 55/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 0.4363 - acc: 0.8861\n",
      "Epoch 56/100\n",
      "12/12 [==============================] - 0s 36ms/step - loss: 0.3972 - acc: 0.8927\n",
      "Epoch 57/100\n",
      "12/12 [==============================] - 0s 29ms/step - loss: 0.3773 - acc: 0.8737\n",
      "Epoch 58/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 0.3735 - acc: 0.8652\n",
      "Epoch 59/100\n",
      "12/12 [==============================] - 0s 39ms/step - loss: 0.3351 - acc: 0.8974\n",
      "Epoch 60/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.3001 - acc: 0.9097\n",
      "Epoch 61/100\n",
      "12/12 [==============================] - 0s 35ms/step - loss: 0.2728 - acc: 0.9215\n",
      "Epoch 62/100\n",
      "12/12 [==============================] - 0s 31ms/step - loss: 0.2761 - acc: 0.9128: 0s - loss: 0.2342 - acc: 0.\n",
      "Epoch 63/100\n",
      "12/12 [==============================] - 0s 39ms/step - loss: 0.2826 - acc: 0.9071\n",
      "Epoch 64/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.1876 - acc: 0.9372\n",
      "Epoch 65/100\n",
      "12/12 [==============================] - 0s 28ms/step - loss: 0.2418 - acc: 0.9163\n",
      "Epoch 66/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.2193 - acc: 0.9254\n",
      "Epoch 67/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 0.2385 - acc: 0.9175\n",
      "Epoch 68/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 0.2542 - acc: 0.9045\n",
      "Epoch 69/100\n",
      "12/12 [==============================] - 0s 35ms/step - loss: 0.2287 - acc: 0.9071\n",
      "Epoch 70/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.2167 - acc: 0.9245\n",
      "Epoch 71/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 0.1879 - acc: 0.9342\n",
      "Epoch 72/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.2314 - acc: 0.9128\n",
      "Epoch 73/100\n",
      "12/12 [==============================] - 0s 40ms/step - loss: 0.2224 - acc: 0.9319\n",
      "Epoch 74/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 0.2177 - acc: 0.9267\n",
      "Epoch 75/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.2174 - acc: 0.9241\n",
      "Epoch 76/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.2491 - acc: 0.9149\n",
      "Epoch 77/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.2436 - acc: 0.9136\n",
      "Epoch 78/100\n",
      "12/12 [==============================] - 0s 28ms/step - loss: 0.1898 - acc: 0.9293\n",
      "Epoch 79/100\n",
      "12/12 [==============================] - 0s 31ms/step - loss: 0.1675 - acc: 0.9307\n",
      "Epoch 80/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 0.1949 - acc: 0.9294\n",
      "Epoch 81/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.1895 - acc: 0.9385\n",
      "Epoch 82/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 0.2656 - acc: 0.9084\n",
      "Epoch 83/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 0.2249 - acc: 0.9232\n",
      "Epoch 84/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 0.2740 - acc: 0.8953\n",
      "Epoch 85/100\n",
      "12/12 [==============================] - 0s 36ms/step - loss: 0.2511 - acc: 0.9019\n",
      "Epoch 86/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.2529 - acc: 0.9084\n",
      "Epoch 87/100\n",
      "12/12 [==============================] - 0s 33ms/step - loss: 0.2575 - acc: 0.8992\n",
      "Epoch 88/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.2617 - acc: 0.9097\n",
      "Epoch 89/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.3072 - acc: 0.9097\n",
      "Epoch 90/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.4338 - acc: 0.8692\n",
      "Epoch 91/100\n",
      "12/12 [==============================] - 0s 27ms/step - loss: 0.5869 - acc: 0.8324\n",
      "Epoch 92/100\n",
      "12/12 [==============================] - 0s 35ms/step - loss: 0.8302 - acc: 0.8206\n",
      "Epoch 93/100\n",
      "12/12 [==============================] - 0s 34ms/step - loss: 1.0682 - acc: 0.7723\n",
      "Epoch 94/100\n",
      "12/12 [==============================] - 0s 30ms/step - loss: 0.8207 - acc: 0.8390\n",
      "Epoch 95/100\n",
      "12/12 [==============================] - 0s 35ms/step - loss: 0.6389 - acc: 0.8260\n",
      "Epoch 96/100\n",
      "12/12 [==============================] - 0s 32ms/step - loss: 0.8993 - acc: 0.8521\n",
      "Epoch 97/100\n",
      " 8/12 [===================>..........] - ETA: 0s - loss: 1.2450 - acc: 0.8379\n",
      "\n",
      "done.\n",
      "Please invoke the Learner.lr_plot() method to visually inspect the loss plot to help identify the maximal learning rate associated with falling loss.\n"
     ]
    }
   ],
   "source": [
    "learner.lr_find(max_epochs=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8ddnJnvIQhYChEDYFRFFA4oLbuB+RVutWmu11VrbWrXa3np/9XbvbXutbdVqq612d629lLqAYkUUFxYBZZF9C1sSlpB9mfn+/piBBgyQQE7OTOb9fDzycObMycybA+adc77nnK855xARkcQV8DuAiIj4S0UgIpLgVAQiIglORSAikuBUBCIiCU5FICKS4JL8DtBZBQUFrrS01O8YIiJxZcGCBVXOucL2Xou7IigtLWX+/Pl+xxARiStmtuFgr+nQkIhIglMRiIgkOBWBiEiCUxGIiCQ4FYGISIJTEYiIJLiEL4JFm3azaWf9Ub9PKOxoCYVpag3R3BrugmQiIt0j7q4jOFK76pr5zrSlLNiwixNKcrjulEHc8fRCqmqbARgzIIfm1jAfbavhE2OLWVNZS0pSgCEFvQgE4NPjB7G7oZlV22upb25l/OB86ptbmbF0O7vrm3lv3U7qm1sxjNZwmDOGFXD52GIyU5LYuqeR8l31TBiST1ZaEht21LOmspblW2vYUdtE78wUlmyuJistmT5ZqaQkBchMSWJwYSYZyUFCzrF0yx427aznhAG5nDQol8qaJlKTgqSnBFmwYReLN+2mT3Yq14wbyPnHFZGeHMTMcM5hZu1uk72vtYTCGJAUTPjfC0QSksXbxDRlZWXuSC4ou2/GRzz8+pqPLZ84opB31+6gNRSmKDuNrdWNAKQkBRha2IvlW/cc9r1L8zPITk9mYF4Gza1hqhtaWLhp92H3DEYU9aJ3RgrVDS2U5mfS0BJi0656dtQ2kxQwdtQ171t3SGEmuenJrNhWQ11zaL/3SU0KMGZADu9v3E0o7AgGIj/48zNTaGwJ0dgapldqEqlJAWqbWsnLTGHL7gZaQo7koNEScphBcW76vj93dX0LJXkZnD2ykFMG59EnO+2w20FEYpeZLXDOlbX7WqIUQXNrmHnrd1JV28ST720kIyXI/3ziePrlpFPd0EJza5jCrFQaW0IEA0Zza5jM1CSq61t4fUUFTa0hUpOCjB2YSzBgvL6ikqSAMWFIPqUFmR/7vOr6FjburGf9jjp6Z6Rw4sBcXv+oglDYUdw7ndH9c0hPCR4289qqWvrnppOdlgxAY0uIzbsbKMpOo765lVDY0ScrjWAg8tv/7FVVzF23g8qaJrZWN5KaFKR3RjIOMKCuuZXaphD5mSlU1TaRk55M3+w0gkGjfGcDry7fTk56MpkpQSprmvaVTv+cNAIBIzstmaLsVNKSg+RlplDQK5XM1CAD8zIZ2TeLUNhRmJVKTnpyp/+ORMQ7KgLpsJZQmOToIaLGlhBLNlfz3rqdLNuyh2DA2Lankc27GqhvbmVXfUu77xEwyO+VyvjBefTNTmPMgBwmjyoiIyVhjkSKxJxDFYH+z5T9JLcZJ0hLDlJWmkdZad5B19/T2MKO2mbW76hj1fYaUpOCLNlcTUNLiPnrd7FtT+RQW2pSgNHFOTjnyEpLpraplXVVdQwtzKSsNI8Lj+tLaX4mORnakxDpbtojEE81t4aZsXQbry7bzqqKWvIyk6mqaSYtJUhywPhoWw21Ta371u+dkcxx/XOYdGwfxg7szZgBOQcd7BaRjtOhIYlZzjm27WnkteUV1DS2smp7DR9srmZ1RS0AV548gP+8YKQGq0WOkg4NScwyM/rlpPOZUwftW+acY+X2Wp5/v5zHZq/lbwvKGdA7nctO6M9pQwuYMDR/35lRInL0tEcgMW11RS3Tl2xl7vpdvLmqEufg+OIcvnLOUC44rq8OG4l0kA4NSY9QUdPI8ws284uZK2luDXP6sHx+dPnx7Z6+KyL7O1QR6FJSiRt9stL40tlDWfLdC/jeZcfxwaZqLvjlbKYu3Ox3NJG4piKQuJOSFOCG00qZefdZnFiSy53PLOJHLy6jfNfR3zNKJBHp0JDEtabWEN/82wdMXbSFgEFhViqfOGkAt587/LBXboskEh0akh4rNSnIL64+kel3nsm14wfSJyuNX89awxWPzGFdVZ3f8UTigvYIpMeZuWw7dz+3mIbmELedO4xbzxpKSpJ+55HEpj0CSSiTRhXx0h1ncu4xffj5qyu55ME3WV1R43cskZilIpAeqTg3nV9/5iQevf5kync1cO1v32P2ykq/Y4nEJBWB9FhmxgXH9eXvXz6NXqlJfPaJuYz41svMW7/T72giMUVFID3esf2ymX7nmdw5aTjNoTCfevQdFmxQGYjspSKQhJCaFOTOSSOYc8+5lOZncuMT83h+QTmtIc0vLeJZEZjZE2ZWYWZLDvK6mdmDZrbazD4ws5O8yiKyV3FuOn+9+RTyeqVw93OLOf8Xs9m4QxeiSWLzco/gD8CFh3j9ImB49OsW4NceZhHZp39uOjPvOotfX3cSW6obmHjf66yprPU7lohvPCsC59xs4FAHYqcAf3IR7wK5ZtbPqzwibSUHA1x0fD8ev2EcSQHjJy9/RLxdUyPSVfwcIygGNrV5Xh5d9jFmdouZzTez+ZWVOgVQus7pwwr4+gUjeXXZdn46fYXfcUR8ERcT0zjnHgMeg8iVxT7HkR7mixOHUL6rnt+8sYZgAO6ePJKAJr6RBOJnEWwGSto8HxBdJtKtzIzvXTaaxpYwD7++huRggDsnjfA7lki38bMIpgG3mdnTwClAtXNuq495JIEFA8Z9V46hNRTmlzNXkZIU4MtnD/M7lki38KwIzOwp4GygwMzKge8AyQDOud8ALwEXA6uBeuBzXmUR6Qgz4/5PnUhL2HH/KysZW9KbCUPz/Y4l4jndfVTkANUNLXzy129TVdvEK3dOpE92mt+RRI6a7j4q0gk56ck8/OmTqG8KcduTC3VaqfR4KgKRdozsm8V3LhvF3PU7+ceiLX7HEfGUikDkIK4dN5ATBuTwPy8tp7ap1e84Ip5REYgcRCBgfG/KaCpqmnjotVV+xxHxjIpA5BBOLMnlU2UDePytdWzf0+h3HBFPqAhEDuMr5wzDDL47bakGjqVHUhGIHMag/EzunDSCl5ds49HZa/2OI9LlVAQiHfCls4Yy6dgiHnxtFRU1OkQkPYuKQKQDAgHjW5ccS3NrmAdmauBYehYVgUgHDS7I5NOnDOTpeZs0kY30KCoCkU64/bzhpCUFuE9zF0gPoiIQ6YSCXqncMnEo05duY/76Q03AJxI/VAQinfSFiYPpm53G919YptNJpUdQEYh0UkZKEl+bPJwPyqt5c1WV33FEjpqKQOQIXDF2AL0zknlm3qbDrywS41QEIkcgJSnAlBOLeXXZdnbUNvkdR+SoqAhEjtBnTh1ISzjMY2/qamOJbyoCkSM0rE8Wl59YzB/fXs+mnfV+xxE5YioCkaNw1+QRGMYDuk21xDEVgchRKMnL4NIx/Xj5w62avEbilopA5Chdd+og6ppDPKsziCROqQhEjtKJJbmUDerNE3PW0RoK+x1HpNNUBCJd4OYzh1C+q4GZyyv8jiLSaSoCkS4w6dg+FPRK4Z+Lt/gdRaTTVAQiXSApGODSMf15Zdk2tlY3+B1HpFNUBCJd5KYzBtMScjy/oNzvKCKdoiIQ6SIleRlMGJLPcwvKdVdSiSsqApEudFXZADbsqGfuOs1VIPFDRSDShS4a3Y9eqUk8O1+HhyR+qAhEulB6SpD/OKEfL364hS27NWgs8UFFINLFbj5zCI0tYaYv2eZ3FJEOURGIdLGhhb0YlJ/B22s0e5nEBxWBiAfOO6aIN1ZWUlHT6HcUkcNSEYh44PoJg2gJOZ58b6PfUUQOS0Ug4oHBBZmcPbKQv7y7kcaWkN9xRA7J0yIwswvNbIWZrTaze9p5faCZvW5mC83sAzO72Ms8It3pljOHUFXbxPPv61RSiW2eFYGZBYGHgYuAUcC1ZjbqgNXuBZ51zo0FrgEe8SqPSHebMDSf0cXZOjwkMc/LPYLxwGrn3FrnXDPwNDDlgHUckB19nAPo1o3SY5gZV4wdwNIte1hdUet3HJGD8rIIioG2UzaVR5e19V3gM2ZWDrwEfLW9NzKzW8xsvpnNr6ys9CKriCcuHdMPgJc+3OpzEpGD83uw+FrgD865AcDFwJ/N7GOZnHOPOefKnHNlhYWF3R5S5EgVZadRNqi3ikBimpdFsBkoafN8QHRZWzcBzwI4594B0oACDzOJdLuLj+/HR9tqWFupw0MSm7wsgnnAcDMbbGYpRAaDpx2wzkbgPAAzO5ZIEejYj/QoFx3fF9DhIYldnhWBc64VuA2YASwncnbQUjP7vpldFl3tbuALZrYYeAq40elG7tLD9MtJ5+RBvXnxQ917SGJTkpdv7px7icggcNtl327zeBlwupcZRGLBxcf34wcvLGNdVR2DCzL9jiOyH78Hi0USwkWjdXhIYpeKQKQb9M9NZ+zAXF78QEUgsUdFINJNLjm+H8u27mF9VZ3fUUT2oyIQ6SYXHd+PgMFf3t3gdxSR/agIRLpJcW46U04s5sm5G2lq1R1JpXO8PKFSRSDSjS4+vh/1zSEWbNjldxSJI3PX7WTkvdP5/Zx1nry/ikCkG00Ymk9KMMC/llf4HUXiyNrKWppDYfrnpnvy/ioCkW7UKzWJM4YXMH3pNk939aVn2Tu50bjSPE/eX0Ug0s0uPK4v5bsaWLplj99RJE40toYBSEv25ke2ikCkm517bB8Anpu/6TBrikTs3SNISwp68v4qApFuVtArlctP7M+z88upa2r1O47EgYaWECnBAIGAefL+KgIRH1w7fiANLSFeXbbd7ygSB5pawqR6dFgIVAQivhhXmkf/nDSmLjpwig6Rj2tsCZGW7M1hIVARiPgiEDAuPaE/c1ZXUVXb5HcciXH1zSEyUlQEIj3OVScPoCXkeGaeBo3l0GoaW8hK827WABWBiE+GF2VxQkkur2icQA6jprGVrNRkz95fRSDio8nH9mHxpt1U7Gn0O4rEqIbmEPM37KK+xbv7U6kIRHw0eVRkwpqZuuWEHMTeEwqqarwbS1IRiPhoRFEvSvLSeXWZ5jOW9lVGC+CvN5/i2WeoCER8ZGZMPrYvc9bs0MVl0q73N0buVFvq4VzXHSoCM7vDzLIt4nEze9/MzvcslUgCmTSqD82tYd5cVel3FIkx4bBj1grv/110dI/g8865PcD5QG/geuAnnqUSSSDjSvPISU/m1WUaJ5D97ahr7pbP6WgR7L3BxcXAn51zS9ssE5GjkBwMcM7IQv710XZaQ2G/40gM2VrdAMB/XzrK08/paBEsMLNXiBTBDDPLAvQvVqSLTB7Vl131LZq5TPazq74FgBMG5Hj6OR0tgpuAe4Bxzrl6IBn4nGepRBLMxBEFpAQDPDV3o99RJEaEw44bnpgLQHa6dxeTQceLYAKwwjm328w+A9wLVHsXSySxZKUlc9OZg5m6aAsrttX4HUe6WUsozNrKWpZv3cPSLZEfrRVtrhvITouNIvg1UG9mJwB3A2uAP3mWSiQBfe60Usxg+hJdU5Bovv2PJZx7/xtc9MCbXPLgW6zcXsOGHXX7XvfyPkPQ8SJodZEJVqcAv3LOPQxkeRdLJPH0yU5jdP8c5qyp8juKdKOG5hBPzd3/xoNf+ssCNu6sByJTm3p551HoeBHUmNl/ETlt9EUzCxAZJxCRLnTG8ALmrtvJyu06PJQo2jtBYE1lHU/P20TA4MFrx2Lm7UmaHS2Cq4EmItcTbAMGAPd5lkokQd10xmCCAWPqQk1Ykyj2/ua/1+8/Nw6IFETYQUqS9zeA6NAnRH/4/xXIMbNLgUbnnMYIRLpYQa9UThuar3GCBFLTGDlF9J3/Opfpd57JOSP7dHuGjt5i4lPAXOAq4FPAe2Z2pZfBRBLV+aOKWFtVx+qKWr+jSDeobWolYNA3O41j+mYD8M0LjyEYMJ67dUK3ZOjoPse3iFxDcINz7rPAeOC/vYslkrgmjSoC4BXdkTQhTFu8hbBjv3GAL509lDX/czHjSvO6JUNHiyDgnGt7I5QdnfheEemEfjnpjBmQw6uauazHC4cdG3bUH35Fj3X0h/l0M5thZjea2Y3Ai8BL3sUSSWyTjy1ikWYu6/Eqa72bbKYzOjpY/A3gMWBM9Osx59w3D/d9Znahma0ws9Vmds9B1vmUmS0zs6Vm9mRnwov0VJOPK8I5zVzW0+09Y+jxG8p8zdHhy9Wcc88Dz3d0fTMLAg8Dk4FyYJ6ZTXPOLWuzznDgv4DTnXO7zKz7h8tFYtDIoqx9M5d9+pSBfscRj2yKFsGgfO8mnemIQ+4RmFmNme1p56vGzPYc5r3HA6udc2udc83A00SuTG7rC8DDzrldAAeMQ4gkLDPj/FF9mbN6B7WauazH2rQzcpvpAb3Tfc1xyCJwzmU557Lb+cpyzmUf5r2LgbbXTZdHl7U1AhhhZnPM7F0zu7C9NzKzW8xsvpnNr6zULE6SGCaPKqI5FOaNbpihSvyxaVc9RdmppCV7ewuJw/H7zJ8kYDhwNnAt8Fszyz1wJefcY865MudcWWFhYTdHFPFH2aDe9M5I1sT2PdiSzdUMKejldwxPi2AzUNLm+YDosrbKgWnOuRbn3DpgJZFiEEl4ScEA5x1bxGvLK2hqDfkdR7pYKOz4aFsN40p7+x3F0yKYBww3s8FmlgJcA0w7YJ2pRPYGMLMCIoeK1nqYSSSuXDKmHzVNrcxeqTuS9jQNLZFy7+XxLaY7wrMicM61ArcBM4DlwLPOuaVm9n0zuyy62gxgh5ktA14HvuGc2+FVJpF4c8awAnIzknnhgy1+R5EuVt8cOQkg3efxAejE6aNHwjn3EgdceOac+3abxw64K/olIgdIDga4aHRf/rFoCw3NIdI9vi+9dJ/G5si07+kpPXiPQES6xqVj+lPfHOL1FTq7uiepb4mdPQIVgUiMO3VIPlmpScxZrXGCnqShOTJG4PXsYx2hIhCJccGAMaYkh0WbdvsdRbrQrvpmQEUgIh100sDeLN2yh4UbPz6tocSnBRt2kRQwjh+Q43cUFYFIPLjpjMFkpyXx53c3+B1Fusj6HfXkZqSQEQODxf4nEJHDys1I4Zxj+vDGikrCYUcg4O1k5uKthuYQL36w1e8Y+2iPQCROnHtMH3bUNbO4XGMF8W7vPMWxQkUgEifOGlFIwOBfH+k00ni396riE0s+dms1X6gIROJEbkYKJw/qzavLthO5FlPi1d4i+OLEIT4niVARiMSRy8cW89G2GhbqVNK4Vh+9hiAtBk4dBRWBSFy5dEx/eqUm8eBrq/yOIkehMVoEsXBVMagIROJKTnoyt541hFkrKvdNcyjxpz6GrioGFYFI3JlyYmSivz+9s97XHHLkdjdEzhrKTkv2OUmEikAkzpTkZXD6sHx+++Y65q3f6XccOQJbdkfmKu6bk+ZzkggVgUgc+sGU0QA8+obmcYpH2/c00jsj2fe5ivdSEYjEoSGFvbhodF9mLt/Oxh0aK4g3NY2t5KTHxmEhUBGIxK3/vPAYQGMF8aiuqZXM1Ni5w4+KQCRODS7I5NIx/fjjO+t1BlGcqVURiEhXufWsoZgZP3pxud9RpBNqm1rJUhGISFcYXZzD1WUlvLGyksbobQsktrWEwmzZ3UBOhsYIRKSLTBpVRENLiLfXaCrLeLBhRx276ls4bWiB31H2URGIxLlTBueRn5nCb2bpVNJ4UFkTmaKyX4xcQwAqApG4l5Yc5MvnDGPu+p0s2Vztdxw5jK3VkYvJCnql+pzk31QEIj3AlScPIDUpwFNzN/odRQ7jn4u30DsjmYF5GX5H2UdFINID5KQnc8nx/Zi2aAv1za1+x5GDcM7x5qoqLjuhP+kxcsM5UBGI9BhXjyuhpqmVfy7e4ncUOYhHZ6+lNezon5vud5T9qAhEeojxg/MYlJ/Bq8s0lWWsWrZlDwAXHNfX5yT7UxGI9BBmxmlD8/nXR9tZU1nrdxxpR8g5hhRkUlqQ6XeU/agIRHqQL04cSlIwwM9mrPA7irRj6+6GmLn1dFsqApEepLQgk1vOHML0pdtYX1Xndxw5wLbqRvrlxNb4AKgIRHqc6ycMwoBn52/yO4q0saO2iW17GinJUxGIiMeKstM4e2QfHpm1RnsFMeRP72wg7OCi0f38jvIxKgKRHujmMwcDMHP5dp+TyF6vLtvO6OJsRvbN8jvKx6gIRHqg04YWMKQwU0UQQ6obWhhZlO13jHZ5WgRmdqGZrTCz1WZ2zyHW+6SZOTMr8zKPSCK56uQS3l27k1krdF1BLNhd30xuDN16ui3PisDMgsDDwEXAKOBaMxvVznpZwB3Ae15lEUlEn50wiIF5GXz9ucU45/yOk9CaW8PUNYfIjaF5itvyco9gPLDaObfWOdcMPA1MaWe9HwA/BRo9zCKScDJTk7hmfAlVtc38bUG533ESWnVDC0Di7REAxUDb89fKo8v2MbOTgBLn3Ise5hBJWJ8/PTJo/PS8Tdor8FF1Q2QOgpyMFJ+TtM+3wWIzCwA/B+7uwLq3mNl8M5tfWVnpfTiRHiItOchXzx3Ggg27+NdHGivwy+766B5BAh4a2gyUtHk+ILpsryxgNDDLzNYDpwLT2hswds495pwrc86VFRYWehhZpOe5/bzhZKUlMX3JNr+jJKzVFZF7P+VlJt4ewTxguJkNNrMU4Bpg2t4XnXPVzrkC51ypc64UeBe4zDk338NMIgknORhg4vBCZq2spDUU9jtOQlpcvpuc9GRG9Uuw00edc63AbcAMYDnwrHNuqZl938wu8+pzReTjLh9bTGVNE394e73fURJSbVOI3hnJBALmd5R2JXn55s65l4CXDlj27YOse7aXWUQS2XnH9KFsUG9++OJyRhRlMXGEDrF2p7qmVjJTPf1xe1R0ZbFIAggEjD/dNJ7S/Aw++8RcFmzY6XekhFLb1EovFYGI+C0jJYnf3RA5F+Mv72qS++5UpyIQkVgxrE8W1586iH8u3sJ7a3f4HSdhVNU2kRuj1xCAikAk4XzjwpEMzMvgK0++z9bqBr/j9Hh7GlvYvqeJYX16+R3loFQEIgkmOy2Zxz57MjWNrdw3XVNaem3vNQQqAhGJKcP6ZHHj6aX8feFmfvzSct1+wkMqAhGJWV8/fyQleek8Onstb66q8jtOj7WmopaUYICS3rE3ReVeKgKRBJUcDDDzrrMo6JXKE3PW+R2nx1pdUcvggkySgrH74zZ2k4mI51KTglx/6iBmrahkneY39sTqytqYPiwEKgKRhHf1uBICBs/N33T4laVTGltCbNpZz1AVgYjEsr45aUweVcSf3tlAi25K16XeWlVF2MExMThhfVsqAhHhirHF1Da18uYqzffRlZZsqQbg3GP6+Jzk0FQEIsK5xxRRnJvOAzNX6VTSLrSzrpmc9GTSkoN+RzkkFYGIkJIU4PbzhrG4vJoHX1vtd5weIRx2PD1vU8zOU9yWikBEAPjESQMA+MXMlSzZXO1zmvj3wodbaW4NMyg/0+8oh6UiEBEgcl3BtNtOB+DSh95ibWWtz4ni2+1PLQTg/qtO8DnJ4akIRGSfMQNyGVIY+Q323Pvf0J7BEZq1ogKAIYWZFGal+pzm8FQEIrKfp75wKueMjMxgduPv5xIOa/C4s6Yt3gLA728c53OSjlERiMh+irLT+P3nxvPpUwZSVdvMpQ+9pTLopPJdDZQN6h0X4wOgIhCRg/j2paNISw6wbOseTvvJv/i/heV+R4oLs1dWMnfdzrgpAVARiMhBpCUHWfTt87libDHb9jTytWcWU7GnEYANO+q465lFbNpZ73PK2NLYEuIbf1sMwLXjS3xO03EqAhE5qLTkIL+4+kS+eeExAFz92Lss2VzNWffN4u8LN3PH0wt9Thg7WkNhLn3oLbbvaeL+q06grDTP70gdpiIQkcP60tlDufWsoayrquPSh94CYGhhJu9v3M2c1ZrLAGDG0u37JqH5xEnFPqfpHBWBiHTIf14wkpvPGEy/nDQeu/5knvniBHLSk7nud+8x5eE51De3+h3RV6HorTmGFGZiZj6n6RyLt/uKlJWVufnz5/sdQ0SAJZur9+0hADxy3UlMOraIlKTE+x3zikfmsHDjbuZ9a1JMXjtgZgucc2XtvZZ4f1si0mVGF+ew7PsXcP2pgwD48l/fZ8S9L/P03I0JdfO615ZvZ+HG3QDkZab4nKbzVAQiclQyUpL4weWjWfTtyftm4rrn7x9y618W0Jog8xvc9MfIUYrHbygjGIivw0IASX4HEJGeITcjhZl3ncUH5bu59rF3mbF0O5N+/gZTv3I6uRnx91vygeqaWlm6ZQ8765q54LgifvbKCl5bXkFNY2RsJD8zhTOHF/qc8shojEBEupxzjq8+tZAXPtjKkIJMpt85MW7HDZxzhB1c/vAcPjzEvZdm3DmRkTE8E9mhxgi0RyAiXc7M+NWnT6KuaS6vr6hkxL0vc9HovowuzuGWiUP4oHw33/q/JRzTN4sff2IM6SmxM3FLc2uYzbsb6JeThnPw0+kf8Ye317e77vx7J3HDE3M5oSQ3pkvgcLRHICKeqWtq5frH3+P96EBqe24/bzg3TBjEnsZWBhd8/LYMCzbs4pVl27hr8ghSgoF9h2T++PnxFGWndVnWzbsbWFtZy/WPz2339eSg8cA1Y7lodF+272miMCs1rsYDDrVHoCIQEc+9taqKxpYQD89azcKNu+mfk8avrjuJR15fzczlFfvWu+T4fgwuyGRnfTPnjypi6sLNTF0UuZNn2aDeVDe0sCp60daZwwt44sZxJAXsqM/bX19Vx9k/m7XfslMG5/Heup0ALPneBfRKje8DKCoCEYkZobDb95v06oparvzN2+yubznk93z57KE8MmsNAMW56UwcUcBTczcBkJoU4BsXjOTmM4cccaapCzdz5zOLyM1I5ppxA7n7/BEkBwNs2llPdUMLo4tzjvi9Y4WKQERi2rqqOgbmZfDLmSvJSU9md30Lj7+1DogMwg7Mz2DJ5lnzoGIAAAozSURBVGpCYUdx73TSk4P8/f1yHnhtNVW1TQA8fcupnDok/7Cf1dAcoqq2iU276qnY00RNYwv/O2MFjS0hPvzuBTE/0fyRUhGISNypqGkkKzX5sAPJK7bVcNEDswk7+MdXTueEktyDrrtxRz0XPjCb+ubQx1770RWjue6UQUedO1b5dmWxmV1oZivMbLWZ3dPO63eZ2TIz+8DMXjOznvu3ICKd0icrrUNnE43sm8XXJo0AYMrDc/jRi8v43ZtreXt1FU++F7nCORx2rNxew5SH39qvBIb36UVJXjq3TBzSo0vgcDwb/TCzIPAwMBkoB+aZ2TTn3LI2qy0Eypxz9Wb2JeB/gau9yiQiPdNXzhnGwPwM7nh6Eb99c91+r62qqOH3c9bve377ecO58bRSstOSSArG57UNXc3LYfDxwGrn3FoAM3samALsKwLn3Ott1n8X+IyHeUSkhwoEjCknFnPSwN7cO3UJyUFjZN8sfjt73X4lcO34Eu6aPMK/oDHKyyIoBja1eV4OnHKI9W8CXvYwj4j0cCV5Gfzx8+P3Pb/q5BLufGYRvTOS+eU1Y8lJT/YxXeyKiRNjzewzQBlw1kFevwW4BWDgwIHdmExE4llpQSZTv3K63zFinpcHyDYDbSftHBBdth8zmwR8C7jMOdfU3hs55x5zzpU558oKC+Pzpk4iIrHKyyKYBww3s8FmlgJcA0xru4KZjQUeJVICFe28h4iIeMyzInDOtQK3ATOA5cCzzrmlZvZ9M7ssutp9QC/gOTNbZGbTDvJ2IiLiEU/HCJxzLwEvHbDs220eT/Ly80VE5PB0Eq2ISIJTEYiIJDgVgYhIglMRiIgkuLi7+6iZVQIb/M7RxQqAKr9DxCltu6Oj7Xd04mn7DXLOtXshVtwVQU9kZvMPdntYOTRtu6Oj7Xd0esr206EhEZEEpyIQEUlwKoLY8JjfAeKYtt3R0fY7Oj1i+2mMQEQkwWmPQEQkwakIREQSnIpARCTBqQhimJkFzOxHZvaQmd3gd554ZGaZZjbfzC71O0u8MbPLzey3ZvaMmZ3vd55YF/239sfoNrvO7zydoSLwiJk9YWYVZrbkgOUXmtkKM1ttZvcc5m2mEJnZrYXInM8Jo4u2H8A3gWe9SRm7umL7OeemOue+ANwKXO1l3ljVye34CeBv0W122cfeLIbprCGPmNlEoBb4k3NudHRZEFgJTCbyg30ecC0QBH58wFt8Pvq1yzn3qJn9zTl3ZXfl91sXbb8TgHwgDahyzr3QPen91xXbb++sgWZ2P/BX59z73RQ/ZnRyO04BXnbOLTKzJ51zn/YpdqfFxOT1PZFzbraZlR6weDyw2jm3FsDMngamOOd+DHzs0IWZlQPN0ach79LGni7afmcDmcAooMHMXnLOhb3MHSu6aPsZ8BMiP9wSrgSgc9uRSCkMABYRZ0dbVATdqxjY1OZ5OXDKIdb/O/CQmZ0JzPYyWJzo1PZzzn0LwMxuJLJHkBAlcAid/ff3VWASkGNmw5xzv/EyXBw52HZ8EPiVmV0C/NOPYEdKRRDDnHP1wE1+54h3zrk/+J0hHjnnHiTyw006wDlXB3zO7xxHIq52X3qAzUBJm+cDosukY7T9jo62X9focdtRRdC95gHDzWywmaUA1wDTfM4UT7T9jo62X9focdtRReARM3sKeAcYaWblZnaTc64VuA2YASwHnnXOLfUzZ6zS9js62n5dI1G2o04fFRFJcNojEBFJcCoCEZEEpyIQEUlwKgIRkQSnIhARSXAqAhGRBKciEM+ZWW03fMZlHbwtdVd+5tlmdtoRfN9YM3s8+vhGM/tV16frPDMrPfB2y+2sU2hm07srk3QPFYHEjejtf9vlnJvmnPuJB595qPtxnQ10ugiA/0ec3sPHOVcJbDWz0/3OIl1HRSDdysy+YWbzzOwDM/tem+VTzWyBmS01s1vaLK81s/vNbDEwwczWm9n3zOx9M/vQzI6JrrfvN2sz+4OZPWhmb5vZWjO7Mro8YGaPmNlHZvaqmb2097UDMs4ys1+a2XzgDjP7DzN7z8wWmtlMMyuK3pr4VuBrZrbIzM6M/rb8fPTPN6+9H5ZmlgWMcc4tbue1UjP7V3TbvGZmA6PLh5rZu9E/7w/b28OyyOxYL5rZYjNbYmZXR5ePi26HxWY218yyop/zZnQbvt/eXo2ZBc3svjZ/V19s8/JUIK5m4JLDcM7pS1+efgG10f+eDzwGGJFfQl4AJkZfy4v+Nx1YAuRHnzvgU23eaz3w1ejjLwO/iz6+EfhV9PEfgOeinzGKyL3jAa4EXoou7wvsAq5sJ+8s4JE2z3vz76vwbwbujz7+LvD1Nus9CZwRfTwQWN7Oe58DPN/medvc/wRuiD7+PDA1+vgF4Nro41v3bs8D3veTwG/bPM8BUoC1wLjosmwidxzOANKiy4YD86OPS4El0ce3APdGH6cC84HB0efFwId+/7vSV9d96TbU0p3Oj34tjD7vReQH0WzgdjO7Irq8JLp8B5EJeZ4/4H3+Hv3vAiLTA7ZnqovMP7DMzIqiy84Anosu32Zmrx8i6zNtHg8AnjGzfkR+uK47yPdMAkZF5nMBINvMejnn2v4G3w+oPMj3T2jz5/kz8L9tll8effwk8LN2vvdD4H4z+ynwgnPuTTM7HtjqnJsH4JzbA5G9ByL3zT+RyPYd0c77nQ+MabPHlEPk72QdUAH0P8ifQeKQikC6kwE/ds49ut/CyExik4AJzrl6M5tFZHpJgEbn3IGzszVF/xvi4P+Gm9o8toOscyh1bR4/BPzcOTctmvW7B/meAHCqc67xEO/bwL//bF3GObfSzE4CLgZ+aGavAf93kNW/BmwnMpVnAGgvrxHZ85rRzmtpRP4c0kNojEC60wzg82bWC8DMis2sD5HfNndFS+AY4FSPPn8O8MnoWEERkcHejsjh3/ebv6HN8hogq83zV4jM6gVA9DfuAy0Hhh3kc94mcktjiByDfzP6+F0ih35o8/p+zKw/UO+c+wtwH3ASsALoZ2bjoutkRQe/c4jsKYSB64nMWXygGcCXzCw5+r0jonsSENmDOOTZRRJfVATSbZxzrxA5tPGOmX0I/I3ID9LpQJKZLScyR+67HkV4nsi0gsuAvwDvA9Ud+L7vAs+Z2QKgqs3yfwJX7B0sBm4HyqKDq8uIHM/fj3PuIyJTP2Yd+BqREvmcmX1A5Af0HdHldwJ3RZcPO0jm44G5ZrYI+A7wQ+dcM3A1kelOFwOvEvlt/hHghuiyY9h/72ev3xHZTu9HTyl9lH/vfZ0DvNjO90ic0m2oJaHsPWZvZvnAXOB059y2bs7wNaDGOfe7Dq6fATQ455yZXUNk4HiKpyEPnWc2kUnvd/mVQbqWxggk0bxgZrlEBn1/0N0lEPVr4KpOrH8ykcFdA3YTOaPIF2ZWSGS8RCXQg2iPQEQkwWmMQEQkwakIREQSnIpARCTBqQhERBKcikBEJMGpCEREEtz/B7Rex7EtugjrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.lr_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 4: Train the Model\n",
    "We will train the model using `autofit`, which uses a triangular learning rate policy.  The training will automatically stop when the validation loss no longer improves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "early_stopping automatically enabled at patience=5\n",
      "reduce_on_plateau automatically enabled at patience=2\n",
      "\n",
      "\n",
      "begin training using triangular learning rate policy with max lr of 0.01...\n",
      "Epoch 1/1024\n",
      "13/13 [==============================] - 6s 484ms/step - loss: 1.0057 - acc: 0.4807 - val_loss: 0.8324 - val_acc: 0.6990\n",
      "Epoch 2/1024\n",
      "13/13 [==============================] - 6s 425ms/step - loss: 0.8001 - acc: 0.7077 - val_loss: 0.6512 - val_acc: 0.7795\n",
      "Epoch 3/1024\n",
      "13/13 [==============================] - 6s 438ms/step - loss: 0.6322 - acc: 0.8045 - val_loss: 0.5574 - val_acc: 0.7875\n",
      "Epoch 4/1024\n",
      "13/13 [==============================] - 6s 430ms/step - loss: 0.5251 - acc: 0.8237 - val_loss: 0.5077 - val_acc: 0.8106\n",
      "Epoch 5/1024\n",
      "13/13 [==============================] - 6s 476ms/step - loss: 0.4407 - acc: 0.8600 - val_loss: 0.5061 - val_acc: 0.8086\n",
      "Epoch 6/1024\n",
      "13/13 [==============================] - 6s 454ms/step - loss: 0.3857 - acc: 0.8697 - val_loss: 0.5033 - val_acc: 0.8046\n",
      "Epoch 7/1024\n",
      "13/13 [==============================] - 6s 453ms/step - loss: 0.3682 - acc: 0.8528 - val_loss: 0.4966 - val_acc: 0.8058\n",
      "Epoch 8/1024\n",
      "13/13 [==============================] - 6s 462ms/step - loss: 0.3110 - acc: 0.8938 - val_loss: 0.4791 - val_acc: 0.8254\n",
      "Epoch 9/1024\n",
      "13/13 [==============================] - 6s 444ms/step - loss: 0.2822 - acc: 0.9035 - val_loss: 0.4873 - val_acc: 0.8160\n",
      "Epoch 10/1024\n",
      "13/13 [==============================] - 6s 443ms/step - loss: 0.2734 - acc: 0.9035 - val_loss: 0.4955 - val_acc: 0.8101\n",
      "\n",
      "Epoch 00010: Reducing Max LR on Plateau: new max lr will be 0.005 (if not early_stopping).\n",
      "Epoch 11/1024\n",
      "13/13 [==============================] - 6s 435ms/step - loss: 0.2361 - acc: 0.9264 - val_loss: 0.4898 - val_acc: 0.8214\n",
      "Epoch 12/1024\n",
      "13/13 [==============================] - 6s 498ms/step - loss: 0.2292 - acc: 0.9155 - val_loss: 0.5074 - val_acc: 0.8174\n",
      "\n",
      "Epoch 00012: Reducing Max LR on Plateau: new max lr will be 0.0025 (if not early_stopping).\n",
      "Epoch 13/1024\n",
      "13/13 [==============================] - 6s 442ms/step - loss: 0.1969 - acc: 0.9421 - val_loss: 0.5203 - val_acc: 0.8132\n",
      "Restoring model weights from the end of the best epoch\n",
      "Epoch 00013: early stopping\n",
      "Weights from best epoch have been loaded into model.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f673538af98>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.autofit(0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate\n",
    "\n",
    "#### Validate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                precision    recall  f1-score   support\n",
      "\n",
      "Diabetes_Mellitus-Experimental       0.76      0.82      0.79      3113\n",
      "      Diabetes_Mellitus-Type_1       0.84      0.81      0.82      5943\n",
      "      Diabetes_Mellitus-Type_2       0.85      0.84      0.85      5930\n",
      "\n",
      "                      accuracy                           0.83     14986\n",
      "                     macro avg       0.82      0.82      0.82     14986\n",
      "                  weighted avg       0.83      0.83      0.83     14986\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[2553,  362,  198],\n",
       "       [ 440, 4815,  688],\n",
       "       [ 359,  572, 4999]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learner.validate(class_names=preproc.get_classes())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a Predictor Object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = ktrain.get_predictor(learner.model, preproc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transductive Inference:  Making Predictions for Unlabeled Nodes in Original Training Graph\n",
    "In transductive inference, we make predictions for unlabeled nodes whose features are visible during training. Making predictions on validation nodes in the training graph is transductive inference.\n",
    "\n",
    "Let's see how well our prediction is for the first validation example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.04122107, 0.9422023 , 0.0165766 ]], dtype=float32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p.predict_transductive(val_data.ids[0:1], return_proba=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 0.])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_data[0][1][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make predictions for all validation nodes and visually compare some of them with ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = p.predict_transductive(val_data.ids, return_proba=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true = preproc.df[preproc.df.index.isin(val_data.ids)]['target'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ground Truth</th>\n",
       "      <th>Predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Diabetes_Mellitus-Type_1</td>\n",
       "      <td>Diabetes_Mellitus-Type_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Diabetes_Mellitus-Type_2</td>\n",
       "      <td>Diabetes_Mellitus-Type_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Diabetes_Mellitus-Type_1</td>\n",
       "      <td>Diabetes_Mellitus-Type_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Diabetes_Mellitus-Type_1</td>\n",
       "      <td>Diabetes_Mellitus-Type_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Diabetes_Mellitus-Experimental</td>\n",
       "      <td>Diabetes_Mellitus-Experimental</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Ground Truth                       Predicted\n",
       "0        Diabetes_Mellitus-Type_1        Diabetes_Mellitus-Type_1\n",
       "1        Diabetes_Mellitus-Type_2        Diabetes_Mellitus-Type_1\n",
       "2        Diabetes_Mellitus-Type_1        Diabetes_Mellitus-Type_1\n",
       "3        Diabetes_Mellitus-Type_1        Diabetes_Mellitus-Type_1\n",
       "4  Diabetes_Mellitus-Experimental  Diabetes_Mellitus-Experimental"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.DataFrame(zip(y_true, y_pred), columns=['Ground Truth', 'Predicted']).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inductive Inference:  Making Predictions for New Nodes Not in the Original Training Graph\n",
    "In inductive inference, we make predictions for entirely new nodes that were not present in the traning graph.  The features or attributes of these nodes were **not** visible during training.  We consider a graph where the heldout nodes are added back into the training graph, which yields the original graph of 19,717 nodes. This graph, `G_complete` was returned as the last return value of `graph_nodes_from_csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = p.predict_inductive(df_holdout, G_complete, return_proba=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true = df_holdout['target'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8303322343393356"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "(y_true == np.array(y_pred)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With an **83.03%** accuracy, we see that inductive performance is quite good and comparable to transductive performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}