DeepL4Patent Pipeline for Multi-label Patent Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning based Pipeline with Multichannel Inputs for Multi-label Patent Classification\n",
    "\n",
    "This notebook describes a deep learning pipeline for automatic patent classification with multichannel inputs.  A neural network model is trained with multichannel inputs namely embeddings of different segments of patent texts, and sparse linear\n",
    "input of different metadata. <br> <br>\n",
    "",
    "\n",
    "<br>\n",
    "In this notebook the classification task is a multi-class classification. The basic outline is:  <br>  <br>\n",
    "\n",
    "\n",
    "- load the patent dataset  <br>\n",
    "- apply preprocessing tasks  <br>\n",
    "- apply Tokenization process  <br>\n",
    "- Load a pretrained word embeddings model  <br>\n",
    "- prepare the embedding matrix for patent texts   <br>\n",
    "- concatenated deep layers <br>\n",
    "- train a deep neural network on the data  <br>\n",
    "- Fit the model and show the results  <br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Loading patent dataset\n",
    "We established the training, and test datasets that are domain-specific datasets and related to information technology domain.\n",
    "The total number of extracted records in the datasets is about 430,000  patents filed between 1978 and 2016. Each patent document contains a  patent number, issued date, patent type, and list of citations, classification codes, a list of inventors, a list of assignees, title, abstract, technical field, background,  summary of invention, and independent claim. <br>\n",
    "The data det is availabe in https://www.kaggle.com/darshmso/it-patent-dataset \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "unique Count:  ID       429711\n",
      "TI       419219\n",
      "AB       429700\n",
      "TECHF    278387\n",
      "BACKG    398251\n",
      "SUMM     363176\n",
      "CLMS     222262\n",
      "ICM      429711\n",
      "AY       429691\n",
      "IPCs     429711\n",
      "REF       25318\n",
      "PA       301442\n",
      "INV      429359\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>TI</th>\n",
       "      <th>AB</th>\n",
       "      <th>TECHF</th>\n",
       "      <th>BACKG</th>\n",
       "      <th>SUMM</th>\n",
       "      <th>CLMS</th>\n",
       "      <th>ICM</th>\n",
       "      <th>AY</th>\n",
       "      <th>IPCs</th>\n",
       "      <th>REF</th>\n",
       "      <th>PA</th>\n",
       "      <th>INV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>EP2000017943-0</td>\n",
       "      <td>recognition disk-shaped medium</td>\n",
       "      <td>recognition disk-like medium multimedia applic...</td>\n",
       "      <td>recognition disk-shaped medium multimedia appl...</td>\n",
       "      <td>identification labels adapted interpreted opti...</td>\n",
       "      <td>overcome drawbacks noted conventional types id...</td>\n",
       "      <td></td>\n",
       "      <td>G06K0019-06</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G06K0019-06, G06K0007-10]</td>\n",
       "      <td></td>\n",
       "      <td>Video_System_Italia_S_r_l</td>\n",
       "      <td>Tassello_Stefano</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EP2003016733-0</td>\n",
       "      <td>optical pickup recording reproducing</td>\n",
       "      <td>optical pickup reproducing optical recording m...</td>\n",
       "      <td>optical pickup recording reproducing optical p...</td>\n",
       "      <td>recently practical short wavelength red laser ...</td>\n",
       "      <td>provide pickup recording reproducing optical r...</td>\n",
       "      <td>optical pickup recording reproducing optical m...</td>\n",
       "      <td>G11B0007-135</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G11B0007-135, G11B0007-125]</td>\n",
       "      <td></td>\n",
       "      <td>Konica_Minolta_Opto_Inc</td>\n",
       "      <td>Arai_Norikazu Kojima_Toshiyuki Kiriki_Toshihik...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>EP2011009984-0</td>\n",
       "      <td>large capacity sales mediation</td>\n",
       "      <td>animation sales mediation animation sales medi...</td>\n",
       "      <td>large capacity sales large capacity sales medi...</td>\n",
       "      <td>recent years distributing music network rapidl...</td>\n",
       "      <td>implemented consideration problems provide ani...</td>\n",
       "      <td>large capacity sales mediation terminal large ...</td>\n",
       "      <td>G07F0017-16</td>\n",
       "      <td>2001</td>\n",
       "      <td>[G07F0017-16, G06Q0030-06, G06Q0020-10, G06Q00...</td>\n",
       "      <td>[JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...</td>\n",
       "      <td>NEC_Corporation</td>\n",
       "      <td>Maeda_Koji</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PCT1997010546-0</td>\n",
       "      <td>bridge client-server environment</td>\n",
       "      <td>software bridge introduced client client-serve...</td>\n",
       "      <td>bridge client-server environment distributed c...</td>\n",
       "      <td>overview object-oriented programming developme...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>G06F009-46</td>\n",
       "      <td>1996</td>\n",
       "      <td>[G06F009-46, G06F009-44, G06F0009-44, G06F0009...</td>\n",
       "      <td></td>\n",
       "      <td>INTERNATIONAL_BUSINESS_MACHINES_CORPORATION CO...</td>\n",
       "      <td>COLYER_ADRIAN_MARK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PCT1998021641-0</td>\n",
       "      <td>sections operating rates</td>\n",
       "      <td>core clocked perform operations clock frequenc...</td>\n",
       "      <td>sections operating rates high speed processors...</td>\n",
       "      <td>illustrates microprocessor microprocessor incl...</td>\n",
       "      <td>microprocessor levels sub-core clocked frequen...</td>\n",
       "      <td></td>\n",
       "      <td>G06F001-32</td>\n",
       "      <td>1997</td>\n",
       "      <td>[G06F001-32, G06F0001-08, G06F0009-30, G06F000...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>SAGER_DAVID_J FLETCHER_THOMAS_D HINTON_GLENN_J...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                ID                                    TI  \\\n",
       "0   EP2000017943-0        recognition disk-shaped medium   \n",
       "1   EP2003016733-0  optical pickup recording reproducing   \n",
       "2   EP2011009984-0        large capacity sales mediation   \n",
       "3  PCT1997010546-0      bridge client-server environment   \n",
       "4  PCT1998021641-0              sections operating rates   \n",
       "\n",
       "                                                  AB  \\\n",
       "0  recognition disk-like medium multimedia applic...   \n",
       "1  optical pickup reproducing optical recording m...   \n",
       "2  animation sales mediation animation sales medi...   \n",
       "3  software bridge introduced client client-serve...   \n",
       "4  core clocked perform operations clock frequenc...   \n",
       "\n",
       "                                               TECHF  \\\n",
       "0  recognition disk-shaped medium multimedia appl...   \n",
       "1  optical pickup recording reproducing optical p...   \n",
       "2  large capacity sales large capacity sales medi...   \n",
       "3  bridge client-server environment distributed c...   \n",
       "4  sections operating rates high speed processors...   \n",
       "\n",
       "                                               BACKG  \\\n",
       "0  identification labels adapted interpreted opti...   \n",
       "1  recently practical short wavelength red laser ...   \n",
       "2  recent years distributing music network rapidl...   \n",
       "3  overview object-oriented programming developme...   \n",
       "4  illustrates microprocessor microprocessor incl...   \n",
       "\n",
       "                                                SUMM  \\\n",
       "0  overcome drawbacks noted conventional types id...   \n",
       "1  provide pickup recording reproducing optical r...   \n",
       "2  implemented consideration problems provide ani...   \n",
       "3  bridge client distributed object-oriented brid...   \n",
       "4  microprocessor levels sub-core clocked frequen...   \n",
       "\n",
       "                                                CLMS           ICM    AY  \\\n",
       "0                                                      G06K0019-06  2000   \n",
       "1  optical pickup recording reproducing optical m...  G11B0007-135  2000   \n",
       "2  large capacity sales mediation terminal large ...   G07F0017-16  2001   \n",
       "3  bridge client distributed object-oriented brid...    G06F009-46  1996   \n",
       "4                                                       G06F001-32  1997   \n",
       "\n",
       "                                                IPCs  \\\n",
       "0                         [G06K0019-06, G06K0007-10]   \n",
       "1                       [G11B0007-135, G11B0007-125]   \n",
       "2  [G07F0017-16, G06Q0030-06, G06Q0020-10, G06Q00...   \n",
       "3  [G06F009-46, G06F009-44, G06F0009-44, G06F0009...   \n",
       "4  [G06F001-32, G06F0001-08, G06F0009-30, G06F000...   \n",
       "\n",
       "                                                 REF  \\\n",
       "0                                                      \n",
       "1                                                      \n",
       "2  [JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...   \n",
       "3                                                      \n",
       "4                                                      \n",
       "\n",
       "                                                  PA  \\\n",
       "0                          Video_System_Italia_S_r_l   \n",
       "1                            Konica_Minolta_Opto_Inc   \n",
       "2                                    NEC_Corporation   \n",
       "3  INTERNATIONAL_BUSINESS_MACHINES_CORPORATION CO...   \n",
       "4                                                      \n",
       "\n",
       "                                                 INV  \n",
       "0                                   Tassello_Stefano  \n",
       "1  Arai_Norikazu Kojima_Toshiyuki Kiriki_Toshihik...  \n",
       "2                                         Maeda_Koji  \n",
       "3                                 COLYER_ADRIAN_MARK  \n",
       "4  SAGER_DAVID_J FLETCHER_THOMAS_D HINTON_GLENN_J...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"../datasets/allITPatTextWith_Metadata.csv\",  encoding = \"ISO-8859-1\", error_bad_lines=False)\n",
    "df.columns =['ID','TI','AB','TECHF','BACKG','SUMM','CLMS','ICM','AY','IPCs','REF','PA','INV']\n",
    "\n",
    "#filter out rows with empty texts\n",
    "df.dropna(subset=['IPCs'], inplace=True)\n",
    "print(\"unique Count: \", df.count())\n",
    "\n",
    "\n",
    "df.fillna(value='', inplace=True)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'YEAR')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA44AAAGfCAYAAAAQ4oyDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xu4XHV59//3TcI5yEEwIgGCBZ6qpKJE0Gp/JioQxQo+tYqPIiiKrVoPRUusTwuiKLai1kO1KCh4iueSchAQiT4egACiISASIQgRgsoxiGDg/v2xvpHJZO+1Z2bP2nt25v26rrn27O/MfOb+zprTPWvNmshMJEmSJEkazSaTXYAkSZIkabDZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkqa0iPhcRLx3kq47IuKzEXFnRFw2GTVIkjQRbBwlSX0VESsj4vaI2Lpl7LURsWQSy2rKs4ADgVmZuX/7iRFxVEQ8FBFryuHG0mjuPfGl9l9EzIyI30bEvLbx0yNi0SSVJUlqgI2jJKkJ04C3THYR3YqIaV1eZHdgZWbeV3OeH2fmDGBb4HnA/cAVEbFPj2VOmoiY3vp/Zq4G3gZ8OiK2LOd5LvBC4B+avG5J0sSycZQkNeHfgbdHxHbtJ0TE7IjI1kYgIpZExGvL8aMi4ocR8eGIuCsiboiIvyzjN5e1mUe2xe4YERdGxL0R8b2I2L0l+8/LaXdExHUR8dKW0z4XEZ+MiHMj4j5g/gj1Pi4iFpfLr4iI15Xxo4HPAM8oaxPfXXeDZOZDmfnLzHwD8D3ghJbreFFELC/zXRIRT2g5bdeI+GZE/CYifhcRHy/je5a53l3W+n1lpOttub2PiYhfR8StEfH2ltM3iYiFEfHLkv/ViNih7bJHR8SvgO+OMK/PA9cBJ5bm8b+AN2fmb0rGrIj4Vqn/xoh4Y8t1PyMiLinzvjUiPhoRm5bTppfrfkNErAB+Xnf7SpKaZeMoSWrC5cAS4O1jnG80BwA/Ax4NfAlYBDwN2BN4JfDxiJjRcv5XAO8BdgSuAr4IUDaXvbBkPAY4HPjPiHhiy2X/D3ASsA3wgxFqWQTcAjwOeAnwvoh4TmaeBvwdZY1iZh7fxfy+CfxVqXFv4MvAW4GdgHOB/4mIzcoa0LOBm4DZwC6lHsp8LwC2B2YBHxvjOucDewEHAcdFxPPK+D8AhwHPLnO8E/hE22WfDTwBOHiU7L8DXlNquzozF5W5bVLqX1pqPxB4R1krCbCWas30jsAzgQXA69uyX0S17OeMMT9JUoNsHCVJTflX4B8iYqceLntjZn42Mx8CvgLsCpyYmQ9k5gXAg1RN5DrnZOb3M/MB4F1UawF3pdpkcmXJWpuZPwG+Afxty2XPyswfZubDmfmH1iJKxjOB4zLzD5l5FdVaxlf1MKdWvwZ2KMdfVuq/MDP/CHwQ2BL4S2B/qmbuHZl5X6lhXXP7R6pNZR/XNj6ad5eMZcBngZeX8b8D3pWZt5Tb7wTgJW2bhp5QLnv/SMGZeQvV8n4e8PctJz0DeFRmvi8zH8zMFcBpVA08mbk0My8ty+YG4FSqJrXV+zLzztGuW5I0MWwcJUmNyMyrqdY2Lezh4qtbjt9f8trHWtc43txyvWuAO6gart2BA8qmkHdFxF1UaycfO9JlR/A44I7MvLdl7CaqtWfjsUupcd113NRS/8Olpl2oGuabMnPtCBn/BARwWdnM9TVjXGfrPG8q1wvVbfStltvnWuAhYOYolx3NcuDOzLy1ZWx3YLe22/+fKLd/2Yz4nIi4LSLuAU6kWvs4Wt2SpEniF80lSU06HrgSOKVlbN2OZLYC7inHWxu5Xuy67kjZhHUHqrV6NwPfy8wDay6bNaf9GtghIrZpaR53A1aNs94XA/+v5Tr+tBlmRATVfFYBD1A1XtPbm8fMvA1Y933LZwHfiYjvl7V6I9mVR74nuFu5Xqhuo9dk5g/bLxARs9ddXTeTa3EzcH1mPmGU0/8LuAR4WWauKd+9fGHbeXq9bklSH7nGUZLUmNLEfAV4c8vYb6iaoldGxLSypuzPxnlVL4iIZ0XEZlTf/bskM2+mWuO5d0QcERGblsPTWnc+M0b9NwM/At4fEVtExF8ARwNf6LbAMtc9IuJjwDxg3c50vgocEhHPLTuGOZaqYfwRcBlwK3ByRGxdanhmyfvbiJhVMu6karAerinhXyJiq4h4EvBqquUC8CngpHU7FIqInSLi0G7nN4ofAw9GxLGl9mkRMSci9iunbwPcDdxXlkn79xslSQPCxlGS1LQTga3bxl4HvAP4HfAkqiZpPL5EtXbzDmA/qh3oUNYSHkT1nbpfA7cBHwA27yL75VQ7pvk18C3g+Mz8TheXf0ZErKFau7oEeBTwtPJdQzLzulLvx4DfAn8N/HX5TuBD5f89gV9R7aTnZSX3acClJXsx8JbyPcHRfA9YAVwEfLB8VxTgP8rlL4iIe6nWAB7QxfxGVdaSvoDqu5ory/z+i+o2gKpJPhK4t4yPuGdYSdLki0y3AJEkaWNVNje9Edh0lO9KSpI0Jtc4SpIkSZJq2ThKkiRJkmq5qaokSZIkqZZrHCVJkiRJtWwcJUmSJEm1pk92AZNpxx13zNmzZ495vvvuu4+tt27fk3zv+p3XROYw1jiMc24ic9Dzmsi0xuHIayJzGGscxjk3kTmMNQ7jnJvIHPS8JjKHscZu8q644orfZuZOY54xM4f2sN9++2UnLr744o7O16l+5zWROYw1DuOcm8gc9LwmMq1xOPKayBzGGodxzk1kDmONwzjnJjIHPa+JzGGssZs84PLsoHdyU1VJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUq3pk12AJEmSpOE0e+E5G4wdO2ctR7WNrzz5kIkqSaNwjaMkSZIkqZaNoyRJkiSplo2jJEmSJKmWjaMkSZIkqZaNoyRJkiSplo2jJEmSJKmWjaMkSZIkqZa/4yhJkiRpTJ3+5iL4u4sbI9c4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSak1I4xgRW0TEZRHx04hYHhHvLuN7RMSlEbEiIr4SEZuV8c3L/yvK6bNbst5Zxq+LiINbxheUsRURsXAi5iVJkiRJw2Ci1jg+ADwnM58M7AssiIinAx8APpyZewJ3AkeX8x8N3FnGP1zOR0Q8ETgceBKwAPjPiJgWEdOATwDPB54IvLycV5IkSZI0ThPSOGZlTfl303JI4DnA18v4GcBh5fih5X/K6c+NiCjjizLzgcy8EVgB7F8OKzLzhsx8EFhUzitJkiRJGqfIzIm5omqt4BXAnlRrB/8duKSsVSQidgXOy8x9IuJqYEFm3lJO+yVwAHBCucwXyvhpwHnlKhZk5mvL+BHAAZn5phHqOAY4BmDmzJn7LVq0aMza16xZw4wZM3qdeuN5TWQOY43DOOcmMgc9r4lMaxyOvCYyh7HGYZxzE5nDWOMwzrmJzPHkLVt19wZjM7eE1fdveN45u2zb18xO80aysS+X8ebNnz//isycO+YZM3NCD8B2wMXAs6jWEq4b3xW4uhy/GpjVctovgR2BjwOvbBk/DXhJOXymZfwI4ONj1bLffvtlJy6++OKOztepfuc1kTmMNQ7jnJvIHPS8JjKtcTjymsgcxhqHcc5NZA5jjcM45yYyx5O3+3Fnb3D46Bf+e8TxfmeOx8a+XMabB1yeHfRx0ztsWvsmM++KiIuBZwDbRcT0zFwLzAJWlbOtomokb4mI6cC2wO9axtdpvcxo45IkSdLQmb3wnA3Gjp2zlqPaxleefMhElaQpbKL2qrpTRGxXjm8JHAhcS7Xm8SXlbEcCZ5Xji8v/lNO/W7rhxcDhZa+rewB7AZcBS4G9yl5aN6Pagc7i5mcmSZIkSRu/iVrjuDNwRvme4ybAVzPz7Ii4BlgUEe8FfkK16Snl7+cjYgVwB1UjSGYuj4ivAtcAa4E3ZuZDABHxJuB8YBpwemYun6C5SZIkSdJGbUIax8z8GfCUEcZvoNojavv4H4C/HSXrJOCkEcbPBc4dd7GSJEmSpPVM1O84SpIkSZKmKBtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSrQn5HUdJkiRJatrshedsMHbsnLUc1Ta+8uRDJqqkjYZrHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtaZPdgGSJEnSsJu98JwRx4+ds5aj2k5befIhE1GStB7XOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJqTUjjGBG7RsTFEXFNRCyPiLeU8RMiYlVEXFUOL2i5zDsjYkVEXBcRB7eMLyhjKyJiYcv4HhFxaRn/SkRsNhFzkyRJkqSN3UStcVwLHJuZTwSeDrwxIp5YTvtwZu5bDucClNMOB54ELAD+MyKmRcQ04BPA84EnAi9vyflAydoTuBM4eoLmJkmSJEkbtQlpHDPz1sy8shy/F7gW2KXmIocCizLzgcy8EVgB7F8OKzLzhsx8EFgEHBoRATwH+Hq5/BnAYc3MRpIkSZKGS2TmxF5hxGzg+8A+wD8CRwH3AJdTrZW8MyI+DlySmV8olzkNOK9ELMjM15bxI4ADgBPK+fcs47sC52XmPiNc/zHAMQAzZ87cb9GiRWPWvGbNGmbMmNHbhCcgr4nMYaxxGOfcROag5zWRaY3DkddE5jDWOIxzbiJzGGvc2Oe8bNXdI47P3BJW37/+2Jxdtu05cyLypkKNneaNZpDuO+PNmz9//hWZOXes800fd1VdiIgZwDeAt2bmPRHxSeA9QJa/pwCvabKGzDwVOBVg7ty5OW/evDEvs2TJEjo5X6f6nddE5jDWOIxzbiJz0POayLTG4chrInMYaxzGOTeROYw1buxzPmrhOSOOHztnLacsW/8t+8pXdHYdI2VORN5UqLHTvNEM0n1nIvJgAhvHiNiUqmn8YmZ+EyAzV7ec/mng7PLvKmDXlovPKmOMMv47YLuImJ6Za9vOL0mSJEkah4naq2oApwHXZuaHWsZ3bjnbi4Gry/HFwOERsXlE7AHsBVwGLAX2KntQ3YxqBzqLs9re9mLgJeXyRwJnNTknSZIkSRoWE7XG8ZnAEcCyiLiqjP0z1V5R96XaVHUl8HqAzFweEV8FrqHaI+sbM/MhgIh4E3A+MA04PTOXl7zjgEUR8V7gJ1SNqiRJkiRpnCakcczMHwAxwknn1lzmJOCkEcbPHelymXkD1V5XJUmSJEl9NFG/4yhJkiRJmqJsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbWmd3KmiNgWeDAz74+ITYBXAQ9l5ucbrU6SJEkDb/bCczYYO3bOWo5qG1958iETVZKkPut0jeM5wJxy/ATgfcB7I+J9TRQlSZIkSRocnTaOTwCuKMdfARwIPAt4ZRNFSZIkSZIGR0ebqgLTMvOhiNgd2CwzlwNExPbNlSZJkiRJGgSdNo7LIuL/ArsBFwBExM7AvU0VJkmSJEkaDJ1uqvpm4PnAnsCJZexAShM5lojYNSIujohrImJ5RLyljO8QERdGxPXl7/ZlPCLioxGxIiJ+FhFPbck6spz/+og4smV8v4hYVi7z0YiIDucmSZIkSarR6RrHqzLzma0DmXlmRHS6V9W1wLGZeWVEbANcEREXAkcBF2XmyRGxEFgIHEfVpO5VDgcAnwQOiIgdgOOBuUCWnMWZeWc5z+uAS4FzgQXAeR3WJ0mSJHXMPclq2HS6xvHuUcZ/18mFM/PWzLyyHL8XuBbYBTgUOKOc7QzgsHL8UODMrFwCbFc2jT0YuDAz7yjN4oXAgnLaozLzksxM4MyWLEmSJEnSOHTaOG6w2Wevm4JGxGzgKVRrBmdm5q3lpNuAmeX4LsDNLRe7pYzVjd8ywrgkSZIkaZyiWkE3yokRp5ajR/LImsF1Hg9snZnP6PjKImYA3wNOysxvRsRdmbldy+l3Zub2EXE2cHJm/qCMX0S1Ces8YIvMfG8Z/xfgfmBJOf/zyvhfAcdl5gtHqOEY4BiAmTNn7rdo0aIx616zZg0zZszodJoTntdE5jDWOIxzbiJz0POayLTG4chrInMYaxzGOTeROWg1Llu14cZpM7eE1fevPzZnl217yoeNf84j5TWRORF5U6HG8dwXYfAf093kzZ8//4rMnDvW+cb6juOm5W+0HAd4mGqN4Wc6qgaIiE2BbwBfzMxvluHVEbFzZt5aNje9vYyvAnZtufisMraKqnlsHV9SxmeNcP4NZOapwKkAc+fOzXnz5o10tvUsWbKETs7XqX7nNZE5jDUO45ybyBz0vCYyrXE48prIHMYah3HOTWQOWo3t3+uD6vt+pyxb/63mylf0lg8b/5xHymsicyLypkKN47kvwuA/ppt4vNQ2jpn5aoCIuCYz/73XKymbtZ4GXJuZH2o5aTHV2syTy9+zWsbfFBGLqHaOc3dpLs8H3tfy+5EHAe/MzDsi4p6IeDpVQ/sq4GO91itJkiRJekRHe1UdT9NYPBM4gur3IK8qY/9M1TB+NSKOBm4CXlpOOxd4AbAC+D3w6lLHHRHxHmBpOd+JmXlHOf4G4HPAllR7U3WPqpIkacK5t01JG6OOGseI2JtqDd5cYJvW0zJzs7EuX76rONrOdJ47wvkTeOMoWacDp48wfjmwz1i1SJIkSZK60+nvOH6Oak+lRwD3NVaNJEmSJGngdNo47gM8OzP/2GQxkiRJkqTB02nj+HPgMYyyp1JJkqR2nX7XD/y+nyQNuk4bx88C34iIfwNuaz0hM3/U96okSZIkSQOj08bxE+Xv19vGE5jWv3IkSZIkSYOm05/j2KTpQiRJkiRJg6nTNY4AREQAj83MWxuqR5IkaVT+RqIkTY6O1iRGxIyIOA24H1hRxg6LiOObLE6SJEmSNPk6XeN4CjATeCbwnTK2FHgf8O4G6pIkSRKuZZU0GDptHF8IPDEz746IBMjMVRHxuOZKkyRJkiQNgk53erMJ1WaqfxIRM4A1fa9IkiRJkjRQOm0cfwC8s23sH4CL+1uOJEmSJGnQdLqp6j8C342IVwIzImIZsBnwnMYqkyRJkiQNhE5/x/HmiNiH6ruOewA3AWdn5v31l5QkSRpc7nhGkjrTUeMYEUdk5ueBb7SNvyIzv9hIZZIkSZKkgdDpdxw/Mcr4x/pViCRJkiRpMHXaOMYGAxGzgbX9LEaSJEmSNHhqN1WNiD8CCUyLiAfbTp4GfLKpwiRJkiRJg2Gs7zg+j2pt47nA81vGHwZuy8zrmypMkiRJkjQYahvHzPweQET8WWbeOjElSZIkSf3lHnTVK+87lU5/juPWiHg08DRgJ1q+85iZZzZUmyRJkoaQb9SlwdPpz3E8j+qnOB4EtgPuKn9vBGwcJUmSJGkj1lHjCJwMnJiZp0TEnZm5U0T8K7CmwdokSZLUZyOtzQPX6Emq1+nPcewFfKQcX7eZ6geAt/a9IkmSJEnSQOm0cfw9sHk5/ruI2A3YDNi+kaokSZIkSQOj08bxR8Bh5fh5wGLgO8CPmyhKkiRJkjQ4Ov2O4yt5pMl8O3AssA3woSaKkiRJkiQNjk4bxy0y806AzPwDcFJzJUmSJEmSBkntpqoR8RcRsRL4bURcHxFPmJiyJEmSJEmDYqzvOP47cBnwIuCnwPsbr0iSJEmSNFDG2lT1KcCfZea9EfEjYNkE1CRJkiRJGiBjrXHcIjPvBSjfcdyq+ZIkSZIkSYNkrDWOm0TEM4Ao/09r+5/M/FFTxUmSJEmSJt9YjeNWwA/bxlr/T2BaXyuSJEmSJA2U2sYxM8falFWSJEmStJGzMZQkSZIk1bJxlCRJkiTVsnGUJEmSJNWycZQkSZIk1Rq1cYyIpS3Hj5+YciRJkiRJg6ZujeNeEbHu9xqPHc+VRMTpEXF7RFzdMnZCRKyKiKvK4QUtp70zIlZExHURcXDL+IIytiIiFraM7xERl5bxr0TEZuOpV5IkSZL0iLqf47gU+H5EXAtsERGnjnSmzDymg+v5HPBx4My28Q9n5gdbByLiicDhwJOAxwHfiYi9y8mfAA4EbgGWRsTizLwG+EDJWhQRnwKOBj7ZQV2SJEmSpDHUrXE8HDgXWLfWcdNRDmPKzO8Dd3RY06HAosx8IDNvBFYA+5fDisy8ITMfBBYBh5a1os8Bvl4ufwZwWIfXJUmSJEkaQ2Tm2Geq1uy9aFxXFDEbODsz9yn/nwAcBdwDXA4cm5l3RsTHgUsy8wvlfKcB55WYBZn52jJ+BHAAcEI5/55lfFfgvHXXM0IdxwDHAMycOXO/RYsWjVn7mjVrmDFjRtdznqi8JjKHscZhnHMTmYOe10SmNQ5HXhOZg1bjslV3bzA2c0tYff/6Y3N22baveU1kTlbeVKhxpLypUKPLZTBr9DHdW2a7yXx9mT9//hWZOXes89Vtqvon65rGiJgJ7ArcnJmrO6pkdJ8E3gNk+XsK8JpxZo4pM08FTgWYO3duzps3b8zLLFmyhE7O16l+5zWROYw1DuOcm8gc9LwmMq1xOPKayBy0Go9aeM4GY8fOWcspy9Z/u7DyFZ3ld5rXROZk5U2FGkfKmwo1ulwGs0Yf071ltpsKry8d/RxHRGwXEWcDtwKXAb+OiP+JiB16veLMXJ2ZD2Xmw8CnqTZFBVhF1ZyuM6uMjTb+O2C7iJjeNi5JkiRJ6oNOf8fxI+Xvn1N9r/EJVGsKP9TrFUfEzi3/vhhYt8fVxcDhEbF5ROwB7EXVrC6l2tPrHmWvqYcDi7Pa1vZi4CXl8kcCZ/ValyRJkiRpfR1tqgocBDwhM9dt4PuLiDgSuKaTC0fEl4F5wI4RcQtwPDAvIvalakBXAq8HyMzlEfHVkr0WeGNmPlRy3gScD0wDTs/M5eUqjgMWRcR7gZ8Ap3U4L0mSJEnSGDptHIOqwWv1MI/scbVWZr58hOFRm7vMPAk4aYTxc6n29No+fgOPbOoqSZIkSeqjTjdVvRD4fEQ8PiI2iYjHU/024wWNVSZJkiRJGgidNo5vBTan+k3FPwLXA1sAb2uoLkmSJEnSgOj05zjuABaUHdqs+zmOWxutTJIkSZI0EDr9jiMApVm0YZQkSZKkIdLppqqSJEmSpCFl4yhJkiRJqmXjKEmSJEmqZeMoSZIkSarVceMYEctajv+vZsqRJEmSJA2a2sYxIv4pIv4qIrYCZrWc9ONmy5IkSZIkDYqx1jjuBLwfWA1sHREnR8QCIBqvTJIkSZI0EGobx8x8R2Y+C3g08AfgPuAdwDYR8e2IeO0E1ChJkiRJmkRjbar60Yh4ObArsDYz35OZzwXWAB8D/nICapQkSZIkTaLpY5z+S+CvgZOAR0XEl4CLATLzHOCcZsuTJEmSJE222sYxM/9j3fGIuAv4NvAcqk1VlwPfyMx/bbZESZIkSdJk6uZ3HDMzz8zMo4C7gZcCaxupSpIkSZI0MMbaVLXVoS3HIzOXA8v7XI8kSZIkacB0vMYxM7/fcnz7ZsqRJEmSJA2abjZVlSRJkiQNIRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUq0JaRwj4vSIuD0irm4Z2yEiLoyI68vf7ct4RMRHI2JFRPwsIp7acpkjy/mvj4gjW8b3i4hl5TIfjYiYiHlJkiRJ0jCYqDWOnwMWtI0tBC7KzL2Ai8r/AM8H9iqHY4BPQtVoAscDBwD7A8evazbLeV7Xcrn265IkSZIk9WhCGsfM/D5wR9vwocAZ5fgZwGEt42dm5RJgu4jYGTgYuDAz78jMO4ELgQXltEdl5iWZmcCZLVmSJEmSpHGKqteagCuKmA2cnZn7lP/vysztyvEA7szM7SLibODkzPxBOe0i4DhgHrBFZr63jP8LcD+wpJz/eWX8r4DjMvOFo9RxDNWaTGbOnLnfokWLxqx9zZo1zJgxo7eJT0BeE5nDWOMwzrmJzEHPayLTGocjr4nMQatx2aq7NxibuSWsvn/9sTm7bNvXvCYyJytvKtQ4Ut5UqNHlMpg1+pjuLbPdZL6+zJ8//4rMnDvW+aaPu6o+yMyMiAnpYDPzVOBUgLlz5+a8efPGvMySJUvo5Hyd6ndeE5nDWOMwzrmJzEHPayLTGocjr4nM8ebNXnjOBmPHznmIU35w33pjK08+pKO8o0bMW8spy9Z/u7DyFfP6mtdE5mTlTYUaR8qbCjW6XAazRh/TvWW2G7TXl5Fc5VRXAAAgAElEQVRM5l5VV5fNTCl/by/jq4BdW843q4zVjc8aYVySJEmS1AeT2TguBtbtGfVI4KyW8VeVvas+Hbg7M28FzgcOiojty05xDgLOL6fdExFPL5u8vqolS5IkSZI0ThOyqWpEfJnqO4o7RsQtVHtHPRn4akQcDdwEvLSc/VzgBcAK4PfAqwEy846IeA+wtJzvxMxct8OdN1DtuXVL4LxykCRJkiT1wYQ0jpn58lFOeu4I503gjaPknA6cPsL45cA+46lRkiRJkjSyydxUVZIkSZI0Bdg4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmpNn+wCJEkaBrMXnrPB2LFz1nJU2/jKkw+ZqJIkSeqYaxwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtaZPdgGSJA2i2QvP2WDs2DlrOaptfOXJh0xUSZIkTRrXOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJqTZ/sAiJiJXAv8BCwNjPnRsQOwFeA2cBK4KWZeWdEBPAfwAuA3wNHZeaVJedI4P+W2Pdm5hkTOQ9J0uSZvfCcDcaOnbOWo9rGV558yESVJEnSRmVQ1jjOz8x9M3Nu+X8hcFFm7gVcVP4HeD6wVzkcA3wSoDSaxwMHAPsDx0fE9hNYvyRJkiRttAalcWx3KLBujeEZwGEt42dm5RJgu4jYGTgYuDAz78jMO4ELgQUTXbQkSZIkbYwGoXFM4IKIuCIijiljMzPz1nL8NmBmOb4LcHPLZW8pY6ONS5IkSZLGKTJzcguI2CUzV0XEY6jWFP4DsDgzt2s5z52ZuX1EnA2cnJk/KOMXAccB84AtMvO9ZfxfgPsz84MjXN8xVJu5MnPmzP0WLVo0Zo1r1qxhxowZ45xpc3lNZA5jjcM45yYyBz2viUxrnPy8Zavu3mBs5paw+v71x+bssu2kZQ5jjZ3mTYUaN/blMhVqdLkMZo0+pnvLbDeZr6nz58+/ouUrg6Oa9J3jZOaq8vf2iPgW1XcUV0fEzpl5a9kU9fZy9lXAri0Xn1XGVlE1j63jS0a5vlOBUwHmzp2b8+bNG+ls61myZAmdnK9T/c5rInMYaxzGOTeROeh5TWRa4+Tnte8EB6qd45yybP2XuZWv6Dy/35nDWGOneVOhxo19uUyFGl0ug1mjj+neMtsN0mvqaCZ1U9WI2Doitll3HDgIuBpYDBxZznYkcFY5vhh4VVSeDtxdNmk9HzgoIrYvO8U5qIxJkiRJksZpstc4zgS+Vf3KBtOBL2XmtyNiKfDViDgauAl4aTn/uVQ/xbGC6uc4Xg2QmXdExHuApeV8J2bmHRM3DUmSJEnaeE1q45iZNwBPHmH8d8BzRxhP4I2jZJ0OnN7vGiVJkiRp2A3CXlUlSZIkSQPMxlGSJEmSVMvGUZIkSZJUa7J3jiNJGkKzR9m1efsuz1eefMhElSRJkmq4xlGSJEmSVMvGUZIkSZJUy8ZRkiRJklTLxlGSJEmSVMvGUZIkSZJUy8ZRkiRJklTLn+OQJNUa6aczwJ/PkCRpmLjGUZIkSZJUy8ZRkiRJklTLTVUlaSMz0qalblYqSZLGw8ZRkrrQRFNmoydJkgadjaOkjZpNmSRJ0vj5HUdJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItf45D0sDo9KczwJ/PkCRJmkiucZQkSZIk1bJxlCRJkiTVclNVST3rdNNSNyuVJEma2lzjKEmSJEmqZeMoSZIkSarlpqrSEHHTUkmSJPXCNY6SJEmSpFo2jpIkSZKkWm6qKg0oNyuVJEnSoLBxlPrERk+SJEkbKzdVlSRJkiTVco2jhpJrByVJkqTO2ThqSrDRkyRJkiaPjaP6bqQmD2z0JEmSpKnKxlGuzZMkSZJUy8ZxCrLRkyRJkjSRNqq9qkbEgoi4LiJWRMTCya5HkiRJkjYGG80ax4iYBnwCOBC4BVgaEYsz85rJrMu1g5IkSZKmuo1pjeP+wIrMvCEzHwQWAYdOck2SJEmSNOVtNGscgV2Am1v+vwU4oNsQ1xBKkiRJ0voiMye7hr6IiJcACzLzteX/I4ADMvNNbec7Bjim/Pu/gOs6iN8R+G0fy+13XhOZw1jjMM65icxBz2si0xqHI6+JzGGscRjn3ETmMNY4jHNuInPQ85rIHMYau8nbPTN3GutMG9Max1XAri3/zypj68nMU4FTuwmOiMszc+74ymsur4nMYaxxGOfcROag5zWRaY3DkddE5jDWOIxzbiJzGGscxjk3kTnoeU1kDmONTcx5Y/qO41Jgr4jYIyI2Aw4HFk9yTZIkSZI05W00axwzc21EvAk4H5gGnJ6Zyye5LEmSJEma8jaaxhEgM88Fzm0guqtNWychr4nMYaxxGOfcROag5zWRaY3DkddE5jDWOIxzbiJzGGscxjk3kTnoeU1kDmONfZ/zRrNzHEmSJElSMzam7zhKkiRJkhpg4yhJkiRJqmXjKEmSJEmqtVHtHEeSJA2miNgBIDPvGOTMQdfvOQ/rchnk2zEiZgK7lH9XZebqAc0c6PtOv+c8VZZLk9w5zij6uSAjIoD9W/OAy7IPN/4gP8AazPSJapymwnIZ5Nuxicd0Q5kDfd/p95yHdbn0s8aI2BZ4J3AY8BgggduBs4CTM/OuLvN2A/4NeC5wFxDAo4DvAgszc2UPNfY1s4E59zWvZPZ7zkO3XBqqsd95+wKfAralehwDzCrZb8jMK7vJayJzitx3+j3ngV8uEyYzPbQcgH2BS4Brge+Uw8/L2FN7yDsIWAGcB3ymHL5dxg7qscbdgEXAb4DrS9btZWz2ZM+5oduxr3OeCrfjEC+Xgb4dG3pM9zVzitx3+j3nYV0u/a7xfOA44LEtY48tYxf0kPdj4GXAtJaxacDhwCU9zrmvmQ3Mua95Dc156JbLVLgdgauAA0YYfzrw0x7n3NfMKXLf6fecB365TNRh0gsYtEMDd7ZrGeENNLAHcG2PNQ70A6yh29EnquFZLgN9Ozb0mO5r5hS57/R7zsO6XPpd43W9nFZzmet7OW0iMxuYc1/zGprz0C2XqXA7jpG3ooE5d505Re47EznngVguE3XwO44b2jozL20fzMxLImLrHvKmA7eMML4K2LSHPIAdM/MrrQOZ+RCwKCLe00Nev+fcRGa/59xEZr/nPKzLZdBvxyYe0/3OnAr3nX7PeViXS79rvCki/gk4I8smtGXT2qOAm3vIuyIi/hM4o+XyuwJHAj/pIa+JzH7Pud950P85D+NyaaLGfuedFxHnAGe25b2KakuCXvQ7cyrcd/o956mwXCaEjeOG+r0gTweWRsSitrzDgdN6rHHQH2BNZPpENf68JjKbWC6Dfjs28Zjud+ZUuO/0e87Dulz6XePLgIXA9yLiMWVsNbAYeGkPea8CjgbezSPfwbwF+J8e62sis99z7nce9H/Ow7hcmqixr3mZ+eaIeD5wKOt/Z/kTmXluD/U1kTnw951+z3mKLJcJ4c5xRjDKglzc64KMiCeMkndNj3mbUT3AWjP/9ADLzAd6yOzrnPud2dCcB/52HMbl0u8aG8rr62O6icxBv++UvH7PeViXS9/nLUlSOxtHSZI0poh4avZxT38R8cLMPLtfeU1kNjDnvuaVzH7PeeiWS8kc6NsxIo7JzFP7lddE5hS57/R7zgO/XPppk8kuYCqJiGP6nHdCP/NK5gv7nNfXOTeR2e85N5HZwJyHdbkM9O3Y0GO6r5lT5L5zwiDnNZHZ0HI5oc+Rf9/nvKf1Oa+JzH7Pud950P85D+NygcG/HaPPeU1kToX7Tr/nPBWWS9/YOHan3wvyij7nweA/wJrI9IlqMDObWC6Dfjs28Zjud+ZUuO/0e87Dulz6WmNmvq7Pecf3M6+JzAbm3Ne8ktnvOQ/dcimZA307ZuZ/9TOvicwpct/p95wHfrn0k5uqSpKkP4nqh9cXsP53Js/PHn5wfYzrOTAzL+zxso8CdsrMX7aN/0Vm/qyHvMcCZOZtEbET8FdUP/mwvJf6Rsh/X2b+cz+ySt4ewFOAazLz5z1cfjfg9sz8Q0QE1Z5KnwpcA3w6M9f2kPkiqt9X/EO3l63J/P+A1Zl5XUQ8E3gG1c/MnDOOzBlU9+9dgYeAX1DV/XCPeX/OyN8xvrbXGke5nldn5md7vOyfU9V3aWauaRlfkJld75grIvYHMjOXRsQTqW7Pn/dzpy4RcWZmvqpPWc8C9geuzswLerj8AVT3u3siYkuqnTate7y8LzPv7iHzzcC3MrPXPQRPChvHNhGxA/Am4NdUe3L6Z8oTFdWd484eMucDf8P6T1KfycwV46jTJyqfqCb9iarsHOdw4NeZ+Z2I+D/AX1I9Xk7NzD/2mPt44H+z/mPmS5l5T495BwOHsf7j5axe7odjXM+/ZuaJPV72YGAWcFFmrmwZf01mnt5lVgB/CyTwdeA5VM8XPwc+1esbpBGu57uZ+ZweL7tjZv625f9XUh4vVG9cu3pxiogXA9/LzDvKG/9TKG+sgWMzc6SfrBgr80PANzLzh91edpS8vr++lNy+vcZExKuA44ELqB4nUN0vDwTenZln9lLjKNf1q8zcrYfLvRT4CHA71U+OHJWZS8tpV2bmU7vMez3V82sAH6Bqoq4GngX8W2Z2tVfHiPho+xBwBNXedMnMN3eTVzL/OzMPK8cPpZr/Eqrn2/dn5ue6zLsa2D8zfx8RHwD+DPhvqucKMvM1PdR4P3AfcB7wZaoPGx7qNqcl7yNUzwnTgfOB55bsZwM/ycx39JD5UuDtwM+A+cCPqLa+mwO8stsPHSLiOODlwCIe+VmcWVSvi4sy8+Rua6y5rl4fL28G3kj1PLMv8JbMPKuc1svj5Xjg+VTL5ULgAOBiqueI8zPzpB5qXNw+RLV8vguQmS/qMu+yzNy/HH8d1fy/BRwE/E+3yyUilgNPzsy1EXEq8Huq19bnlvH/3U1eybyb6vHyS6rHy9cy8zfd5kw0G8c2EXEusAx4FPCEcvyrVA+IJ2fmoV3mvR94LHAR1RvXG6le1N9A9Ubhaz3U6BOVT1QD8UQVEV+kWiZbAXcBM4BvlhojM4/sIfPNwAuB7wMvoPpZj7uAFwNvyMwlXeZ9BNib6k1b6+PlVVQ/wPuWbmusua5eHy/vo3qTeiXw18BHMvNj5bReHi//CTwG2Ay4B9icahf2h1B9et/1nCOi/Q1VUN2u1wFk5l90mfeneUXE/6Vaw/MlqmV/S2a+rcu8azLzieX4V4BLgK8BzwNekZkHdpNXcn4D3ATsBHwF+HJm9vozM31/fSmZfX2NiYjrgAPa1y5GxPZUHwLu3WVe+/Psn04CnpOZXf9+ZURcBTw/M28tHyieCbwzM78VET/JzKd0mbeM6jVlS6rlvWdZ87g9cHFm7ttl3s3A96ia73WbIH+QqmEhM8/oJq9k/mleEfEjqvv0jRGxI9WHTU/uMq/18XIF8LR1HyhFxE+7zVtXI1Xj+RKq9yP7UL0Gfjkzv9dD3vKSsSXVhxi7lEZ3U6rGcZ8eMn8GPL3k7Ah8MTMPjoi/oPpQ7S+7zPsF8KT2D0nLh6rLM3OvHuob8SRg78zcvJu8krkMeEZmromI2VTvIz6fmf8xjsfLvlSvK7cBs1o+4L6029eCknkl1Yd8n6H6wDOo3qMcDtDt/aft8bIUeEFm/iaq38u9JDPndJl3bWY+YV2tra/JEXFVt88R62oE9qN6jXoZ8CKqrxd8GfhmZt7bbeaEyEwPLQfgqvI3gFUjndZl3rKW49OBH5bj21Otieqlxl8Am44wvhnVG+Fu8342ymEZ8ECPNS4DZpTjs4HLqZpHqJ7we8mbRtWg3AM8qoxvCfysxxqvBL4AzKP6BHMecGs5/uwe8n7Scnwp1WZUAFu33g+6yLu2tdbx3hfX1Uj16epBVGs8fkP1+3FHAtv0ct8pf6dT/b7WtPJ/jGO5LGvJ2QpYUo7v1uN95xejjEePj5d7RjncC6wdx5ynl+PbAecCH26/X3WTV/5uCvwO2KxlOfW6XBaXx8ufA7uXx/XN5fjuvdwXW45fCWzdUnMvj5frWo5f0XZaz4+X8ndv4F+A5VRrbY+nehPXbV5fX19al3XL8h3XawzV68u2I4xv2+Pj5U6qDyye3XaYR/UhxrjmXP7fmeoN15tpe67sMO/KluM/He1+2kXeNlRrBL8EPK6M3dDLXEep8bI+1Hg+VeMO8I11j2Hg0e23QS81lv8fW5bJj4Gbe8i7uvzdotyPtiz/T6PaRLen+w6PrDTZsu15qJfHy89Hev4rz4vX9ZC3mqop273tMJtqy55e5ry87f8ZVK/7H+rleaftNvtJ22m9Po9tAryNasXAvmWs58cM8NPyHPho4PLR6u8i72vAq8vxzwJzy/G9gaU91tj+eNmUqnn8MvCbXufe9GE6ardJ+ZRxG2BGRMzOzJUR8WiqxqxbD0fEDpl5B/A4qic8MvPOsjlZLx4uWTe1je9cTuvWTOBgqifmVkG1GUcvNsmyeWq5/eYBX4+I3aGnnUCszWqTl99HxC+zbLKYmfdHRK+b3c0F3gK8C3hHZl4VEfdnD5+MFuvuO5tQvTD9ptR4X0R0/X0R4Op4ZFPhn0bE3My8PCL2BnraBLQqJx+m+iT8gvLJ7fOp1mB/kGrNSjc2KZ+sbk3V5G0L3EH1SeSmPdYI1Rvgh0rOjFL4r0q93fpDRDwty6ZsLZ4G9PJdnLuoPp1f3X5CWdPQi+lZvlOUmXdFxF8Dp0bE1+jteWdd1h8jYmlmPlj+X9vr4yUzX1Q2Bz0V+GBmLo6IP2Zm+/NQp7aMiKdQPV6mZeZ9LTX3snnbkog4EXh/Of7irNZAzQe63qy7yFLTL4D3AO8payZeTtXc79llXr9fX6D/rzEnAVdGxAVUHwxA9aHNgVS3QbcuAX4/0vNqWbvZi3sj4s+yfL8xqzWP86g2tXxSD3kZEZtmtdbokJb6tqCHnQhmtabgrRGxH/DFiDinl5w2T46Ie6hePzePiJ3LvDejLPMuvRY4M6o9794NXFXW5G4H/GOPNa53f8vM24CPAh8tr/3dOici/h9V4/gZ4KsRcQnVBw/f77HGc4FvR8T3qb7u8jX402bkvTxe3gpcFBHXs/7jZU+qzdK7dTbVh+5XtZ8QEUt6yANYHRH7rsvMas3jC4HTqTbR7daDEbFVZv6eao3Zuvq2pbf3oJT3JR8ur3kfjojVMK4eZVuqD5OC6vG97vEyg96W82uB/yhbx/wW+HF5vb+5nNaL9sfLH6k+oF0cEVv1mNm8ye5cB+1A9YZgdTn8DfAdqk9AVgHH9JD3MqoG70LgV8AhZXwnqu9s9VLjAmAF1bb+p5bDt8vYgh7yTgOeNcppvdb4XcqnRi1j06k2KXqoh7xLga3K8U1axrelh0+Y27JnUb14fBz41ThyVgI3UG0qdgOwcxmfQW+f6m0LfI5qs9JLqZrFG6g2gXpyjzWO+knbutu3y7y3lZpuovpk+SLg01Sf6h7fY41voVrj/WmqT3PXfcq3E/D9HvKeWm6/aygNM9Um1JcA+/WQ916q7waNdNoHepzz2Yywlrtc18M95J1HWePfNv5Y2tZW9JC9NdUn1WdRbVLaa87FbYd1j5cNPiHuMG9T4ITyPPsrqjcw91Kt9dmtxxq7/mR6jLy+vr6UzCZeY7an2kTs2HI4HNi+n7fFOG/HJwN7jXIfeEUPebsx8lY8uwDPG2etQfW1hS80dFtsR7UZYq+XfwLV95//hmpz3U3GkTWvgfk9g2rTUqi+h/l24KXjrPMFJefAlrFNgM17zNsEeHq5Df+mHJ/WxPLusb5ZwGNHOe2ZPeSNeDsBOwJz+lTzIVSb2vf7ttgK2GMcl39Uef7ZD5g5zlq63mplEA5+x3EEETGNao3R2oiYTrXZwKrMvLXHvB2AxwMrsk97pYuITai+NN66s4+lOY4vovdTRMyiWkt42winPTO73NlERGyemQ+MML4j1RvOZb1X+6esQ6ieRPu257uSuxXVE8yNPV7+UcAeVI33LTnCmq4usvbOau1J30TE4wAy89cRsR3V9vq/yszLxpH5JKo3NFdnD3sMHCXzsbQ8Xka6b06W8t0QMvP+EU7bJTNXbXipnq5na6pNQm/vQ9aTqd6wfmr8la2XO43qjcnvx5GxLdVa3N+Ns5YZ2bJjr37o9+tLyWziNWYm6z9een7eaSKvicxBz2si0xr7lznCdfT1+aOh5yNrHLC8pjL7xcaxTfS4K++JymvJ3Q24J6vN2mZTbXZ5bfa46/BR8n6emVf3ucaeM4exxmGcc4OZc2nZ6+R4G9J+51njYOYNW40RsS/wKaotHm6hWmM2i2oT7Tdk5pVd5j0F+GTJa91La095I9TYnvn32eUOjCY4r9c5192Og1JjXzOnQo1jXFdPO0ubqLwmMoexxqkw536ycWwT1XdrbqDaY+mXM/OaQcormQuB1wMP8Mie2n5ItXnEaZn5ocnMs8bBzBviGp9N9dMMd1FtXvJDqk3x/ggckV3+NEm/86xxMPOGuMargNdn5qVt408H/iu733tnX/OmQo3DOOchrnG074MG8K7M3GEy85rIHMYap8KcJ4qNY5uodo97BNV3UV5G9dMFX6b6mYuVk51XMpdTrYHZiup7dY/PR3YzfGl2uYvqfudZ42DmDXGNPwEOKhl7AB/KzBdHxIFUO0U6aDLzrHEw84a4xutzlJ8QiIgVmdnVDoH6nTcVahzGOQ9xjX8A/p2yQ7I2b8vM7SYzzxoHM6+pzIngXlU3lFltDvcu4F1R/UbU4f9/e/cXamlVxnH8+0wjmB7IHJtTMFFEOBIkQ1A3ijlSlBQWVqMGMXXRTdQIBkVQBNJFF90UGFk0WYTVZDUGWZiUgYT0Z/pjBkaFWFHWODPChGbW08Xa4W7P7N2cc/ba73pd3w+8OLPfc37zey5GZp1377WAe6I8Ot7QGT8V8qBsLvNYRDwBPEbZbp8su3duIm7peXZsM6/Xjs/Ip86qfIiytTmZ+d0oZzwOnWfHNvN67fjtKLuAfoGndol8PuXc0+80kDeGjj3O3GvHI8DhzPzp7I2I2Mxum8vOq5HZY8cxzLwSPnGcEXMOQ43yL9bLcguHkC4jb/K9t1C2bj+Xcij8k5T/4V1BOYtv35B5dmwzr+OOBynHKnyPckbSnzLzhiibFh3JzIuGzLNjm3m9dpxkXknZaXN687VvZuYdG82qkTeGjj3O3GPHiNgNHJv64c30vfXc4KY7y86zY5t5tTJXwYXjjIh4a2be2mreJHM78BbKPxZuo2yhfR3lp8035eQ8tKHy7NhmXscdzwLeCbyEcijwwcz8V5SdTHfmBs8hXHaeHdvM67WjJEnzuHCUJEkARDnK5AOUJzLrlB/g/JVybudHc4PHfSw7bwwde5zZjrwR2Nlanh3bzKuVuQrbhi7QmohYi4gbI+L+iHg0Iv4WEfdGxNtbyPs/mftbyLNjm3l2jF8t+e/0UvLs2GZerx2BQ8BxYG9mnp+ZO4C9lF1bDzWQN4aOPc7ce8fLZ/KON5JnxzbzamVW5xPHGRFxO/AN4C5gH+VzVl8GPkj57MiGDodfdp4d++nY48xj6NjjzGPo2OPMlTo+kJm7N3pvVXlj6NjjzHZsM8+ObebVylyJzPSauoBfzPz+x5P/bqMcQD5onh376djjzGPo2OPMY+jY48yVOt4JvA9Yn3ptHXg/cNfQeWPo2OPMdmwzz45t5tXKXMXlW1VP9feIuBQgIq4CjgFk5r+BzZwFsOw8O/bTsceZx9Cxx5nH0LHHmWtkXgPsAH4QEccj4hhwN3A+5Ynm0Hlj6NjjzHZsM8+ObebVyqxv6JVraxdwMfAjynuM7wEunLz+HODA0Hl27KdjjzOPoWOPM4+hY48zV8y8CHgVsDbz+mtbyBtDxx5ntmObeXZsM69WZu1r8AJjuoB3tJxnx3YzW8+zY5t5dmwz7+ncETgAPAAcBh4E3jB178jQeWPo2OPMdmwzz45t5tXKXMU1eIExXcBDLefZsd3M1vPs2GaeHdvMezp3BO5j8tNv4IXAT4DrJ7//2dB5Y+jY48x2bDPPjm3m1cpcxbUd/Y+I+OW8W5QPrQ6aVyPTjsvJbD2vRmaPHXucuUZm63k1MsfQEdiWmScBMvPBiLgcuC0iXsDmPjO57LwxdOxxZju2mWfHNvNqZVbnwvFU68BrKJ8XmRbADxvIq5Fpx+Vktp5XI7PHjj3OXCOz9bwamWPo+HBE7MnMnwNk5smIeD1wEHhpA3lj6NjjzHZsM8+ObebVyqxv6EeerV3AZ4FL59y7deg8O/bTsceZx9Cxx5nH0LHHmSt13AU8d869S4bOG0PHHme2Y5t5dmwzr1bmKq6YFJQkSZIk6bQ8x1GSJEmStJALR0mSJEnSQi4cJUmSJEkLuXCUJGkLIuKLEfG5mddeGRGPRMTNEfHPiDg5dZ2Y+dqIiN9HxK9Pk313RDw++b6jEfH1iHhe7ZkkSZrlwlGSpK25HrgyIl4NEBFnA58B3gv8GfhKZq5NXefNfP9lwE7gRRHx8tPkvzsz14AXA2vAx2oNIknSPC4cJUnagsx8BHgP8OmIOBf4MPC7zLzlDCP2A7cDd0x+Pe/POQEcBvZsqbAkSZuwfegCkiSNXWZ+NSKuBb4EXCz+drUAAAFkSURBVMIZLu4i4hzgzcC1wDOBmyPihsx84jRfuwO4Gvjt0opLknSGfOIoSdJyvAu4ArgxM/8w9fq+iDgxdX1/6t7VwD+AO4FvAWcBr5vJ/UREPAocBS6gPN2UJGmlXDhKkrQEmfkwZXF3/8ytQ5l53tS1d+re/sn9JzPzceBrnPp21QOZ+SzgYuDZwK5KI0iSNJdvVZUkaQARsYvyhPIVEfGmycvnAGdHxAWZeXT66zPzvoj4CHBTRLwsM3PFlSVJHfOJoyRJw3gb8BtgN+UzkXuAC4E/AtfN+Z7PA+vAVasoKEnSf7lwlCSprmtmznE8GRE7KW9J/WRm/mX6Aj7FnN1VJ5vmfBz40OrqS5IE4TtdJEmSJEmL+MRRkiRJkrSQC0dJkiRJ0kIuHCVJkiRJC7lwlCRJkiQt5MJRkiRJkrSQC0dJkiRJ0kIuHCVJkiRJC7lwlCRJkiQt5MJRkiRJkrTQfwCXMkWw4uVqEAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df_AY = df.groupby(['AY'])\n",
    "df_AY = df_AY.size().reset_index(name='Docs')\n",
    "\n",
    "\n",
    "df_AY.plot(x='AY', y='Docs', kind='bar', legend=False, grid=True, figsize=(15, 6))\n",
    "plt.title(\"Number of Docs per Year\")\n",
    "plt.ylabel('# of Patents', fontsize=13)\n",
    "plt.xlabel('YEAR', fontsize=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applying preprocessing tasks on metadat of patent\n",
    "Converting the metatadata such as inventors and assignees of each patent into a python list, then apply preprocessing task on each element in the list in order to remove undesired tokens.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.57 s, sys: 93.6 ms, total: 2.67 s\n",
      "Wall time: 2.67 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#preprocess of list fields\n",
    "#convert all IPCs in df into one list\n",
    "def toList(s):\n",
    "    \"\"\"\n",
    "    this method is to convert the list of IPCs in each row from a string to a python List\n",
    "    \"\"\"\n",
    "    s  = s.translate ({ord(c): \" \" for c in \"[]\"})\n",
    "    ss= []\n",
    "    for cls in s.strip().split(','):\n",
    "        ss.append(cls.strip())\n",
    "    return ss\n",
    "\n",
    "#apply toList method on all rows in the DF\n",
    "df['PA'] = df['PA'].map(lambda pa :   toList(pa))\n",
    "df['INV'] = df['INV'].map(lambda inv :   toList(inv))\n",
    "\n",
    "df.head()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>TI</th>\n",
       "      <th>AB</th>\n",
       "      <th>TECHF</th>\n",
       "      <th>BACKG</th>\n",
       "      <th>SUMM</th>\n",
       "      <th>CLMS</th>\n",
       "      <th>ICM</th>\n",
       "      <th>AY</th>\n",
       "      <th>IPCs</th>\n",
       "      <th>REF</th>\n",
       "      <th>PA</th>\n",
       "      <th>INV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>EP2000017943-0</td>\n",
       "      <td>recognition disk-shaped medium</td>\n",
       "      <td>recognition disk-like medium multimedia applic...</td>\n",
       "      <td>recognition disk-shaped medium multimedia appl...</td>\n",
       "      <td>identification labels adapted interpreted opti...</td>\n",
       "      <td>overcome drawbacks noted conventional types id...</td>\n",
       "      <td></td>\n",
       "      <td>G06K0019-06</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G06K0019-06, G06K0007-10]</td>\n",
       "      <td></td>\n",
       "      <td>[Video_System_Italia_S_r_l]</td>\n",
       "      <td>[Tassello_Stefano]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EP2003016733-0</td>\n",
       "      <td>optical pickup recording reproducing</td>\n",
       "      <td>optical pickup reproducing optical recording m...</td>\n",
       "      <td>optical pickup recording reproducing optical p...</td>\n",
       "      <td>recently practical short wavelength red laser ...</td>\n",
       "      <td>provide pickup recording reproducing optical r...</td>\n",
       "      <td>optical pickup recording reproducing optical m...</td>\n",
       "      <td>G11B0007-135</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G11B0007-135, G11B0007-125]</td>\n",
       "      <td></td>\n",
       "      <td>[Konica_Minolta_Opto_Inc]</td>\n",
       "      <td>[Arai_Norikazu Kojima_Toshiyuki Kiriki_Toshihi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>EP2011009984-0</td>\n",
       "      <td>large capacity sales mediation</td>\n",
       "      <td>animation sales mediation animation sales medi...</td>\n",
       "      <td>large capacity sales large capacity sales medi...</td>\n",
       "      <td>recent years distributing music network rapidl...</td>\n",
       "      <td>implemented consideration problems provide ani...</td>\n",
       "      <td>large capacity sales mediation terminal large ...</td>\n",
       "      <td>G07F0017-16</td>\n",
       "      <td>2001</td>\n",
       "      <td>[G07F0017-16, G06Q0030-06, G06Q0020-10, G06Q00...</td>\n",
       "      <td>[JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...</td>\n",
       "      <td>[NEC_Corporation]</td>\n",
       "      <td>[Maeda_Koji]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PCT1997010546-0</td>\n",
       "      <td>bridge client-server environment</td>\n",
       "      <td>software bridge introduced client client-serve...</td>\n",
       "      <td>bridge client-server environment distributed c...</td>\n",
       "      <td>overview object-oriented programming developme...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>G06F009-46</td>\n",
       "      <td>1996</td>\n",
       "      <td>[G06F009-46, G06F009-44, G06F0009-44, G06F0009...</td>\n",
       "      <td></td>\n",
       "      <td>[INTERNATIONAL_BUSINESS_MACHINES_CORPORATION C...</td>\n",
       "      <td>[COLYER_ADRIAN_MARK]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PCT1998021641-0</td>\n",
       "      <td>sections operating rates</td>\n",
       "      <td>core clocked perform operations clock frequenc...</td>\n",
       "      <td>sections operating rates high speed processors...</td>\n",
       "      <td>illustrates microprocessor microprocessor incl...</td>\n",
       "      <td>microprocessor levels sub-core clocked frequen...</td>\n",
       "      <td></td>\n",
       "      <td>G06F001-32</td>\n",
       "      <td>1997</td>\n",
       "      <td>[G06F001-32, G06F0001-08, G06F0009-30, G06F000...</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>[SAGER_DAVID_J FLETCHER_THOMAS_D HINTON_GLENN_...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                ID                                    TI  \\\n",
       "0   EP2000017943-0        recognition disk-shaped medium   \n",
       "1   EP2003016733-0  optical pickup recording reproducing   \n",
       "2   EP2011009984-0        large capacity sales mediation   \n",
       "3  PCT1997010546-0      bridge client-server environment   \n",
       "4  PCT1998021641-0              sections operating rates   \n",
       "\n",
       "                                                  AB  \\\n",
       "0  recognition disk-like medium multimedia applic...   \n",
       "1  optical pickup reproducing optical recording m...   \n",
       "2  animation sales mediation animation sales medi...   \n",
       "3  software bridge introduced client client-serve...   \n",
       "4  core clocked perform operations clock frequenc...   \n",
       "\n",
       "                                               TECHF  \\\n",
       "0  recognition disk-shaped medium multimedia appl...   \n",
       "1  optical pickup recording reproducing optical p...   \n",
       "2  large capacity sales large capacity sales medi...   \n",
       "3  bridge client-server environment distributed c...   \n",
       "4  sections operating rates high speed processors...   \n",
       "\n",
       "                                               BACKG  \\\n",
       "0  identification labels adapted interpreted opti...   \n",
       "1  recently practical short wavelength red laser ...   \n",
       "2  recent years distributing music network rapidl...   \n",
       "3  overview object-oriented programming developme...   \n",
       "4  illustrates microprocessor microprocessor incl...   \n",
       "\n",
       "                                                SUMM  \\\n",
       "0  overcome drawbacks noted conventional types id...   \n",
       "1  provide pickup recording reproducing optical r...   \n",
       "2  implemented consideration problems provide ani...   \n",
       "3  bridge client distributed object-oriented brid...   \n",
       "4  microprocessor levels sub-core clocked frequen...   \n",
       "\n",
       "                                                CLMS           ICM    AY  \\\n",
       "0                                                      G06K0019-06  2000   \n",
       "1  optical pickup recording reproducing optical m...  G11B0007-135  2000   \n",
       "2  large capacity sales mediation terminal large ...   G07F0017-16  2001   \n",
       "3  bridge client distributed object-oriented brid...    G06F009-46  1996   \n",
       "4                                                       G06F001-32  1997   \n",
       "\n",
       "                                                IPCs  \\\n",
       "0                         [G06K0019-06, G06K0007-10]   \n",
       "1                       [G11B0007-135, G11B0007-125]   \n",
       "2  [G07F0017-16, G06Q0030-06, G06Q0020-10, G06Q00...   \n",
       "3  [G06F009-46, G06F009-44, G06F0009-44, G06F0009...   \n",
       "4  [G06F001-32, G06F0001-08, G06F0009-30, G06F000...   \n",
       "\n",
       "                                                 REF  \\\n",
       "0                                                      \n",
       "1                                                      \n",
       "2  [JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...   \n",
       "3                                                      \n",
       "4                                                      \n",
       "\n",
       "                                                  PA  \\\n",
       "0                        [Video_System_Italia_S_r_l]   \n",
       "1                          [Konica_Minolta_Opto_Inc]   \n",
       "2                                  [NEC_Corporation]   \n",
       "3  [INTERNATIONAL_BUSINESS_MACHINES_CORPORATION C...   \n",
       "4                                                 []   \n",
       "\n",
       "                                                 INV  \n",
       "0                                 [Tassello_Stefano]  \n",
       "1  [Arai_Norikazu Kojima_Toshiyuki Kiriki_Toshihi...  \n",
       "2                                       [Maeda_Koji]  \n",
       "3                               [COLYER_ADRIAN_MARK]  \n",
       "4  [SAGER_DAVID_J FLETCHER_THOMAS_D HINTON_GLENN_...  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>TI</th>\n",
       "      <th>AB</th>\n",
       "      <th>TECHF</th>\n",
       "      <th>BACKG</th>\n",
       "      <th>SUMM</th>\n",
       "      <th>CLMS</th>\n",
       "      <th>ICM</th>\n",
       "      <th>AY</th>\n",
       "      <th>IPCs</th>\n",
       "      <th>REF</th>\n",
       "      <th>PA</th>\n",
       "      <th>INV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>EP2000017943-0</td>\n",
       "      <td>recognition disk-shaped medium</td>\n",
       "      <td>recognition disk-like medium multimedia applic...</td>\n",
       "      <td>recognition disk-shaped medium multimedia appl...</td>\n",
       "      <td>identification labels adapted interpreted opti...</td>\n",
       "      <td>overcome drawbacks noted conventional types id...</td>\n",
       "      <td></td>\n",
       "      <td>G06K0019-06</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G06K]</td>\n",
       "      <td></td>\n",
       "      <td>[Video_System_Italia_S_r_l]</td>\n",
       "      <td>[Tassello_Stefano]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EP2003016733-0</td>\n",
       "      <td>optical pickup recording reproducing</td>\n",
       "      <td>optical pickup reproducing optical recording m...</td>\n",
       "      <td>optical pickup recording reproducing optical p...</td>\n",
       "      <td>recently practical short wavelength red laser ...</td>\n",
       "      <td>provide pickup recording reproducing optical r...</td>\n",
       "      <td>optical pickup recording reproducing optical m...</td>\n",
       "      <td>G11B0007-135</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G11B]</td>\n",
       "      <td></td>\n",
       "      <td>[Konica_Minolta_Opto_Inc]</td>\n",
       "      <td>[Arai_Norikazu Kojima_Toshiyuki Kiriki_Toshihi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>EP2011009984-0</td>\n",
       "      <td>large capacity sales mediation</td>\n",
       "      <td>animation sales mediation animation sales medi...</td>\n",
       "      <td>large capacity sales large capacity sales medi...</td>\n",
       "      <td>recent years distributing music network rapidl...</td>\n",
       "      <td>implemented consideration problems provide ani...</td>\n",
       "      <td>large capacity sales mediation terminal large ...</td>\n",
       "      <td>G07F0017-16</td>\n",
       "      <td>2001</td>\n",
       "      <td>[G07F, G06Q, H04L, G06F]</td>\n",
       "      <td>[JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...</td>\n",
       "      <td>[NEC_Corporation]</td>\n",
       "      <td>[Maeda_Koji]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PCT1997010546-0</td>\n",
       "      <td>bridge client-server environment</td>\n",
       "      <td>software bridge introduced client client-serve...</td>\n",
       "      <td>bridge client-server environment distributed c...</td>\n",
       "      <td>overview object-oriented programming developme...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>G06F009-46</td>\n",
       "      <td>1996</td>\n",
       "      <td>[G06F]</td>\n",
       "      <td></td>\n",
       "      <td>[INTERNATIONAL_BUSINESS_MACHINES_CORPORATION C...</td>\n",
       "      <td>[COLYER_ADRIAN_MARK]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PCT1998021641-0</td>\n",
       "      <td>sections operating rates</td>\n",
       "      <td>core clocked perform operations clock frequenc...</td>\n",
       "      <td>sections operating rates high speed processors...</td>\n",
       "      <td>illustrates microprocessor microprocessor incl...</td>\n",
       "      <td>microprocessor levels sub-core clocked frequen...</td>\n",
       "      <td></td>\n",
       "      <td>G06F001-32</td>\n",
       "      <td>1997</td>\n",
       "      <td>[G06F]</td>\n",
       "      <td></td>\n",
       "      <td>[]</td>\n",
       "      <td>[SAGER_DAVID_J FLETCHER_THOMAS_D HINTON_GLENN_...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                ID                                    TI  \\\n",
       "0   EP2000017943-0        recognition disk-shaped medium   \n",
       "1   EP2003016733-0  optical pickup recording reproducing   \n",
       "2   EP2011009984-0        large capacity sales mediation   \n",
       "3  PCT1997010546-0      bridge client-server environment   \n",
       "4  PCT1998021641-0              sections operating rates   \n",
       "\n",
       "                                                  AB  \\\n",
       "0  recognition disk-like medium multimedia applic...   \n",
       "1  optical pickup reproducing optical recording m...   \n",
       "2  animation sales mediation animation sales medi...   \n",
       "3  software bridge introduced client client-serve...   \n",
       "4  core clocked perform operations clock frequenc...   \n",
       "\n",
       "                                               TECHF  \\\n",
       "0  recognition disk-shaped medium multimedia appl...   \n",
       "1  optical pickup recording reproducing optical p...   \n",
       "2  large capacity sales large capacity sales medi...   \n",
       "3  bridge client-server environment distributed c...   \n",
       "4  sections operating rates high speed processors...   \n",
       "\n",
       "                                               BACKG  \\\n",
       "0  identification labels adapted interpreted opti...   \n",
       "1  recently practical short wavelength red laser ...   \n",
       "2  recent years distributing music network rapidl...   \n",
       "3  overview object-oriented programming developme...   \n",
       "4  illustrates microprocessor microprocessor incl...   \n",
       "\n",
       "                                                SUMM  \\\n",
       "0  overcome drawbacks noted conventional types id...   \n",
       "1  provide pickup recording reproducing optical r...   \n",
       "2  implemented consideration problems provide ani...   \n",
       "3  bridge client distributed object-oriented brid...   \n",
       "4  microprocessor levels sub-core clocked frequen...   \n",
       "\n",
       "                                                CLMS           ICM    AY  \\\n",
       "0                                                      G06K0019-06  2000   \n",
       "1  optical pickup recording reproducing optical m...  G11B0007-135  2000   \n",
       "2  large capacity sales mediation terminal large ...   G07F0017-16  2001   \n",
       "3  bridge client distributed object-oriented brid...    G06F009-46  1996   \n",
       "4                                                       G06F001-32  1997   \n",
       "\n",
       "                       IPCs  \\\n",
       "0                    [G06K]   \n",
       "1                    [G11B]   \n",
       "2  [G07F, G06Q, H04L, G06F]   \n",
       "3                    [G06F]   \n",
       "4                    [G06F]   \n",
       "\n",
       "                                                 REF  \\\n",
       "0                                                      \n",
       "1                                                      \n",
       "2  [JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...   \n",
       "3                                                      \n",
       "4                                                      \n",
       "\n",
       "                                                  PA  \\\n",
       "0                        [Video_System_Italia_S_r_l]   \n",
       "1                          [Konica_Minolta_Opto_Inc]   \n",
       "2                                  [NEC_Corporation]   \n",
       "3  [INTERNATIONAL_BUSINESS_MACHINES_CORPORATION C...   \n",
       "4                                                 []   \n",
       "\n",
       "                                                 INV  \n",
       "0                                 [Tassello_Stefano]  \n",
       "1  [Arai_Norikazu Kojima_Toshiyuki Kiriki_Toshihi...  \n",
       "2                                       [Maeda_Koji]  \n",
       "3                               [COLYER_ADRIAN_MARK]  \n",
       "4  [SAGER_DAVID_J FLETCHER_THOMAS_D HINTON_GLENN_...  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def IPCtoList(s):\n",
    "    \"\"\"\n",
    "    this method is to convert the list of IPCs in each row from a string to a python List\n",
    "    \"\"\"\n",
    "    s  = s.translate ({ord(c): \" \" for c in \"[]\"})\n",
    "    ss= []\n",
    "    for cls in s.strip().split(','):\n",
    "        ss.append(cls.strip()[:4])\n",
    "    return list(set(ss))\n",
    "\n",
    "df['IPCs'] = df['IPCs'].map(lambda ipcs :   IPCtoList(ipcs))\n",
    "\n",
    "df.head()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.59 s, sys: 41.7 ms, total: 5.64 s\n",
      "Wall time: 5.65 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "def metadataPreprocessing(input):\n",
    "    newInput=' '\n",
    "    for item in input:\n",
    "        item = item.translate ({ord(c): \" \" for c in \"!@#$%^&*()'[]{};:,./<>?\\|`~°=\\\"+\"})\n",
    "        itms=' '\n",
    "        for itm in item.split():\n",
    "            itms= itms +' '+itm.strip()\n",
    "        newInput = newInput + ' '+ itms.strip().replace(' ','_')\n",
    "    return newInput.strip()\n",
    "\n",
    "df['PA'] = df['PA'].map(lambda pa :   metadataPreprocessing(pa))\n",
    "df['INV'] = df['INV'].map(lambda inv :   metadataPreprocessing(inv))\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Applying preprocessing tasks on texts of patent\n",
    "A simple preprocessing tasks such as tokenization, stopword removal, lemmatization, and converting letters into lower case are performed on each text section of each patent document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#preprocessing \n",
    "standardStopwordFile = \"sources/stopwords/stopwords-all.txt\"\n",
    "\n",
    "#loading terms from a file to a set\n",
    "def get_terms_from_file(filePath):\n",
    "    terms = set(line.strip() for line in open(filePath))\n",
    "    return terms\n",
    "\n",
    "#remove undiserd terms\n",
    "def remove_terms(termSet, phrase):\n",
    "    newPhrase = \"\"\n",
    "    for term in phrase.split():\n",
    "        if term.strip() not in termSet and len(term.strip())>2:\n",
    "            newPhrase = newPhrase + \" \" + term.strip()\n",
    "\n",
    "\n",
    "\n",
    "def clean_texts(doc):\n",
    "    #Remove punctuation from texts\n",
    "    doc = doc.translate ({ord(c): ' ' for c in \"0123456789!@#$%^&*()'/[]{};:,./<>?\\|`~°=\\\"+\"})\n",
    "    # split into tokens by white space\n",
    "    tokens = doc.lower().strip().split()\n",
    "    \n",
    "    # filter out stop words\n",
    "    stop_words = get_terms_from_file(standardStopwordFile)\n",
    "    #generalStopwords = get_terms_from_file(generalWordsFile)\n",
    "\n",
    "    \n",
    "    tokens = [w.strip('-')  for w in tokens if  w not in stop_words ]\n",
    "    # filter out short and long  tokens\n",
    "    output = [word for word in tokens if len(word.strip()) > 2 and len(word) < 30 ]\n",
    "    output = \" \".join(output)\n",
    "    #apply stemming\n",
    "    #output = stem_text(output)\n",
    "    return output\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 0 ns, sys: 5 µs, total: 5 µs\n",
      "Wall time: 8.82 µs\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "apply simple preprocessing on text\n",
    "df['TI'] = df['TI'].map(lambda line : clean_texts(line))\n",
    "df['AB'] = df['AB'].map(lambda line : clean_texts(line))\n",
    "df['TECHF'] = df['TECHF'].map(lambda line : clean_texts(line))\n",
    "df['BACKG'] = df['BACKG'].map(lambda line : clean_texts(line))\n",
    "df['SUMM'] = df['SUMM'].map(lambda line : clean_texts(line))\n",
    "df['CLMS'] = df['CLMS'].map(lambda line : clean_texts(line))\n",
    "\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Applying preprocessing tasks on patent labels ( IPC codes)\n",
    "The main IPC codes is considered to be the labels for the patent documents. we only consider the subclass level of the IPC code. \n",
    "Each label/class appears at least more than 3000 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of all IPCs in the dataset is:  774686\n",
      "The number of unique filtered IPCs to be used as labels is:  31\n"
     ]
    }
   ],
   "source": [
    "import collections\n",
    "\n",
    "#get a list of all unique IPCs\n",
    "IPCsDF = pd.DataFrame(df['IPCs'])\n",
    "ipcsList = [ipc for ipcs in IPCsDF['IPCs']  for ipc in ipcs]\n",
    "\n",
    "print('The number of all IPCs in the dataset is: ', len(ipcsList))\n",
    "\n",
    "\n",
    "IPCsDic = collections.Counter(ipcsList)\n",
    "uniqueIPCsList = [x[0] for x in IPCsDic.items() if x[1] >= 3000]\n",
    "\n",
    "print('The number of unique filtered IPCs to be used as labels is: ',len(uniqueIPCsList))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>TI</th>\n",
       "      <th>AB</th>\n",
       "      <th>TECHF</th>\n",
       "      <th>BACKG</th>\n",
       "      <th>SUMM</th>\n",
       "      <th>CLMS</th>\n",
       "      <th>ICM</th>\n",
       "      <th>AY</th>\n",
       "      <th>IPCs</th>\n",
       "      <th>REF</th>\n",
       "      <th>PA</th>\n",
       "      <th>INV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>EP2000017943-0</td>\n",
       "      <td>recognition disk-shaped medium</td>\n",
       "      <td>recognition disk-like medium multimedia applic...</td>\n",
       "      <td>recognition disk-shaped medium multimedia appl...</td>\n",
       "      <td>identification labels adapted interpreted opti...</td>\n",
       "      <td>overcome drawbacks noted conventional types id...</td>\n",
       "      <td></td>\n",
       "      <td>G06K0019-06</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G06K]</td>\n",
       "      <td></td>\n",
       "      <td>Video_System_Italia_S_r_l</td>\n",
       "      <td>Tassello_Stefano</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EP2003016733-0</td>\n",
       "      <td>optical pickup recording reproducing</td>\n",
       "      <td>optical pickup reproducing optical recording m...</td>\n",
       "      <td>optical pickup recording reproducing optical p...</td>\n",
       "      <td>recently practical short wavelength red laser ...</td>\n",
       "      <td>provide pickup recording reproducing optical r...</td>\n",
       "      <td>optical pickup recording reproducing optical m...</td>\n",
       "      <td>G11B0007-135</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G11B]</td>\n",
       "      <td></td>\n",
       "      <td>Konica_Minolta_Opto_Inc</td>\n",
       "      <td>Arai_Norikazu_Kojima_Toshiyuki_Kiriki_Toshihik...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>EP2011009984-0</td>\n",
       "      <td>large capacity sales mediation</td>\n",
       "      <td>animation sales mediation animation sales medi...</td>\n",
       "      <td>large capacity sales large capacity sales medi...</td>\n",
       "      <td>recent years distributing music network rapidl...</td>\n",
       "      <td>implemented consideration problems provide ani...</td>\n",
       "      <td>large capacity sales mediation terminal large ...</td>\n",
       "      <td>G07F0017-16</td>\n",
       "      <td>2001</td>\n",
       "      <td>[G07F, G06Q, H04L, G06F]</td>\n",
       "      <td>[JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...</td>\n",
       "      <td>NEC_Corporation</td>\n",
       "      <td>Maeda_Koji</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PCT1997010546-0</td>\n",
       "      <td>bridge client-server environment</td>\n",
       "      <td>software bridge introduced client client-serve...</td>\n",
       "      <td>bridge client-server environment distributed c...</td>\n",
       "      <td>overview object-oriented programming developme...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>G06F009-46</td>\n",
       "      <td>1996</td>\n",
       "      <td>[G06F]</td>\n",
       "      <td></td>\n",
       "      <td>INTERNATIONAL_BUSINESS_MACHINES_CORPORATION_CO...</td>\n",
       "      <td>COLYER_ADRIAN_MARK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PCT1998021641-0</td>\n",
       "      <td>sections operating rates</td>\n",
       "      <td>core clocked perform operations clock frequenc...</td>\n",
       "      <td>sections operating rates high speed processors...</td>\n",
       "      <td>illustrates microprocessor microprocessor incl...</td>\n",
       "      <td>microprocessor levels sub-core clocked frequen...</td>\n",
       "      <td></td>\n",
       "      <td>G06F001-32</td>\n",
       "      <td>1997</td>\n",
       "      <td>[G06F]</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>SAGER_DAVID_J_FLETCHER_THOMAS_D_HINTON_GLENN_J...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                ID                                    TI  \\\n",
       "0   EP2000017943-0        recognition disk-shaped medium   \n",
       "1   EP2003016733-0  optical pickup recording reproducing   \n",
       "2   EP2011009984-0        large capacity sales mediation   \n",
       "3  PCT1997010546-0      bridge client-server environment   \n",
       "4  PCT1998021641-0              sections operating rates   \n",
       "\n",
       "                                                  AB  \\\n",
       "0  recognition disk-like medium multimedia applic...   \n",
       "1  optical pickup reproducing optical recording m...   \n",
       "2  animation sales mediation animation sales medi...   \n",
       "3  software bridge introduced client client-serve...   \n",
       "4  core clocked perform operations clock frequenc...   \n",
       "\n",
       "                                               TECHF  \\\n",
       "0  recognition disk-shaped medium multimedia appl...   \n",
       "1  optical pickup recording reproducing optical p...   \n",
       "2  large capacity sales large capacity sales medi...   \n",
       "3  bridge client-server environment distributed c...   \n",
       "4  sections operating rates high speed processors...   \n",
       "\n",
       "                                               BACKG  \\\n",
       "0  identification labels adapted interpreted opti...   \n",
       "1  recently practical short wavelength red laser ...   \n",
       "2  recent years distributing music network rapidl...   \n",
       "3  overview object-oriented programming developme...   \n",
       "4  illustrates microprocessor microprocessor incl...   \n",
       "\n",
       "                                                SUMM  \\\n",
       "0  overcome drawbacks noted conventional types id...   \n",
       "1  provide pickup recording reproducing optical r...   \n",
       "2  implemented consideration problems provide ani...   \n",
       "3  bridge client distributed object-oriented brid...   \n",
       "4  microprocessor levels sub-core clocked frequen...   \n",
       "\n",
       "                                                CLMS           ICM    AY  \\\n",
       "0                                                      G06K0019-06  2000   \n",
       "1  optical pickup recording reproducing optical m...  G11B0007-135  2000   \n",
       "2  large capacity sales mediation terminal large ...   G07F0017-16  2001   \n",
       "3  bridge client distributed object-oriented brid...    G06F009-46  1996   \n",
       "4                                                       G06F001-32  1997   \n",
       "\n",
       "                       IPCs  \\\n",
       "0                    [G06K]   \n",
       "1                    [G11B]   \n",
       "2  [G07F, G06Q, H04L, G06F]   \n",
       "3                    [G06F]   \n",
       "4                    [G06F]   \n",
       "\n",
       "                                                 REF  \\\n",
       "0                                                      \n",
       "1                                                      \n",
       "2  [JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...   \n",
       "3                                                      \n",
       "4                                                      \n",
       "\n",
       "                                                  PA  \\\n",
       "0                          Video_System_Italia_S_r_l   \n",
       "1                            Konica_Minolta_Opto_Inc   \n",
       "2                                    NEC_Corporation   \n",
       "3  INTERNATIONAL_BUSINESS_MACHINES_CORPORATION_CO...   \n",
       "4                                                      \n",
       "\n",
       "                                                 INV  \n",
       "0                                   Tassello_Stefano  \n",
       "1  Arai_Norikazu_Kojima_Toshiyuki_Kiriki_Toshihik...  \n",
       "2                                         Maeda_Koji  \n",
       "3                                 COLYER_ADRIAN_MARK  \n",
       "4  SAGER_DAVID_J_FLETCHER_THOMAS_D_HINTON_GLENN_J...  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#update the dataset (DF) according to te unique filtered IPCs \n",
    "def filterOutIPCs(docIPCList, uniqueALlIPCList):\n",
    "    \"\"\"\n",
    "    this method is to filter out all IPcs according to te unique filtered IPCs \n",
    "    \"\"\"\n",
    "    \n",
    "    newIPCList= []\n",
    "    for cls in docIPCList:\n",
    "        if cls in uniqueALlIPCList:\n",
    "            newIPCList.append(cls.strip())\n",
    "    if not newIPCList:\n",
    "        return None\n",
    "    \n",
    "    return newIPCList\n",
    "\n",
    "#apply filterOutIPCs method on all rows in the DF\n",
    "df['IPCs'] = df['IPCs'].map(lambda ipcs :   filterOutIPCs(ipcs, uniqueIPCsList))\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "unique Count:  ID       420787\n",
      "TI       420787\n",
      "AB       420787\n",
      "TECHF    420787\n",
      "BACKG    420787\n",
      "SUMM     420787\n",
      "CLMS     420787\n",
      "ICM      420787\n",
      "AY       420787\n",
      "IPCs     420787\n",
      "REF      420787\n",
      "PA       420787\n",
      "INV      420787\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "#again filter out rows with empty texts\n",
    "df.dropna(subset=['IPCs'], inplace=True)\n",
    "print(\"unique Count: \", df.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Randomly reorder a dataset by rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>TI</th>\n",
       "      <th>AB</th>\n",
       "      <th>TECHF</th>\n",
       "      <th>BACKG</th>\n",
       "      <th>SUMM</th>\n",
       "      <th>CLMS</th>\n",
       "      <th>ICM</th>\n",
       "      <th>AY</th>\n",
       "      <th>IPCs</th>\n",
       "      <th>REF</th>\n",
       "      <th>PA</th>\n",
       "      <th>INV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>28536</th>\n",
       "      <td>PCT2000077740-0</td>\n",
       "      <td>geometric compression three-dimensional graphics</td>\n",
       "      <td>compressing geometry capable compressing regul...</td>\n",
       "      <td>geometric compression three-dimensional graphi...</td>\n",
       "      <td>three-dimensional graphics systems employing l...</td>\n",
       "      <td>problems outlined solved compressing geometry ...</td>\n",
       "      <td></td>\n",
       "      <td>G06T009-00</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G06T]</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>DEERING_MICHAEL_F</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28135</th>\n",
       "      <td>PCT2013133985-0</td>\n",
       "      <td>entity augmentation latent relational</td>\n",
       "      <td>directed providing augmenting entity-attribute...</td>\n",
       "      <td></td>\n",
       "      <td>entity augmentation latent relational workers ...</td>\n",
       "      <td>provided introduce representative concepts sim...</td>\n",
       "      <td>computing environment performed augmentation a...</td>\n",
       "      <td>G06F0017-00</td>\n",
       "      <td>2013</td>\n",
       "      <td>[G06F]</td>\n",
       "      <td></td>\n",
       "      <td>MICROSOFT_CORPORATION</td>\n",
       "      <td>GANJAM_Kris_K_CHAKRABARTI_Kaushik_YAKOUT_Moham...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>185746</th>\n",
       "      <td>PCT2003010617-0</td>\n",
       "      <td>methods equipment analysing biological signals...</td>\n",
       "      <td>concerns representing analysing pressure varia...</td>\n",
       "      <td></td>\n",
       "      <td>processes equipment analysis biological signal...</td>\n",
       "      <td>aims addressing providing representation analy...</td>\n",
       "      <td>system representation analysis pressure variab...</td>\n",
       "      <td>G06F</td>\n",
       "      <td>2002</td>\n",
       "      <td>[G06F, A61B]</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>LEMAIRE_JEAN-JACQUES</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39935</th>\n",
       "      <td>PCT2002003177-0</td>\n",
       "      <td>identifying persons seeking computers networks</td>\n",
       "      <td>verifying identity seeking directly network pr...</td>\n",
       "      <td>identifying persons seeking computers networks...</td>\n",
       "      <td>security violations databases constitute major...</td>\n",
       "      <td>primarily serve positively identify seeking di...</td>\n",
       "      <td>verifying identity seeking gain local communic...</td>\n",
       "      <td>G06F001-00</td>\n",
       "      <td>2001</td>\n",
       "      <td>[G06F, H04L]</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>DOR_EREZ_DRACH_ZIPORA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31628</th>\n",
       "      <td>PCT2003098393-0</td>\n",
       "      <td>routing securely share network host utilizing ...</td>\n",
       "      <td>providing selectively share network routing ho...</td>\n",
       "      <td></td>\n",
       "      <td>routing securely sharenetwork host utilizing h...</td>\n",
       "      <td>provide restricting host accessing network int...</td>\n",
       "      <td>claimed node wireless ad-hoc communications ne...</td>\n",
       "      <td>G06F</td>\n",
       "      <td>2003</td>\n",
       "      <td>[G06F, H04W, H04L]</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>SCHMIDT_JEFFREY_GUTIERREZ_PHILIP_BARKER_JR_CHA...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     ID                                                 TI  \\\n",
       "28536   PCT2000077740-0   geometric compression three-dimensional graphics   \n",
       "28135   PCT2013133985-0              entity augmentation latent relational   \n",
       "185746  PCT2003010617-0  methods equipment analysing biological signals...   \n",
       "39935   PCT2002003177-0     identifying persons seeking computers networks   \n",
       "31628   PCT2003098393-0  routing securely share network host utilizing ...   \n",
       "\n",
       "                                                       AB  \\\n",
       "28536   compressing geometry capable compressing regul...   \n",
       "28135   directed providing augmenting entity-attribute...   \n",
       "185746  concerns representing analysing pressure varia...   \n",
       "39935   verifying identity seeking directly network pr...   \n",
       "31628   providing selectively share network routing ho...   \n",
       "\n",
       "                                                    TECHF  \\\n",
       "28536   geometric compression three-dimensional graphi...   \n",
       "28135                                                       \n",
       "185746                                                      \n",
       "39935   identifying persons seeking computers networks...   \n",
       "31628                                                       \n",
       "\n",
       "                                                    BACKG  \\\n",
       "28536   three-dimensional graphics systems employing l...   \n",
       "28135   entity augmentation latent relational workers ...   \n",
       "185746  processes equipment analysis biological signal...   \n",
       "39935   security violations databases constitute major...   \n",
       "31628   routing securely sharenetwork host utilizing h...   \n",
       "\n",
       "                                                     SUMM  \\\n",
       "28536   problems outlined solved compressing geometry ...   \n",
       "28135   provided introduce representative concepts sim...   \n",
       "185746  aims addressing providing representation analy...   \n",
       "39935   primarily serve positively identify seeking di...   \n",
       "31628   provide restricting host accessing network int...   \n",
       "\n",
       "                                                     CLMS          ICM    AY  \\\n",
       "28536                                                       G06T009-00  2000   \n",
       "28135   computing environment performed augmentation a...  G06F0017-00  2013   \n",
       "185746  system representation analysis pressure variab...         G06F  2002   \n",
       "39935   verifying identity seeking gain local communic...   G06F001-00  2001   \n",
       "31628   claimed node wireless ad-hoc communications ne...         G06F  2003   \n",
       "\n",
       "                      IPCs REF                     PA  \\\n",
       "28536               [G06T]                              \n",
       "28135               [G06F]      MICROSOFT_CORPORATION   \n",
       "185746        [G06F, A61B]                              \n",
       "39935         [G06F, H04L]                              \n",
       "31628   [G06F, H04W, H04L]                              \n",
       "\n",
       "                                                      INV  \n",
       "28536                                   DEERING_MICHAEL_F  \n",
       "28135   GANJAM_Kris_K_CHAKRABARTI_Kaushik_YAKOUT_Moham...  \n",
       "185746                               LEMAIRE_JEAN-JACQUES  \n",
       "39935                               DOR_EREZ_DRACH_ZIPORA  \n",
       "31628   SCHMIDT_JEFFREY_GUTIERREZ_PHILIP_BARKER_JR_CHA...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "df = shuffle(df)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# split the dataset into train and test datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(336629,)\n",
      "(84158,)\n"
     ]
    }
   ],
   "source": [
    "# lets take n% data as training and remaining m% for test.\n",
    "train_size = int(len(df) * .8)\n",
    "\n",
    "train_TI = df['TI'][:train_size]\n",
    "train_AB = df['AB'][:train_size]\n",
    "train_TECHF = df['TECHF'][:train_size]\n",
    "train_BACKG = df['BACKG'][:train_size]\n",
    "train_SUMM = df['SUMM'][:train_size]\n",
    "train_CLMS = df['CLMS'][:train_size]\n",
    "train_ICM= df['IPCs'][:train_size]\n",
    "train_ID= df['ID'][:train_size]\n",
    "\n",
    "test_TI = df['TI'][train_size:]\n",
    "test_AB = df['AB'][train_size:]\n",
    "test_TECHF = df['TECHF'][train_size:]\n",
    "test_BACKG = df['BACKG'][train_size:]\n",
    "test_SUMM = df['SUMM'][train_size:]\n",
    "test_CLMS = df['CLMS'][train_size:]\n",
    "test_ICM = df['IPCs'][train_size:]\n",
    "test_ID = df['ID'][train_size:]\n",
    "\n",
    "\n",
    "#metadata\n",
    "train_pa_series = df['PA'][:train_size]\n",
    "test_pa_series = df['PA'][train_size:]\n",
    "\n",
    "train_inv_series = df['INV'][:train_size]\n",
    "test_inv_series = df['INV'][train_size:]\n",
    "\n",
    "\n",
    "print(train_AB.shape)\n",
    "print(test_AB.shape)\n",
    "\n",
    "#free up some memory space\n",
    "#df.iloc[0:0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Applying tokenization process \n",
    "For texts of each segment, a Keras tokenization process is used for breaking the text into individual words, and  set the sequence length of each segment according to the length of each.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.utils import to_categorical\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.preprocessing.text import one_hot\n",
    "from sklearn.preprocessing import LabelBinarizer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying Keras tokenization on Metadata of patent(Inventors, Assignees), and convert the related text into One-hot that encodes a text into a list of word indexes of size n."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 91324 words in PA\n",
      "Found 264576 words in INV\n"
     ]
    }
   ],
   "source": [
    "#PA\n",
    "pa_inv_vocab_size = 2000\n",
    "pa_tokenizer = Tokenizer(num_words=pa_inv_vocab_size,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "pa_tokenizer.fit_on_texts(train_pa_series)\n",
    "train_pa_one_hot =pa_tokenizer.texts_to_matrix(train_pa_series)\n",
    "test_pa_one_hot =pa_tokenizer.texts_to_matrix(test_pa_series)\n",
    "\n",
    "\n",
    "#INV\n",
    "inv_tokenizer = Tokenizer(num_words=pa_inv_vocab_size,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "inv_tokenizer.fit_on_texts(train_inv_series)\n",
    "train_inv_one_hot =inv_tokenizer.texts_to_matrix(train_inv_series)\n",
    "test_inv_one_hot =inv_tokenizer.texts_to_matrix(test_inv_series)\n",
    "\n",
    "\n",
    "print('Found %s words in PA' % len(pa_tokenizer.word_index))\n",
    "print('Found %s words in INV' % len(inv_tokenizer.word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "- Transform each text in Titles (train and test datasets) into a sequence of integers. <br>\n",
    "- set the sequence length.<br>\n",
    "- Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 10.3 s, sys: 64.3 ms, total: 10.3 s\n",
      "Wall time: 10.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#Title\n",
    "TI_tokenizer = Tokenizer(num_words=10000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "TI_tokenizer.fit_on_texts(train_TI)\n",
    "encoded_train_TI = TI_tokenizer.texts_to_sequences(train_TI)\n",
    "encoded_test_TI = TI_tokenizer.texts_to_sequences(test_TI)\n",
    "#convert all sequences in a list into the same length\n",
    "TI_train = pad_sequences(encoded_train_TI,  maxlen=20, padding='post')\n",
    "TI_test = pad_sequences(encoded_test_TI,  maxlen=20, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Abstrcat (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 28.7 s, sys: 311 ms, total: 29 s\n",
      "Wall time: 29.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#Abstract\n",
    "AB_tokenizer = Tokenizer(num_words=50000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "AB_tokenizer.fit_on_texts(train_AB)\n",
    "encoded_train_AB = AB_tokenizer.texts_to_sequences(train_AB)\n",
    "encoded_test_AB = AB_tokenizer.texts_to_sequences(test_AB)\n",
    "#convert all sequences in a list into the same length\n",
    "AB_train = pad_sequences(encoded_train_AB,  maxlen=100, padding='post')\n",
    "AB_test = pad_sequences(encoded_test_AB,  maxlen=100, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Technical Field (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 18.2 s, sys: 145 ms, total: 18.3 s\n",
      "Wall time: 18.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#TECHNICAL_FIELD\n",
    "TECHF_tokenizer = Tokenizer(num_words=20000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "TECHF_tokenizer.fit_on_texts(train_TECHF)\n",
    "encoded_train_TECHF = TECHF_tokenizer.texts_to_sequences(train_TECHF)\n",
    "encoded_test_TECHF = TECHF_tokenizer.texts_to_sequences(test_TECHF)\n",
    "#convert all sequences in a list into the same length\n",
    "TECHF_train = pad_sequences(encoded_train_TECHF,  maxlen=30, padding='post')\n",
    "TECHF_test = pad_sequences(encoded_test_TECHF,  maxlen=30, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Background (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 3s, sys: 1 s, total: 2min 4s\n",
      "Wall time: 2min 5s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#BACKGROUND\n",
    "BACKG_tokenizer = Tokenizer(num_words=50000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "BACKG_tokenizer.fit_on_texts(train_BACKG)\n",
    "encoded_train_BACKG = BACKG_tokenizer.texts_to_sequences(train_BACKG)\n",
    "encoded_test_BACKG = BACKG_tokenizer.texts_to_sequences(test_BACKG)\n",
    "#convert all sequences in a list into the same length\n",
    "BACKG_train = pad_sequences(encoded_train_BACKG,  maxlen=100, padding='post')\n",
    "BACKG_test = pad_sequences(encoded_test_BACKG,  maxlen=100, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Summary (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 43s, sys: 1.19 s, total: 2min 44s\n",
      "Wall time: 2min 44s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#SUMMARY\n",
    "SUMM_tokenizer = Tokenizer(num_words=50000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "SUMM_tokenizer.fit_on_texts(train_SUMM)\n",
    "encoded_train_SUMM = SUMM_tokenizer.texts_to_sequences(train_SUMM)\n",
    "encoded_test_SUMM = SUMM_tokenizer.texts_to_sequences(test_SUMM)\n",
    "#convert all sequences in a list into the same length\n",
    "SUMM_train = pad_sequences(encoded_train_SUMM,  maxlen=100, padding='post')\n",
    "SUMM_test = pad_sequences(encoded_test_SUMM,  maxlen=100, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Independent Claim (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 39.5 s, sys: 845 ms, total: 40.3 s\n",
      "Wall time: 40.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#CLAIMS\n",
    "CLMS_tokenizer = Tokenizer(num_words=50000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "CLMS_tokenizer.fit_on_texts(train_CLMS)\n",
    "encoded_train_CLMS = CLMS_tokenizer.texts_to_sequences(train_CLMS)\n",
    "encoded_test_CLMS = CLMS_tokenizer.texts_to_sequences(test_CLMS)\n",
    "#convert all sequences in a list into the same length\n",
    "CLMS_train = pad_sequences(encoded_train_CLMS,  maxlen=100, padding='post')\n",
    "CLMS_test = pad_sequences(encoded_test_CLMS,  maxlen=100, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Representing the labels/classes in the numeric format by scikit-learn - MultiLabelBinarizer class. <br>\n",
    "Convert 1-dimensional class arrays to n-dimensional(#classes) class matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 655 ms, sys: 70.8 ms, total: 726 ms\n",
      "Wall time: 725 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "\n",
    "multiLabelBinarizer =  MultiLabelBinarizer(classes=uniqueIPCsList)\n",
    "\n",
    "\n",
    "y_train = multiLabelBinarizer.fit_transform(train_ICM)\n",
    "y_test = multiLabelBinarizer.fit_transform(test_ICM)\n",
    "\n",
    "num_classes = len(uniqueIPCsList)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  load the whole embeddings model into memory and get matrix\n",
    "We load a pre-trained word2vec word embedding model that was trained on five million patents (Titles and abstracts) <br>\n",
    "\n",
    "The Embeddings model is availabel <a href=https://www.kaggle.com/darshmso/w2vec-patent-domain > here.</a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def load_embedding_model(filePath):\n",
    "    embeddings_index = dict()\n",
    "    f = open(filePath, encoding='utf8')\n",
    "    for line in f:\n",
    "        values = line.split()\n",
    "        word = values[0]\n",
    "        coefs = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = coefs\n",
    "        \n",
    "    return embeddings_index\n",
    "\n",
    "def create_embedding_matrix(tokenizer, embeddings_index, vocab_size_embbs, dim_size):\n",
    "    embeddings_matrix = np.zeros((vocab_size_embbs, dim_size))\n",
    "    for word, i in tokenizer.word_index.items():\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if embedding_vector is not None:\n",
    "            embeddings_matrix[i] = embedding_vector[0:dim_size]\n",
    "    \n",
    "    return embeddings_matrix\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Loading the whole embeddings into memory and get matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 28s, sys: 2.6 s, total: 1min 30s\n",
      "Wall time: 1min 30s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "embedding_index = load_embedding_model('../models/w2v/phrase/patWordPhrase2VecModel.txt')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Creating TITLE embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 84.6 ms, sys: 54.9 ms, total: 139 ms\n",
      "Wall time: 138 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(TI_tokenizer.word_index) + 1\n",
    "\n",
    "TI_embeddings_matrix = create_embedding_matrix(TI_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              20)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Creating ABSTRACT embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 240 ms, sys: 109 ms, total: 349 ms\n",
      "Wall time: 349 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(AB_tokenizer.word_index) + 1\n",
    "AB_embeddings_matrix = create_embedding_matrix(AB_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              100)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating TECHNICAL_FIELD embedding Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 160 ms, sys: 25 ms, total: 185 ms\n",
      "Wall time: 184 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(TECHF_tokenizer.word_index) + 1\n",
    "TECHF_embeddings_matrix = create_embedding_matrix(TECHF_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating BACKGROUND embedding Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 556 ms, sys: 192 ms, total: 749 ms\n",
      "Wall time: 749 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(BACKG_tokenizer.word_index) + 1\n",
    "BACKG_embeddings_matrix = create_embedding_matrix(BACKG_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating SUMMARY embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 523 ms, sys: 165 ms, total: 688 ms\n",
      "Wall time: 689 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(SUMM_tokenizer.word_index) + 1\n",
    "SUMM_embeddings_matrix = create_embedding_matrix(SUMM_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating INDEPENDENT CLAIMS embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 176 ms, sys: 50.9 ms, total: 227 ms\n",
      "Wall time: 226 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(CLMS_tokenizer.word_index) + 1\n",
    "CLMS_embeddings_matrix = create_embedding_matrix(CLMS_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a Deep Layer for each Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, Input, Embedding, BatchNormalization, ELU, Concatenate\n",
    "from keras.layers import LSTM, Conv1D, MaxPooling1D\n",
    "from keras.layers.merge import concatenate\n",
    "from keras.layers.core import Dropout\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for Title Embeddings<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.65 s, sys: 8.49 s, total: 13.1 s\n",
      "Wall time: 4.02 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#TITLE \n",
    "sequence_len =20\n",
    "dropout_pct =  0.3\n",
    "\n",
    "TI_embedding_layer_input = Input(shape=(sequence_len,), name='TI_embed_input')\n",
    "TI_embedding_layer = Embedding(input_dim=len(TI_tokenizer.word_index) + 1,\n",
    "                        output_dim=20, # Dimension of the dense embedding\n",
    "                        weights=[TI_embeddings_matrix],\n",
    "                        input_length=20)(TI_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "TI_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_TI')(TI_embedding_layer)\n",
    "\n",
    "TI_deep = Dense(300, activation=None)(TI_deep)\n",
    "TI_deep = Dropout(dropout_pct)(TI_deep)\n",
    "TI_deep = BatchNormalization()(TI_deep)\n",
    "TI_deep = ELU()(TI_deep)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for Abstract Embeddings<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.42 s, sys: 6.4 s, total: 9.82 s\n",
      "Wall time: 754 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#Abstract \n",
    "sequence_len =100\n",
    "dropout_pct =  0.3\n",
    "\n",
    "AB_embedding_layer_input = Input(shape=(sequence_len,), name='AB_embed_input')\n",
    "AB_embedding_layer = Embedding(input_dim=len(AB_tokenizer.word_index) + 1,\n",
    "                        output_dim=100, # Dimension of the dense embedding\n",
    "                        weights=[AB_embeddings_matrix],\n",
    "                        input_length=100)(AB_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "AB_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_AB')(AB_embedding_layer)\n",
    "\n",
    "AB_deep = Dense(300, activation=None)(AB_deep)\n",
    "AB_deep = Dropout(dropout_pct)(AB_deep)\n",
    "AB_deep = BatchNormalization()(AB_deep)\n",
    "AB_deep = ELU()(AB_deep)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for TECHNICAL-Field Embeddings<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.56 s, sys: 6.17 s, total: 9.72 s\n",
      "Wall time: 667 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#TECHNICAL-Field \n",
    "sequence_len =30\n",
    "dropout_pct =  0.3\n",
    "\n",
    "TECHF_embedding_layer_input = Input(shape=(sequence_len,), name='TECHF_embed_input')\n",
    "TECHF_embedding_layer = Embedding(input_dim=len(TECHF_tokenizer.word_index) + 1,\n",
    "                        output_dim=30, # Dimension of the dense embedding\n",
    "                        weights=[TECHF_embeddings_matrix],\n",
    "                        input_length=30)(TECHF_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "TECHF_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_TECHF')(TECHF_embedding_layer)\n",
    "\n",
    "TECHF_deep = Dense(300, activation=None)(TECHF_deep)\n",
    "TECHF_deep = Dropout(dropout_pct)(TECHF_deep)\n",
    "TECHF_deep = BatchNormalization()(TECHF_deep)\n",
    "TECHF_deep = ELU()(TECHF_deep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for BACKGROUND Embeddings<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.81 s, sys: 7.12 s, total: 10.9 s\n",
      "Wall time: 1.85 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#BACKGROUND \n",
    "sequence_len =100\n",
    "dropout_pct =  0.3\n",
    "\n",
    "BACKG_embedding_layer_input = Input(shape=(sequence_len,), name='BACKG_embed_input')\n",
    "BACKG_embedding_layer = Embedding(input_dim=len(BACKG_tokenizer.word_index) + 1,\n",
    "                        output_dim=100, # Dimension of the dense embedding\n",
    "                        weights=[BACKG_embeddings_matrix],\n",
    "                        input_length=100)(BACKG_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "BACKG_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_BACK')(BACKG_embedding_layer)\n",
    "\n",
    "BACKG_deep = Dense(300, activation=None)(BACKG_deep)\n",
    "BACKG_deep = Dropout(dropout_pct)(BACKG_deep)\n",
    "BACKG_deep = BatchNormalization()(BACKG_deep)\n",
    "BACKG_deep = ELU()(BACKG_deep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for Summary Embeddings<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.85 s, sys: 6.52 s, total: 10.4 s\n",
      "Wall time: 1.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#SUMMARY\n",
    "sequence_len =100\n",
    "dropout_pct =  0.3\n",
    "\n",
    "SUMM_embedding_layer_input = Input(shape=(sequence_len,), name='SUMM_embed_input')\n",
    "SUMM_embedding_layer = Embedding(input_dim=len(SUMM_tokenizer.word_index) + 1,\n",
    "                        output_dim=100, # Dimension of the dense embedding\n",
    "                        weights=[SUMM_embeddings_matrix],\n",
    "                        input_length=100)(SUMM_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "SUMM_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_SUMM')(SUMM_embedding_layer)\n",
    "\n",
    "SUMM_deep = Dense(300, activation=None)(SUMM_deep)\n",
    "SUMM_deep = Dropout(dropout_pct)(SUMM_deep)\n",
    "SUMM_deep = BatchNormalization()(SUMM_deep)\n",
    "SUMM_deep = ELU()(SUMM_deep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for Independent Claim Embeddings<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.75 s, sys: 6.27 s, total: 10 s\n",
      "Wall time: 948 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#CLAIMS \n",
    "sequence_len =100\n",
    "dropout_pct =  0.4\n",
    "\n",
    "\n",
    "CLMS_embedding_layer_input = Input(shape=(sequence_len,), name='CLMS_embed_input')\n",
    "CLMS_embedding_layer = Embedding(input_dim=len(CLMS_tokenizer.word_index) + 1,\n",
    "                        output_dim=100, # Dimension of the dense embedding\n",
    "                        weights=[CLMS_embeddings_matrix],\n",
    "                        input_length=100)(CLMS_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "CLMS_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_CLMS')(CLMS_embedding_layer)\n",
    "\n",
    "CLMS_deep = Dense(300, activation=None)(CLMS_deep)\n",
    "CLMS_deep = Dropout(dropout_pct)(CLMS_deep)\n",
    "CLMS_deep = BatchNormalization()(CLMS_deep)\n",
    "CLMS_deep = ELU()(CLMS_deep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layers for one-hot vectors of Inventors and Assignees<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pa_input and inv_input layers are finished\n"
     ]
    }
   ],
   "source": [
    "dropout_pct =  0.3\n",
    "pa_input = Input(shape=(train_pa_one_hot.shape[1],), name='pa_input') \n",
    "pas = Dense(32,input_dim=train_pa_one_hot.shape[1], activation=None)(pa_input) \n",
    "pas = Dropout(dropout_pct)(pas)\n",
    "pas = BatchNormalization()(pas)\n",
    "pas = ELU()(pas)\n",
    "\n",
    "#inv\n",
    "inv_input = Input(shape=(train_inv_one_hot.shape[1],), name='inv_input') \n",
    "invs = Dense(32,input_dim=train_inv_one_hot.shape[1], activation=None)(pa_input) \n",
    "invs = Dropout(dropout_pct)(invs)\n",
    "invs = BatchNormalization()(invs)\n",
    "\n",
    "print('pa_input and inv_input layers are finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train Model\n",
    "The following cells specify the neural network architecture and hyperparameters. <br>\n",
    "\n",
    "The model is generally composed of:<br>\n",
    "\n",
    "- contacting sequential word embeddings ofpatent text segments into a fully-connected layer\n",
    "-  binary_crossentropy is used for calculatinges the mean accuracy rate across all predictions for multi-label classification problems <br>\n",
    "- Compile the  the Network. <br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "TI_embed_input (InputLayer)     (None, 20)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "AB_embed_input (InputLayer)     (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "TECHF_embed_input (InputLayer)  (None, 30)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "BACKG_embed_input (InputLayer)  (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "SUMM_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "CLMS_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 20, 20)       899240      TI_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 100, 100)     15099300    AB_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 30, 30)       4542330     TECHF_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_4 (Embedding)         (None, 100, 100)     75638800    BACKG_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_5 (Embedding)         (None, 100, 100)     66417600    SUMM_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "embedding_6 (Embedding)         (None, 100, 100)     14941900    CLMS_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TI (LSTM)                  (None, 64)           21760       embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_AB (LSTM)                  (None, 64)           42240       embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TECHF (LSTM)               (None, 64)           24320       embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_BACK (LSTM)                (None, 64)           42240       embedding_4[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_SUMM (LSTM)                (None, 64)           42240       embedding_5[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_CLMS (LSTM)                (None, 64)           42240       embedding_6[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 300)          19500       LSTM_TI[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 300)          19500       LSTM_AB[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 300)          19500       LSTM_TECHF[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 300)          19500       LSTM_BACK[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 300)          19500       LSTM_SUMM[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 300)          19500       LSTM_CLMS[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 300)          0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 300)          0           dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 300)          0           dense_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 300)          0           dense_4[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, 300)          0           dense_5[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_6 (Dropout)             (None, 300)          0           dense_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 300)          1200        dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 300)          1200        dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 300)          1200        dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 300)          1200        dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 300)          1200        dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, 300)          1200        dropout_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_1 (ELU)                     (None, 300)          0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_2 (ELU)                     (None, 300)          0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_3 (ELU)                     (None, 300)          0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_4 (ELU)                     (None, 300)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_5 (ELU)                     (None, 300)          0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_6 (ELU)                     (None, 300)          0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "concatenated_layer (Concatenate (None, 1800)         0           elu_1[0][0]                      \n",
      "                                                                 elu_2[0][0]                      \n",
      "                                                                 elu_3[0][0]                      \n",
      "                                                                 elu_4[0][0]                      \n",
      "                                                                 elu_5[0][0]                      \n",
      "                                                                 elu_6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dense_9 (Dense)                 (None, 128)          230528      concatenated_layer[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_9 (Dropout)             (None, 128)          0           dense_9[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, 128)          512         dropout_9[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_8 (ELU)                     (None, 128)          0           batch_normalization_9[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dense_10 (Dense)                (None, 31)           3999        elu_8[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 178,113,449\n",
      "Trainable params: 178,109,593\n",
      "Non-trainable params: 3,856\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import keras_metrics as km\n",
    "#contacting \n",
    "model_inputs_to_concat = [TI_deep, AB_deep, TECHF_deep, BACKG_deep, SUMM_deep, CLMS_deep] #invs , pas, invs\n",
    "final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)\n",
    "\n",
    "output = Dense(128, activation=None)(final_layer)\n",
    "output = Dropout(dropout_pct)(output)\n",
    "output = BatchNormalization()(output)\n",
    "output = ELU()(output)\n",
    "output = Dense(num_classes, activation='sigmoid')(output)\n",
    "\n",
    "model = Model(inputs=[TI_embedding_layer_input,\n",
    "                      AB_embedding_layer_input,\n",
    "                      TECHF_embedding_layer_input,\n",
    "                      BACKG_embedding_layer_input,\n",
    "                     SUMM_embedding_layer_input,\n",
    "                     CLMS_embedding_layer_input,\n",
    "                     ],\n",
    "              outputs=output, name='model')\n",
    "\n",
    "# Calculate precision for the second label.\n",
    "precision = km.binary_precision(label=1)\n",
    "\n",
    "# Calculate recall for the first label.\n",
    "recall = km.binary_recall(label=0)\n",
    "\n",
    "model.compile(loss='binary_crossentropy',\n",
    "                      optimizer='adam',\n",
    "                      metrics=['accuracy', precision, recall])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train / Fit the Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_circles\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from matplotlib import pyplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Train the neural network on  multichannel inputs namely deep layers of patent text segments \n",
    " and deep layers of patent metadata.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras_metrics as km\n",
    "\n",
    "#contacting two input models\n",
    "model_inputs_to_concat = [TI_deep, AB_deep, TECHF_deep, BACKG_deep, SUMM_deep, CLMS_deep, pas, invs] \n",
    "final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)\n",
    "\n",
    "output = Dense(128, activation=None)(final_layer)\n",
    "output = Dropout(dropout_pct)(output)\n",
    "output = BatchNormalization()(output)\n",
    "output = ELU()(output)\n",
    "output = Dense(num_classes, activation='sigmoid')(output)\n",
    "\n",
    "model2 =Model(inputs=[ TI_embedding_layer_input,\n",
    "                      AB_embedding_layer_input,\n",
    "                      TECHF_embedding_layer_input,\n",
    "                      BACKG_embedding_layer_input,\n",
    "                     SUMM_embedding_layer_input,\n",
    "                     CLMS_embedding_layer_input,\n",
    "                     pa_input,\n",
    "                      inv_input],\n",
    "              outputs=output, name='model')\n",
    "model2.compile(loss='binary_crossentropy',\n",
    "                      optimizer='adam',\n",
    "                       metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the model/network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "batch_size= 500 \n",
    "num_epochs = 10\n",
    "\n",
    "\n",
    "history2 = model2.fit(x={'TI_embed_input': TI_train,\n",
    "                         'AB_embed_input': AB_train,\n",
    "             'TECHF_embed_input': TECHF_train,\n",
    "             'BACKG_embed_input': BACKG_train,\n",
    "             'SUMM_embed_input': SUMM_train,\n",
    "             'CLMS_embed_input': CLMS_train,\n",
    "             'pa_input': train_pa_one_hot,\n",
    "             'inv_input': train_inv_one_hot\n",
    "            },\n",
    "          y=y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=num_epochs,\n",
    "          validation_data=\n",
    "          ({'TI_embed_input': TI_test,\n",
    "            'AB_embed_input': AB_test,\n",
    "            'TECHF_embed_input': TECHF_test,\n",
    "             'BACKG_embed_input': BACKG_test,\n",
    "             'SUMM_embed_input': SUMM_test,\n",
    "            'CLMS_embed_input': CLMS_test,\n",
    "            'pa_input': test_pa_one_hot,\n",
    "            'inv_input': test_inv_one_hot\n",
    "            },\n",
    "           y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation of Multi-label Classification \n",
    "We actually used the code availab el <a href=https://github.com/tensorflow/tensorflow/issues/28074 >here </a> as a metrics for multi-label classification for using Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "class MetricsAtTopK:\n",
    "    def __init__(self, k):\n",
    "        self.k = k\n",
    "\n",
    "    def _get_prediction_tensor(self, y_pred):\n",
    "        \"\"\"Takes y_pred and creates a tensor of same shape with 1 in indices where, the values are in top_k\n",
    "        \"\"\"\n",
    "        topk_values, topk_indices = tf.nn.top_k(y_pred, k=self.k, sorted=False, name=\"topk\")\n",
    "        # the topk_indices are along last axis (1). Add indices for axis=0\n",
    "        ii, _ = tf.meshgrid(tf.range(tf.shape(y_pred)[0]), tf.range(self.k), indexing='ij')\n",
    "        index_tensor = tf.reshape(tf.stack([ii, topk_indices], axis=-1), shape=(-1, 2))\n",
    "        prediction_tensor = tf.sparse_to_dense(sparse_indices=index_tensor,\n",
    "                                               output_shape=tf.shape(y_pred),\n",
    "                                               default_value=0,\n",
    "                                               sparse_values=1.0,\n",
    "                                               validate_indices=False\n",
    "                                               )\n",
    "        prediction_tensor = tf.cast(prediction_tensor, K.floatx())\n",
    "        return prediction_tensor\n",
    "\n",
    "    def true_positives_at_k(self, y_true, y_pred):\n",
    "        prediction_tensor = self._get_prediction_tensor(y_pred=y_pred)\n",
    "        true_positive = K.sum(tf.multiply(prediction_tensor, y_true))\n",
    "        return true_positive\n",
    "\n",
    "    def false_positives_at_k(self, y_true, y_pred):\n",
    "        prediction_tensor = self._get_prediction_tensor(y_pred=y_pred)\n",
    "        true_positive = K.sum(tf.multiply(prediction_tensor, y_true))\n",
    "        c2 = K.sum(prediction_tensor)  # TP + FP\n",
    "        false_positive = c2 - true_positive\n",
    "        return false_positive\n",
    "\n",
    "    def false_negatives_at_k(self, y_true, y_pred):\n",
    "        prediction_tensor = self._get_prediction_tensor(y_pred=y_pred)\n",
    "        true_positive = K.sum(tf.multiply(prediction_tensor, y_true))\n",
    "        c3 = K.sum(y_true)  # TP + FN\n",
    "        false_negative = c3 - true_positive\n",
    "        return false_negative\n",
    "\n",
    "    def precision_at_k(self, y_true, y_pred):\n",
    "        prediction_tensor = self._get_prediction_tensor(y_pred=y_pred)\n",
    "        true_positive = K.sum(tf.multiply(prediction_tensor, y_true))\n",
    "        c2 = K.sum(prediction_tensor)  # TP + FP\n",
    "        return true_positive/(c2+K.epsilon())\n",
    "\n",
    "    def recall_at_k(self, y_true, y_pred):\n",
    "        prediction_tensor = self._get_prediction_tensor(y_pred=y_pred)\n",
    "        true_positive = K.sum(tf.multiply(prediction_tensor, y_true))\n",
    "        c3 = K.sum(y_true)  # TP + FN\n",
    "        return true_positive/(c3+K.epsilon())\n",
    "\n",
    "    def f1_at_k(self, y_true, y_pred):\n",
    "        precision = self.precision_at_k(y_true=y_true, y_pred=y_pred)\n",
    "        recall = self.recall_at_k(y_true=y_true, y_pred=y_pred)\n",
    "        f1 = (2*precision*recall)/(precision+recall+K.epsilon())\n",
    "        return f1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-45-4cb8bf0984c2>:19: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "TI_embed_input (InputLayer)     (None, 20)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "AB_embed_input (InputLayer)     (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "TECHF_embed_input (InputLayer)  (None, 30)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "BACKG_embed_input (InputLayer)  (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "SUMM_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "CLMS_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 20, 20)       899240      TI_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 100, 100)     15099300    AB_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 30, 30)       4542330     TECHF_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_4 (Embedding)         (None, 100, 100)     75638800    BACKG_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_5 (Embedding)         (None, 100, 100)     66417600    SUMM_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "embedding_6 (Embedding)         (None, 100, 100)     14941900    CLMS_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TI (LSTM)                  (None, 64)           21760       embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_AB (LSTM)                  (None, 64)           42240       embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TECHF (LSTM)               (None, 64)           24320       embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_BACK (LSTM)                (None, 64)           42240       embedding_4[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_SUMM (LSTM)                (None, 64)           42240       embedding_5[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_CLMS (LSTM)                (None, 64)           42240       embedding_6[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 300)          19500       LSTM_TI[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 300)          19500       LSTM_AB[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 300)          19500       LSTM_TECHF[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 300)          19500       LSTM_BACK[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 300)          19500       LSTM_SUMM[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 300)          19500       LSTM_CLMS[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 300)          0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 300)          0           dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 300)          0           dense_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 300)          0           dense_4[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, 300)          0           dense_5[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_6 (Dropout)             (None, 300)          0           dense_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 300)          1200        dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 300)          1200        dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 300)          1200        dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 300)          1200        dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 300)          1200        dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, 300)          1200        dropout_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_1 (ELU)                     (None, 300)          0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_2 (ELU)                     (None, 300)          0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_3 (ELU)                     (None, 300)          0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_4 (ELU)                     (None, 300)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_5 (ELU)                     (None, 300)          0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_6 (ELU)                     (None, 300)          0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "concatenated_layer (Concatenate (None, 1800)         0           elu_1[0][0]                      \n",
      "                                                                 elu_2[0][0]                      \n",
      "                                                                 elu_3[0][0]                      \n",
      "                                                                 elu_4[0][0]                      \n",
      "                                                                 elu_5[0][0]                      \n",
      "                                                                 elu_6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dense_9 (Dense)                 (None, 128)          230528      concatenated_layer[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_9 (Dropout)             (None, 128)          0           dense_9[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, 128)          512         dropout_9[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_8 (ELU)                     (None, 128)          0           batch_normalization_9[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dense_10 (Dense)                (None, 31)           3999        elu_8[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 178,113,449\n",
      "Trainable params: 178,109,593\n",
      "Non-trainable params: 3,856\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "### Usage:\n",
    "metrics = MetricsAtTopK(k=5)\n",
    "# model definition\n",
    "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc', \n",
    "                                                                     metrics.true_positives_at_k, \n",
    "                                                                     metrics.false_positives_at_k,\n",
    "                                                                     metrics.false_negatives_at_k,\n",
    "                                                                     metrics.recall_at_k,\n",
    "                                                                     metrics.precision_at_k,\n",
    "                                                                     metrics.f1_at_k,\n",
    "                                                                    ])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 336629 samples, validate on 84158 samples\n",
      "Epoch 1/10\n",
      "336629/336629 [==============================] - 457s 1ms/step - loss: 0.0804 - acc: 0.9710 - true_positives_at_k: 715.3929 - false_positives_at_k: 1783.8963 - false_negatives_at_k: 72.7587 - recall_at_k: 0.9078 - precision_at_k: 0.2862 - f1_at_k: 0.4352 - val_loss: 0.0820 - val_acc: 0.9695 - val_true_positives_at_k: 725.5607 - val_false_positives_at_k: 1771.2290 - val_false_negatives_at_k: 65.9321 - val_recall_at_k: 0.9168 - val_precision_at_k: 0.2906 - val_f1_at_k: 0.4413\n",
      "Epoch 2/10\n",
      "336629/336629 [==============================] - 442s 1ms/step - loss: 0.0794 - acc: 0.9712 - true_positives_at_k: 717.1670 - false_positives_at_k: 1782.1222 - false_negatives_at_k: 70.9945 - recall_at_k: 0.9100 - precision_at_k: 0.2869 - f1_at_k: 0.4363 - val_loss: 0.0806 - val_acc: 0.9701 - val_true_positives_at_k: 726.1545 - val_false_positives_at_k: 1770.6351 - val_false_negatives_at_k: 65.3383 - val_recall_at_k: 0.9176 - val_precision_at_k: 0.2908 - val_f1_at_k: 0.4416\n",
      "Epoch 3/10\n",
      "336629/336629 [==============================] - 431s 1ms/step - loss: 0.0787 - acc: 0.9714 - true_positives_at_k: 718.0541 - false_positives_at_k: 1781.2350 - false_negatives_at_k: 70.1019 - recall_at_k: 0.9111 - precision_at_k: 0.2873 - f1_at_k: 0.4368 - val_loss: 0.0788 - val_acc: 0.9708 - val_true_positives_at_k: 727.7470 - val_false_positives_at_k: 1769.0426 - val_false_negatives_at_k: 63.7457 - val_recall_at_k: 0.9196 - val_precision_at_k: 0.2915 - val_f1_at_k: 0.4426\n",
      "Epoch 4/10\n",
      "336629/336629 [==============================] - 433s 1ms/step - loss: 0.0779 - acc: 0.9717 - true_positives_at_k: 719.7074 - false_positives_at_k: 1779.5817 - false_negatives_at_k: 68.4409 - recall_at_k: 0.9132 - precision_at_k: 0.2880 - f1_at_k: 0.4378 - val_loss: 0.0796 - val_acc: 0.9703 - val_true_positives_at_k: 728.2896 - val_false_positives_at_k: 1768.5001 - val_false_negatives_at_k: 63.2032 - val_recall_at_k: 0.9203 - val_precision_at_k: 0.2917 - val_f1_at_k: 0.4429\n",
      "Epoch 5/10\n",
      "336629/336629 [==============================] - 441s 1ms/step - loss: 0.0773 - acc: 0.9719 - true_positives_at_k: 720.8135 - false_positives_at_k: 1778.4756 - false_negatives_at_k: 67.3403 - recall_at_k: 0.9146 - precision_at_k: 0.2884 - f1_at_k: 0.4385 - val_loss: 0.0792 - val_acc: 0.9706 - val_true_positives_at_k: 729.8874 - val_false_positives_at_k: 1766.9022 - val_false_negatives_at_k: 61.6053 - val_recall_at_k: 0.9223 - val_precision_at_k: 0.2923 - val_f1_at_k: 0.4439\n",
      "Epoch 6/10\n",
      "336629/336629 [==============================] - 435s 1ms/step - loss: 0.0766 - acc: 0.9721 - true_positives_at_k: 721.9088 - false_positives_at_k: 1777.3804 - false_negatives_at_k: 66.2572 - recall_at_k: 0.9160 - precision_at_k: 0.2888 - f1_at_k: 0.4391 - val_loss: 0.0785 - val_acc: 0.9706 - val_true_positives_at_k: 730.1588 - val_false_positives_at_k: 1766.6308 - val_false_negatives_at_k: 61.3339 - val_recall_at_k: 0.9226 - val_precision_at_k: 0.2924 - val_f1_at_k: 0.4441\n",
      "Epoch 7/10\n",
      "336629/336629 [==============================] - 429s 1ms/step - loss: 0.0761 - acc: 0.9723 - true_positives_at_k: 722.7028 - false_positives_at_k: 1776.5863 - false_negatives_at_k: 65.4543 - recall_at_k: 0.9171 - precision_at_k: 0.2892 - f1_at_k: 0.4396 - val_loss: 0.0779 - val_acc: 0.9709 - val_true_positives_at_k: 730.9493 - val_false_positives_at_k: 1765.8403 - val_false_negatives_at_k: 60.5434 - val_recall_at_k: 0.9236 - val_precision_at_k: 0.2927 - val_f1_at_k: 0.4445\n",
      "Epoch 8/10\n",
      "336629/336629 [==============================] - 430s 1ms/step - loss: 0.0754 - acc: 0.9725 - true_positives_at_k: 723.8539 - false_positives_at_k: 1775.4353 - false_negatives_at_k: 64.3165 - recall_at_k: 0.9185 - precision_at_k: 0.2896 - f1_at_k: 0.4403 - val_loss: 0.0777 - val_acc: 0.9708 - val_true_positives_at_k: 731.8108 - val_false_positives_at_k: 1764.9788 - val_false_negatives_at_k: 59.6819 - val_recall_at_k: 0.9247 - val_precision_at_k: 0.2931 - val_f1_at_k: 0.4451\n",
      "Epoch 9/10\n",
      "336629/336629 [==============================] - 433s 1ms/step - loss: 0.0748 - acc: 0.9727 - true_positives_at_k: 725.0860 - false_positives_at_k: 1774.2031 - false_negatives_at_k: 63.0601 - recall_at_k: 0.9201 - precision_at_k: 0.2901 - f1_at_k: 0.4411 - val_loss: 0.0768 - val_acc: 0.9714 - val_true_positives_at_k: 731.9650 - val_false_positives_at_k: 1764.8246 - val_false_negatives_at_k: 59.5278 - val_recall_at_k: 0.9249 - val_precision_at_k: 0.2932 - val_f1_at_k: 0.4452\n",
      "Epoch 10/10\n",
      "336629/336629 [==============================] - 430s 1ms/step - loss: 0.0743 - acc: 0.9728 - true_positives_at_k: 725.7604 - false_positives_at_k: 1773.5287 - false_negatives_at_k: 62.3846 - recall_at_k: 0.9209 - precision_at_k: 0.2904 - f1_at_k: 0.4415 - val_loss: 0.0766 - val_acc: 0.9713 - val_true_positives_at_k: 732.6188 - val_false_positives_at_k: 1764.1708 - val_false_negatives_at_k: 58.8739 - val_recall_at_k: 0.9257 - val_precision_at_k: 0.2934 - val_f1_at_k: 0.4455\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batch_size= 500 \n",
    "num_epochs = 10\n",
    "\n",
    "history = model.fit(x={'TI_embed_input': TI_train,\n",
    "                       'AB_embed_input': AB_train,\n",
    "             'TECHF_embed_input': TECHF_train,\n",
    "             'BACKG_embed_input': BACKG_train,\n",
    "             'SUMM_embed_input': SUMM_train,\n",
    "             'CLMS_embed_input': CLMS_train\n",
    "             \n",
    "            },\n",
    "          y=y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=num_epochs,\n",
    "          validation_data=\n",
    "          ({'TI_embed_input': TI_test,\n",
    "            'AB_embed_input': AB_test,\n",
    "            'TECHF_embed_input': TECHF_test,\n",
    "             'BACKG_embed_input': BACKG_test,\n",
    "             'SUMM_embed_input': SUMM_test,\n",
    "            'CLMS_embed_input': CLMS_test\n",
    "            },\n",
    "           y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "TI_embed_input (InputLayer)     (None, 20)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "AB_embed_input (InputLayer)     (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "TECHF_embed_input (InputLayer)  (None, 30)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "BACKG_embed_input (InputLayer)  (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "SUMM_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "CLMS_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 20, 20)       899240      TI_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 100, 100)     15099300    AB_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 30, 30)       4542330     TECHF_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_4 (Embedding)         (None, 100, 100)     75638800    BACKG_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_5 (Embedding)         (None, 100, 100)     66417600    SUMM_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "embedding_6 (Embedding)         (None, 100, 100)     14941900    CLMS_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TI (LSTM)                  (None, 64)           21760       embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_AB (LSTM)                  (None, 64)           42240       embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TECHF (LSTM)               (None, 64)           24320       embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_BACK (LSTM)                (None, 64)           42240       embedding_4[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_SUMM (LSTM)                (None, 64)           42240       embedding_5[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_CLMS (LSTM)                (None, 64)           42240       embedding_6[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 300)          19500       LSTM_TI[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 300)          19500       LSTM_AB[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 300)          19500       LSTM_TECHF[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 300)          19500       LSTM_BACK[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 300)          19500       LSTM_SUMM[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 300)          19500       LSTM_CLMS[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 300)          0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 300)          0           dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 300)          0           dense_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 300)          0           dense_4[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, 300)          0           dense_5[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_6 (Dropout)             (None, 300)          0           dense_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 300)          1200        dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 300)          1200        dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 300)          1200        dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 300)          1200        dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 300)          1200        dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, 300)          1200        dropout_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_1 (ELU)                     (None, 300)          0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_2 (ELU)                     (None, 300)          0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_3 (ELU)                     (None, 300)          0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_4 (ELU)                     (None, 300)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_5 (ELU)                     (None, 300)          0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_6 (ELU)                     (None, 300)          0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "concatenated_layer (Concatenate (None, 1800)         0           elu_1[0][0]                      \n",
      "                                                                 elu_2[0][0]                      \n",
      "                                                                 elu_3[0][0]                      \n",
      "                                                                 elu_4[0][0]                      \n",
      "                                                                 elu_5[0][0]                      \n",
      "                                                                 elu_6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dense_9 (Dense)                 (None, 128)          230528      concatenated_layer[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_9 (Dropout)             (None, 128)          0           dense_9[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, 128)          512         dropout_9[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_8 (ELU)                     (None, 128)          0           batch_normalization_9[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dense_10 (Dense)                (None, 31)           3999        elu_8[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 178,113,449\n",
      "Trainable params: 178,109,593\n",
      "Non-trainable params: 3,856\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "#contacting two input models\n",
    "model_inputs_to_concat = [TI_deep, AB_deep, TECHF_deep, BACKG_deep, SUMM_deep, CLMS_deep, pas, invs] \n",
    "final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)\n",
    "\n",
    "output = Dense(128, activation=None)(final_layer)\n",
    "output = Dropout(dropout_pct)(output)\n",
    "output = BatchNormalization()(output)\n",
    "output = ELU()(output)\n",
    "output = Dense(num_classes, activation='sigmoid')(output)\n",
    "\n",
    "model2 =Model(inputs=[ TI_embedding_layer_input,\n",
    "                      AB_embedding_layer_input,\n",
    "                      TECHF_embedding_layer_input,\n",
    "                      BACKG_embedding_layer_input,\n",
    "                     SUMM_embedding_layer_input,\n",
    "                     CLMS_embedding_layer_input,\n",
    "                     pa_input,\n",
    "                      inv_input],\n",
    "              outputs=output, name='model')\n",
    "\n",
    "model2.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc', \n",
    "                                                                     metrics.true_positives_at_k, \n",
    "                                                                     metrics.false_positives_at_k,\n",
    "                                                                     metrics.false_negatives_at_k,\n",
    "                                                                     metrics.recall_at_k,\n",
    "                                                                     metrics.precision_at_k,\n",
    "                                                                     metrics.f1_at_k,\n",
    "                                                                    ])\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 336629 samples, validate on 84158 samples\n",
      "Epoch 1/10\n",
      "336629/336629 [==============================] - 467s 1ms/step - loss: 0.1251 - acc: 0.9578 - true_positives_at_k: 691.7435 - false_positives_at_k: 1807.5457 - false_negatives_at_k: 96.3883 - recall_at_k: 0.8778 - precision_at_k: 0.2768 - f1_at_k: 0.4208 - val_loss: 0.0775 - val_acc: 0.9724 - val_true_positives_at_k: 730.3311 - val_false_positives_at_k: 1766.4585 - val_false_negatives_at_k: 61.1616 - val_recall_at_k: 0.9228 - val_precision_at_k: 0.2925 - val_f1_at_k: 0.4442\n",
      "Epoch 2/10\n",
      "336629/336629 [==============================] - 440s 1ms/step - loss: 0.0769 - acc: 0.9725 - true_positives_at_k: 722.5263 - false_positives_at_k: 1776.7629 - false_negatives_at_k: 65.6055 - recall_at_k: 0.9169 - precision_at_k: 0.2891 - f1_at_k: 0.4395 - val_loss: 0.0764 - val_acc: 0.9721 - val_true_positives_at_k: 732.9118 - val_false_positives_at_k: 1763.8778 - val_false_negatives_at_k: 58.5809 - val_recall_at_k: 0.9261 - val_precision_at_k: 0.2935 - val_f1_at_k: 0.4457\n",
      "Epoch 3/10\n",
      "336629/336629 [==============================] - 443s 1ms/step - loss: 0.0750 - acc: 0.9729 - true_positives_at_k: 725.9024 - false_positives_at_k: 1773.3868 - false_negatives_at_k: 62.2316 - recall_at_k: 0.9211 - precision_at_k: 0.2904 - f1_at_k: 0.4416 - val_loss: 0.0761 - val_acc: 0.9719 - val_true_positives_at_k: 733.8840 - val_false_positives_at_k: 1762.9056 - val_false_negatives_at_k: 57.6088 - val_recall_at_k: 0.9273 - val_precision_at_k: 0.2939 - val_f1_at_k: 0.4463\n",
      "Epoch 4/10\n",
      "336629/336629 [==============================] - 445s 1ms/step - loss: 0.0737 - acc: 0.9733 - true_positives_at_k: 727.9184 - false_positives_at_k: 1771.3708 - false_negatives_at_k: 60.2123 - recall_at_k: 0.9237 - precision_at_k: 0.2913 - f1_at_k: 0.4428 - val_loss: 0.0758 - val_acc: 0.9720 - val_true_positives_at_k: 734.3214 - val_false_positives_at_k: 1762.4682 - val_false_negatives_at_k: 57.1713 - val_recall_at_k: 0.9279 - val_precision_at_k: 0.2941 - val_f1_at_k: 0.4466\n",
      "Epoch 5/10\n",
      "336629/336629 [==============================] - 444s 1ms/step - loss: 0.0728 - acc: 0.9735 - true_positives_at_k: 729.2505 - false_positives_at_k: 1770.0386 - false_negatives_at_k: 58.9033 - recall_at_k: 0.9254 - precision_at_k: 0.2918 - f1_at_k: 0.4436 - val_loss: 0.0762 - val_acc: 0.9717 - val_true_positives_at_k: 734.5988 - val_false_positives_at_k: 1762.1908 - val_false_negatives_at_k: 56.8939 - val_recall_at_k: 0.9282 - val_precision_at_k: 0.2942 - val_f1_at_k: 0.4468\n",
      "Epoch 6/10\n",
      "336629/336629 [==============================] - 439s 1ms/step - loss: 0.0721 - acc: 0.9737 - true_positives_at_k: 730.2518 - false_positives_at_k: 1769.0374 - false_negatives_at_k: 57.8800 - recall_at_k: 0.9267 - precision_at_k: 0.2922 - f1_at_k: 0.4442 - val_loss: 0.0762 - val_acc: 0.9716 - val_true_positives_at_k: 735.2167 - val_false_positives_at_k: 1761.5729 - val_false_negatives_at_k: 56.2761 - val_recall_at_k: 0.9290 - val_precision_at_k: 0.2945 - val_f1_at_k: 0.4471\n",
      "Epoch 7/10\n",
      "336629/336629 [==============================] - 433s 1ms/step - loss: 0.0714 - acc: 0.9739 - true_positives_at_k: 731.2858 - false_positives_at_k: 1768.0034 - false_negatives_at_k: 56.8658 - recall_at_k: 0.9279 - precision_at_k: 0.2926 - f1_at_k: 0.4449 - val_loss: 0.0757 - val_acc: 0.9718 - val_true_positives_at_k: 735.2664 - val_false_positives_at_k: 1761.5232 - val_false_negatives_at_k: 56.2263 - val_recall_at_k: 0.9291 - val_precision_at_k: 0.2945 - val_f1_at_k: 0.4472\n",
      "Epoch 8/10\n",
      "336629/336629 [==============================] - 434s 1ms/step - loss: 0.0709 - acc: 0.9740 - true_positives_at_k: 732.2356 - false_positives_at_k: 1767.0535 - false_negatives_at_k: 55.9116 - recall_at_k: 0.9291 - precision_at_k: 0.2930 - f1_at_k: 0.4454 - val_loss: 0.0756 - val_acc: 0.9719 - val_true_positives_at_k: 735.1792 - val_false_positives_at_k: 1761.6104 - val_false_negatives_at_k: 56.3136 - val_recall_at_k: 0.9290 - val_precision_at_k: 0.2944 - val_f1_at_k: 0.4471\n",
      "Epoch 9/10\n",
      "336629/336629 [==============================] - 432s 1ms/step - loss: 0.0703 - acc: 0.9742 - true_positives_at_k: 732.9241 - false_positives_at_k: 1766.3650 - false_negatives_at_k: 55.2275 - recall_at_k: 0.9300 - precision_at_k: 0.2933 - f1_at_k: 0.4459 - val_loss: 0.0758 - val_acc: 0.9717 - val_true_positives_at_k: 736.2902 - val_false_positives_at_k: 1760.4994 - val_false_negatives_at_k: 55.2026 - val_recall_at_k: 0.9304 - val_precision_at_k: 0.2949 - val_f1_at_k: 0.4478\n",
      "Epoch 10/10\n",
      "336629/336629 [==============================] - 433s 1ms/step - loss: 0.0698 - acc: 0.9743 - true_positives_at_k: 733.5962 - false_positives_at_k: 1765.6929 - false_negatives_at_k: 54.5499 - recall_at_k: 0.9309 - precision_at_k: 0.2935 - f1_at_k: 0.4463 - val_loss: 0.0764 - val_acc: 0.9715 - val_true_positives_at_k: 735.6266 - val_false_positives_at_k: 1761.1630 - val_false_negatives_at_k: 55.8661 - val_recall_at_k: 0.9295 - val_precision_at_k: 0.2946 - val_f1_at_k: 0.4474\n",
      "CPU times: user 2h 19min 33s, sys: 21min 35s, total: 2h 41min 9s\n",
      "Wall time: 1h 13min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batch_size= 500 \n",
    "num_epochs = 10\n",
    "\n",
    "\n",
    "history2 = model2.fit(x={'TI_embed_input': TI_train,\n",
    "                         'AB_embed_input': AB_train,\n",
    "             'TECHF_embed_input': TECHF_train,\n",
    "             'BACKG_embed_input': BACKG_train,\n",
    "             'SUMM_embed_input': SUMM_train,\n",
    "             'CLMS_embed_input': CLMS_train,\n",
    "             'pa_input': train_pa_one_hot,\n",
    "             'inv_input': train_inv_one_hot\n",
    "            },\n",
    "          y=y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=num_epochs,\n",
    "          validation_data=\n",
    "          ({'TI_embed_input': TI_test,\n",
    "            'AB_embed_input': AB_test,\n",
    "            'TECHF_embed_input': TECHF_test,\n",
    "             'BACKG_embed_input': BACKG_test,\n",
    "             'SUMM_embed_input': SUMM_test,\n",
    "            'CLMS_embed_input': CLMS_test,\n",
    "            'pa_input': test_pa_one_hot,\n",
    "            'inv_input': test_inv_one_hot\n",
    "            },\n",
    "           y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score = model.evaluate(x={'TI_embed_input': TI_test,\n",
    "            'AB_embed_input': AB_test,\n",
    "            'TECHF_embed_input': TECHF_test,\n",
    "             'BACKG_embed_input': BACKG_test,\n",
    "             'SUMM_embed_input': SUMM_test,\n",
    "            'CLMS_embed_input': CLMS_test\n",
    "            },\n",
    "          y= y_test,\n",
    "                       batch_size=batch_size, verbose=1)\n",
    " \n",
    "print('Test accuracy:', score[1])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}