DeepL4Patent Pipeline for Multi-class Patent Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning based Pipeline with Multichannel Inputs for Multi-class Patent Classification\n",
    "\n",
    "This notebook describes a deep learning pipeline for automatic patent classification with multichannel inputs.  A neural network model is trained with multichannel inputs namely embeddings of different segments of patent texts, and sparse linear\n",
    "input of different metadata. <br> <br>\n",
    "",
    "\n",
    "<br>\n",
    "In this notebook the classification task is a multi-class classification. The basic outline is:  <br>  <br>\n",
    "\n",
    "- load the patent dataset  <br>\n",
    "- apply preprocessing tasks  <br>\n",
    "- apply Tokenization process  <br>\n",
    "- Load a pretrained word embeddings model  <br>\n",
    "- prepare the embedding matrix for patent texts   <br>\n",
    "- concatenated deep layers <br>\n",
    "- train a deep neural network on the data  <br>\n",
    "- Fit the model and show the results  <br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Loading patent dataset\n",
    "We established the training, and test datasets that are domain-specific datasets and related to the information technology domain.\n",
    "The total number of extracted records in the datasets is about 430,000  patents filed between 1978 and 2016. Each patent document contains a  patent number, issued date, patent type, and list of citations, classification codes, a list of inventors, a list of assignees, title, abstract, technical field, background,  summary of invention, and independent claim. <br>\n",
    "The data det is availabe in https://www.kaggle.com/darshmso/it-patent-dataset \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>TI</th>\n",
       "      <th>AB</th>\n",
       "      <th>TECHF</th>\n",
       "      <th>BACKG</th>\n",
       "      <th>SUMM</th>\n",
       "      <th>CLMS</th>\n",
       "      <th>ICM</th>\n",
       "      <th>AY</th>\n",
       "      <th>IPC</th>\n",
       "      <th>REF</th>\n",
       "      <th>PA</th>\n",
       "      <th>INV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>EP2000017943-0</td>\n",
       "      <td>Recognition code, particularly for a disk-shap...</td>\n",
       "      <td>A recognition code, particularly for a disk-li...</td>\n",
       "      <td>[1] The present invention relates to a recogni...</td>\n",
       "      <td>[2] It is known that identification labels, ad...</td>\n",
       "      <td>[5] The aim of the present invention is to ove...</td>\n",
       "      <td></td>\n",
       "      <td>G06K0019-06</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G06K0019-06, G06K0007-10]</td>\n",
       "      <td></td>\n",
       "      <td>[Video System Italia S.r.l.]</td>\n",
       "      <td>[Tassello  Stefano]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EP2003016733-0</td>\n",
       "      <td>Optical pickup apparatus for recording/reprodu...</td>\n",
       "      <td>An optical pickup apparatus for reproducing in...</td>\n",
       "      <td>[1] The present invention relates to an optica...</td>\n",
       "      <td>BACKGROUND OF THE INVENTION   [2] Recently, as...</td>\n",
       "      <td>SUMMARY OF THE INVENTION[11] An object of the ...</td>\n",
       "      <td>['An optical pickup apparatus for recording an...</td>\n",
       "      <td>G11B0007-135</td>\n",
       "      <td>2000</td>\n",
       "      <td>[G11B0007-135, G11B0007-125]</td>\n",
       "      <td></td>\n",
       "      <td>[Konica Minolta Opto  Inc.]</td>\n",
       "      <td>[Arai  Norikazu, Kojima  Toshiyuki, Kiriki  To...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>EP2011009984-0</td>\n",
       "      <td>Large capacity data sales mediation system, se...</td>\n",
       "      <td>An animation data sales mediation method, an a...</td>\n",
       "      <td>[1] The present invention relates to large cap...</td>\n",
       "      <td>BACKGROUND OF THE INVENTION   Description of t...</td>\n",
       "      <td>SUMMARY OF THE INVENTION[20] The present inven...</td>\n",
       "      <td>['A large capacity data sales mediation system...</td>\n",
       "      <td>G07F0017-16</td>\n",
       "      <td>2001</td>\n",
       "      <td>[G07F0017-16, G06Q0030-06, G06Q0020-10, G06Q00...</td>\n",
       "      <td>[JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...</td>\n",
       "      <td>[NEC Corporation]</td>\n",
       "      <td>[Maeda Koji]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PCT1997010546-0</td>\n",
       "      <td>BRIDGE FOR A CLIENT-SERVER ENVIRONMENT</td>\n",
       "      <td>A software bridge (300) is introduced between ...</td>\n",
       "      <td>1 BRIDGE FOR A CLIENT-SERVER ENVIRONMENT  Fiel...</td>\n",
       "      <td>Background of the Invention Overview of Object...</td>\n",
       "      <td>45 Disclosure of the Invention Accordingly the...</td>\n",
       "      <td>['1. A bridge (300) for use between a client (...</td>\n",
       "      <td>G06F009-46</td>\n",
       "      <td>1996</td>\n",
       "      <td>[G06F009-46, G06F009-44, G06F0009-44, G06F0009...</td>\n",
       "      <td></td>\n",
       "      <td>[INTERNATIONAL BUSINESS MACHINES CORPORATION, ...</td>\n",
       "      <td>[COLYER  ADRIAN  MARK]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PCT1998021641-0</td>\n",
       "      <td>PROCESSOR HAVING SECTIONS OPERATING AT DIFFERE...</td>\n",
       "      <td>A processor (250) including a first execution ...</td>\n",
       "      <td>PROCESSOR HAVING SECTIONS OPERATING AT DIFFERE...</td>\n",
       "      <td>Background of the Prior Art Fig. 1 illustrates...</td>\n",
       "      <td>SUMMARY OF THE INVENTION The invention provide...</td>\n",
       "      <td></td>\n",
       "      <td>G06F001-32</td>\n",
       "      <td>1997</td>\n",
       "      <td>[G06F001-32, G06F0001-08, G06F0009-30, G06F000...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>[SAGER DAVID J, FLETCHER THOMAS D, HINTON GLEN...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                ID                                                 TI  \\\n",
       "0   EP2000017943-0  Recognition code, particularly for a disk-shap...   \n",
       "1   EP2003016733-0  Optical pickup apparatus for recording/reprodu...   \n",
       "2   EP2011009984-0  Large capacity data sales mediation system, se...   \n",
       "3  PCT1997010546-0             BRIDGE FOR A CLIENT-SERVER ENVIRONMENT   \n",
       "4  PCT1998021641-0  PROCESSOR HAVING SECTIONS OPERATING AT DIFFERE...   \n",
       "\n",
       "                                                  AB  \\\n",
       "0  A recognition code, particularly for a disk-li...   \n",
       "1  An optical pickup apparatus for reproducing in...   \n",
       "2  An animation data sales mediation method, an a...   \n",
       "3  A software bridge (300) is introduced between ...   \n",
       "4  A processor (250) including a first execution ...   \n",
       "\n",
       "                                               TECHF  \\\n",
       "0  [1] The present invention relates to a recogni...   \n",
       "1  [1] The present invention relates to an optica...   \n",
       "2  [1] The present invention relates to large cap...   \n",
       "3  1 BRIDGE FOR A CLIENT-SERVER ENVIRONMENT  Fiel...   \n",
       "4  PROCESSOR HAVING SECTIONS OPERATING AT DIFFERE...   \n",
       "\n",
       "                                               BACKG  \\\n",
       "0  [2] It is known that identification labels, ad...   \n",
       "1  BACKGROUND OF THE INVENTION   [2] Recently, as...   \n",
       "2  BACKGROUND OF THE INVENTION   Description of t...   \n",
       "3  Background of the Invention Overview of Object...   \n",
       "4  Background of the Prior Art Fig. 1 illustrates...   \n",
       "\n",
       "                                                SUMM  \\\n",
       "0  [5] The aim of the present invention is to ove...   \n",
       "1  SUMMARY OF THE INVENTION[11] An object of the ...   \n",
       "2  SUMMARY OF THE INVENTION[20] The present inven...   \n",
       "3  45 Disclosure of the Invention Accordingly the...   \n",
       "4  SUMMARY OF THE INVENTION The invention provide...   \n",
       "\n",
       "                                                CLMS           ICM    AY  \\\n",
       "0                                                      G06K0019-06  2000   \n",
       "1  ['An optical pickup apparatus for recording an...  G11B0007-135  2000   \n",
       "2  ['A large capacity data sales mediation system...   G07F0017-16  2001   \n",
       "3  ['1. A bridge (300) for use between a client (...    G06F009-46  1996   \n",
       "4                                                       G06F001-32  1997   \n",
       "\n",
       "                                                 IPC  \\\n",
       "0                         [G06K0019-06, G06K0007-10]   \n",
       "1                       [G11B0007-135, G11B0007-125]   \n",
       "2  [G07F0017-16, G06Q0030-06, G06Q0020-10, G06Q00...   \n",
       "3  [G06F009-46, G06F009-44, G06F0009-44, G06F0009...   \n",
       "4  [G06F001-32, G06F0001-08, G06F0009-30, G06F000...   \n",
       "\n",
       "                                                 REF  \\\n",
       "0                                                      \n",
       "1                                                      \n",
       "2  [JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...   \n",
       "3                                                      \n",
       "4                                                      \n",
       "\n",
       "                                                  PA  \\\n",
       "0                       [Video System Italia S.r.l.]   \n",
       "1                        [Konica Minolta Opto  Inc.]   \n",
       "2                                  [NEC Corporation]   \n",
       "3  [INTERNATIONAL BUSINESS MACHINES CORPORATION, ...   \n",
       "4                                                      \n",
       "\n",
       "                                                 INV  \n",
       "0                                [Tassello  Stefano]  \n",
       "1  [Arai  Norikazu, Kojima  Toshiyuki, Kiriki  To...  \n",
       "2                                       [Maeda Koji]  \n",
       "3                             [COLYER  ADRIAN  MARK]  \n",
       "4  [SAGER DAVID J, FLETCHER THOMAS D, HINTON GLEN...  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"../datasets/allITPatTextWith_Metadata.csv\",  encoding = \"ISO-8859-1\", error_bad_lines=False)\n",
    "df.columns =['ID','TI','AB','TECHF','BACKG','SUMM','CLMS','ICM','AY','IPC','REF','PA','INV']\n",
    "\n",
    "df.dropna(subset=['ICM'], inplace=True)\n",
    "\n",
    "\n",
    "df.fillna(value='', inplace=True)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'YEAR')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA44AAAGfCAYAAAAQ4oyDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xu4XHV59//3TcI5yEEwIgGCBZ6qpKJE0Gp/JioQxQo+tYqPIiiKrVoPRUusTwuiKLai1kO1KCh4iueSchAQiT4egACiISASIQgRgsoxiGDg/v2xvpHJZO+1Z2bP2nt25v26rrn27O/MfOb+zprTPWvNmshMJEmSJEkazSaTXYAkSZIkabDZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkqa0iPhcRLx3kq47IuKzEXFnRFw2GTVIkjQRbBwlSX0VESsj4vaI2Lpl7LURsWQSy2rKs4ADgVmZuX/7iRFxVEQ8FBFryuHG0mjuPfGl9l9EzIyI30bEvLbx0yNi0SSVJUlqgI2jJKkJ04C3THYR3YqIaV1eZHdgZWbeV3OeH2fmDGBb4HnA/cAVEbFPj2VOmoiY3vp/Zq4G3gZ8OiK2LOd5LvBC4B+avG5J0sSycZQkNeHfgbdHxHbtJ0TE7IjI1kYgIpZExGvL8aMi4ocR8eGIuCsiboiIvyzjN5e1mUe2xe4YERdGxL0R8b2I2L0l+8/LaXdExHUR8dKW0z4XEZ+MiHMj4j5g/gj1Pi4iFpfLr4iI15Xxo4HPAM8oaxPfXXeDZOZDmfnLzHwD8D3ghJbreFFELC/zXRIRT2g5bdeI+GZE/CYifhcRHy/je5a53l3W+n1lpOttub2PiYhfR8StEfH2ltM3iYiFEfHLkv/ViNih7bJHR8SvgO+OMK/PA9cBJ5bm8b+AN2fmb0rGrIj4Vqn/xoh4Y8t1PyMiLinzvjUiPhoRm5bTppfrfkNErAB+Xnf7SpKaZeMoSWrC5cAS4O1jnG80BwA/Ax4NfAlYBDwN2BN4JfDxiJjRcv5XAO8BdgSuAr4IUDaXvbBkPAY4HPjPiHhiy2X/D3ASsA3wgxFqWQTcAjwOeAnwvoh4TmaeBvwdZY1iZh7fxfy+CfxVqXFv4MvAW4GdgHOB/4mIzcoa0LOBm4DZwC6lHsp8LwC2B2YBHxvjOucDewEHAcdFxPPK+D8AhwHPLnO8E/hE22WfDTwBOHiU7L8DXlNquzozF5W5bVLqX1pqPxB4R1krCbCWas30jsAzgQXA69uyX0S17OeMMT9JUoNsHCVJTflX4B8iYqceLntjZn42Mx8CvgLsCpyYmQ9k5gXAg1RN5DrnZOb3M/MB4F1UawF3pdpkcmXJWpuZPwG+Afxty2XPyswfZubDmfmH1iJKxjOB4zLzD5l5FdVaxlf1MKdWvwZ2KMdfVuq/MDP/CHwQ2BL4S2B/qmbuHZl5X6lhXXP7R6pNZR/XNj6ad5eMZcBngZeX8b8D3pWZt5Tb7wTgJW2bhp5QLnv/SMGZeQvV8n4e8PctJz0DeFRmvi8zH8zMFcBpVA08mbk0My8ty+YG4FSqJrXV+zLzztGuW5I0MWwcJUmNyMyrqdY2Lezh4qtbjt9f8trHWtc43txyvWuAO6gart2BA8qmkHdFxF1UaycfO9JlR/A44I7MvLdl7CaqtWfjsUupcd113NRS/8Olpl2oGuabMnPtCBn/BARwWdnM9TVjXGfrPG8q1wvVbfStltvnWuAhYOYolx3NcuDOzLy1ZWx3YLe22/+fKLd/2Yz4nIi4LSLuAU6kWvs4Wt2SpEniF80lSU06HrgSOKVlbN2OZLYC7inHWxu5Xuy67kjZhHUHqrV6NwPfy8wDay6bNaf9GtghIrZpaR53A1aNs94XA/+v5Tr+tBlmRATVfFYBD1A1XtPbm8fMvA1Y933LZwHfiYjvl7V6I9mVR74nuFu5Xqhuo9dk5g/bLxARs9ddXTeTa3EzcH1mPmGU0/8LuAR4WWauKd+9fGHbeXq9bklSH7nGUZLUmNLEfAV4c8vYb6iaoldGxLSypuzPxnlVL4iIZ0XEZlTf/bskM2+mWuO5d0QcERGblsPTWnc+M0b9NwM/At4fEVtExF8ARwNf6LbAMtc9IuJjwDxg3c50vgocEhHPLTuGOZaqYfwRcBlwK3ByRGxdanhmyfvbiJhVMu6karAerinhXyJiq4h4EvBqquUC8CngpHU7FIqInSLi0G7nN4ofAw9GxLGl9mkRMSci9iunbwPcDdxXlkn79xslSQPCxlGS1LQTga3bxl4HvAP4HfAkqiZpPL5EtXbzDmA/qh3oUNYSHkT1nbpfA7cBHwA27yL75VQ7pvk18C3g+Mz8TheXf0ZErKFau7oEeBTwtPJdQzLzulLvx4DfAn8N/HX5TuBD5f89gV9R7aTnZSX3acClJXsx8JbyPcHRfA9YAVwEfLB8VxTgP8rlL4iIe6nWAB7QxfxGVdaSvoDqu5ory/z+i+o2gKpJPhK4t4yPuGdYSdLki0y3AJEkaWNVNje9Edh0lO9KSpI0Jtc4SpIkSZJq2ThKkiRJkmq5qaokSZIkqZZrHCVJkiRJtWwcJUmSJEm1pk92AZNpxx13zNmzZ495vvvuu4+tt27fk3zv+p3XROYw1jiMc24ic9Dzmsi0xuHIayJzGGscxjk3kTmMNQ7jnJvIHPS8JjKHscZu8q644orfZuZOY54xM4f2sN9++2UnLr744o7O16l+5zWROYw1DuOcm8gc9LwmMq1xOPKayBzGGodxzk1kDmONwzjnJjIHPa+JzGGssZs84PLsoHdyU1VJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUq3pk12AJEmSpOE0e+E5G4wdO2ctR7WNrzz5kIkqSaNwjaMkSZIkqZaNoyRJkiSplo2jJEmSJKmWjaMkSZIkqZaNoyRJkiSplo2jJEmSJKmWjaMkSZIkqZa/4yhJkiRpTJ3+5iL4u4sbI9c4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSak1I4xgRW0TEZRHx04hYHhHvLuN7RMSlEbEiIr4SEZuV8c3L/yvK6bNbst5Zxq+LiINbxheUsRURsXAi5iVJkiRJw2Ci1jg+ADwnM58M7AssiIinAx8APpyZewJ3AkeX8x8N3FnGP1zOR0Q8ETgceBKwAPjPiJgWEdOATwDPB54IvLycV5IkSZI0ThPSOGZlTfl303JI4DnA18v4GcBh5fih5X/K6c+NiCjjizLzgcy8EVgB7F8OKzLzhsx8EFhUzitJkiRJGqfIzIm5omqt4BXAnlRrB/8duKSsVSQidgXOy8x9IuJqYEFm3lJO+yVwAHBCucwXyvhpwHnlKhZk5mvL+BHAAZn5phHqOAY4BmDmzJn7LVq0aMza16xZw4wZM3qdeuN5TWQOY43DOOcmMgc9r4lMaxyOvCYyh7HGYZxzE5nDWOMwzrmJzPHkLVt19wZjM7eE1fdveN45u2zb18xO80aysS+X8ebNnz//isycO+YZM3NCD8B2wMXAs6jWEq4b3xW4uhy/GpjVctovgR2BjwOvbBk/DXhJOXymZfwI4ONj1bLffvtlJy6++OKOztepfuc1kTmMNQ7jnJvIHPS8JjKtcTjymsgcxhqHcc5NZA5jjcM45yYyx5O3+3Fnb3D46Bf+e8TxfmeOx8a+XMabB1yeHfRx0ztsWvsmM++KiIuBZwDbRcT0zFwLzAJWlbOtomokb4mI6cC2wO9axtdpvcxo45IkSdLQmb3wnA3Gjp2zlqPaxleefMhElaQpbKL2qrpTRGxXjm8JHAhcS7Xm8SXlbEcCZ5Xji8v/lNO/W7rhxcDhZa+rewB7AZcBS4G9yl5aN6Pagc7i5mcmSZIkSRu/iVrjuDNwRvme4ybAVzPz7Ii4BlgUEe8FfkK16Snl7+cjYgVwB1UjSGYuj4ivAtcAa4E3ZuZDABHxJuB8YBpwemYun6C5SZIkSdJGbUIax8z8GfCUEcZvoNojavv4H4C/HSXrJOCkEcbPBc4dd7GSJEmSpPVM1O84SpIkSZKmKBtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSrQn5HUdJkiRJatrshedsMHbsnLUc1Ta+8uRDJqqkjYZrHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtaZPdgGSJEnSsJu98JwRx4+ds5aj2k5befIhE1GStB7XOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJqTUjjGBG7RsTFEXFNRCyPiLeU8RMiYlVEXFUOL2i5zDsjYkVEXBcRB7eMLyhjKyJiYcv4HhFxaRn/SkRsNhFzkyRJkqSN3UStcVwLHJuZTwSeDrwxIp5YTvtwZu5bDucClNMOB54ELAD+MyKmRcQ04BPA84EnAi9vyflAydoTuBM4eoLmJkmSJEkbtQlpHDPz1sy8shy/F7gW2KXmIocCizLzgcy8EVgB7F8OKzLzhsx8EFgEHBoRATwH+Hq5/BnAYc3MRpIkSZKGS2TmxF5hxGzg+8A+wD8CRwH3AJdTrZW8MyI+DlySmV8olzkNOK9ELMjM15bxI4ADgBPK+fcs47sC52XmPiNc/zHAMQAzZ87cb9GiRWPWvGbNGmbMmNHbhCcgr4nMYaxxGOfcROag5zWRaY3DkddE5jDWOIxzbiJzGGvc2Oe8bNXdI47P3BJW37/+2Jxdtu05cyLypkKNneaNZpDuO+PNmz9//hWZOXes800fd1VdiIgZwDeAt2bmPRHxSeA9QJa/pwCvabKGzDwVOBVg7ty5OW/evDEvs2TJEjo5X6f6nddE5jDWOIxzbiJz0POayLTG4chrInMYaxzGOTeROYw1buxzPmrhOSOOHztnLacsW/8t+8pXdHYdI2VORN5UqLHTvNEM0n1nIvJgAhvHiNiUqmn8YmZ+EyAzV7ec/mng7PLvKmDXlovPKmOMMv47YLuImJ6Za9vOL0mSJEkah4naq2oApwHXZuaHWsZ3bjnbi4Gry/HFwOERsXlE7AHsBVwGLAX2KntQ3YxqBzqLs9re9mLgJeXyRwJnNTknSZIkSRoWE7XG8ZnAEcCyiLiqjP0z1V5R96XaVHUl8HqAzFweEV8FrqHaI+sbM/MhgIh4E3A+MA04PTOXl7zjgEUR8V7gJ1SNqiRJkiRpnCakcczMHwAxwknn1lzmJOCkEcbPHelymXkD1V5XJUmSJEl9NFG/4yhJkiRJmqJsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbWmd3KmiNgWeDAz74+ITYBXAQ9l5ucbrU6SJEkDb/bCczYYO3bOWo5qG1958iETVZKkPut0jeM5wJxy/ATgfcB7I+J9TRQlSZIkSRocnTaOTwCuKMdfARwIPAt4ZRNFSZIkSZIGR0ebqgLTMvOhiNgd2CwzlwNExPbNlSZJkiRJGgSdNo7LIuL/ArsBFwBExM7AvU0VJkmSJEkaDJ1uqvpm4PnAnsCJZexAShM5lojYNSIujohrImJ5RLyljO8QERdGxPXl7/ZlPCLioxGxIiJ+FhFPbck6spz/+og4smV8v4hYVi7z0YiIDucmSZIkSarR6RrHqzLzma0DmXlmRHS6V9W1wLGZeWVEbANcEREXAkcBF2XmyRGxEFgIHEfVpO5VDgcAnwQOiIgdgOOBuUCWnMWZeWc5z+uAS4FzgQXAeR3WJ0mSJHXMPclq2HS6xvHuUcZ/18mFM/PWzLyyHL8XuBbYBTgUOKOc7QzgsHL8UODMrFwCbFc2jT0YuDAz7yjN4oXAgnLaozLzksxM4MyWLEmSJEnSOHTaOG6w2Wevm4JGxGzgKVRrBmdm5q3lpNuAmeX4LsDNLRe7pYzVjd8ywrgkSZIkaZyiWkE3yokRp5ajR/LImsF1Hg9snZnP6PjKImYA3wNOysxvRsRdmbldy+l3Zub2EXE2cHJm/qCMX0S1Ces8YIvMfG8Z/xfgfmBJOf/zyvhfAcdl5gtHqOEY4BiAmTNn7rdo0aIx616zZg0zZszodJoTntdE5jDWOIxzbiJz0POayLTG4chrInMYaxzGOTeROWg1Llu14cZpM7eE1fevPzZnl217yoeNf84j5TWRORF5U6HG8dwXYfAf093kzZ8//4rMnDvW+cb6juOm5W+0HAd4mGqN4Wc6qgaIiE2BbwBfzMxvluHVEbFzZt5aNje9vYyvAnZtufisMraKqnlsHV9SxmeNcP4NZOapwKkAc+fOzXnz5o10tvUsWbKETs7XqX7nNZE5jDUO45ybyBz0vCYyrXE48prIHMYah3HOTWQOWo3t3+uD6vt+pyxb/63mylf0lg8b/5xHymsicyLypkKN47kvwuA/ppt4vNQ2jpn5aoCIuCYz/73XKymbtZ4GXJuZH2o5aTHV2syTy9+zWsbfFBGLqHaOc3dpLs8H3tfy+5EHAe/MzDsi4p6IeDpVQ/sq4GO91itJkiRJekRHe1UdT9NYPBM4gur3IK8qY/9M1TB+NSKOBm4CXlpOOxd4AbAC+D3w6lLHHRHxHmBpOd+JmXlHOf4G4HPAllR7U3WPqpIkacK5t01JG6OOGseI2JtqDd5cYJvW0zJzs7EuX76rONrOdJ47wvkTeOMoWacDp48wfjmwz1i1SJIkSZK60+nvOH6Oak+lRwD3NVaNJEmSJGngdNo47gM8OzP/2GQxkiRJkqTB02nj+HPgMYyyp1JJkqR2nX7XD/y+nyQNuk4bx88C34iIfwNuaz0hM3/U96okSZIkSQOj08bxE+Xv19vGE5jWv3IkSZIkSYOm05/j2KTpQiRJkiRJg6nTNY4AREQAj83MWxuqR5IkaVT+RqIkTY6O1iRGxIyIOA24H1hRxg6LiOObLE6SJEmSNPk6XeN4CjATeCbwnTK2FHgf8O4G6pIkSRKuZZU0GDptHF8IPDEz746IBMjMVRHxuOZKkyRJkiQNgk53erMJ1WaqfxIRM4A1fa9IkiRJkjRQOm0cfwC8s23sH4CL+1uOJEmSJGnQdLqp6j8C342IVwIzImIZsBnwnMYqkyRJkiQNhE5/x/HmiNiH6ruOewA3AWdn5v31l5QkSRpc7nhGkjrTUeMYEUdk5ueBb7SNvyIzv9hIZZIkSZKkgdDpdxw/Mcr4x/pViCRJkiRpMHXaOMYGAxGzgbX9LEaSJEmSNHhqN1WNiD8CCUyLiAfbTp4GfLKpwiRJkiRJg2Gs7zg+j2pt47nA81vGHwZuy8zrmypMkiRJkjQYahvHzPweQET8WWbeOjElSZIkSf3lHnTVK+87lU5/juPWiHg08DRgJ1q+85iZZzZUmyRJkoaQb9SlwdPpz3E8j+qnOB4EtgPuKn9vBGwcJUmSJGkj1lHjCJwMnJiZp0TEnZm5U0T8K7CmwdokSZLUZyOtzQPX6Emq1+nPcewFfKQcX7eZ6geAt/a9IkmSJEnSQOm0cfw9sHk5/ruI2A3YDNi+kaokSZIkSQOj08bxR8Bh5fh5wGLgO8CPmyhKkiRJkjQ4Ov2O4yt5pMl8O3AssA3woSaKkiRJkiQNjk4bxy0y806AzPwDcFJzJUmSJEmSBkntpqoR8RcRsRL4bURcHxFPmJiyJEmSJEmDYqzvOP47cBnwIuCnwPsbr0iSJEmSNFDG2lT1KcCfZea9EfEjYNkE1CRJkiRJGiBjrXHcIjPvBSjfcdyq+ZIkSZIkSYNkrDWOm0TEM4Ao/09r+5/M/FFTxUmSJEmSJt9YjeNWwA/bxlr/T2BaXyuSJEmSJA2U2sYxM8falFWSJEmStJGzMZQkSZIk1bJxlCRJkiTVsnGUJEmSJNWycZQkSZIk1Rq1cYyIpS3Hj5+YciRJkiRJg6ZujeNeEbHu9xqPHc+VRMTpEXF7RFzdMnZCRKyKiKvK4QUtp70zIlZExHURcXDL+IIytiIiFraM7xERl5bxr0TEZuOpV5IkSZL0iLqf47gU+H5EXAtsERGnjnSmzDymg+v5HPBx4My28Q9n5gdbByLiicDhwJOAxwHfiYi9y8mfAA4EbgGWRsTizLwG+EDJWhQRnwKOBj7ZQV2SJEmSpDHUrXE8HDgXWLfWcdNRDmPKzO8Dd3RY06HAosx8IDNvBFYA+5fDisy8ITMfBBYBh5a1os8Bvl4ufwZwWIfXJUmSJEkaQ2Tm2Geq1uy9aFxXFDEbODsz9yn/nwAcBdwDXA4cm5l3RsTHgUsy8wvlfKcB55WYBZn52jJ+BHAAcEI5/55lfFfgvHXXM0IdxwDHAMycOXO/RYsWjVn7mjVrmDFjRtdznqi8JjKHscZhnHMTmYOe10SmNQ5HXhOZg1bjslV3bzA2c0tYff/6Y3N22baveU1kTlbeVKhxpLypUKPLZTBr9DHdW2a7yXx9mT9//hWZOXes89Vtqvon65rGiJgJ7ArcnJmrO6pkdJ8E3gNk+XsK8JpxZo4pM08FTgWYO3duzps3b8zLLFmyhE7O16l+5zWROYw1DuOcm8gc9LwmMq1xOPKayBy0Go9aeM4GY8fOWcspy9Z/u7DyFZ3ld5rXROZk5U2FGkfKmwo1ulwGs0Yf071ltpsKry8d/RxHRGwXEWcDtwKXAb+OiP+JiB16veLMXJ2ZD2Xmw8CnqTZFBVhF1ZyuM6uMjTb+O2C7iJjeNi5JkiRJ6oNOf8fxI+Xvn1N9r/EJVGsKP9TrFUfEzi3/vhhYt8fVxcDhEbF5ROwB7EXVrC6l2tPrHmWvqYcDi7Pa1vZi4CXl8kcCZ/ValyRJkiRpfR1tqgocBDwhM9dt4PuLiDgSuKaTC0fEl4F5wI4RcQtwPDAvIvalakBXAq8HyMzlEfHVkr0WeGNmPlRy3gScD0wDTs/M5eUqjgMWRcR7gZ8Ap3U4L0mSJEnSGDptHIOqwWv1MI/scbVWZr58hOFRm7vMPAk4aYTxc6n29No+fgOPbOoqSZIkSeqjTjdVvRD4fEQ8PiI2iYjHU/024wWNVSZJkiRJGgidNo5vBTan+k3FPwLXA1sAb2uoLkmSJEnSgOj05zjuABaUHdqs+zmOWxutTJIkSZI0EDr9jiMApVm0YZQkSZKkIdLppqqSJEmSpCFl4yhJkiRJqmXjKEmSJEmqZeMoSZIkSarVceMYEctajv+vZsqRJEmSJA2a2sYxIv4pIv4qIrYCZrWc9ONmy5IkSZIkDYqx1jjuBLwfWA1sHREnR8QCIBqvTJIkSZI0EGobx8x8R2Y+C3g08AfgPuAdwDYR8e2IeO0E1ChJkiRJmkRjbar60Yh4ObArsDYz35OZzwXWAB8D/nICapQkSZIkTaLpY5z+S+CvgZOAR0XEl4CLATLzHOCcZsuTJEmSJE222sYxM/9j3fGIuAv4NvAcqk1VlwPfyMx/bbZESZIkSdJk6uZ3HDMzz8zMo4C7gZcCaxupSpIkSZI0MMbaVLXVoS3HIzOXA8v7XI8kSZIkacB0vMYxM7/fcnz7ZsqRJEmSJA2abjZVlSRJkiQNIRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUq0JaRwj4vSIuD0irm4Z2yEiLoyI68vf7ct4RMRHI2JFRPwsIp7acpkjy/mvj4gjW8b3i4hl5TIfjYiYiHlJkiRJ0jCYqDWOnwMWtI0tBC7KzL2Ai8r/AM8H9iqHY4BPQtVoAscDBwD7A8evazbLeV7Xcrn265IkSZIk9WhCGsfM/D5wR9vwocAZ5fgZwGEt42dm5RJgu4jYGTgYuDAz78jMO4ELgQXltEdl5iWZmcCZLVmSJEmSpHGKqteagCuKmA2cnZn7lP/vysztyvEA7szM7SLibODkzPxBOe0i4DhgHrBFZr63jP8LcD+wpJz/eWX8r4DjMvOFo9RxDNWaTGbOnLnfokWLxqx9zZo1zJgxo7eJT0BeE5nDWOMwzrmJzEHPayLTGocjr4nMQatx2aq7NxibuSWsvn/9sTm7bNvXvCYyJytvKtQ4Ut5UqNHlMpg1+pjuLbPdZL6+zJ8//4rMnDvW+aaPu6o+yMyMiAnpYDPzVOBUgLlz5+a8efPGvMySJUvo5Hyd6ndeE5nDWOMwzrmJzEHPayLTGocjr4nM8ebNXnjOBmPHznmIU35w33pjK08+pKO8o0bMW8spy9Z/u7DyFfP6mtdE5mTlTYUaR8qbCjW6XAazRh/TvWW2G7TXl5Fc5VRXAAAgAElEQVRM5l5VV5fNTCl/by/jq4BdW843q4zVjc8aYVySJEmS1AeT2TguBtbtGfVI4KyW8VeVvas+Hbg7M28FzgcOiojty05xDgLOL6fdExFPL5u8vqolS5IkSZI0ThOyqWpEfJnqO4o7RsQtVHtHPRn4akQcDdwEvLSc/VzgBcAK4PfAqwEy846IeA+wtJzvxMxct8OdN1DtuXVL4LxykCRJkiT1wYQ0jpn58lFOeu4I503gjaPknA6cPsL45cA+46lRkiRJkjSyydxUVZIkSZI0Bdg4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmpNn+wCJEkaBrMXnrPB2LFz1nJU2/jKkw+ZqJIkSeqYaxwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtWwcJUmSJEm1bBwlSZIkSbVsHCVJkiRJtaZPdgGSJA2i2QvP2WDs2DlrOaptfOXJh0xUSZIkTRrXOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJq2ThKkiRJkmrZOEqSJEmSatk4SpIkSZJqTZ/sAiJiJXAv8BCwNjPnRsQOwFeA2cBK4KWZeWdEBPAfwAuA3wNHZeaVJedI4P+W2Pdm5hkTOQ9J0uSZvfCcDcaOnbOWo9rGV558yESVJEnSRmVQ1jjOz8x9M3Nu+X8hcFFm7gVcVP4HeD6wVzkcA3wSoDSaxwMHAPsDx0fE9hNYvyRJkiRttAalcWx3KLBujeEZwGEt42dm5RJgu4jYGTgYuDAz78jMO4ELgQUTXbQkSZIkbYwGoXFM4IKIuCIijiljMzPz1nL8NmBmOb4LcHPLZW8pY6ONS5IkSZLGKTJzcguI2CUzV0XEY6jWFP4DsDgzt2s5z52ZuX1EnA2cnJk/KOMXAccB84AtMvO9ZfxfgPsz84MjXN8xVJu5MnPmzP0WLVo0Zo1r1qxhxowZ45xpc3lNZA5jjcM45yYyBz2viUxrnPy8Zavu3mBs5paw+v71x+bssu2kZQ5jjZ3mTYUaN/blMhVqdLkMZo0+pnvLbDeZr6nz58+/ouUrg6Oa9J3jZOaq8vf2iPgW1XcUV0fEzpl5a9kU9fZy9lXAri0Xn1XGVlE1j63jS0a5vlOBUwHmzp2b8+bNG+ls61myZAmdnK9T/c5rInMYaxzGOTeROeh5TWRa4+Tnte8EB6qd45yybP2XuZWv6Dy/35nDWGOneVOhxo19uUyFGl0ug1mjj+neMtsN0mvqaCZ1U9WI2Doitll3HDgIuBpYDBxZznYkcFY5vhh4VVSeDtxdNmk9HzgoIrYvO8U5qIxJkiRJksZpstc4zgS+Vf3KBtOBL2XmtyNiKfDViDgauAl4aTn/uVQ/xbGC6uc4Xg2QmXdExHuApeV8J2bmHRM3DUmSJEnaeE1q45iZNwBPHmH8d8BzRxhP4I2jZJ0OnN7vGiVJkiRp2A3CXlUlSZIkSQPMxlGSJEmSVMvGUZIkSZJUa7J3jiNJGkKzR9m1efsuz1eefMhElSRJkmq4xlGSJEmSVMvGUZIkSZJUy8ZRkiRJklTLxlGSJEmSVMvGUZIkSZJUy8ZRkiRJklTLn+OQJNUa6aczwJ/PkCRpmLjGUZIkSZJUy8ZRkiRJklTLTVUlaSMz0qalblYqSZLGw8ZRkrrQRFNmoydJkgadjaOkjZpNmSRJ0vj5HUdJkiRJUi0bR0mSJElSLRtHSZIkSVItG0dJkiRJUi0bR0mSJElSLRtHSZIkSVItf45D0sDo9KczwJ/PkCRJmkiucZQkSZIk1bJxlCRJkiTVclNVST3rdNNSNyuVJEma2lzjKEmSJEmqZeMoSZIkSarlpqrSEHHTUkmSJPXCNY6SJEmSpFo2jpIkSZKkWm6qKg0oNyuVJEnSoLBxlPrERk+SJEkbKzdVlSRJkiTVco2jhpJrByVJkqTO2ThqSrDRkyRJkiaPjaP6bqQmD2z0JEmSpKnKxlGuzZMkSZJUy8ZxCrLRkyRJkjSRNqq9qkbEgoi4LiJWRMTCya5HkiRJkjYGG80ax4iYBnwCOBC4BVgaEYsz85rJrMu1g5IkSZKmuo1pjeP+wIrMvCEzHwQWAYdOck2SJEmSNOVtNGscgV2Am1v+vwU4oNsQ1xBKkiRJ0voiMye7hr6IiJcACzLzteX/I4ADMvNNbec7Bjim/Pu/gOs6iN8R+G0fy+13XhOZw1jjMM65icxBz2si0xqHI6+JzGGscRjn3ETmMNY4jHNuInPQ85rIHMYau8nbPTN3GutMG9Max1XAri3/zypj68nMU4FTuwmOiMszc+74ymsur4nMYaxxGOfcROag5zWRaY3DkddE5jDWOIxzbiJzGGscxjk3kTnoeU1kDmONTcx5Y/qO41Jgr4jYIyI2Aw4HFk9yTZIkSZI05W00axwzc21EvAk4H5gGnJ6Zyye5LEmSJEma8jaaxhEgM88Fzm0guqtNWychr4nMYaxxGOfcROag5zWRaY3DkddE5jDWOIxzbiJzGGscxjk3kTnoeU1kDmONfZ/zRrNzHEmSJElSMzam7zhKkiRJkhpg4yhJkiRJqmXjKEmSJEmqtVHtHEeSJA2miNgBIDPvGOTMQdfvOQ/rchnk2zEiZgK7lH9XZebqAc0c6PtOv+c8VZZLk9w5zij6uSAjIoD9W/OAy7IPN/4gP8AazPSJapymwnIZ5Nuxicd0Q5kDfd/p95yHdbn0s8aI2BZ4J3AY8BgggduBs4CTM/OuLvN2A/4NeC5wFxDAo4DvAgszc2UPNfY1s4E59zWvZPZ7zkO3XBqqsd95+wKfAralehwDzCrZb8jMK7vJayJzitx3+j3ngV8uEyYzPbQcgH2BS4Brge+Uw8/L2FN7yDsIWAGcB3ymHL5dxg7qscbdgEXAb4DrS9btZWz2ZM+5oduxr3OeCrfjEC+Xgb4dG3pM9zVzitx3+j3nYV0u/a7xfOA44LEtY48tYxf0kPdj4GXAtJaxacDhwCU9zrmvmQ3Mua95Dc156JbLVLgdgauAA0YYfzrw0x7n3NfMKXLf6fecB365TNRh0gsYtEMDd7ZrGeENNLAHcG2PNQ70A6yh29EnquFZLgN9Ozb0mO5r5hS57/R7zsO6XPpd43W9nFZzmet7OW0iMxuYc1/zGprz0C2XqXA7jpG3ooE5d505Re47EznngVguE3XwO44b2jozL20fzMxLImLrHvKmA7eMML4K2LSHPIAdM/MrrQOZ+RCwKCLe00Nev+fcRGa/59xEZr/nPKzLZdBvxyYe0/3OnAr3nX7PeViXS79rvCki/gk4I8smtGXT2qOAm3vIuyIi/hM4o+XyuwJHAj/pIa+JzH7Pud950P85D+NyaaLGfuedFxHnAGe25b2KakuCXvQ7cyrcd/o956mwXCaEjeOG+r0gTweWRsSitrzDgdN6rHHQH2BNZPpENf68JjKbWC6Dfjs28Zjud+ZUuO/0e87Dulz6XePLgIXA9yLiMWVsNbAYeGkPea8CjgbezSPfwbwF+J8e62sis99z7nce9H/Ow7hcmqixr3mZ+eaIeD5wKOt/Z/kTmXluD/U1kTnw951+z3mKLJcJ4c5xRjDKglzc64KMiCeMkndNj3mbUT3AWjP/9ADLzAd6yOzrnPud2dCcB/52HMbl0u8aG8rr62O6icxBv++UvH7PeViXS9/nLUlSOxtHSZI0poh4avZxT38R8cLMPLtfeU1kNjDnvuaVzH7PeeiWS8kc6NsxIo7JzFP7lddE5hS57/R7zgO/XPppk8kuYCqJiGP6nHdCP/NK5gv7nNfXOTeR2e85N5HZwJyHdbkM9O3Y0GO6r5lT5L5zwiDnNZHZ0HI5oc+Rf9/nvKf1Oa+JzH7Pud950P85D+NygcG/HaPPeU1kToX7Tr/nPBWWS9/YOHan3wvyij7nweA/wJrI9IlqMDObWC6Dfjs28Zjud+ZUuO/0e87Dulz6WmNmvq7Pecf3M6+JzAbm3Ne8ktnvOQ/dcimZA307ZuZ/9TOvicwpct/p95wHfrn0k5uqSpKkP4nqh9cXsP53Js/PHn5wfYzrOTAzL+zxso8CdsrMX7aN/0Vm/qyHvMcCZOZtEbET8FdUP/mwvJf6Rsh/X2b+cz+ySt4ewFOAazLz5z1cfjfg9sz8Q0QE1Z5KnwpcA3w6M9f2kPkiqt9X/EO3l63J/P+A1Zl5XUQ8E3gG1c/MnDOOzBlU9+9dgYeAX1DV/XCPeX/OyN8xvrbXGke5nldn5md7vOyfU9V3aWauaRlfkJld75grIvYHMjOXRsQTqW7Pn/dzpy4RcWZmvqpPWc8C9geuzswLerj8AVT3u3siYkuqnTate7y8LzPv7iHzzcC3MrPXPQRPChvHNhGxA/Am4NdUe3L6Z8oTFdWd484eMucDf8P6T1KfycwV46jTJyqfqCb9iarsHOdw4NeZ+Z2I+D/AX1I9Xk7NzD/2mPt44H+z/mPmS5l5T495BwOHsf7j5axe7odjXM+/ZuaJPV72YGAWcFFmrmwZf01mnt5lVgB/CyTwdeA5VM8XPwc+1esbpBGu57uZ+ZweL7tjZv625f9XUh4vVG9cu3pxiogXA9/LzDvKG/9TKG+sgWMzc6SfrBgr80PANzLzh91edpS8vr++lNy+vcZExKuA44ELqB4nUN0vDwTenZln9lLjKNf1q8zcrYfLvRT4CHA71U+OHJWZS8tpV2bmU7vMez3V82sAH6Bqoq4GngX8W2Z2tVfHiPho+xBwBNXedMnMN3eTVzL/OzMPK8cPpZr/Eqrn2/dn5ue6zLsa2D8zfx8RHwD+DPhvqucKMvM1PdR4P3AfcB7wZaoPGx7qNqcl7yNUzwnTgfOB55bsZwM/ycx39JD5UuDtwM+A+cCPqLa+mwO8stsPHSLiOODlwCIe+VmcWVSvi4sy8+Rua6y5rl4fL28G3kj1PLMv8JbMPKuc1svj5Xjg+VTL5ULgAOBiqueI8zPzpB5qXNw+RLV8vguQmS/qMu+yzNy/HH8d1fy/BRwE/E+3yyUilgNPzsy1EXEq8Huq19bnlvH/3U1eybyb6vHyS6rHy9cy8zfd5kw0G8c2EXEusAx4FPCEcvyrVA+IJ2fmoV3mvR94LHAR1RvXG6le1N9A9Ubhaz3U6BOVT1QD8UQVEV+kWiZbAXcBM4BvlhojM4/sIfPNwAuB7wMvoPpZj7uAFwNvyMwlXeZ9BNib6k1b6+PlVVQ/wPuWbmusua5eHy/vo3qTeiXw18BHMvNj5bReHi//CTwG2Ay4B9icahf2h1B9et/1nCOi/Q1VUN2u1wFk5l90mfeneUXE/6Vaw/MlqmV/S2a+rcu8azLzieX4V4BLgK8BzwNekZkHdpNXcn4D3ATsBHwF+HJm9vozM31/fSmZfX2NiYjrgAPa1y5GxPZUHwLu3WVe+/Psn04CnpOZXf9+ZURcBTw/M28tHyieCbwzM78VET/JzKd0mbeM6jVlS6rlvWdZ87g9cHFm7ttl3s3A96ia73WbIH+QqmEhM8/oJq9k/mleEfEjqvv0jRGxI9WHTU/uMq/18XIF8LR1HyhFxE+7zVtXI1Xj+RKq9yP7UL0Gfjkzv9dD3vKSsSXVhxi7lEZ3U6rGcZ8eMn8GPL3k7Ah8MTMPjoi/oPpQ7S+7zPsF8KT2D0nLh6rLM3OvHuob8SRg78zcvJu8krkMeEZmromI2VTvIz6fmf8xjsfLvlSvK7cBs1o+4L6029eCknkl1Yd8n6H6wDOo3qMcDtDt/aft8bIUeEFm/iaq38u9JDPndJl3bWY+YV2tra/JEXFVt88R62oE9qN6jXoZ8CKqrxd8GfhmZt7bbeaEyEwPLQfgqvI3gFUjndZl3rKW49OBH5bj21Otieqlxl8Am44wvhnVG+Fu8342ymEZ8ECPNS4DZpTjs4HLqZpHqJ7we8mbRtWg3AM8qoxvCfysxxqvBL4AzKP6BHMecGs5/uwe8n7Scnwp1WZUAFu33g+6yLu2tdbx3hfX1Uj16epBVGs8fkP1+3FHAtv0ct8pf6dT/b7WtPJ/jGO5LGvJ2QpYUo7v1uN95xejjEePj5d7RjncC6wdx5ynl+PbAecCH26/X3WTV/5uCvwO2KxlOfW6XBaXx8ufA7uXx/XN5fjuvdwXW45fCWzdUnMvj5frWo5f0XZaz4+X8ndv4F+A5VRrbY+nehPXbV5fX19al3XL8h3XawzV68u2I4xv2+Pj5U6qDyye3XaYR/UhxrjmXP7fmeoN15tpe67sMO/KluM/He1+2kXeNlRrBL8EPK6M3dDLXEep8bI+1Hg+VeMO8I11j2Hg0e23QS81lv8fW5bJj4Gbe8i7uvzdotyPtiz/T6PaRLen+w6PrDTZsu15qJfHy89Hev4rz4vX9ZC3mqop273tMJtqy55e5ry87f8ZVK/7H+rleaftNvtJ22m9Po9tAryNasXAvmWs58cM8NPyHPho4PLR6u8i72vAq8vxzwJzy/G9gaU91tj+eNmUqnn8MvCbXufe9GE6ardJ+ZRxG2BGRMzOzJUR8WiqxqxbD0fEDpl5B/A4qic8MvPOsjlZLx4uWTe1je9cTuvWTOBgqifmVkG1GUcvNsmyeWq5/eYBX4+I3aGnnUCszWqTl99HxC+zbLKYmfdHRK+b3c0F3gK8C3hHZl4VEfdnD5+MFuvuO5tQvTD9ptR4X0R0/X0R4Op4ZFPhn0bE3My8PCL2BnraBLQqJx+m+iT8gvLJ7fOp1mB/kGrNSjc2KZ+sbk3V5G0L3EH1SeSmPdYI1Rvgh0rOjFL4r0q93fpDRDwty6ZsLZ4G9PJdnLuoPp1f3X5CWdPQi+lZvlOUmXdFxF8Dp0bE1+jteWdd1h8jYmlmPlj+X9vr4yUzX1Q2Bz0V+GBmLo6IP2Zm+/NQp7aMiKdQPV6mZeZ9LTX3snnbkog4EXh/Of7irNZAzQe63qy7yFLTL4D3AO8payZeTtXc79llXr9fX6D/rzEnAVdGxAVUHwxA9aHNgVS3QbcuAX4/0vNqWbvZi3sj4s+yfL8xqzWP86g2tXxSD3kZEZtmtdbokJb6tqCHnQhmtabgrRGxH/DFiDinl5w2T46Ie6hePzePiJ3LvDejLPMuvRY4M6o9794NXFXW5G4H/GOPNa53f8vM24CPAh8tr/3dOici/h9V4/gZ4KsRcQnVBw/f77HGc4FvR8T3qb7u8jX402bkvTxe3gpcFBHXs/7jZU+qzdK7dTbVh+5XtZ8QEUt6yANYHRH7rsvMas3jC4HTqTbR7daDEbFVZv6eao3Zuvq2pbf3oJT3JR8ur3kfjojVMK4eZVuqD5OC6vG97vEyg96W82uB/yhbx/wW+HF5vb+5nNaL9sfLH6k+oF0cEVv1mNm8ye5cB+1A9YZgdTn8DfAdqk9AVgHH9JD3MqoG70LgV8AhZXwnqu9s9VLjAmAF1bb+p5bDt8vYgh7yTgOeNcppvdb4XcqnRi1j06k2KXqoh7xLga3K8U1axrelh0+Y27JnUb14fBz41ThyVgI3UG0qdgOwcxmfQW+f6m0LfI5qs9JLqZrFG6g2gXpyjzWO+knbutu3y7y3lZpuovpk+SLg01Sf6h7fY41voVrj/WmqT3PXfcq3E/D9HvKeWm6/aygNM9Um1JcA+/WQ916q7waNdNoHepzz2Yywlrtc18M95J1HWePfNv5Y2tZW9JC9NdUn1WdRbVLaa87FbYd1j5cNPiHuMG9T4ITyPPsrqjcw91Kt9dmtxxq7/mR6jLy+vr6UzCZeY7an2kTs2HI4HNi+n7fFOG/HJwN7jXIfeEUPebsx8lY8uwDPG2etQfW1hS80dFtsR7UZYq+XfwLV95//hmpz3U3GkTWvgfk9g2rTUqi+h/l24KXjrPMFJefAlrFNgM17zNsEeHq5Df+mHJ/WxPLusb5ZwGNHOe2ZPeSNeDsBOwJz+lTzIVSb2vf7ttgK2GMcl39Uef7ZD5g5zlq63mplEA5+x3EEETGNao3R2oiYTrXZwKrMvLXHvB2AxwMrsk97pYuITai+NN66s4+lOY4vovdTRMyiWkt42winPTO73NlERGyemQ+MML4j1RvOZb1X+6esQ6ieRPu257uSuxXVE8yNPV7+UcAeVI33LTnCmq4usvbOau1J30TE4wAy89cRsR3V9vq/yszLxpH5JKo3NFdnD3sMHCXzsbQ8Xka6b06W8t0QMvP+EU7bJTNXbXipnq5na6pNQm/vQ9aTqd6wfmr8la2XO43qjcnvx5GxLdVa3N+Ns5YZ2bJjr37o9+tLyWziNWYm6z9een7eaSKvicxBz2si0xr7lznCdfT1+aOh5yNrHLC8pjL7xcaxTfS4K++JymvJ3Q24J6vN2mZTbXZ5bfa46/BR8n6emVf3ucaeM4exxmGcc4OZc2nZ6+R4G9J+51njYOYNW40RsS/wKaotHm6hWmM2i2oT7Tdk5pVd5j0F+GTJa91La095I9TYnvn32eUOjCY4r9c5192Og1JjXzOnQo1jXFdPO0ubqLwmMoexxqkw536ycWwT1XdrbqDaY+mXM/OaQcormQuB1wMP8Mie2n5ItXnEaZn5ocnMs8bBzBviGp9N9dMMd1FtXvJDqk3x/ggckV3+NEm/86xxMPOGuMargNdn5qVt408H/iu733tnX/OmQo3DOOchrnG074MG8K7M3GEy85rIHMYap8KcJ4qNY5uodo97BNV3UV5G9dMFX6b6mYuVk51XMpdTrYHZiup7dY/PR3YzfGl2uYvqfudZ42DmDXGNPwEOKhl7AB/KzBdHxIFUO0U6aDLzrHEw84a4xutzlJ8QiIgVmdnVDoH6nTcVahzGOQ9xjX8A/p2yQ7I2b8vM7SYzzxoHM6+pzIngXlU3lFltDvcu4F1R/UbU4f9/e/cXamlVxnH8+0wjmB7IHJtTMFFEOBIkQ1A3ijlSlBQWVqMGMXXRTdQIBkVQBNJFF90UGFk0WYTVZDUGWZiUgYT0Z/pjBkaFWFHWODPChGbW08Xa4W7P7N2cc/ba73pd3w+8OLPfc37zey5GZp1377WAe6I8Ot7QGT8V8qBsLvNYRDwBPEbZbp8su3duIm7peXZsM6/Xjs/Ip86qfIiytTmZ+d0oZzwOnWfHNvN67fjtKLuAfoGndol8PuXc0+80kDeGjj3O3GvHI8DhzPzp7I2I2Mxum8vOq5HZY8cxzLwSPnGcEXMOQ43yL9bLcguHkC4jb/K9t1C2bj+Xcij8k5T/4V1BOYtv35B5dmwzr+OOBynHKnyPckbSnzLzhiibFh3JzIuGzLNjm3m9dpxkXknZaXN687VvZuYdG82qkTeGjj3O3GPHiNgNHJv64c30vfXc4KY7y86zY5t5tTJXwYXjjIh4a2be2mreJHM78BbKPxZuo2yhfR3lp8035eQ8tKHy7NhmXscdzwLeCbyEcijwwcz8V5SdTHfmBs8hXHaeHdvM67WjJEnzuHCUJEkARDnK5AOUJzLrlB/g/JVybudHc4PHfSw7bwwde5zZjrwR2Nlanh3bzKuVuQrbhi7QmohYi4gbI+L+iHg0Iv4WEfdGxNtbyPs/mftbyLNjm3l2jF8t+e/0UvLs2GZerx2BQ8BxYG9mnp+ZO4C9lF1bDzWQN4aOPc7ce8fLZ/KON5JnxzbzamVW5xPHGRFxO/AN4C5gH+VzVl8GPkj57MiGDodfdp4d++nY48xj6NjjzGPo2OPMlTo+kJm7N3pvVXlj6NjjzHZsM8+ObebVylyJzPSauoBfzPz+x5P/bqMcQD5onh376djjzGPo2OPMY+jY48yVOt4JvA9Yn3ptHXg/cNfQeWPo2OPMdmwzz45t5tXKXMXlW1VP9feIuBQgIq4CjgFk5r+BzZwFsOw8O/bTsceZx9Cxx5nH0LHHmWtkXgPsAH4QEccj4hhwN3A+5Ynm0Hlj6NjjzHZsM8+ObebVyqxv6JVraxdwMfAjynuM7wEunLz+HODA0Hl27KdjjzOPoWOPM4+hY48zV8y8CHgVsDbz+mtbyBtDxx5ntmObeXZsM69WZu1r8AJjuoB3tJxnx3YzW8+zY5t5dmwz7+ncETgAPAAcBh4E3jB178jQeWPo2OPMdmwzz45t5tXKXMU1eIExXcBDLefZsd3M1vPs2GaeHdvMezp3BO5j8tNv4IXAT4DrJ7//2dB5Y+jY48x2bDPPjm3m1cpcxbUd/Y+I+OW8W5QPrQ6aVyPTjsvJbD2vRmaPHXucuUZm63k1MsfQEdiWmScBMvPBiLgcuC0iXsDmPjO57LwxdOxxZju2mWfHNvNqZVbnwvFU68BrKJ8XmRbADxvIq5Fpx+Vktp5XI7PHjj3OXCOz9bwamWPo+HBE7MnMnwNk5smIeD1wEHhpA3lj6NjjzHZsM8+ObebVyqxv6EeerV3AZ4FL59y7deg8O/bTsceZx9Cxx5nH0LHHmSt13AU8d869S4bOG0PHHme2Y5t5dmwzr1bmKq6YFJQkSZIk6bQ8x1GSJEmStJALR0mSJEnSQi4cJUmSJEkLuXCUJGkLIuKLEfG5mddeGRGPRMTNEfHPiDg5dZ2Y+dqIiN9HxK9Pk313RDw++b6jEfH1iHhe7ZkkSZrlwlGSpK25HrgyIl4NEBFnA58B3gv8GfhKZq5NXefNfP9lwE7gRRHx8tPkvzsz14AXA2vAx2oNIknSPC4cJUnagsx8BHgP8OmIOBf4MPC7zLzlDCP2A7cDd0x+Pe/POQEcBvZsqbAkSZuwfegCkiSNXWZ+NSKuBb4EXCz+drUAAAFkSURBVMIZLu4i4hzgzcC1wDOBmyPihsx84jRfuwO4Gvjt0opLknSGfOIoSdJyvAu4ArgxM/8w9fq+iDgxdX1/6t7VwD+AO4FvAWcBr5vJ/UREPAocBS6gPN2UJGmlXDhKkrQEmfkwZXF3/8ytQ5l53tS1d+re/sn9JzPzceBrnPp21QOZ+SzgYuDZwK5KI0iSNJdvVZUkaQARsYvyhPIVEfGmycvnAGdHxAWZeXT66zPzvoj4CHBTRLwsM3PFlSVJHfOJoyRJw3gb8BtgN+UzkXuAC4E/AtfN+Z7PA+vAVasoKEnSf7lwlCSprmtmznE8GRE7KW9J/WRm/mX6Aj7FnN1VJ5vmfBz40OrqS5IE4TtdJEmSJEmL+MRRkiRJkrSQC0dJkiRJ0kIuHCVJkiRJC7lwlCRJkiQt5MJRkiRJkrSQC0dJkiRJ0kIuHCVJkiRJC7lwlCRJkiQt5MJRkiRJkrTQfwCXMkWw4uVqEAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "df_AY = df.groupby(['AY'])\n",
    "df_AY = df_AY.size().reset_index(name='Docs')\n",
    "\n",
    "\n",
    "df_AY.plot(x='AY', y='Docs', kind='bar', legend=False, grid=True, figsize=(15, 6))\n",
    "plt.title(\"Number of Docs per Year\")\n",
    "plt.ylabel('# of Patents', fontsize=13)\n",
    "plt.xlabel('YEAR', fontsize=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applying preprocessing tasks on metadata of patents\n",
    "Converting the metatadata such as inventors and assignees of each patent into a python list, then apply preprocessing task on each element in the list in order to remove undesired tokens.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.46 s, sys: 70.6 ms, total: 2.53 s\n",
      "Wall time: 2.53 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#preprocess of list fields\n",
    "#convert all IPCs in df into one list\n",
    "def toList(s):\n",
    "    \"\"\"\n",
    "    this method is to convert the list of IPCs in each row from a string to a python List\n",
    "    \"\"\"\n",
    "    s  = s.translate ({ord(c): \" \" for c in \"[]\"})\n",
    "    ss= []\n",
    "    for cls in s.strip().split(','):\n",
    "        ss.append(cls.strip())\n",
    "    return ss\n",
    "\n",
    "#apply toList method on all rows in the DF\n",
    "df['PA'] = df['PA'].map(lambda pa :   toList(pa))\n",
    "df['INV'] = df['INV'].map(lambda inv :   toList(inv))\n",
    "\n",
    "df.head()  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.31 s, sys: 24.3 ms, total: 5.33 s\n",
      "Wall time: 5.34 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "def metadataPreprocessing(input):\n",
    "    newInput=' '\n",
    "    for item in input:\n",
    "        item = item.translate ({ord(c): \" \" for c in \"!@#$%^&*()'[]{};:,./<>?\\|`~°=\\\"+\"})\n",
    "        itms=' '\n",
    "        for itm in item.split():\n",
    "            itms= itms +' '+itm.strip()\n",
    "        newInput = newInput + ' '+ itms.strip().replace(' ','_')\n",
    "    return newInput.strip()\n",
    "\n",
    "df['PA'] = df['PA'].map(lambda pa :   metadataPreprocessing(pa))\n",
    "df['INV'] = df['INV'].map(lambda inv :   metadataPreprocessing(inv))\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Applying preprocessing tasks on texts of patent\n",
    "A simple preprocessing tasks such as tokenization, stopword removal, lemmatization, and converting letters into lower case are performed on each text section of each patent document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#preprocessing \n",
    "standardStopwordFile = \"sources/stopwords/stopwords-all.txt\"\n",
    "\n",
    "#loading terms from a file to a set\n",
    "def get_terms_from_file(filePath):\n",
    "    terms = set(line.strip() for line in open(filePath))\n",
    "    return terms\n",
    "\n",
    "#remove undiserd terms\n",
    "def remove_terms(termSet, phrase):\n",
    "    newPhrase = \"\"\n",
    "    for term in phrase.split():\n",
    "        if term.strip() not in termSet and len(term.strip())>2:\n",
    "            newPhrase = newPhrase + \" \" + term.strip()\n",
    "\n",
    "\n",
    "\n",
    "def clean_texts(doc):\n",
    "    #Remove punctuation from texts\n",
    "    doc = doc.translate ({ord(c): ' ' for c in \"0123456789!@#$%^&*()'/[]{};:,./<>?\\|`~°=\\\"+\"})\n",
    "    # split into tokens by white space\n",
    "    tokens = doc.lower().strip().split()\n",
    "    \n",
    "    # filter out stop words\n",
    "    stop_words = get_terms_from_file(standardStopwordFile)\n",
    "    #generalStopwords = get_terms_from_file(generalWordsFile)\n",
    "\n",
    "    \n",
    "    tokens = [w.strip('-')  for w in tokens if  w not in stop_words ]\n",
    "    # filter out short and long  tokens\n",
    "    output = [word for word in tokens if len(word.strip()) > 2 and len(word) < 30 ]\n",
    "    output = \" \".join(output)\n",
    "    #apply stemming\n",
    "    #output = stem_text(output)\n",
    "    return output\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 309 µs, sys: 76 µs, total: 385 µs\n",
      "Wall time: 378 µs\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>TI</th>\n",
       "      <th>AB</th>\n",
       "      <th>TECHF</th>\n",
       "      <th>BACKG</th>\n",
       "      <th>SUMM</th>\n",
       "      <th>CLMS</th>\n",
       "      <th>ICM</th>\n",
       "      <th>AY</th>\n",
       "      <th>IPC</th>\n",
       "      <th>REF</th>\n",
       "      <th>PA</th>\n",
       "      <th>INV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>EP2000017943-0</td>\n",
       "      <td>recognition disk-shaped medium</td>\n",
       "      <td>recognition disk-like medium multimedia applic...</td>\n",
       "      <td>recognition disk-shaped medium multimedia appl...</td>\n",
       "      <td>identification labels adapted interpreted opti...</td>\n",
       "      <td>overcome drawbacks noted conventional types id...</td>\n",
       "      <td></td>\n",
       "      <td>G06K</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>[G06K0019-06, G06K0007-10]</td>\n",
       "      <td></td>\n",
       "      <td>Video_System_Italia_S_r_l</td>\n",
       "      <td>Tassello_Stefano</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EP2003016733-0</td>\n",
       "      <td>optical pickup recording reproducing</td>\n",
       "      <td>optical pickup reproducing optical recording m...</td>\n",
       "      <td>optical pickup recording reproducing optical p...</td>\n",
       "      <td>recently practical short wavelength red laser ...</td>\n",
       "      <td>provide pickup recording reproducing optical r...</td>\n",
       "      <td>optical pickup recording reproducing optical m...</td>\n",
       "      <td>G11B</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>[G11B0007-135, G11B0007-125]</td>\n",
       "      <td></td>\n",
       "      <td>Konica_Minolta_Opto_Inc</td>\n",
       "      <td>Arai_Norikazu_Kojima_Toshiyuki_Kiriki_Toshihik...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>EP2011009984-0</td>\n",
       "      <td>large capacity sales mediation</td>\n",
       "      <td>animation sales mediation animation sales medi...</td>\n",
       "      <td>large capacity sales large capacity sales medi...</td>\n",
       "      <td>recent years distributing music network rapidl...</td>\n",
       "      <td>implemented consideration problems provide ani...</td>\n",
       "      <td>large capacity sales mediation terminal large ...</td>\n",
       "      <td>G07F</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>[G07F0017-16, G06Q0030-06, G06Q0020-10, G06Q00...</td>\n",
       "      <td>[JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...</td>\n",
       "      <td>NEC_Corporation</td>\n",
       "      <td>Maeda_Koji</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PCT1997010546-0</td>\n",
       "      <td>bridge client-server environment</td>\n",
       "      <td>software bridge introduced client client-serve...</td>\n",
       "      <td>bridge client-server environment distributed c...</td>\n",
       "      <td>overview object-oriented programming developme...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>bridge client distributed object-oriented brid...</td>\n",
       "      <td>G06F</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>[G06F009-46, G06F009-44, G06F0009-44, G06F0009...</td>\n",
       "      <td></td>\n",
       "      <td>INTERNATIONAL_BUSINESS_MACHINES_CORPORATION_CO...</td>\n",
       "      <td>COLYER_ADRIAN_MARK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PCT1998021641-0</td>\n",
       "      <td>sections operating rates</td>\n",
       "      <td>core clocked perform operations clock frequenc...</td>\n",
       "      <td>sections operating rates high speed processors...</td>\n",
       "      <td>illustrates microprocessor microprocessor incl...</td>\n",
       "      <td>microprocessor levels sub-core clocked frequen...</td>\n",
       "      <td></td>\n",
       "      <td>G06F</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>[G06F001-32, G06F0001-08, G06F0009-30, G06F000...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>SAGER_DAVID_J_FLETCHER_THOMAS_D_HINTON_GLENN_J...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                ID                                    TI  \\\n",
       "0   EP2000017943-0        recognition disk-shaped medium   \n",
       "1   EP2003016733-0  optical pickup recording reproducing   \n",
       "2   EP2011009984-0        large capacity sales mediation   \n",
       "3  PCT1997010546-0      bridge client-server environment   \n",
       "4  PCT1998021641-0              sections operating rates   \n",
       "\n",
       "                                                  AB  \\\n",
       "0  recognition disk-like medium multimedia applic...   \n",
       "1  optical pickup reproducing optical recording m...   \n",
       "2  animation sales mediation animation sales medi...   \n",
       "3  software bridge introduced client client-serve...   \n",
       "4  core clocked perform operations clock frequenc...   \n",
       "\n",
       "                                               TECHF  \\\n",
       "0  recognition disk-shaped medium multimedia appl...   \n",
       "1  optical pickup recording reproducing optical p...   \n",
       "2  large capacity sales large capacity sales medi...   \n",
       "3  bridge client-server environment distributed c...   \n",
       "4  sections operating rates high speed processors...   \n",
       "\n",
       "                                               BACKG  \\\n",
       "0  identification labels adapted interpreted opti...   \n",
       "1  recently practical short wavelength red laser ...   \n",
       "2  recent years distributing music network rapidl...   \n",
       "3  overview object-oriented programming developme...   \n",
       "4  illustrates microprocessor microprocessor incl...   \n",
       "\n",
       "                                                SUMM  \\\n",
       "0  overcome drawbacks noted conventional types id...   \n",
       "1  provide pickup recording reproducing optical r...   \n",
       "2  implemented consideration problems provide ani...   \n",
       "3  bridge client distributed object-oriented brid...   \n",
       "4  microprocessor levels sub-core clocked frequen...   \n",
       "\n",
       "                                                CLMS   ICM      AY  \\\n",
       "0                                                     G06K  2000.0   \n",
       "1  optical pickup recording reproducing optical m...  G11B  2000.0   \n",
       "2  large capacity sales mediation terminal large ...  G07F  2001.0   \n",
       "3  bridge client distributed object-oriented brid...  G06F  1996.0   \n",
       "4                                                     G06F  1997.0   \n",
       "\n",
       "                                                 IPC  \\\n",
       "0                         [G06K0019-06, G06K0007-10]   \n",
       "1                       [G11B0007-135, G11B0007-125]   \n",
       "2  [G07F0017-16, G06Q0030-06, G06Q0020-10, G06Q00...   \n",
       "3  [G06F009-46, G06F009-44, G06F0009-44, G06F0009...   \n",
       "4  [G06F001-32, G06F0001-08, G06F0009-30, G06F000...   \n",
       "\n",
       "                                                 REF  \\\n",
       "0                                                      \n",
       "1                                                      \n",
       "2  [JPHEI033290B, JPHEI08235759B, JPHEI10334048B,...   \n",
       "3                                                      \n",
       "4                                                      \n",
       "\n",
       "                                                  PA  \\\n",
       "0                          Video_System_Italia_S_r_l   \n",
       "1                            Konica_Minolta_Opto_Inc   \n",
       "2                                    NEC_Corporation   \n",
       "3  INTERNATIONAL_BUSINESS_MACHINES_CORPORATION_CO...   \n",
       "4                                                      \n",
       "\n",
       "                                                 INV  \n",
       "0                                   Tassello_Stefano  \n",
       "1  Arai_Norikazu_Kojima_Toshiyuki_Kiriki_Toshihik...  \n",
       "2                                         Maeda_Koji  \n",
       "3                                 COLYER_ADRIAN_MARK  \n",
       "4  SAGER_DAVID_J_FLETCHER_THOMAS_D_HINTON_GLENN_J...  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "apply simple preprocessing on text\n",
    "df['TI'] = df['TI'].map(lambda line : clean_texts(line))\n",
    "df['AB'] = df['AB'].map(lambda line : clean_texts(line))\n",
    "df['TECHF'] = df['TECHF'].map(lambda line : clean_texts(line))\n",
    "df['BACKG'] = df['BACKG'].map(lambda line : clean_texts(line))\n",
    "df['SUMM'] = df['SUMM'].map(lambda line : clean_texts(line))\n",
    "df['CLMS'] = df['CLMS'].map(lambda line : clean_texts(line))\n",
    "\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Applying preprocessing tasks for patent labels (main IPC codes)\n",
    "The main IPC codes is considered to be the labels for the patent documents. we only consider the subclass level of the IPC code. \n",
    "Each label/class has at least more than 500 documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42\n",
      "number of remaining documents in the dataset is:  403726\n",
      "Number of unique labels is:  42\n"
     ]
    }
   ],
   "source": [
    "#process the ICM codes and #related-patents\n",
    "df['ICM'] = df['ICM'].map(lambda icmCode : icmCode[:4])  \n",
    "\n",
    "df_ICMs = df.groupby(['ICM'])\n",
    "df_ICMs = df_ICMs.size().reset_index(name='Docs')\n",
    "\n",
    "print(len(df_ICMs.ICM.unique()))\n",
    "#filter out the rows with #docs less than N documents\n",
    "df_ICMOut =  df_ICMs[df_ICMs['Docs'] >= 500]\n",
    "\n",
    "#filter out rows of the original dataframe df accordding to df_ICMOut\n",
    "ICMList = df_ICMOut['ICM'].tolist()\n",
    "df = df[df.ICM.isin(ICMList)]\n",
    "\n",
    "icmCount = df_ICMs.count().tolist()[0]\n",
    "\n",
    "print( 'number of remaining documents in the dataset is: ',len(df))\n",
    "\n",
    "print('Number of unique labels is: ', len(df.ICM.unique()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Randomly reorder a dataset by rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>TI</th>\n",
       "      <th>AB</th>\n",
       "      <th>TECHF</th>\n",
       "      <th>BACKG</th>\n",
       "      <th>SUMM</th>\n",
       "      <th>CLMS</th>\n",
       "      <th>ICM</th>\n",
       "      <th>AY</th>\n",
       "      <th>IPC</th>\n",
       "      <th>REF</th>\n",
       "      <th>PA</th>\n",
       "      <th>INV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>62459</th>\n",
       "      <td>PCT2012055768-0</td>\n",
       "      <td>controlling rotary speed</td>\n",
       "      <td>controlling rotary speed pulse-width modulated...</td>\n",
       "      <td>concerns drehzahlregeÃ£ÂÃ¢Â¬ lung pulse-far-m...</td>\n",
       "      <td>speed regulation</td>\n",
       "      <td>describe improved speed regulation pulse-far-m...</td>\n",
       "      <td></td>\n",
       "      <td>G06F</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>[G06F0001-20, H05K0007-20]</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>BUSCH_Peter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87947</th>\n",
       "      <td>PCT2011077221-0</td>\n",
       "      <td>electric power supply</td>\n",
       "      <td>electric power supply common power supply stor...</td>\n",
       "      <td>buildings power supply distribute electric pow...</td>\n",
       "      <td>power supply</td>\n",
       "      <td>defined building areas laid power network buil...</td>\n",
       "      <td></td>\n",
       "      <td>H02J</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>[H02J0003-38, H02J0003-46, H02J0007-35, G06Q00...</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>TAKEHARA_Kiyotaka_NAKAKITA_Kenji</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18014</th>\n",
       "      <td>EP2016068104-0</td>\n",
       "      <td>controlling tasks performed computing</td>\n",
       "      <td>controlling tasks includes receiving ordering ...</td>\n",
       "      <td>controlling tasks performed computing</td>\n",
       "      <td>techniques controlling tasks performed computi...</td>\n",
       "      <td>defined independent directed features dependen...</td>\n",
       "      <td>performing tasks storing memory instructions p...</td>\n",
       "      <td>G06F</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>[G06F0009-44, G06F0009-48]</td>\n",
       "      <td></td>\n",
       "      <td>AB_Initio_Technology_LLC</td>\n",
       "      <td>STANFILL_Craig</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84133</th>\n",
       "      <td>PCT2002013564-0</td>\n",
       "      <td>simplifying exchange sim card subscribers mobi...</td>\n",
       "      <td>simplifying exchange sim card subscribers mobi...</td>\n",
       "      <td>simplified sim map participants mobile net con...</td>\n",
       "      <td>portable radio products portable radio service...</td>\n",
       "      <td>consists suggesting simplified sim map partici...</td>\n",
       "      <td>procedures simplified sim map participants mob...</td>\n",
       "      <td>H04Q</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>[H04Q007-38, H04W0008-26]</td>\n",
       "      <td></td>\n",
       "      <td>DETEMOBIL_DEUTSCHE_TELEKOM_MOBILNET_GMBH</td>\n",
       "      <td>REEMTSMA_Jan-Hinnerk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131689</th>\n",
       "      <td>EP2009025035-0</td>\n",
       "      <td>conversion services applications based applica...</td>\n",
       "      <td>converting page-based based configured configu...</td>\n",
       "      <td>transformation applications services network</td>\n",
       "      <td>continually devices today two-way devices mobi...</td>\n",
       "      <td></td>\n",
       "      <td>enabling computing interact generic schema def...</td>\n",
       "      <td>G06F</td>\n",
       "      <td>2005.0</td>\n",
       "      <td>[G06F0017-30, G06F0009-44]</td>\n",
       "      <td>[WO2004059957A]</td>\n",
       "      <td>RESEARCH_IN_MOTION_LIMITED</td>\n",
       "      <td>Shenfield_Michael</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     ID                                                 TI  \\\n",
       "62459   PCT2012055768-0                           controlling rotary speed   \n",
       "87947   PCT2011077221-0                              electric power supply   \n",
       "18014    EP2016068104-0              controlling tasks performed computing   \n",
       "84133   PCT2002013564-0  simplifying exchange sim card subscribers mobi...   \n",
       "131689   EP2009025035-0  conversion services applications based applica...   \n",
       "\n",
       "                                                       AB  \\\n",
       "62459   controlling rotary speed pulse-width modulated...   \n",
       "87947   electric power supply common power supply stor...   \n",
       "18014   controlling tasks includes receiving ordering ...   \n",
       "84133   simplifying exchange sim card subscribers mobi...   \n",
       "131689  converting page-based based configured configu...   \n",
       "\n",
       "                                                    TECHF  \\\n",
       "62459   concerns drehzahlregeÃ£ÂÃ¢Â¬ lung pulse-far-m...   \n",
       "87947   buildings power supply distribute electric pow...   \n",
       "18014               controlling tasks performed computing   \n",
       "84133   simplified sim map participants mobile net con...   \n",
       "131689       transformation applications services network   \n",
       "\n",
       "                                                    BACKG  \\\n",
       "62459                                    speed regulation   \n",
       "87947                                        power supply   \n",
       "18014   techniques controlling tasks performed computi...   \n",
       "84133   portable radio products portable radio service...   \n",
       "131689  continually devices today two-way devices mobi...   \n",
       "\n",
       "                                                     SUMM  \\\n",
       "62459   describe improved speed regulation pulse-far-m...   \n",
       "87947   defined building areas laid power network buil...   \n",
       "18014   defined independent directed features dependen...   \n",
       "84133   consists suggesting simplified sim map partici...   \n",
       "131689                                                      \n",
       "\n",
       "                                                     CLMS   ICM      AY  \\\n",
       "62459                                                      G06F  2011.0   \n",
       "87947                                                      H02J  2010.0   \n",
       "18014   performing tasks storing memory instructions p...  G06F  2014.0   \n",
       "84133   procedures simplified sim map participants mob...  H04Q  2001.0   \n",
       "131689  enabling computing interact generic schema def...  G06F  2005.0   \n",
       "\n",
       "                                                      IPC              REF  \\\n",
       "62459                          [G06F0001-20, H05K0007-20]                    \n",
       "87947   [H02J0003-38, H02J0003-46, H02J0007-35, G06Q00...                    \n",
       "18014                          [G06F0009-44, G06F0009-48]                    \n",
       "84133                           [H04Q007-38, H04W0008-26]                    \n",
       "131689                         [G06F0017-30, G06F0009-44]  [WO2004059957A]   \n",
       "\n",
       "                                              PA  \\\n",
       "62459                                              \n",
       "87947                                              \n",
       "18014                   AB_Initio_Technology_LLC   \n",
       "84133   DETEMOBIL_DEUTSCHE_TELEKOM_MOBILNET_GMBH   \n",
       "131689                RESEARCH_IN_MOTION_LIMITED   \n",
       "\n",
       "                                     INV  \n",
       "62459                        BUSCH_Peter  \n",
       "87947   TAKEHARA_Kiyotaka_NAKAKITA_Kenji  \n",
       "18014                     STANFILL_Craig  \n",
       "84133               REEMTSMA_Jan-Hinnerk  \n",
       "131689                 Shenfield_Michael  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "df = shuffle(df)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# split the dataset into train and test datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(322980,)\n",
      "(80746,)\n"
     ]
    }
   ],
   "source": [
    "# lets take n% data as training and remaining m% for test.\n",
    "train_size = int(len(df) * .8)\n",
    "\n",
    "train_TI = df['TI'][:train_size]\n",
    "train_AB = df['AB'][:train_size]\n",
    "train_TECHF = df['TECHF'][:train_size]\n",
    "train_BACKG = df['BACKG'][:train_size]\n",
    "train_SUMM = df['SUMM'][:train_size]\n",
    "train_CLMS = df['CLMS'][:train_size]\n",
    "train_ICM= df['ICM'][:train_size]\n",
    "train_ID= df['ID'][:train_size]\n",
    "\n",
    "test_TI = df['TI'][train_size:]\n",
    "test_AB = df['AB'][train_size:]\n",
    "test_TECHF = df['TECHF'][train_size:]\n",
    "test_BACKG = df['BACKG'][train_size:]\n",
    "test_SUMM = df['SUMM'][train_size:]\n",
    "test_CLMS = df['CLMS'][train_size:]\n",
    "test_ICM = df['ICM'][train_size:]\n",
    "test_ID = df['ID'][train_size:]\n",
    "\n",
    "\n",
    "#metadata\n",
    "train_pa_series = df['PA'][:train_size]\n",
    "test_pa_series = df['PA'][train_size:]\n",
    "\n",
    "train_inv_series = df['INV'][:train_size]\n",
    "test_inv_series = df['INV'][train_size:]\n",
    "\n",
    "\n",
    "print(train_AB.shape)\n",
    "print(test_AB.shape)\n",
    "\n",
    "#free up some memory space\n",
    "#df.iloc[0:0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Applying tokenization process \n",
    "For texts of each segment, a Keras tokenization process is used for breaking the text into individual words, and  set the sequence length of each segment according to the length of each.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.utils import to_categorical\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.preprocessing.text import one_hot\n",
    "from sklearn.preprocessing import LabelBinarizer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying Keras tokenization on Metadata of patent(Inventors, Assignees), and convert the related text into One-hot that encodes a text into a list of word indexes of size n."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 88383 words in PA\n",
      "Found 253812 words in INV\n"
     ]
    }
   ],
   "source": [
    "#PA\n",
    "pa_inv_vocab_size = 2000\n",
    "pa_tokenizer = Tokenizer(num_words=pa_inv_vocab_size,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "pa_tokenizer.fit_on_texts(train_pa_series)\n",
    "train_pa_one_hot =pa_tokenizer.texts_to_matrix(train_pa_series)\n",
    "test_pa_one_hot =pa_tokenizer.texts_to_matrix(test_pa_series)\n",
    "\n",
    "\n",
    "#INV\n",
    "inv_tokenizer = Tokenizer(num_words=pa_inv_vocab_size,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "inv_tokenizer.fit_on_texts(train_inv_series)\n",
    "train_inv_one_hot =inv_tokenizer.texts_to_matrix(train_inv_series)\n",
    "test_inv_one_hot =inv_tokenizer.texts_to_matrix(test_inv_series)\n",
    "\n",
    "\n",
    "print('Found %s words in PA' % len(pa_tokenizer.word_index))\n",
    "print('Found %s words in INV' % len(inv_tokenizer.word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "-  Transform each text in Titles (train and test datasets) into a sequence of integers. <br>\n",
    "-  set the sequence length.<br>\n",
    "-  Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9.92 s, sys: 51.7 ms, total: 9.97 s\n",
      "Wall time: 9.98 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#Title\n",
    "TI_tokenizer = Tokenizer(num_words=10000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "TI_tokenizer.fit_on_texts(train_TI)\n",
    "encoded_train_TI = TI_tokenizer.texts_to_sequences(train_TI)\n",
    "encoded_test_TI = TI_tokenizer.texts_to_sequences(test_TI)\n",
    "#convert all sequences in a list into the same length\n",
    "TI_train = pad_sequences(encoded_train_TI,  maxlen=20, padding='post')\n",
    "TI_test = pad_sequences(encoded_test_TI,  maxlen=20, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Abstrcat (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27.4 s, sys: 253 ms, total: 27.6 s\n",
      "Wall time: 27.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#Abstract\n",
    "AB_tokenizer = Tokenizer(num_words=50000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "AB_tokenizer.fit_on_texts(train_AB)\n",
    "encoded_train_AB = AB_tokenizer.texts_to_sequences(train_AB)\n",
    "encoded_test_AB = AB_tokenizer.texts_to_sequences(test_AB)\n",
    "#convert all sequences in a list into the same length\n",
    "AB_train = pad_sequences(encoded_train_AB,  maxlen=100, padding='post')\n",
    "AB_test = pad_sequences(encoded_test_AB,  maxlen=100, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Technical Field (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 17.7 s, sys: 175 ms, total: 17.9 s\n",
      "Wall time: 17.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#TECHNICAL_FIELD\n",
    "TECHF_tokenizer = Tokenizer(num_words=20000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "TECHF_tokenizer.fit_on_texts(train_TECHF)\n",
    "encoded_train_TECHF = TECHF_tokenizer.texts_to_sequences(train_TECHF)\n",
    "encoded_test_TECHF = TECHF_tokenizer.texts_to_sequences(test_TECHF)\n",
    "#convert all sequences in a list into the same length\n",
    "TECHF_train = pad_sequences(encoded_train_TECHF,  maxlen=30, padding='post')\n",
    "TECHF_test = pad_sequences(encoded_test_TECHF,  maxlen=30, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Background (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 56s, sys: 1.09 s, total: 1min 57s\n",
      "Wall time: 1min 58s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#BACKGROUND\n",
    "BACKG_tokenizer = Tokenizer(num_words=50000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "BACKG_tokenizer.fit_on_texts(train_BACKG)\n",
    "encoded_train_BACKG = BACKG_tokenizer.texts_to_sequences(train_BACKG)\n",
    "encoded_test_BACKG = BACKG_tokenizer.texts_to_sequences(test_BACKG)\n",
    "#convert all sequences in a list into the same length\n",
    "BACKG_train = pad_sequences(encoded_train_BACKG,  maxlen=100, padding='post')\n",
    "BACKG_test = pad_sequences(encoded_test_BACKG,  maxlen=100, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Summary (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 37s, sys: 1.51 s, total: 2min 38s\n",
      "Wall time: 2min 38s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#SUMMARY\n",
    "SUMM_tokenizer = Tokenizer(num_words=50000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "SUMM_tokenizer.fit_on_texts(train_SUMM)\n",
    "encoded_train_SUMM = SUMM_tokenizer.texts_to_sequences(train_SUMM)\n",
    "encoded_test_SUMM = SUMM_tokenizer.texts_to_sequences(test_SUMM)\n",
    "#convert all sequences in a list into the same length\n",
    "SUMM_train = pad_sequences(encoded_train_SUMM,  maxlen=100, padding='post')\n",
    "SUMM_test = pad_sequences(encoded_test_SUMM,  maxlen=100, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Transform each text in Independent Claim (train and test datasets) into a sequence of integers. <br>\n",
    "set the sequence length.<br>\n",
    "Pads sequences to the same length.<br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 37.8 s, sys: 697 ms, total: 38.5 s\n",
      "Wall time: 38.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#CLAIMS\n",
    "CLMS_tokenizer = Tokenizer(num_words=50000,  filters='!\"#$%&()*+,./:;<=>?@[\\]^`{|}~_', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "CLMS_tokenizer.fit_on_texts(train_CLMS)\n",
    "encoded_train_CLMS = CLMS_tokenizer.texts_to_sequences(train_CLMS)\n",
    "encoded_test_CLMS = CLMS_tokenizer.texts_to_sequences(test_CLMS)\n",
    "#convert all sequences in a list into the same length\n",
    "CLMS_train = pad_sequences(encoded_train_CLMS,  maxlen=100, padding='post')\n",
    "CLMS_test = pad_sequences(encoded_test_CLMS,  maxlen=100, padding='post')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-processing Output Labels / Classes\n",
    " we  need to make sure our labels are represented in the numeric format accepted by neural network model.  We need to convert our labels to one hot vector. <br>\n",
    "\n",
    "scikit-learn has a LabelBinarizer class which makes it easy to build these one-hot vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.54 s, sys: 66.8 ms, total: 2.6 s\n",
      "Wall time: 2.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# \n",
    "encoder = LabelBinarizer()\n",
    "encoder.fit(train_ICM)\n",
    "y_train = encoder.transform(train_ICM)\n",
    "y_test = encoder.transform(test_ICM)\n",
    "\n",
    "#get the unique number of labels in the training set\n",
    "classesList = train_ICM.tolist()\n",
    "classesList =set(classesList)\n",
    "num_classes = len(classesList)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  load the whole embeddings model into memory and get matrix\n",
    "We load a pre-trained word2vec word embedding model that was trained on five million patents (Titles and abstracts)<br>\n",
    "\n",
    "The Embeddings model is availabel <a href=https://www.kaggle.com/darshmso/w2vec-patent-domain > here.</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def load_embedding_model(filePath):\n",
    "    embeddings_index = dict()\n",
    "    f = open(filePath, encoding='utf8')\n",
    "    for line in f:\n",
    "        values = line.split()\n",
    "        word = values[0]\n",
    "        coefs = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = coefs\n",
    "        \n",
    "    return embeddings_index\n",
    "\n",
    "def create_embedding_matrix(tokenizer, embeddings_index, vocab_size_embbs, dim_size):\n",
    "    embeddings_matrix = np.zeros((vocab_size_embbs, dim_size))\n",
    "    for word, i in tokenizer.word_index.items():\n",
    "        embedding_vector = embeddings_index.get(word)\n",
    "        if embedding_vector is not None:\n",
    "            embeddings_matrix[i] = embedding_vector[0:dim_size]\n",
    "    \n",
    "    return embeddings_matrix\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Loading the whole embedding into memory and get matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 28s, sys: 2.77 s, total: 1min 31s\n",
      "Wall time: 1min 31s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "embedding_index = load_embedding_model('../models/w2v/phrase/patWordPhrase2VecModel.txt')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Creating TITLE embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 85 ms, sys: 56.9 ms, total: 142 ms\n",
      "Wall time: 140 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(TI_tokenizer.word_index) + 1\n",
    "\n",
    "TI_embeddings_matrix = create_embedding_matrix(TI_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              20)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Creating ABSTRACT embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 256 ms, sys: 126 ms, total: 382 ms\n",
      "Wall time: 382 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(AB_tokenizer.word_index) + 1\n",
    "AB_embeddings_matrix = create_embedding_matrix(AB_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              100)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating TECHNICAL_FIELD embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 160 ms, sys: 42.9 ms, total: 203 ms\n",
      "Wall time: 202 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(TECHF_tokenizer.word_index) + 1\n",
    "TECHF_embeddings_matrix = create_embedding_matrix(TECHF_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating BACKGROUND embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 639 ms, sys: 225 ms, total: 864 ms\n",
      "Wall time: 865 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(BACKG_tokenizer.word_index) + 1\n",
    "BACKG_embeddings_matrix = create_embedding_matrix(BACKG_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating SUMMARY embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 586 ms, sys: 216 ms, total: 802 ms\n",
      "Wall time: 803 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(SUMM_tokenizer.word_index) + 1\n",
    "SUMM_embeddings_matrix = create_embedding_matrix(SUMM_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating INDEPENDENT CLAIMS embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 197 ms, sys: 62.9 ms, total: 259 ms\n",
      "Wall time: 259 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#vocab_size for embedding\n",
    "vocab_size_embb = len(CLMS_tokenizer.word_index) + 1\n",
    "CLMS_embeddings_matrix = create_embedding_matrix(CLMS_tokenizer,\n",
    "                                              embedding_index,\n",
    "                                              vocab_size_embb,\n",
    "                                              100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a Deep Layer for each Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Dense, Input, Embedding, BatchNormalization, ELU, Concatenate\n",
    "from keras.layers import LSTM, Conv1D, MaxPooling1D\n",
    "from keras.layers.merge import concatenate\n",
    "from keras.layers.core import Dropout\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for Title Embeddings<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.72 s, sys: 9 s, total: 13.7 s\n",
      "Wall time: 4.59 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#TITLE \n",
    "sequence_len =20\n",
    "dropout_pct =  0.3\n",
    "\n",
    "TI_embedding_layer_input = Input(shape=(sequence_len,), name='TI_embed_input')\n",
    "TI_embedding_layer = Embedding(input_dim=len(TI_tokenizer.word_index) + 1,\n",
    "                        output_dim=20, # Dimension of the dense embedding\n",
    "                        weights=[TI_embeddings_matrix],\n",
    "                        input_length=20)(TI_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "TI_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_TI')(TI_embedding_layer)\n",
    "\n",
    "TI_deep = Dense(300, activation=None)(TI_deep)\n",
    "TI_deep = Dropout(dropout_pct)(TI_deep)\n",
    "TI_deep = BatchNormalization()(TI_deep)\n",
    "TI_deep = ELU()(TI_deep)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for Abstract Embeddings<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.3 s, sys: 6.57 s, total: 9.87 s\n",
      "Wall time: 827 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#Abstract \n",
    "sequence_len =100\n",
    "dropout_pct =  0.3\n",
    "\n",
    "AB_embedding_layer_input = Input(shape=(sequence_len,), name='AB_embed_input')\n",
    "AB_embedding_layer = Embedding(input_dim=len(AB_tokenizer.word_index) + 1,\n",
    "                        output_dim=100, # Dimension of the dense embedding\n",
    "                        weights=[AB_embeddings_matrix],\n",
    "                        input_length=100)(AB_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "AB_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_AB')(AB_embedding_layer)\n",
    "\n",
    "AB_deep = Dense(300, activation=None)(AB_deep)\n",
    "AB_deep = Dropout(dropout_pct)(AB_deep)\n",
    "AB_deep = BatchNormalization()(AB_deep)\n",
    "AB_deep = ELU()(AB_deep)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for TECHNICAL-Field Embeddings<br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.52 s, sys: 6.23 s, total: 9.75 s\n",
      "Wall time: 693 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#TECHNICAL-Field \n",
    "sequence_len =30\n",
    "dropout_pct =  0.3\n",
    "\n",
    "TECHF_embedding_layer_input = Input(shape=(sequence_len,), name='TECHF_embed_input')\n",
    "TECHF_embedding_layer = Embedding(input_dim=len(TECHF_tokenizer.word_index) + 1,\n",
    "                        output_dim=30, # Dimension of the dense embedding\n",
    "                        weights=[TECHF_embeddings_matrix],\n",
    "                        input_length=30)(TECHF_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "TECHF_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_TECHF')(TECHF_embedding_layer)\n",
    "\n",
    "TECHF_deep = Dense(300, activation=None)(TECHF_deep)\n",
    "TECHF_deep = Dropout(dropout_pct)(TECHF_deep)\n",
    "TECHF_deep = BatchNormalization()(TECHF_deep)\n",
    "TECHF_deep = ELU()(TECHF_deep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for BACKGROUND Embeddings<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.72 s, sys: 7.52 s, total: 11.2 s\n",
      "Wall time: 2.16 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#BACKGROUND \n",
    "sequence_len =100\n",
    "dropout_pct =  0.3\n",
    "\n",
    "BACKG_embedding_layer_input = Input(shape=(sequence_len,), name='BACKG_embed_input')\n",
    "BACKG_embedding_layer = Embedding(input_dim=len(BACKG_tokenizer.word_index) + 1,\n",
    "                        output_dim=100, # Dimension of the dense embedding\n",
    "                        weights=[BACKG_embeddings_matrix],\n",
    "                        input_length=100)(BACKG_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "BACKG_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_BACK')(BACKG_embedding_layer)\n",
    "\n",
    "BACKG_deep = Dense(300, activation=None)(BACKG_deep)\n",
    "BACKG_deep = Dropout(dropout_pct)(BACKG_deep)\n",
    "BACKG_deep = BatchNormalization()(BACKG_deep)\n",
    "BACKG_deep = ELU()(BACKG_deep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for Summary Embeddings<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.78 s, sys: 6.7 s, total: 10.5 s\n",
      "Wall time: 1.44 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#SUMMARY\n",
    "sequence_len =100\n",
    "dropout_pct =  0.3\n",
    "\n",
    "SUMM_embedding_layer_input = Input(shape=(sequence_len,), name='SUMM_embed_input')\n",
    "SUMM_embedding_layer = Embedding(input_dim=len(SUMM_tokenizer.word_index) + 1,\n",
    "                        output_dim=100, # Dimension of the dense embedding\n",
    "                        weights=[SUMM_embeddings_matrix],\n",
    "                        input_length=100)(SUMM_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "SUMM_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_SUMM')(SUMM_embedding_layer)\n",
    "\n",
    "SUMM_deep = Dense(300, activation=None)(SUMM_deep)\n",
    "SUMM_deep = Dropout(dropout_pct)(SUMM_deep)\n",
    "SUMM_deep = BatchNormalization()(SUMM_deep)\n",
    "SUMM_deep = ELU()(SUMM_deep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layer for Independent Claim Embeddings<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.55 s, sys: 6.46 s, total: 10 s\n",
      "Wall time: 981 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#CLAIMS \n",
    "sequence_len =100\n",
    "dropout_pct =  0.4\n",
    "\n",
    "\n",
    "CLMS_embedding_layer_input = Input(shape=(sequence_len,), name='CLMS_embed_input')\n",
    "CLMS_embedding_layer = Embedding(input_dim=len(CLMS_tokenizer.word_index) + 1,\n",
    "                        output_dim=100, # Dimension of the dense embedding\n",
    "                        weights=[CLMS_embeddings_matrix],\n",
    "                        input_length=100)(CLMS_embedding_layer_input)\n",
    "\n",
    "lstm_size = 64\n",
    "CLMS_deep = LSTM(lstm_size,\n",
    "            dropout=dropout_pct,\n",
    "            recurrent_dropout=dropout_pct,\n",
    "            return_sequences=False,\n",
    "            name='LSTM_CLMS')(CLMS_embedding_layer)\n",
    "\n",
    "CLMS_deep = Dense(300, activation=None)(CLMS_deep)\n",
    "CLMS_deep = Dropout(dropout_pct)(CLMS_deep)\n",
    "CLMS_deep = BatchNormalization()(CLMS_deep)\n",
    "CLMS_deep = ELU()(CLMS_deep)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Creating LSTM deep layers for one-hot vectors of Inventors and Assignees<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pa_input and inv_input layers are finished\n"
     ]
    }
   ],
   "source": [
    "dropout_pct =  0.3\n",
    "pa_input = Input(shape=(train_pa_one_hot.shape[1],), name='pa_input') \n",
    "pas = Dense(32,input_dim=train_pa_one_hot.shape[1], activation=None)(pa_input) \n",
    "pas = Dropout(dropout_pct)(pas)\n",
    "pas = BatchNormalization()(pas)\n",
    "pas = ELU()(pas)\n",
    "\n",
    "#inv\n",
    "inv_input = Input(shape=(train_inv_one_hot.shape[1],), name='inv_input') \n",
    "invs = Dense(32,input_dim=train_inv_one_hot.shape[1], activation=None)(pa_input) \n",
    "invs = Dropout(dropout_pct)(invs)\n",
    "invs = BatchNormalization()(invs)\n",
    "\n",
    "print('pa_input and inv_input layers are finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train Model\n",
    "The following cells specify the neural network architecture and hyperparameters. <br>\n",
    "\n",
    "The model is generally composed of:<br>\n",
    "\n",
    "- contacting sequential word embeddings ofpatent text segments into a fully-connected layer\n",
    "-  categorical_accuracy is used for calculatinges the mean accuracy rate across all predictions for multiclass classification problems <br>\n",
    "- Compile the  the Network. <br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "TI_embed_input (InputLayer)     (None, 20)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "AB_embed_input (InputLayer)     (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "TECHF_embed_input (InputLayer)  (None, 30)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "BACKG_embed_input (InputLayer)  (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "SUMM_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "CLMS_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 20, 20)       860380      TI_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 100, 100)     14372200    AB_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 30, 30)       4292250     TECHF_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_4 (Embedding)         (None, 100, 100)     71163800    BACKG_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_5 (Embedding)         (None, 100, 100)     63000400    SUMM_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "embedding_6 (Embedding)         (None, 100, 100)     14205800    CLMS_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TI (LSTM)                  (None, 64)           21760       embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_AB (LSTM)                  (None, 64)           42240       embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TECHF (LSTM)               (None, 64)           24320       embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_BACK (LSTM)                (None, 64)           42240       embedding_4[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_SUMM (LSTM)                (None, 64)           42240       embedding_5[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_CLMS (LSTM)                (None, 64)           42240       embedding_6[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 300)          19500       LSTM_TI[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 300)          19500       LSTM_AB[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 300)          19500       LSTM_TECHF[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 300)          19500       LSTM_BACK[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 300)          19500       LSTM_SUMM[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 300)          19500       LSTM_CLMS[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 300)          0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 300)          0           dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 300)          0           dense_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 300)          0           dense_4[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, 300)          0           dense_5[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_6 (Dropout)             (None, 300)          0           dense_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 300)          1200        dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 300)          1200        dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 300)          1200        dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 300)          1200        dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 300)          1200        dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, 300)          1200        dropout_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_1 (ELU)                     (None, 300)          0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_2 (ELU)                     (None, 300)          0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_3 (ELU)                     (None, 300)          0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_4 (ELU)                     (None, 300)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_5 (ELU)                     (None, 300)          0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_6 (ELU)                     (None, 300)          0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "concatenated_layer (Concatenate (None, 1800)         0           elu_1[0][0]                      \n",
      "                                                                 elu_2[0][0]                      \n",
      "                                                                 elu_3[0][0]                      \n",
      "                                                                 elu_4[0][0]                      \n",
      "                                                                 elu_5[0][0]                      \n",
      "                                                                 elu_6[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dense_9 (Dense)                 (None, 128)          230528      concatenated_layer[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_9 (Dropout)             (None, 128)          0           dense_9[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_9 (BatchNor (None, 128)          512         dropout_9[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_8 (ELU)                     (None, 128)          0           batch_normalization_9[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dense_10 (Dense)                (None, 42)           5418        elu_8[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 168,470,528\n",
      "Trainable params: 168,466,672\n",
      "Non-trainable params: 3,856\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import keras_metrics as km\n",
    "#contacting \n",
    "model_inputs_to_concat = [TI_deep, AB_deep, TECHF_deep, BACKG_deep, SUMM_deep, CLMS_deep] \n",
    "final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)\n",
    "\n",
    "output = Dense(128, activation=None)(final_layer)\n",
    "output = Dropout(dropout_pct)(output)\n",
    "output = BatchNormalization()(output)\n",
    "output = ELU()(output)\n",
    "output = Dense(num_classes, activation='softmax')(output)\n",
    "\n",
    "model = Model(inputs=[TI_embedding_layer_input,\n",
    "                      AB_embedding_layer_input,\n",
    "                      TECHF_embedding_layer_input,\n",
    "                      BACKG_embedding_layer_input,\n",
    "                     SUMM_embedding_layer_input,\n",
    "                     CLMS_embedding_layer_input,\n",
    "                     ],\n",
    "              outputs=output, name='model')\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adam',\n",
    "                      metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train / Fit the Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 322980 samples, validate on 80746 samples\n",
      "Epoch 1/20\n",
      "322980/322980 [==============================] - 461s 1ms/step - loss: 1.7969 - acc: 0.5258 - precision: 0.4037 - recall: 0.0159 - val_loss: 1.4179 - val_acc: 0.5861 - val_precision: 0.4080 - val_recall: 0.2372\n",
      "Epoch 2/20\n",
      "322980/322980 [==============================] - 450s 1ms/step - loss: 1.4392 - acc: 0.5846 - precision: 0.4834 - recall: 0.0867 - val_loss: 1.3088 - val_acc: 0.6032 - val_precision: 0.4631 - val_recall: 0.3163\n",
      "Epoch 3/20\n",
      "322980/322980 [==============================] - 442s 1ms/step - loss: 1.3511 - acc: 0.6010 - precision: 0.5133 - recall: 0.1648 - val_loss: 1.2841 - val_acc: 0.6077 - val_precision: 0.4656 - val_recall: 0.4984\n",
      "Epoch 4/20\n",
      "322980/322980 [==============================] - 436s 1ms/step - loss: 1.2965 - acc: 0.6114 - precision: 0.5304 - recall: 0.1961 - val_loss: 1.2290 - val_acc: 0.6202 - val_precision: 0.4645 - val_recall: 0.5178\n",
      "Epoch 5/20\n",
      "322980/322980 [==============================] - 437s 1ms/step - loss: 1.2574 - acc: 0.6187 - precision: 0.5558 - recall: 0.2380 - val_loss: 1.2182 - val_acc: 0.6196 - val_precision: 0.4795 - val_recall: 0.5171\n",
      "Epoch 6/20\n",
      "322980/322980 [==============================] - 435s 1ms/step - loss: 1.2228 - acc: 0.6279 - precision: 0.5753 - recall: 0.2593 - val_loss: 1.1861 - val_acc: 0.6263 - val_precision: 0.4966 - val_recall: 0.5031\n",
      "Epoch 7/20\n",
      "322980/322980 [==============================] - 434s 1ms/step - loss: 1.1940 - acc: 0.6339 - precision: 0.5902 - recall: 0.2802 - val_loss: 1.1726 - val_acc: 0.6324 - val_precision: 0.5043 - val_recall: 0.5054\n",
      "Epoch 8/20\n",
      "322980/322980 [==============================] - 434s 1ms/step - loss: 1.1690 - acc: 0.6408 - precision: 0.6120 - recall: 0.3113 - val_loss: 1.1574 - val_acc: 0.6371 - val_precision: 0.4842 - val_recall: 0.5574\n",
      "Epoch 9/20\n",
      "322980/322980 [==============================] - 435s 1ms/step - loss: 1.1481 - acc: 0.6457 - precision: 0.6196 - recall: 0.3223 - val_loss: 1.1421 - val_acc: 0.6415 - val_precision: 0.4863 - val_recall: 0.5930\n",
      "Epoch 10/20\n",
      "322980/322980 [==============================] - 437s 1ms/step - loss: 1.1270 - acc: 0.6521 - precision: 0.6216 - recall: 0.3383 - val_loss: 1.1276 - val_acc: 0.6453 - val_precision: 0.5068 - val_recall: 0.5791\n",
      "Epoch 11/20\n",
      "322980/322980 [==============================] - 432s 1ms/step - loss: 1.1120 - acc: 0.6548 - precision: 0.6170 - recall: 0.3414 - val_loss: 1.1175 - val_acc: 0.6457 - val_precision: 0.4831 - val_recall: 0.6093\n",
      "Epoch 12/20\n",
      "322980/322980 [==============================] - 435s 1ms/step - loss: 1.0955 - acc: 0.6596 - precision: 0.6327 - recall: 0.3540 - val_loss: 1.1093 - val_acc: 0.6488 - val_precision: 0.5224 - val_recall: 0.5612\n",
      "Epoch 13/20\n",
      "322980/322980 [==============================] - 434s 1ms/step - loss: 1.0773 - acc: 0.6640 - precision: 0.6332 - recall: 0.3620 - val_loss: 1.1003 - val_acc: 0.6518 - val_precision: 0.5293 - val_recall: 0.5395\n",
      "Epoch 14/20\n",
      "322980/322980 [==============================] - 433s 1ms/step - loss: 1.0620 - acc: 0.6685 - precision: 0.6425 - recall: 0.3740 - val_loss: 1.0957 - val_acc: 0.6537 - val_precision: 0.5643 - val_recall: 0.5000\n",
      "Epoch 15/20\n",
      "322980/322980 [==============================] - 436s 1ms/step - loss: 1.0482 - acc: 0.6722 - precision: 0.6532 - recall: 0.3841 - val_loss: 1.1027 - val_acc: 0.6500 - val_precision: 0.5298 - val_recall: 0.5659\n",
      "Epoch 16/20\n",
      "322980/322980 [==============================] - 435s 1ms/step - loss: 1.0337 - acc: 0.6762 - precision: 0.6497 - recall: 0.3894 - val_loss: 1.1017 - val_acc: 0.6509 - val_precision: 0.5416 - val_recall: 0.5550\n",
      "Epoch 17/20\n",
      "322980/322980 [==============================] - 435s 1ms/step - loss: 1.0209 - acc: 0.6794 - precision: 0.6581 - recall: 0.4033 - val_loss: 1.0901 - val_acc: 0.6570 - val_precision: 0.5553 - val_recall: 0.5527\n",
      "Epoch 18/20\n",
      "322980/322980 [==============================] - 437s 1ms/step - loss: 1.0099 - acc: 0.6825 - precision: 0.6607 - recall: 0.4151 - val_loss: 1.0883 - val_acc: 0.6583 - val_precision: 0.5466 - val_recall: 0.5496\n",
      "Epoch 19/20\n",
      "322980/322980 [==============================] - 434s 1ms/step - loss: 0.9975 - acc: 0.6861 - precision: 0.6691 - recall: 0.4127 - val_loss: 1.0892 - val_acc: 0.6559 - val_precision: 0.5141 - val_recall: 0.6085\n",
      "Epoch 20/20\n",
      "322980/322980 [==============================] - 437s 1ms/step - loss: 0.9839 - acc: 0.6898 - precision: 0.6718 - recall: 0.4218 - val_loss: 1.0795 - val_acc: 0.6591 - val_precision: 0.5321 - val_recall: 0.5977\n",
      "CPU times: user 4h 30min 29s, sys: 41min 49s, total: 5h 12min 18s\n",
      "Wall time: 2h 26min 11s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batch_size= 500 \n",
    "num_epochs = 20\n",
    "\n",
    "history = model.fit(x={'TI_embed_input': TI_train,\n",
    "                       'AB_embed_input': AB_train,\n",
    "             'TECHF_embed_input': TECHF_train,\n",
    "             'BACKG_embed_input': BACKG_train,\n",
    "             'SUMM_embed_input': SUMM_train,\n",
    "             'CLMS_embed_input': CLMS_train\n",
    "             \n",
    "            },\n",
    "          y=y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=num_epochs,\n",
    "          validation_data=\n",
    "          ({'TI_embed_input': TI_test,\n",
    "            'AB_embed_input': AB_test,\n",
    "            'TECHF_embed_input': TECHF_test,\n",
    "             'BACKG_embed_input': BACKG_test,\n",
    "             'SUMM_embed_input': SUMM_test,\n",
    "            'CLMS_embed_input': CLMS_test\n",
    "            },\n",
    "           y_test))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_circles\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from matplotlib import pyplot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Train the neural network on  multichannel inputs namely deep layers of patent text segments \n",
    " and deep layers of patent metadata.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "TI_embed_input (InputLayer)     (None, 20)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "AB_embed_input (InputLayer)     (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "TECHF_embed_input (InputLayer)  (None, 30)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "BACKG_embed_input (InputLayer)  (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "SUMM_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "CLMS_embed_input (InputLayer)   (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 20, 20)       860380      TI_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 100, 100)     14372200    AB_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 30, 30)       4292250     TECHF_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_4 (Embedding)         (None, 100, 100)     71163800    BACKG_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "embedding_5 (Embedding)         (None, 100, 100)     63000400    SUMM_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "embedding_6 (Embedding)         (None, 100, 100)     14205800    CLMS_embed_input[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TI (LSTM)                  (None, 64)           21760       embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_AB (LSTM)                  (None, 64)           42240       embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TECHF (LSTM)               (None, 64)           24320       embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_BACK (LSTM)                (None, 64)           42240       embedding_4[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_SUMM (LSTM)                (None, 64)           42240       embedding_5[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_CLMS (LSTM)                (None, 64)           42240       embedding_6[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "pa_input (InputLayer)           (None, 2000)         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 300)          19500       LSTM_TI[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 300)          19500       LSTM_AB[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 300)          19500       LSTM_TECHF[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_4 (Dense)                 (None, 300)          19500       LSTM_BACK[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 300)          19500       LSTM_SUMM[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 300)          19500       LSTM_CLMS[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_7 (Dense)                 (None, 32)           64032       pa_input[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 300)          0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 300)          0           dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 300)          0           dense_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_4 (Dropout)             (None, 300)          0           dense_4[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_5 (Dropout)             (None, 300)          0           dense_5[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_6 (Dropout)             (None, 300)          0           dense_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_7 (Dropout)             (None, 32)           0           dense_7[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_8 (Dense)                 (None, 32)           64032       pa_input[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 300)          1200        dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 300)          1200        dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 300)          1200        dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_4 (BatchNor (None, 300)          1200        dropout_4[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_5 (BatchNor (None, 300)          1200        dropout_5[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_6 (BatchNor (None, 300)          1200        dropout_6[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_7 (BatchNor (None, 32)           128         dropout_7[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_8 (Dropout)             (None, 32)           0           dense_8[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "elu_1 (ELU)                     (None, 300)          0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_2 (ELU)                     (None, 300)          0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_3 (ELU)                     (None, 300)          0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_4 (ELU)                     (None, 300)          0           batch_normalization_4[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_5 (ELU)                     (None, 300)          0           batch_normalization_5[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_6 (ELU)                     (None, 300)          0           batch_normalization_6[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_7 (ELU)                     (None, 32)           0           batch_normalization_7[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_8 (BatchNor (None, 32)           128         dropout_8[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "concatenated_layer (Concatenate (None, 1864)         0           elu_1[0][0]                      \n",
      "                                                                 elu_2[0][0]                      \n",
      "                                                                 elu_3[0][0]                      \n",
      "                                                                 elu_4[0][0]                      \n",
      "                                                                 elu_5[0][0]                      \n",
      "                                                                 elu_6[0][0]                      \n",
      "                                                                 elu_7[0][0]                      \n",
      "                                                                 batch_normalization_8[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dense_11 (Dense)                (None, 128)          238720      concatenated_layer[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_10 (Dropout)            (None, 128)          0           dense_11[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_10 (BatchNo (None, 128)          512         dropout_10[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "elu_9 (ELU)                     (None, 128)          0           batch_normalization_10[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_12 (Dense)                (None, 42)           5418        elu_9[0][0]                      \n",
      "==================================================================================================\n",
      "Total params: 168,607,040\n",
      "Trainable params: 168,603,056\n",
      "Non-trainable params: 3,984\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import keras_metrics as km\n",
    "\n",
    "#contacting two input models\n",
    "model_inputs_to_concat = [TI_deep, AB_deep, TECHF_deep, BACKG_deep, SUMM_deep, CLMS_deep, pas, invs] \n",
    "final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)\n",
    "\n",
    "output = Dense(128, activation=None)(final_layer)\n",
    "output = Dropout(dropout_pct)(output)\n",
    "output = BatchNormalization()(output)\n",
    "output = ELU()(output)\n",
    "output = Dense(num_classes, activation='softmax')(output)\n",
    "\n",
    "model2 =Model(inputs=[ TI_embedding_layer_input,\n",
    "                      AB_embedding_layer_input,\n",
    "                      TECHF_embedding_layer_input,\n",
    "                      BACKG_embedding_layer_input,\n",
    "                     SUMM_embedding_layer_input,\n",
    "                     CLMS_embedding_layer_input,\n",
    "                     pa_input,\n",
    "                      inv_input],\n",
    "              outputs=output, name='model')\n",
    "model2.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adam',\n",
    "                       metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the model/network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 322980 samples, validate on 80746 samples\n",
      "Epoch 1/20\n",
      "322980/322980 [==============================] - 477s 1ms/step - loss: 1.1486 - acc: 0.6703 - precision: 0.6459 - recall: 0.3769 - val_loss: 1.1171 - val_acc: 0.6656 - val_precision: 0.5228 - val_recall: 0.5860\n",
      "Epoch 2/20\n",
      "322980/322980 [==============================] - 458s 1ms/step - loss: 0.9977 - acc: 0.6915 - precision: 0.6642 - recall: 0.4410 - val_loss: 1.1003 - val_acc: 0.6631 - val_precision: 0.5672 - val_recall: 0.5465\n",
      "Epoch 3/20\n",
      "322980/322980 [==============================] - 456s 1ms/step - loss: 0.9683 - acc: 0.6973 - precision: 0.6755 - recall: 0.4547 - val_loss: 1.1008 - val_acc: 0.6602 - val_precision: 0.5137 - val_recall: 0.6256\n",
      "Epoch 4/20\n",
      "322980/322980 [==============================] - 457s 1ms/step - loss: 0.9485 - acc: 0.7034 - precision: 0.6799 - recall: 0.4739 - val_loss: 1.0926 - val_acc: 0.6635 - val_precision: 0.5520 - val_recall: 0.5597\n",
      "Epoch 5/20\n",
      "322980/322980 [==============================] - 455s 1ms/step - loss: 0.9302 - acc: 0.7064 - precision: 0.6908 - recall: 0.4783 - val_loss: 1.0930 - val_acc: 0.6619 - val_precision: 0.5792 - val_recall: 0.5527\n",
      "Epoch 6/20\n",
      "322980/322980 [==============================] - 455s 1ms/step - loss: 0.9148 - acc: 0.7116 - precision: 0.6986 - recall: 0.4866 - val_loss: 1.0966 - val_acc: 0.6638 - val_precision: 0.5312 - val_recall: 0.6132\n",
      "Epoch 7/20\n",
      "322980/322980 [==============================] - 455s 1ms/step - loss: 0.9020 - acc: 0.7142 - precision: 0.7100 - recall: 0.5018 - val_loss: 1.0997 - val_acc: 0.6658 - val_precision: 0.5691 - val_recall: 0.5519\n",
      "Epoch 8/20\n",
      "322980/322980 [==============================] - 458s 1ms/step - loss: 0.8861 - acc: 0.7190 - precision: 0.7091 - recall: 0.5085 - val_loss: 1.1060 - val_acc: 0.6609 - val_precision: 0.5159 - val_recall: 0.6411\n",
      "Epoch 9/20\n",
      "322980/322980 [==============================] - 459s 1ms/step - loss: 0.8749 - acc: 0.7219 - precision: 0.7071 - recall: 0.5159 - val_loss: 1.1131 - val_acc: 0.6623 - val_precision: 0.5739 - val_recall: 0.5659\n",
      "Epoch 10/20\n",
      "322980/322980 [==============================] - 456s 1ms/step - loss: 0.8610 - acc: 0.7259 - precision: 0.7244 - recall: 0.5311 - val_loss: 1.1124 - val_acc: 0.6625 - val_precision: 0.5277 - val_recall: 0.5984\n",
      "Epoch 11/20\n",
      "322980/322980 [==============================] - 456s 1ms/step - loss: 0.8498 - acc: 0.7294 - precision: 0.7169 - recall: 0.5297 - val_loss: 1.1149 - val_acc: 0.6629 - val_precision: 0.5151 - val_recall: 0.6233\n",
      "Epoch 12/20\n",
      "322980/322980 [==============================] - 457s 1ms/step - loss: 0.8385 - acc: 0.7326 - precision: 0.7341 - recall: 0.5464 - val_loss: 1.1318 - val_acc: 0.6592 - val_precision: 0.5396 - val_recall: 0.6016\n",
      "Epoch 13/20\n",
      "322980/322980 [==============================] - 456s 1ms/step - loss: 0.8265 - acc: 0.7363 - precision: 0.7358 - recall: 0.5442 - val_loss: 1.1238 - val_acc: 0.6640 - val_precision: 0.5254 - val_recall: 0.6256\n",
      "Epoch 14/20\n",
      "322980/322980 [==============================] - 458s 1ms/step - loss: 0.8163 - acc: 0.7398 - precision: 0.7403 - recall: 0.5518 - val_loss: 1.1302 - val_acc: 0.6630 - val_precision: 0.5292 - val_recall: 0.6109\n",
      "Epoch 15/20\n",
      "322980/322980 [==============================] - 455s 1ms/step - loss: 0.8062 - acc: 0.7424 - precision: 0.7469 - recall: 0.5697 - val_loss: 1.1476 - val_acc: 0.6583 - val_precision: 0.5322 - val_recall: 0.6217\n",
      "Epoch 16/20\n",
      "322980/322980 [==============================] - 454s 1ms/step - loss: 0.7955 - acc: 0.7451 - precision: 0.7447 - recall: 0.5630 - val_loss: 1.1380 - val_acc: 0.6619 - val_precision: 0.5096 - val_recall: 0.6395\n",
      "Epoch 17/20\n",
      "322980/322980 [==============================] - 457s 1ms/step - loss: 0.7869 - acc: 0.7479 - precision: 0.7542 - recall: 0.5822 - val_loss: 1.1472 - val_acc: 0.6610 - val_precision: 0.5431 - val_recall: 0.6008\n",
      "Epoch 18/20\n",
      "322980/322980 [==============================] - 456s 1ms/step - loss: 0.7772 - acc: 0.7499 - precision: 0.7585 - recall: 0.5835 - val_loss: 1.1474 - val_acc: 0.6605 - val_precision: 0.5310 - val_recall: 0.6101\n",
      "Epoch 19/20\n",
      "322980/322980 [==============================] - 457s 1ms/step - loss: 0.7677 - acc: 0.7528 - precision: 0.7596 - recall: 0.5895 - val_loss: 1.1585 - val_acc: 0.6595 - val_precision: 0.5471 - val_recall: 0.5992\n",
      "Epoch 20/20\n",
      "322980/322980 [==============================] - 457s 1ms/step - loss: 0.7587 - acc: 0.7559 - precision: 0.7598 - recall: 0.5956 - val_loss: 1.1719 - val_acc: 0.6592 - val_precision: 0.5000 - val_recall: 0.6612\n",
      "CPU times: user 4h 18min 26s, sys: 35min 55s, total: 4h 54min 21s\n",
      "Wall time: 2h 32min 58s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batch_size= 500 \n",
    "num_epochs = 20\n",
    "\n",
    "\n",
    "history2 = model2.fit(x={'TI_embed_input': TI_train,\n",
    "                         'AB_embed_input': AB_train,\n",
    "             'TECHF_embed_input': TECHF_train,\n",
    "             'BACKG_embed_input': BACKG_train,\n",
    "             'SUMM_embed_input': SUMM_train,\n",
    "             'CLMS_embed_input': CLMS_train,\n",
    "             'pa_input': train_pa_one_hot,\n",
    "             'inv_input': train_inv_one_hot\n",
    "            },\n",
    "          y=y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=num_epochs,\n",
    "          validation_data=\n",
    "          ({'TI_embed_input': TI_test,\n",
    "            'AB_embed_input': AB_test,\n",
    "            'TECHF_embed_input': TECHF_test,\n",
    "             'BACKG_embed_input': BACKG_test,\n",
    "             'SUMM_embed_input': SUMM_test,\n",
    "            'CLMS_embed_input': CLMS_test,\n",
    "            'pa_input': test_pa_one_hot,\n",
    "            'inv_input': test_inv_one_hot\n",
    "            },\n",
    "           y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Train the neural network on  multichannel inputs namely deep layers of Title and Technical Field.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "TI_embed_input (InputLayer)     (None, 20)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "TECHF_embed_input (InputLayer)  (None, 30)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 20, 20)       860380      TI_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 30, 30)       4292250     TECHF_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TI (LSTM)                  (None, 64)           21760       embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TECHF (LSTM)               (None, 64)           24320       embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 300)          19500       LSTM_TI[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 300)          19500       LSTM_TECHF[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 300)          0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 300)          0           dense_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 300)          1200        dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 300)          1200        dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_1 (ELU)                     (None, 300)          0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_3 (ELU)                     (None, 300)          0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "concatenated_layer (Concatenate (None, 600)          0           elu_1[0][0]                      \n",
      "                                                                 elu_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dense_13 (Dense)                (None, 64)           38464       concatenated_layer[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_11 (Dropout)            (None, 64)           0           dense_13[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_11 (BatchNo (None, 64)           256         dropout_11[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "elu_10 (ELU)                    (None, 64)           0           batch_normalization_11[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_14 (Dense)                (None, 42)           2730        elu_10[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 5,281,560\n",
      "Trainable params: 5,280,232\n",
      "Non-trainable params: 1,328\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "#contacting two input models\n",
    "model_inputs_to_concat = [TI_deep,  TECHF_deep] #invs , pas, invs\n",
    "final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)\n",
    "\n",
    "output = Dense(64, activation=None)(final_layer)\n",
    "output = Dropout(dropout_pct)(output)\n",
    "output = BatchNormalization()(output)\n",
    "output = ELU()(output)\n",
    "output = Dense(num_classes, activation='softmax')(output)\n",
    "\n",
    "model3 =Model(inputs=[ TI_embedding_layer_input,\n",
    "                      TECHF_embedding_layer_input\n",
    "                      ],\n",
    "              outputs=output, name='model')\n",
    "model3.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adam',\n",
    "                       metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])\n",
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Fit the model <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 322980 samples, validate on 80746 samples\n",
      "Epoch 1/20\n",
      "322980/322980 [==============================] - 55s 169us/step - loss: 1.6390 - acc: 0.5718 - precision: 0.5691 - recall: 0.1416 - val_loss: 1.4379 - val_acc: 0.5923 - val_precision: 0.4446 - val_recall: 0.3953\n",
      "Epoch 2/20\n",
      "322980/322980 [==============================] - 48s 148us/step - loss: 1.4377 - acc: 0.5924 - precision: 0.5717 - recall: 0.1956 - val_loss: 1.4040 - val_acc: 0.5979 - val_precision: 0.4448 - val_recall: 0.3907\n",
      "Epoch 3/20\n",
      "322980/322980 [==============================] - 49s 151us/step - loss: 1.4060 - acc: 0.5983 - precision: 0.5748 - recall: 0.2081 - val_loss: 1.3956 - val_acc: 0.5963 - val_precision: 0.4803 - val_recall: 0.3690\n",
      "Epoch 4/20\n",
      "322980/322980 [==============================] - 50s 154us/step - loss: 1.3865 - acc: 0.6016 - precision: 0.5945 - recall: 0.2233 - val_loss: 1.3819 - val_acc: 0.6000 - val_precision: 0.4606 - val_recall: 0.3946\n",
      "Epoch 5/20\n",
      "322980/322980 [==============================] - 50s 154us/step - loss: 1.3730 - acc: 0.6040 - precision: 0.6068 - recall: 0.2356 - val_loss: 1.3735 - val_acc: 0.5998 - val_precision: 0.4962 - val_recall: 0.3566\n",
      "Epoch 6/20\n",
      "322980/322980 [==============================] - 49s 153us/step - loss: 1.3603 - acc: 0.6065 - precision: 0.5837 - recall: 0.2305 - val_loss: 1.3637 - val_acc: 0.6011 - val_precision: 0.4698 - val_recall: 0.4047\n",
      "Epoch 7/20\n",
      "322980/322980 [==============================] - 49s 153us/step - loss: 1.3499 - acc: 0.6073 - precision: 0.5961 - recall: 0.2405 - val_loss: 1.3650 - val_acc: 0.6002 - val_precision: 0.4836 - val_recall: 0.3992\n",
      "Epoch 8/20\n",
      "322980/322980 [==============================] - 49s 151us/step - loss: 1.3402 - acc: 0.6093 - precision: 0.6108 - recall: 0.2466 - val_loss: 1.3604 - val_acc: 0.6014 - val_precision: 0.4942 - val_recall: 0.3969\n",
      "Epoch 9/20\n",
      "322980/322980 [==============================] - 50s 153us/step - loss: 1.3316 - acc: 0.6114 - precision: 0.5910 - recall: 0.2463 - val_loss: 1.3503 - val_acc: 0.6038 - val_precision: 0.4753 - val_recall: 0.4101\n",
      "Epoch 10/20\n",
      "322980/322980 [==============================] - 50s 154us/step - loss: 1.3236 - acc: 0.6129 - precision: 0.6087 - recall: 0.2479 - val_loss: 1.3531 - val_acc: 0.6031 - val_precision: 0.4776 - val_recall: 0.4124\n",
      "Epoch 11/20\n",
      "322980/322980 [==============================] - 49s 153us/step - loss: 1.3166 - acc: 0.6142 - precision: 0.6174 - recall: 0.2534 - val_loss: 1.3507 - val_acc: 0.6020 - val_precision: 0.4833 - val_recall: 0.4271\n",
      "Epoch 12/20\n",
      "322980/322980 [==============================] - 49s 152us/step - loss: 1.3109 - acc: 0.6155 - precision: 0.6112 - recall: 0.2573 - val_loss: 1.3436 - val_acc: 0.6034 - val_precision: 0.4709 - val_recall: 0.4333\n",
      "Epoch 13/20\n",
      "322980/322980 [==============================] - 49s 152us/step - loss: 1.3074 - acc: 0.6161 - precision: 0.6021 - recall: 0.2628 - val_loss: 1.3403 - val_acc: 0.6028 - val_precision: 0.4965 - val_recall: 0.3868\n",
      "Epoch 14/20\n",
      "322980/322980 [==============================] - 50s 154us/step - loss: 1.2992 - acc: 0.6174 - precision: 0.6207 - recall: 0.2649 - val_loss: 1.3351 - val_acc: 0.6016 - val_precision: 0.4840 - val_recall: 0.4109\n",
      "Epoch 15/20\n",
      "322980/322980 [==============================] - 50s 154us/step - loss: 1.2940 - acc: 0.6190 - precision: 0.6332 - recall: 0.2733 - val_loss: 1.3331 - val_acc: 0.6066 - val_precision: 0.4887 - val_recall: 0.4178\n",
      "Epoch 16/20\n",
      "322980/322980 [==============================] - 50s 154us/step - loss: 1.2878 - acc: 0.6203 - precision: 0.6106 - recall: 0.2745 - val_loss: 1.3304 - val_acc: 0.6046 - val_precision: 0.4750 - val_recall: 0.4558\n",
      "Epoch 17/20\n",
      "322980/322980 [==============================] - 50s 154us/step - loss: 1.2837 - acc: 0.6217 - precision: 0.6437 - recall: 0.2908 - val_loss: 1.3326 - val_acc: 0.6035 - val_precision: 0.4982 - val_recall: 0.4302\n",
      "Epoch 18/20\n",
      "322980/322980 [==============================] - 50s 154us/step - loss: 1.2779 - acc: 0.6220 - precision: 0.6195 - recall: 0.2807 - val_loss: 1.3252 - val_acc: 0.6066 - val_precision: 0.4739 - val_recall: 0.4574\n",
      "Epoch 19/20\n",
      "322980/322980 [==============================] - 50s 153us/step - loss: 1.2743 - acc: 0.6231 - precision: 0.6211 - recall: 0.2885 - val_loss: 1.3255 - val_acc: 0.6066 - val_precision: 0.4754 - val_recall: 0.4574\n",
      "Epoch 20/20\n",
      "322980/322980 [==============================] - 50s 154us/step - loss: 1.2684 - acc: 0.6242 - precision: 0.6299 - recall: 0.2955 - val_loss: 1.3252 - val_acc: 0.6068 - val_precision: 0.4866 - val_recall: 0.4225\n",
      "CPU times: user 31min 30s, sys: 4min 9s, total: 35min 40s\n",
      "Wall time: 16min 47s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batch_size= 500 \n",
    "num_epochs = 20\n",
    "\n",
    "\n",
    "history3 = model3.fit(x={'TI_embed_input': TI_train,\n",
    "             'TECHF_embed_input': TECHF_train\n",
    "            },\n",
    "          y=y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=num_epochs,\n",
    "          validation_data=\n",
    "          ({'TI_embed_input': TI_test,\n",
    "            'TECHF_embed_input': TECHF_test\n",
    "            },\n",
    "           y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Train the neural network on  multichannel inputs namely deep layers of Title, abstrcat and Technical Field.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "TI_embed_input (InputLayer)     (None, 20)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "AB_embed_input (InputLayer)     (None, 100)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "TECHF_embed_input (InputLayer)  (None, 30)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 20, 20)       860380      TI_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_2 (Embedding)         (None, 100, 100)     14372200    AB_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 30, 30)       4292250     TECHF_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TI (LSTM)                  (None, 64)           21760       embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_AB (LSTM)                  (None, 64)           42240       embedding_2[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TECHF (LSTM)               (None, 64)           24320       embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 300)          19500       LSTM_TI[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 300)          19500       LSTM_AB[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 300)          19500       LSTM_TECHF[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 300)          0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 300)          0           dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 300)          0           dense_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 300)          1200        dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_2 (BatchNor (None, 300)          1200        dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 300)          1200        dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "elu_1 (ELU)                     (None, 300)          0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_2 (ELU)                     (None, 300)          0           batch_normalization_2[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_3 (ELU)                     (None, 300)          0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "concatenated_layer (Concatenate (None, 900)          0           elu_1[0][0]                      \n",
      "                                                                 elu_2[0][0]                      \n",
      "                                                                 elu_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dense_15 (Dense)                (None, 128)          115328      concatenated_layer[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_12 (Dropout)            (None, 128)          0           dense_15[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_12 (BatchNo (None, 128)          512         dropout_12[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "elu_11 (ELU)                    (None, 128)          0           batch_normalization_12[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_16 (Dense)                (None, 42)           5418        elu_11[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 19,796,508\n",
      "Trainable params: 19,794,452\n",
      "Non-trainable params: 2,056\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import keras_metrics as km\n",
    "#contacting two input models\n",
    "model_inputs_to_concat = [TI_deep, AB_deep, TECHF_deep] #invs , pas, invs\n",
    "final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)\n",
    "\n",
    "output = Dense(128, activation=None)(final_layer)\n",
    "output = Dropout(dropout_pct)(output)\n",
    "output = BatchNormalization()(output)\n",
    "output = ELU()(output)\n",
    "output = Dense(num_classes, activation='softmax')(output)\n",
    "\n",
    "model4 = Model(inputs=[TI_embedding_layer_input,\n",
    "                      AB_embedding_layer_input,\n",
    "                      TECHF_embedding_layer_input\n",
    "                     ],\n",
    "              outputs=output, name='model')\n",
    "model4.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adam',\n",
    "                       metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])\n",
    "model4.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Fit the model <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 322980 samples, validate on 80746 samples\n",
      "Epoch 1/20\n",
      "322980/322980 [==============================] - 154s 476us/step - loss: 1.2702 - acc: 0.6409 - precision: 0.6272 - recall: 0.2992 - val_loss: 1.2700 - val_acc: 0.6334 - val_precision: 0.4969 - val_recall: 0.4310\n",
      "Epoch 2/20\n",
      "322980/322980 [==============================] - 144s 444us/step - loss: 1.1308 - acc: 0.6586 - precision: 0.6388 - recall: 0.3405 - val_loss: 1.2481 - val_acc: 0.6326 - val_precision: 0.4970 - val_recall: 0.4550\n",
      "Epoch 3/20\n",
      "322980/322980 [==============================] - 144s 446us/step - loss: 1.1071 - acc: 0.6636 - precision: 0.6475 - recall: 0.3466 - val_loss: 1.2487 - val_acc: 0.6304 - val_precision: 0.4724 - val_recall: 0.4783\n",
      "Epoch 4/20\n",
      "322980/322980 [==============================] - 144s 445us/step - loss: 1.0942 - acc: 0.6660 - precision: 0.6503 - recall: 0.3613 - val_loss: 1.2582 - val_acc: 0.6254 - val_precision: 0.5224 - val_recall: 0.4240\n",
      "Epoch 5/20\n",
      "322980/322980 [==============================] - 143s 442us/step - loss: 1.0822 - acc: 0.6679 - precision: 0.6558 - recall: 0.3578 - val_loss: 1.2403 - val_acc: 0.6320 - val_precision: 0.4879 - val_recall: 0.4845\n",
      "Epoch 6/20\n",
      "322980/322980 [==============================] - 142s 440us/step - loss: 1.0718 - acc: 0.6711 - precision: 0.6543 - recall: 0.3568 - val_loss: 1.2360 - val_acc: 0.6327 - val_precision: 0.5181 - val_recall: 0.4333\n",
      "Epoch 7/20\n",
      "322980/322980 [==============================] - 142s 440us/step - loss: 1.0597 - acc: 0.6740 - precision: 0.6602 - recall: 0.3698 - val_loss: 1.2417 - val_acc: 0.6318 - val_precision: 0.5057 - val_recall: 0.4488\n",
      "Epoch 8/20\n",
      "322980/322980 [==============================] - 142s 440us/step - loss: 1.0484 - acc: 0.6758 - precision: 0.6675 - recall: 0.3770 - val_loss: 1.2222 - val_acc: 0.6323 - val_precision: 0.4901 - val_recall: 0.4783\n",
      "Epoch 9/20\n",
      "322980/322980 [==============================] - 141s 437us/step - loss: 1.0384 - acc: 0.6791 - precision: 0.6712 - recall: 0.3859 - val_loss: 1.2195 - val_acc: 0.6329 - val_precision: 0.5317 - val_recall: 0.4225\n",
      "Epoch 10/20\n",
      "322980/322980 [==============================] - 142s 439us/step - loss: 1.0297 - acc: 0.6809 - precision: 0.6706 - recall: 0.3848 - val_loss: 1.2314 - val_acc: 0.6285 - val_precision: 0.5012 - val_recall: 0.4752\n",
      "Epoch 11/20\n",
      "322980/322980 [==============================] - 142s 438us/step - loss: 1.0216 - acc: 0.6835 - precision: 0.6728 - recall: 0.3973 - val_loss: 1.2032 - val_acc: 0.6378 - val_precision: 0.5172 - val_recall: 0.4791\n",
      "Epoch 12/20\n",
      "322980/322980 [==============================] - 142s 440us/step - loss: 1.0145 - acc: 0.6857 - precision: 0.6747 - recall: 0.3933 - val_loss: 1.2031 - val_acc: 0.6360 - val_precision: 0.5252 - val_recall: 0.4605\n",
      "Epoch 13/20\n",
      "322980/322980 [==============================] - 143s 442us/step - loss: 1.0079 - acc: 0.6867 - precision: 0.6760 - recall: 0.3926 - val_loss: 1.2067 - val_acc: 0.6378 - val_precision: 0.4909 - val_recall: 0.5000\n",
      "Epoch 14/20\n",
      "322980/322980 [==============================] - 143s 442us/step - loss: 1.0010 - acc: 0.6886 - precision: 0.6860 - recall: 0.4064 - val_loss: 1.2042 - val_acc: 0.6367 - val_precision: 0.4984 - val_recall: 0.4899\n",
      "Epoch 15/20\n",
      "322980/322980 [==============================] - 142s 440us/step - loss: 0.9927 - acc: 0.6908 - precision: 0.6840 - recall: 0.4049 - val_loss: 1.2199 - val_acc: 0.6332 - val_precision: 0.5073 - val_recall: 0.4860\n",
      "Epoch 16/20\n",
      "322980/322980 [==============================] - 143s 441us/step - loss: 0.9882 - acc: 0.6927 - precision: 0.6885 - recall: 0.4118 - val_loss: 1.2210 - val_acc: 0.6338 - val_precision: 0.4786 - val_recall: 0.5295\n",
      "Epoch 17/20\n",
      "322980/322980 [==============================] - 143s 444us/step - loss: 0.9817 - acc: 0.6935 - precision: 0.6852 - recall: 0.4162 - val_loss: 1.2064 - val_acc: 0.6358 - val_precision: 0.4996 - val_recall: 0.5008\n",
      "Epoch 18/20\n",
      "322980/322980 [==============================] - 144s 446us/step - loss: 0.9744 - acc: 0.6948 - precision: 0.6907 - recall: 0.4174 - val_loss: 1.2164 - val_acc: 0.6370 - val_precision: 0.5190 - val_recall: 0.4651\n",
      "Epoch 19/20\n",
      "322980/322980 [==============================] - 144s 445us/step - loss: 0.9711 - acc: 0.6964 - precision: 0.6815 - recall: 0.4162 - val_loss: 1.2233 - val_acc: 0.6333 - val_precision: 0.5044 - val_recall: 0.4837\n",
      "Epoch 20/20\n",
      "322980/322980 [==============================] - 143s 444us/step - loss: 0.9646 - acc: 0.6986 - precision: 0.7014 - recall: 0.4276 - val_loss: 1.2057 - val_acc: 0.6394 - val_precision: 0.5194 - val_recall: 0.4767\n",
      "CPU times: user 1h 25min 44s, sys: 10min 32s, total: 1h 36min 16s\n",
      "Wall time: 48min 7s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batch_size= 500 \n",
    "num_epochs = 20\n",
    "\n",
    "\n",
    "history4 = model4.fit(x={'TI_embed_input': TI_train,\n",
    "                         'AB_embed_input': AB_train,\n",
    "             'TECHF_embed_input': TECHF_train\n",
    "            },\n",
    "          y=y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=num_epochs,\n",
    "          validation_data=\n",
    "          ({'TI_embed_input': TI_test,\n",
    "            'AB_embed_input': AB_test,\n",
    "            'TECHF_embed_input': TECHF_test\n",
    "            },\n",
    "           y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>Train the neural network on  multichannel inputs namely deep layers of Title, Technical Field, Inventors, and Assignees.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "TI_embed_input (InputLayer)     (None, 20)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "TECHF_embed_input (InputLayer)  (None, 30)           0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 20, 20)       860380      TI_embed_input[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 30, 30)       4292250     TECHF_embed_input[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TI (LSTM)                  (None, 64)           21760       embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "LSTM_TECHF (LSTM)               (None, 64)           24320       embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "pa_input (InputLayer)           (None, 2000)         0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 300)          19500       LSTM_TI[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 300)          19500       LSTM_TECHF[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_7 (Dense)                 (None, 32)           64032       pa_input[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 300)          0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_3 (Dropout)             (None, 300)          0           dense_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dropout_7 (Dropout)             (None, 32)           0           dense_7[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_8 (Dense)                 (None, 32)           64032       pa_input[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_1 (BatchNor (None, 300)          1200        dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_3 (BatchNor (None, 300)          1200        dropout_3[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_7 (BatchNor (None, 32)           128         dropout_7[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dropout_8 (Dropout)             (None, 32)           0           dense_8[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "elu_1 (ELU)                     (None, 300)          0           batch_normalization_1[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_3 (ELU)                     (None, 300)          0           batch_normalization_3[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "elu_7 (ELU)                     (None, 32)           0           batch_normalization_7[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_8 (BatchNor (None, 32)           128         dropout_8[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "concatenated_layer (Concatenate (None, 664)          0           elu_1[0][0]                      \n",
      "                                                                 elu_3[0][0]                      \n",
      "                                                                 elu_7[0][0]                      \n",
      "                                                                 batch_normalization_8[0][0]      \n",
      "__________________________________________________________________________________________________\n",
      "dense_17 (Dense)                (None, 128)          85120       concatenated_layer[0][0]         \n",
      "__________________________________________________________________________________________________\n",
      "dropout_13 (Dropout)            (None, 128)          0           dense_17[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_13 (BatchNo (None, 128)          512         dropout_13[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "elu_12 (ELU)                    (None, 128)          0           batch_normalization_13[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dense_18 (Dense)                (None, 42)           5418        elu_12[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 5,459,480\n",
      "Trainable params: 5,457,896\n",
      "Non-trainable params: 1,584\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#contacting two input models\n",
    "model_inputs_to_concat = [TI_deep, TECHF_deep,  pas, invs] #invs , pas, invs\n",
    "final_layer =  Concatenate(name='concatenated_layer')(model_inputs_to_concat)\n",
    "\n",
    "output = Dense(128, activation=None)(final_layer)\n",
    "output = Dropout(dropout_pct)(output)\n",
    "output = BatchNormalization()(output)\n",
    "output = ELU()(output)\n",
    "output = Dense(num_classes, activation='softmax')(output)\n",
    "\n",
    "model5 =Model(inputs=[ TI_embedding_layer_input,\n",
    "                      TECHF_embedding_layer_input,\n",
    "                     pa_input,\n",
    "                      inv_input],\n",
    "              outputs=output, name='model')\n",
    "model5.compile(loss='categorical_crossentropy',\n",
    "                      optimizer='adam',\n",
    "                       metrics=['accuracy', km.categorical_precision(), km.categorical_recall()])\n",
    "model5.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br> Fit the mode. <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 322980 samples, validate on 80746 samples\n",
      "Epoch 1/20\n",
      "322980/322980 [==============================] - 64s 199us/step - loss: 1.3491 - acc: 0.6212 - precision: 0.6471 - recall: 0.3138 - val_loss: 1.3344 - val_acc: 0.6147 - val_precision: 0.5265 - val_recall: 0.4163\n",
      "Epoch 2/20\n",
      "322980/322980 [==============================] - 57s 175us/step - loss: 1.1993 - acc: 0.6403 - precision: 0.6491 - recall: 0.3595 - val_loss: 1.3275 - val_acc: 0.6149 - val_precision: 0.4900 - val_recall: 0.4760\n",
      "Epoch 3/20\n",
      "322980/322980 [==============================] - 57s 176us/step - loss: 1.1797 - acc: 0.6449 - precision: 0.6636 - recall: 0.3673 - val_loss: 1.3184 - val_acc: 0.6148 - val_precision: 0.4844 - val_recall: 0.4922\n",
      "Epoch 4/20\n",
      "322980/322980 [==============================] - 57s 176us/step - loss: 1.1670 - acc: 0.6459 - precision: 0.6683 - recall: 0.3721 - val_loss: 1.3128 - val_acc: 0.6121 - val_precision: 0.4762 - val_recall: 0.5109\n",
      "Epoch 5/20\n",
      "322980/322980 [==============================] - 57s 176us/step - loss: 1.1590 - acc: 0.6477 - precision: 0.6568 - recall: 0.3691 - val_loss: 1.3131 - val_acc: 0.6136 - val_precision: 0.4748 - val_recall: 0.5116\n",
      "Epoch 6/20\n",
      "322980/322980 [==============================] - 57s 175us/step - loss: 1.1516 - acc: 0.6499 - precision: 0.6660 - recall: 0.3760 - val_loss: 1.3159 - val_acc: 0.6087 - val_precision: 0.4960 - val_recall: 0.4767\n",
      "Epoch 7/20\n",
      "322980/322980 [==============================] - 56s 174us/step - loss: 1.1457 - acc: 0.6506 - precision: 0.6722 - recall: 0.3754 - val_loss: 1.3140 - val_acc: 0.6105 - val_precision: 0.4820 - val_recall: 0.4984\n",
      "Epoch 8/20\n",
      "322980/322980 [==============================] - 57s 176us/step - loss: 1.1411 - acc: 0.6523 - precision: 0.6750 - recall: 0.3870 - val_loss: 1.3076 - val_acc: 0.6151 - val_precision: 0.4780 - val_recall: 0.4969\n",
      "Epoch 9/20\n",
      "322980/322980 [==============================] - 56s 175us/step - loss: 1.1369 - acc: 0.6526 - precision: 0.6716 - recall: 0.3796 - val_loss: 1.3074 - val_acc: 0.6145 - val_precision: 0.4984 - val_recall: 0.4760\n",
      "Epoch 10/20\n",
      "322980/322980 [==============================] - 56s 174us/step - loss: 1.1321 - acc: 0.6535 - precision: 0.6698 - recall: 0.3865 - val_loss: 1.3029 - val_acc: 0.6154 - val_precision: 0.5307 - val_recall: 0.4550\n",
      "Epoch 11/20\n",
      "322980/322980 [==============================] - 56s 174us/step - loss: 1.1279 - acc: 0.6541 - precision: 0.6709 - recall: 0.3807 - val_loss: 1.3111 - val_acc: 0.6118 - val_precision: 0.4770 - val_recall: 0.5147\n",
      "Epoch 12/20\n",
      "322980/322980 [==============================] - 56s 174us/step - loss: 1.1241 - acc: 0.6558 - precision: 0.6803 - recall: 0.3930 - val_loss: 1.3136 - val_acc: 0.6104 - val_precision: 0.4904 - val_recall: 0.4961\n",
      "Epoch 13/20\n",
      "322980/322980 [==============================] - 56s 173us/step - loss: 1.1209 - acc: 0.6559 - precision: 0.6813 - recall: 0.3961 - val_loss: 1.3116 - val_acc: 0.6092 - val_precision: 0.5016 - val_recall: 0.4775\n",
      "Epoch 14/20\n",
      "322980/322980 [==============================] - 56s 174us/step - loss: 1.1163 - acc: 0.6576 - precision: 0.6797 - recall: 0.3897 - val_loss: 1.3161 - val_acc: 0.6078 - val_precision: 0.4904 - val_recall: 0.4961\n",
      "Epoch 15/20\n",
      "322980/322980 [==============================] - 56s 173us/step - loss: 1.1142 - acc: 0.6572 - precision: 0.6739 - recall: 0.3971 - val_loss: 1.3016 - val_acc: 0.6185 - val_precision: 0.5050 - val_recall: 0.4721\n",
      "Epoch 16/20\n",
      "322980/322980 [==============================] - 56s 175us/step - loss: 1.1108 - acc: 0.6579 - precision: 0.6771 - recall: 0.3988 - val_loss: 1.3052 - val_acc: 0.6122 - val_precision: 0.5194 - val_recall: 0.4659\n",
      "Epoch 17/20\n",
      "322980/322980 [==============================] - 56s 174us/step - loss: 1.1073 - acc: 0.6590 - precision: 0.6783 - recall: 0.3982 - val_loss: 1.3134 - val_acc: 0.6093 - val_precision: 0.5268 - val_recall: 0.4488\n",
      "Epoch 18/20\n",
      "322980/322980 [==============================] - 56s 174us/step - loss: 1.1046 - acc: 0.6594 - precision: 0.6838 - recall: 0.3971 - val_loss: 1.3110 - val_acc: 0.6111 - val_precision: 0.4739 - val_recall: 0.5217\n",
      "Epoch 19/20\n",
      "322980/322980 [==============================] - 56s 175us/step - loss: 1.1017 - acc: 0.6599 - precision: 0.6771 - recall: 0.4013 - val_loss: 1.3129 - val_acc: 0.6111 - val_precision: 0.5062 - val_recall: 0.4752\n",
      "Epoch 20/20\n",
      "322980/322980 [==============================] - 57s 175us/step - loss: 1.0984 - acc: 0.6609 - precision: 0.6905 - recall: 0.4064 - val_loss: 1.3069 - val_acc: 0.6124 - val_precision: 0.5111 - val_recall: 0.4806\n",
      "CPU times: user 34min 23s, sys: 4min 20s, total: 38min 44s\n",
      "Wall time: 19min 13s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "batch_size= 500 \n",
    "num_epochs = 20\n",
    "\n",
    "\n",
    "history5 = model5.fit(x={'TI_embed_input': TI_train,\n",
    "             'TECHF_embed_input': TECHF_train,\n",
    "             'pa_input': train_pa_one_hot,\n",
    "             'inv_input': train_inv_one_hot\n",
    "            },\n",
    "          y=y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=num_epochs,\n",
    "          validation_data=\n",
    "          ({'TI_embed_input': TI_test,\n",
    "            'TECHF_embed_input': TECHF_test,\n",
    "            'pa_input': test_pa_one_hot,\n",
    "            'inv_input': test_inv_one_hot\n",
    "            },\n",
    "           y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}