notebooks/04_One_Hidden_Layer_NN_Save_Restore.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing an one hidden layer Neural Network with save and restore\n",
    "\n",
    "\n",
    "![one_layer_network](../images/one_layer_network.png)\n",
    "\n",
    "\n",
    "The progress of the model can be saved during and after training. This means that a model can be resumed where it left off and avoid long training times. Saving also means that you can share your model and others can recreate your work.\n",
    "\n",
    "We will illustrate how to create an one hidden layer NN, save it and make predictions with trained model after reload it.\n",
    "\n",
    "We will use the iris data for this exercise.\n",
    "\n",
    "We will build a one-hidden layer neural network  to predict the fourth attribute, Petal Width from the other three (Sepal length, Sepal width, Petal length).\n",
    "\n",
    "There are several differences with respect to the example before in order to ilustrate more Tensorflow possibilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n",
      "/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n",
      "/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n",
      "/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from sklearn.datasets import load_iris\n",
    "from tensorflow.python.framework import ops\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ingest raw data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimensions of dataset\n",
      "n= 150 p= 3\n"
     ]
    }
   ],
   "source": [
    "# Before getting into pandas dataframes we will load an example dataset from sklearn library \n",
    "# type(data) #iris is a bunch instance which is inherited from dictionary\n",
    "data = load_iris() #load iris dataset\n",
    "\n",
    "# We get a pandas dataframe to better visualize the datasets\n",
    "df = pd.DataFrame(data.data, columns=data.feature_names)\n",
    "\n",
    "X_raw = np.array([x[0:3] for x in data.data])\n",
    "y_raw = np.array([x[3] for x in data.data])\n",
    "\n",
    "# Dimensions of dataset\n",
    "print(\"Dimensions of dataset\")\n",
    "n = X_raw.shape[0]\n",
    "p = X_raw.shape[1]\n",
    "print(\"n=\",n,\"p=\",p)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target_names', 'feature_names', 'target', 'DESCR'])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.keys() #keys of the dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(150, 3)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_raw.shape # Array 150x3. Each element is a 3-dimensional data point: sepal length, sepal width, petal length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(150,)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_raw.shape # Vector 150. Each element is a 1-dimensional (scalar) data point: petal width"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5.4</td>\n",
       "      <td>3.9</td>\n",
       "      <td>1.7</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.4</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.4</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>4.4</td>\n",
       "      <td>2.9</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>5.4</td>\n",
       "      <td>3.7</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>4.8</td>\n",
       "      <td>3.4</td>\n",
       "      <td>1.6</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>4.8</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>4.3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.1</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>5.8</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>5.7</td>\n",
       "      <td>4.4</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>5.4</td>\n",
       "      <td>3.9</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>5.7</td>\n",
       "      <td>3.8</td>\n",
       "      <td>1.7</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.8</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>5.4</td>\n",
       "      <td>3.4</td>\n",
       "      <td>1.7</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.7</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.3</td>\n",
       "      <td>1.7</td>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>4.8</td>\n",
       "      <td>3.4</td>\n",
       "      <td>1.9</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.6</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.4</td>\n",
       "      <td>1.6</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>5.2</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>5.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.6</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>6.9</td>\n",
       "      <td>3.2</td>\n",
       "      <td>5.7</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>5.6</td>\n",
       "      <td>2.8</td>\n",
       "      <td>4.9</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>7.7</td>\n",
       "      <td>2.8</td>\n",
       "      <td>6.7</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.7</td>\n",
       "      <td>4.9</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.3</td>\n",
       "      <td>5.7</td>\n",
       "      <td>2.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>7.2</td>\n",
       "      <td>3.2</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>6.2</td>\n",
       "      <td>2.8</td>\n",
       "      <td>4.8</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>6.1</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.9</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>6.4</td>\n",
       "      <td>2.8</td>\n",
       "      <td>5.6</td>\n",
       "      <td>2.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>7.2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.8</td>\n",
       "      <td>1.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>7.4</td>\n",
       "      <td>2.8</td>\n",
       "      <td>6.1</td>\n",
       "      <td>1.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>7.9</td>\n",
       "      <td>3.8</td>\n",
       "      <td>6.4</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>6.4</td>\n",
       "      <td>2.8</td>\n",
       "      <td>5.6</td>\n",
       "      <td>2.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.8</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>6.1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>5.6</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>7.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.1</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>6.3</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.6</td>\n",
       "      <td>2.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>6.4</td>\n",
       "      <td>3.1</td>\n",
       "      <td>5.5</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>6.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.8</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>6.9</td>\n",
       "      <td>3.1</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.1</td>\n",
       "      <td>5.6</td>\n",
       "      <td>2.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>6.9</td>\n",
       "      <td>3.1</td>\n",
       "      <td>5.1</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>5.8</td>\n",
       "      <td>2.7</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>6.8</td>\n",
       "      <td>3.2</td>\n",
       "      <td>5.9</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.3</td>\n",
       "      <td>5.7</td>\n",
       "      <td>2.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)\n",
       "0                  5.1               3.5                1.4               0.2\n",
       "1                  4.9               3.0                1.4               0.2\n",
       "2                  4.7               3.2                1.3               0.2\n",
       "3                  4.6               3.1                1.5               0.2\n",
       "4                  5.0               3.6                1.4               0.2\n",
       "5                  5.4               3.9                1.7               0.4\n",
       "6                  4.6               3.4                1.4               0.3\n",
       "7                  5.0               3.4                1.5               0.2\n",
       "8                  4.4               2.9                1.4               0.2\n",
       "9                  4.9               3.1                1.5               0.1\n",
       "10                 5.4               3.7                1.5               0.2\n",
       "11                 4.8               3.4                1.6               0.2\n",
       "12                 4.8               3.0                1.4               0.1\n",
       "13                 4.3               3.0                1.1               0.1\n",
       "14                 5.8               4.0                1.2               0.2\n",
       "15                 5.7               4.4                1.5               0.4\n",
       "16                 5.4               3.9                1.3               0.4\n",
       "17                 5.1               3.5                1.4               0.3\n",
       "18                 5.7               3.8                1.7               0.3\n",
       "19                 5.1               3.8                1.5               0.3\n",
       "20                 5.4               3.4                1.7               0.2\n",
       "21                 5.1               3.7                1.5               0.4\n",
       "22                 4.6               3.6                1.0               0.2\n",
       "23                 5.1               3.3                1.7               0.5\n",
       "24                 4.8               3.4                1.9               0.2\n",
       "25                 5.0               3.0                1.6               0.2\n",
       "26                 5.0               3.4                1.6               0.4\n",
       "27                 5.2               3.5                1.5               0.2\n",
       "28                 5.2               3.4                1.4               0.2\n",
       "29                 4.7               3.2                1.6               0.2\n",
       "..                 ...               ...                ...               ...\n",
       "120                6.9               3.2                5.7               2.3\n",
       "121                5.6               2.8                4.9               2.0\n",
       "122                7.7               2.8                6.7               2.0\n",
       "123                6.3               2.7                4.9               1.8\n",
       "124                6.7               3.3                5.7               2.1\n",
       "125                7.2               3.2                6.0               1.8\n",
       "126                6.2               2.8                4.8               1.8\n",
       "127                6.1               3.0                4.9               1.8\n",
       "128                6.4               2.8                5.6               2.1\n",
       "129                7.2               3.0                5.8               1.6\n",
       "130                7.4               2.8                6.1               1.9\n",
       "131                7.9               3.8                6.4               2.0\n",
       "132                6.4               2.8                5.6               2.2\n",
       "133                6.3               2.8                5.1               1.5\n",
       "134                6.1               2.6                5.6               1.4\n",
       "135                7.7               3.0                6.1               2.3\n",
       "136                6.3               3.4                5.6               2.4\n",
       "137                6.4               3.1                5.5               1.8\n",
       "138                6.0               3.0                4.8               1.8\n",
       "139                6.9               3.1                5.4               2.1\n",
       "140                6.7               3.1                5.6               2.4\n",
       "141                6.9               3.1                5.1               2.3\n",
       "142                5.8               2.7                5.1               1.9\n",
       "143                6.8               3.2                5.9               2.3\n",
       "144                6.7               3.3                5.7               2.5\n",
       "145                6.7               3.0                5.2               2.3\n",
       "146                6.3               2.5                5.0               1.9\n",
       "147                6.5               3.0                5.2               2.0\n",
       "148                6.2               3.4                5.4               2.3\n",
       "149                5.9               3.0                5.1               1.8\n",
       "\n",
       "[150 rows x 4 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic pre-process data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# Leave in blanck intentionally\n",
    "#"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of samples:  150 \n",
      "Samples in train set:  105 \n",
      "Samples in test set:  45\n",
      "X_train.shape =  (105, 3) y_train.shape = (105,) \n",
      "X_test.shape =   (45, 3) y_test.shape =  (45,)\n"
     ]
    }
   ],
   "source": [
    "# split into train and test sets\n",
    "\n",
    "# Total samples\n",
    "nsamples = n\n",
    "\n",
    "# Splitting into train (70%) and test (30%) sets\n",
    "split = 70 # training split% ; test (100-split)%\n",
    "jindex = nsamples*split//100 # Index for slicing the samples\n",
    "\n",
    "# Samples in train\n",
    "nsamples_train = jindex\n",
    "\n",
    "# Samples in test\n",
    "nsamples_test = nsamples - nsamples_train\n",
    "print(\"Total number of samples: \",nsamples,\"\\nSamples in train set: \", nsamples_train,\n",
    "      \"\\nSamples in test set: \",nsamples_test)\n",
    "\n",
    "# Here are train and test samples\n",
    "X_train = X_raw[:jindex, :]\n",
    "y_train = y_raw[:jindex]\n",
    "\n",
    "X_test = X_raw[jindex:, :]\n",
    "y_test = y_raw[jindex:]\n",
    "\n",
    "print(\"X_train.shape = \", X_train.shape, \"y_train.shape =\", y_train.shape, \"\\nX_test.shape =  \",\n",
    "      X_test.shape, \"y_test.shape = \", y_test.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transform features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**\n",
    "\n",
    "Becareful not to write `X_test_std = sc.fit_transform(X_test)` instead of `X_test_std = sc.transform(X_test)`. In this case, it wouldn't make a great difference since the mean and standard deviation of the test set should be (quite) similar to the training set. However, this is not always the case in Forex market data, as has been well stablished in literature. The correct way is to re-use parameters from the training set if we are doing any kind of transformation. So, the test set should basically stand for \"new, unseen\" data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scale data\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "sc = StandardScaler()\n",
    "X_train_std = sc.fit_transform(X_train)\n",
    "X_test_std = sc.transform(X_test)\n",
    "\n",
    "y_train_std = sc.fit_transform(y_train.reshape(-1, 1))\n",
    "y_test_std = sc.transform(y_test.reshape(-1, 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implement the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clears the default graph stack and resets the global default graph\n",
    "ops.reset_default_graph()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Placeholders\n",
      "Initializers\n"
     ]
    }
   ],
   "source": [
    "# make results reproducible\n",
    "seed = 2\n",
    "tf.set_random_seed(seed)\n",
    "np.random.seed(seed)  \n",
    "\n",
    "\n",
    "# Parameters\n",
    "learning_rate = 0.005\n",
    "batch_size = 50\n",
    "n_features = X_train.shape[1]#  Number of features in training data\n",
    "epochs = 1000\n",
    "display_step = 50\n",
    "model_path = \"/tmp/model.ckpt\"\n",
    "n_classes = 1\n",
    "\n",
    "# Network Parameters\n",
    "# See figure of the model\n",
    "d0 = D = n_features # Layer 0 (Input layer number of features)\n",
    "d1 = 10 # Layer 1 (1st hidden layer number of features. Selected 10 for this example)\n",
    "d2 = C = 1 # Layer 2 (Output layer)\n",
    "\n",
    "# tf Graph input\n",
    "print(\"Placeholders\")\n",
    "X = tf.placeholder(dtype=tf.float32, shape=[None, n_features], name=\"X\")\n",
    "y = tf.placeholder(dtype=tf.float32, shape=[None,n_classes], name=\"y\")\n",
    "\n",
    "\n",
    "# Initializers\n",
    "print(\"Initializers\")\n",
    "sigma = 1\n",
    "weight_initializer = tf.variance_scaling_initializer(mode=\"fan_avg\", distribution=\"uniform\", scale=sigma)\n",
    "bias_initializer = tf.zeros_initializer()\n",
    "\n",
    "# Create model\n",
    "def onelayer_perceptron(X, variables):\n",
    "    # Hidden layer with ReLU activation\n",
    "    layer_1 = tf.nn.relu(tf.add(tf.matmul(X, variables['W1']), variables['bias1']))\n",
    "    # Output layer with ReLU activation\n",
    "    out_layer = tf.nn.relu(tf.add(tf.matmul(layer_1, variables['W2']), variables['bias2']))\n",
    "    return out_layer\n",
    "\n",
    "# Store layers weight & bias\n",
    "variables = {\n",
    "    'W1': tf.Variable(weight_initializer([n_features, d1]), name=\"W1\"), # inputs -> hidden neurons\n",
    "    'bias1': tf.Variable(bias_initializer([d1]), name=\"bias1\"), # one biases for each hidden neurons\n",
    "    'W2': tf.Variable(weight_initializer([d1, d2]), name=\"W2\"), # hidden inputs -> 1 output\n",
    "    'bias2': tf.Variable(bias_initializer([d2]), name=\"bias2\") # 1 bias for the output\n",
    "}\n",
    "\n",
    "# Construct model\n",
    "y_hat = onelayer_perceptron(X, variables)\n",
    "\n",
    "# Define loss and optimizer\n",
    "loss = tf.reduce_mean(tf.square(y - y_hat)) # MSE\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss) # Train step\n",
    "\n",
    "# Initialize the variables (i.e. assign their default value)\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "# 'Saver' op to save and restore all the variables\n",
    "saver = tf.train.Saver()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model  and Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting 1st session...\n",
      "Epoch: 0050 Lost= 0.599382699\n",
      "Epoch: 0100 Lost= 0.200652853\n",
      "Epoch: 0150 Lost= 0.082070500\n",
      "Epoch: 0200 Lost= 0.046969157\n",
      "Epoch: 0250 Lost= 0.033277217\n",
      "Epoch: 0300 Lost= 0.029509921\n",
      "Epoch: 0350 Lost= 0.046582703\n",
      "Epoch: 0400 Lost= 0.051407199\n",
      "Epoch: 0450 Lost= 0.080046408\n",
      "Epoch: 0500 Lost= 0.032044422\n",
      "Epoch: 0550 Lost= 0.028484538\n",
      "Epoch: 0600 Lost= 0.030885572\n",
      "Epoch: 0650 Lost= 0.053837571\n",
      "Epoch: 0700 Lost= 0.030355027\n",
      "Epoch: 0750 Lost= 0.030203044\n",
      "Epoch: 0800 Lost= 0.021480566\n",
      "Epoch: 0850 Lost= 0.011752291\n",
      "Epoch: 0900 Lost= 0.040840883\n",
      "Epoch: 0950 Lost= 0.035907771\n",
      "Epoch: 1000 Lost= 0.042663313\n",
      "Model saved in file: /tmp/model.ckpt\n",
      "First Optimization Finished!\n"
     ]
    }
   ],
   "source": [
    "# Running first session\n",
    "print(\"Starting 1st session...\")\n",
    "with tf.Session() as sess:\n",
    "\n",
    "    # Writer to record image, scalar, histogram and graph for display in tensorboard\n",
    "    writer = tf.summary.FileWriter(\"/tmp/tensorflow_logs\", sess.graph)  # create writer\n",
    "    writer.add_graph(sess.graph)\n",
    "\n",
    "    # Run the initializer\n",
    "    sess.run(init)\n",
    "\n",
    "    # Training cycle\n",
    "    train_loss = []\n",
    "    test_loss = []\n",
    "    \n",
    "    for epoch in range(epochs):\n",
    "        rand_index = np.random.choice(len(X_train), size=batch_size)\n",
    "        X_rand = X_train[rand_index]\n",
    "        y_rand = np.transpose([y_train[rand_index]])\n",
    "        sess.run(optimizer, feed_dict={X: X_rand, y: y_rand})\n",
    "\n",
    "        train_temp_loss = sess.run(loss, feed_dict={X: X_rand, y: y_rand})\n",
    "        train_loss.append(np.sqrt(train_temp_loss))\n",
    "    \n",
    "        test_temp_loss = sess.run(loss, feed_dict={X: X_test, y: np.transpose([y_test])})\n",
    "        test_loss.append(np.sqrt(test_temp_loss))\n",
    "        if (epoch+1) % display_step == 0:\n",
    "            print(\"Epoch:\", '%04d' % (epoch+1), \"Lost=\", \\\n",
    "                \"{:.9f}\".format(train_temp_loss))\n",
    "\n",
    "    # Close writer\n",
    "    writer.flush()\n",
    "    writer.close()\n",
    "        \n",
    "    # Save model weights to disk\n",
    "    save_path = saver.save(sess, model_path)\n",
    "    print(\"Model saved in file: %s\" % save_path)\n",
    "    print(\"First Optimization Finished!\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4FOX2wPHvMfQamo2OFwuiIEYUKwKiWFAUO6hYEBuWiwVB4Oq14g8roiCICFdsoFgQLGAHCb1Jb6ETegmknN8f7+xmN9kkm2Q3G8L5PM882XnnnZkzu5s5+77TRFUxxhhj8nJUrAMwxhhzeLCEYYwxJiyWMIwxxoTFEoYxxpiwWMIwxhgTFksYxhhjwmIJwxwWRKSWiPwjIuWLcJ0PicjLRbW+kkhELhCRJbGOw0SGJQwTNhFZLSLtYrT6p4CRqnrAi2WqiKiINAusJCLjvfLW3ni8iIwQkU0iskdElorIUwH1VUT2icjegOEJb/Iw4FYRObpoNjE7ESkjIv1EZIkX53oRmSgi7WMVU2689/NfvnFV/U1VT4plTCZyLGGYYk9EygK3A6OzTFoK3BZQrwbQCtgaUOc1oBJwClAV6Agsz7KcZqpaKWB4BUBVU4CJgeuIFhEplcOkz4GrvRiqAQ2BN4Aroh1TVrnEaI4QljBMRIjIPSKyXES2i8gEETneKxcReU1EtojIbhGZLyJNvWmXi8gi75f/ehHplcPizwZ2qmpSlvIxwI0iEueN3wyMBw4F1DkL+J+q7lDVDFX9R1U/z8emTSWXnbP3i7qniKwUkW0iMlBEjgqYfqeILBaRHSIySUTqZ5n3ARFZBiwLsex2wCXA1ao6XVUPecP3qvpwQL3jReQLEdkqIqtEpGfAtAEi8qmIjPLe54UikpCPeT8XkdEishu4Q0RaishfIrJTRDaKyNsiUsar/6s361yvpXajiLQWkaSAZZ7itQ53erF0DJg2UkQGi8i3XqzTReSEPD8hU2QsYZhCE5E2wIvADcBxwBpgrDe5PXAhcCLuF/4NQLI3bThwr6pWBpoCP+ewitOAUP3gG4BF3jrA/QoflaXONOB5EekmIo3zt2UALAaa5VGnE5AAtMC1Bu4EEJGrgaeBa4FawG/Ax1nmvQaXEJuEWG47YHqIROnnJaevgblAbaAt8IiIXBpQrSPu84gHJgBv52Peq3GtnHhcgk4HHgVq4lpzbYH7AVT1Qm8eX4vtkyyxlvbWNxk4GngIGCMigV1WNwH/wbWmlgPP57TtpuhZwjCRcCswQlVnqepBoDfQSkQaAKlAZeBkQFR1sapu9OZLBZqISBWvBTArh+XHA3tymDYKuE1ETgbiVfWvLNMfwu3oHgQWea2gDlnqzPJ+8fqGwB3mHlyiy83LqrpdVdcCr+NaOgA9gBe9bU4DXgCaB7YyvOnbfcdmsqgJbPKNiEh1L75dIpLiFZ8F1FLVZ73Wx0rcsZebApbzu6p+p6rpwEdkJsBw5v1LVb/0WmcHVHWmqk5T1TRVXQ28B1yUx/vjcw6ue/Alb30/A98EvF8A41X1b+/9GgM0D3PZpghYwjCRcDyuVQGAqu7FtSJqezuFt4HBwBYRGSoiVbyq1wGXA2tE5BcRaZXD8nfgkk4o44A2uITwUdaJ3k7uBVU9E6gBfAp8JiLVA6q1UNX4gGFSwLTKwK5ctx7WBbxeg3s/AOoDb/gSEbAdENyv+VDzZpWMa7H5tmW7qsYDZwJlA9ZxfGDCw7VqjglYzqaA1/uBct7xiHDmDYpPRE4UkW/EnUSwG5cEa+ayDYGOB9apakZA2RqC34+ssVYKc9mmCFjCMJGwAbfzAUBEKuJ2zusBVPVNb4fdBNc19bhXPkNVr8Z1T3yJ25mHMs+bLxtV3Y87MH0fIRJGlrq+HVxF3MHjcJyC67LJTd2A1/Vw7we4ne29WZJReVX9MzCsXJb7E3CWiNTJpc46YFWWdVRW1cvziDncebPGNwT4B2isqlVwCUbCWBe496Vu4DEe3Pu1Psz5TYxZwjD5VVpEygUMpXD98t1EpLm4M5pewPW9rxaRs0TkbK//eh+QAmSIO130VhGpqqqpwG4gI4d1/g3Ei0jtHKY/DVzkdZEEEZFnvBjKiEg54GFgJ6GPiYRyES4h5eZxEakmInW95fv67t8FeovIqV4sVUXk+jDXi6pOBqYAX3rvYRnvfTwnoNrfwB4ReVJEyotInIg0FZGzwlhFQeatjPus9nrdgPdlmb4ZaJTDvNNxrYYnRKS0uFOfryLzeJcp5ixhmPz6DjgQMAxQ1R+BZ4AvgI3ACWT2g1fB9YvvwHU/JAMDvWldgdVe10YP3LGQbFT1EDAS6JLD9A2q+nsO8SrwAbAN9wv3EuAKr9vMx3dWj294HcBLMJcDH+b4bjhfATOBOcC3uIP5qOp44GVgrLeNC4Csx0/y0gnXzz8al+hW4d6nS711pANX4vr6V3nb+T55H3cp6Ly9gFtwx3aGkZkcfQYAH3pdXDdkWd8hXILo4K3rHeA2Vf0nr1hN8SD2ACVzOBAR31lGZ+RwgDga63wIqKuqT+RSR3HdM1mv7TCmxLGEYUwhWMIwRxLrkjLGGBMWa2EYY4wJi7UwjDHGhKVE3UysZs2a2qBBg1iHYYwxh42ZM2duU9Va4dQtUQmjQYMGJCYmxjoMY4w5bIjImrxrOdYlZYwxJiyWMIwxxoTFEoYxxpiwlKhjGMaYkiE1NZWkpCRSUlLyrmzCUq5cOerUqUPp0qULvAxLGMaYYicpKYnKlSvToEEDRMK9Ga7JiaqSnJxMUlISDRuGe6Pm7KxLyhhT7KSkpFCjRg1LFhEiItSoUaPQLTZLGMaYYsmSRWRF4v20LimA9evhvffca9+bKgKdO0PTprBqFXz0EdStC7ffDkdZnjXGHHksYQBs2AD//S9kva/WKadkJoz+/V1ZqVLQtWvRx2iMKTLJycm0bdsWgE2bNhEXF0etWu5i6L///psyZcrkuYxu3brx1FNPcdJJJ4W1zvfff58FCxbw+uuvFzzwKItawvCePjYK93xgBYaq6htZ6gjwBu4hNfuBO1R1ljftdqCvV/W/qprXQ2wK7qyzICPEw958CeTiiyEtDU44AcaPt4RhTAlXo0YN5syZA8CAAQOoVKkSvXr1CqqjqqgqR+XQ4/DBBx9EPc6iFs2+lTTg36raBPdIyQdEpEmWOh2Axt7QHfe8YESkOtAfOBtoCfQXkWpRjDW0wO6puDg491yYMaPIwzDGFA/Lly+nSZMm3HrrrZx66qls3LiR7t27k5CQwKmnnsqzzz7rr3v++eczZ84c0tLSiI+P56mnnqJZs2a0atWKLVu2hL3O0aNHc9ppp9G0aVOefvppANLS0ujatau//M033wTgtddeo0mTJpx++ul06RLyAZWFErUWhqpuxD2uE1XdIyKLgdrAooBqVwOj1N1jfZqIxIvIcUBr4AdV3Q4gIj8Al+GeHR07LVvCp5/Ctm1Qs2ZMQzHmSPHII4/4f+1HSvPmzQvc9fPPP/8watQoEhISAHjppZeoXr06aWlpXHzxxXTu3JkmTYJ/G+/atYuLLrqIl156iccee4wRI0bw1FNP5bmupKQk+vbtS2JiIlWrVqVdu3Z888031KpVi23btjF//nwAdu7cCcArr7zCmjVrKFOmjL8skork6K2INADOwD0EPlBtYF3AeJJXllN5qGV3F5FEEUncunVrpEIO7a67YM8eSxbGHMFOOOEEf7IA+Pjjj2nRogUtWrRg8eLFLFq0KNs85cuXp0MH9zj3M888k9WrV4e1runTp9OmTRtq1qxJ6dKlueWWW/j111/517/+xZIlS+jZsyeTJk2ialX3GPZTTz2VLl26MGbMmEJdoJeTqB/0FpFKwBfAI6q6O9LLV9WhwFCAhISE6D4NqnLlqC7eGJNdcTsIXLFiRf/rZcuW8cYbb/D3338THx9Ply5dQl7rEHiQPC4ujrS0tELFUKNGDebNm8fEiRMZPHgwX3zxBUOHDmXSpEn88ssvTJgwgRdeeIF58+YRFxdXqHUFimoLQ0RK45LFGFUdF6LKeqBuwHgdryyn8tgbONCdUWWMOeLt3r2bypUrU6VKFTZu3MikSZMiuvyzzz6bKVOmkJycTFpaGmPHjuWiiy5i69atqCrXX389zz77LLNmzSI9PZ2kpCTatGnDK6+8wrZt29i/f39E44nmWVICDAcWq+qgHKpNAB4UkbG4A9y7VHWjiEwCXgg40N0e6B2tWPNl2jSYPx/69s27rjGmRGvRogVNmjTh5JNPpn79+px33nmFWt7w4cP5/PPP/eOJiYk899xztG7dGlXlqquu4oorrmDWrFncddddqCoiwssvv0xaWhq33HILe/bsISMjg169elE5wr0iUXumt4icD/wGzAd856w+DdQDUNV3vaTyNu6A9n6gm6omevPf6dUHeF5V8zxHLSEhQaP+AKVnn4UBA2D3bqhUKbrrMuYItXjxYk455ZRYh1HihHpfRWSmqibkMEuQaJ4l9TuQ67Xo3tlRD+QwbQQwIgqhFU6zZu76jAUL4JxzYh2NMcYUGbvHRX41a+b+zp0b2ziMMaaIWcLIr/r14V//ArtPvzHmCGP3ksovEVi2LNZRGGNMkbMWhjHGmLBYwiiIn392d7EN82pNY4wpCSxhFESFCrBwoR34NqaESk5Opnnz5jRv3pxjjz2W2rVr+8cPHToU9nJGjBjBpk2bQk7r0qULX375ZaRCLhJ2DKMgTjvNHcuYOxeuvjrW0RhjIiyc25uHY8SIEbRo0YJjjz020iHGhLUwCqJiRXem1Lx5sY7EGFPEPvzwQ1q2bEnz5s25//77ycjICHm78U8++YQ5c+Zw4403ht0yycjI4LHHHqNp06acdtpp/qu+169fz/nnn0/z5s1p2rQpf/75Z463OI8ma2EU1OmnW5eUMUWldevsZTfcAPffD/v3w+WXZ59+xx1u2LbNPW450NSpBQpjwYIFjB8/nj///JNSpUrRvXt3xo4dywknnJDtduPx8fG89dZbvP322zRv3jys5X/22WcsXryYuXPnsnXrVs466ywuvPBCRo8ezVVXXcWTTz5Jeno6Bw4cYObMmSFvcR5NljAK6rLL3K1BMjLsGd/GHCF+/PFHZsyY4b+9+YEDB6hbty6XXnqp/3bjV1xxBe3bty/Q8n///Xduvvlm4uLiOPbYYzn//PNJTEzkrLPO4t577yUlJYVrrrmGZs2aBd3ivDDrzA9LGAV1991uMMZEX24tggoVcp9es2aBWxRZqSp33nknzz33XLZpoW43Hilt2rRh6tSpfPvtt9x222088cQT3HrrrVFdZyj207gwVN2zvo0xR4R27drx6aefsm3bNsCdTbV27dqQtxsHqFy5Mnv27Al7+RdccAFjx44lIyODzZs388cff5CQkMCaNWs49thj6d69O926dWP27Nk5rjOarIVRUIcOwTHHwL//bbc6N+YIcdppp9G/f3/atWtHRkYGpUuX5t133yUuLi7b7cYBunXrxt1330358uX5+++/gx6kBHD33Xfz4IMPAtCwYUN++eUXpk2bxumnn46IMGjQII4++mhGjBjBoEGDKF26NJUrV+ajjz5i3bp1IdcZTVG7vXksFMntzQMdd5w72DZ8eNGt05gjgN3ePDoKe3tz65IqjAYN7GpvY8wRwxJGYTRsCKtWxToKY4wpElFLGCIyQkS2iMiCHKY/LiJzvGGBiKSLSHVv2moRme9NK8I+pnxq2BDWrrUD38ZEQUnqLi8OIvF+RvOg90jc41dHhZqoqgOBgQAichXwqKpuD6hysapui2J8hde2rbsO4+BBKGXnDxgTKeXKlSM5OZkaNWrgnuRsCkNVSU5Oply5coVaTjQf0fqriDQIs/rNwMfRiiVq2rRxgzEmourUqUNSUhJbt26NdSglRrly5ahTp06hlhHzn8UiUgG4DHgwoFiBySKiwHuqGt2rUQpjzx7XwqhZM9aRGFNilC5dmoYNG8Y6DJNFcTjofRXwR5buqPNVtQXQAXhARC7MaWYR6S4iiSKSWOS/RlTh6KPhlVeKdr3GGBMDxSFh3ESW7ihVXe/93QKMB1rmNLOqDlXVBFVNqFWrVlQDzUYEateGdeuKdr3GGBMDMU0YIlIVuAj4KqCsoohU9r0G2gMhz7QqFurWtYRhjDkiRO0Yhoh8DLQGaopIEtAfKA2gqu961ToBk1V1X8CsxwDjvTMjSgH/U9XvoxVnodWpA7/+GusojDEm6qJ5ltTNYdQZiTv9NrBsJdAsOlFFQd26sGEDpKdDXFysozHGmKiJ+VlSh72OHV0rwxKGMaaEs4RRWOec4wZjjCnhisNZUoe39HRYuNB1SxljTAlmCaOwUlKgaVMYOTLWkRhjTFRZwiisihWhVi27zbkxpsSzhBEJ9epBUlKsozDGmKiyhBEJRx8NdpM0Y0wJZwkjEmrVsoRhjCnx7LTaSLj/frjhhlhHYYwxUWUJIxLOPjvWERhjTNRZl1QkbNsG33/vno1hjDEllCWMSJg2DTp0gMWLYx2JMcZEjSWMSPA9h8MOfBtjSjBLGJFgCcMYcwSwhBEJvoSxZUts4zDGmCiyhBEJlSpBuXLWwjDGlGh2Wm0kiMA330DDhrGOxBhjoiZqLQwRGSEiW0Qk5PO4RaS1iOwSkTne0C9g2mUiskRElovIU9GKMaLatoVGjWIdhTHGRE00u6RGApflUec3VW3uDc8CiEgcMBjoADQBbhaRJlGMMzKmTYOvvop1FMYYEzVRSxiq+iuwvQCztgSWq+pKVT0EjAWujmhw0TBkCPTsGesojDEmamJ90LuViMwVkYkicqpXVhtYF1AnySsLSUS6i0iiiCRujeVB5+OPh40bISMjdjEYY0wUxTJhzALqq2oz4C3gy4IsRFWHqmqCqibU8p3eGgv16kFqKmzaFLsYjDEmimKWMFR1t6ru9V5/B5QWkZrAeqBuQNU6XlnxVr+++7tmTWzjMMaYKIlZwhCRY0VEvNctvViSgRlAYxFpKCJlgJuACbGKM2z16rm/a9fGNg5jjImSqF2HISIfA62BmiKSBPQHSgOo6rtAZ+A+EUkDDgA3qaoCaSLyIDAJiANGqOrCaMUZMY0bw+zZ7q8xxpRA4vbRJUNCQoImJibGOgxjjDlsiMhMVU0Ip26sz5IqWcaPh48+inUUxhgTFZYwImnkSBg4MNZRGGNMVFjCiKS6dWHdurzrGWPMYcgSRiTVrQs7d8LevbGOxBhjIs4SRiTV9S4fsVaGMaYEsoQRSZYwjDElmD0PI5JatoRt26B69VhHYowxEWcJI5LKlnWDMcaUQNYlFWkDBsC4cbGOwhhjIs4SRqQNGwbffRfrKIwxJuIsYURatWqwY0esozDGmIizhBFpNWpALB/kZIwxUWIJI9Lq17dnYhhjSiRLGJHWoAFs3w5pabGOxBhjIsoSRqT17w+7d0MpO2PZGFOy2F4t0uLiYh2BMcZEhbUwIm3NGrjtNpgxI9aRGGNMREUtYYjICBHZIiILcph+q4jME5H5IvKniDQLmLbaK58jIofXI/TS0txDlBYW/6fKGmNMfkSzhTESuCyX6auAi1T1NOA5YGiW6ReravNwHx1YbBxzjPu7eXNs4zDGmAiL2jEMVf1VRBrkMv3PgNFpQJ1oxVKkKlWCChUsYRhjSpzicgzjLmBiwLgCk0Vkpoh0z21GEekuIokikri1uFwwd+yxsGlTrKMwxpiIivlZUiJyMS5hnB9QfL6qrheRo4EfROQfVf011PyqOhSvOyshIUGjHnA4Tj4ZjiouudgYYyIjpglDRE4H3gc6qGqyr1xV13t/t4jIeKAlEDJhFEvffhvrCIwxJuJi9jNYROoB44Cuqro0oLyiiFT2vQbaAyHPtDLGGFN0onla7cfAX8BJIpIkIneJSA8R6eFV6QfUAN7JcvrsMcDvIjIX+Bv4VlW/j1acUfHLL3DxxbB+fawjMcaYiInmWVI35zH9buDuEOUrgWbZ5ziMHDwIU6fCypVQu3asozHGmIiwI7PRUL+++2t3rTXGlCCWMKKhbl33d+3a2MZhjDERZAkjGipUcFd8r1gR60iMMSZiLGHkYf369Tz44IPs3bs3fzO2awfVq0cnKGOMiYGwDnqLyAlAkqoeFJHWwOnAKFXdGc3gioPPP/+cwYMHU758eQYOHBj+jKNHRy8oY4yJgXBbGF8A6SLyL9xV1XWB/0UtqiJ23XXX0aBBg2zlb7/9NtOmTQNgxYoVzJw5M38LPngQNm6MQITGGBN7opr33TREZJaqthCRx4EUVX1LRGar6hnRDzF8CQkJmpiY/7uhiwgAaWlpvPnmm9x3333s37+fGjVqZKubnp7OUeHe9qNrV/j9d1i1Kt8xGWNMURCRmeHeFTzcFkaqiNwM3A5845WVLkhwxdkHH3zAY489Rs+ePVm6dGnIOnv27Al/gXXquIv3MjIiFKExxsROuAmjG9AKeF5VV4lIQ+Cj6IVVdAJbWDt3ukMyw4YNo2PHjiHr33PPPeEvvG5dSE2FLVsKFaMxxhQHYSUMVV2kqj1V9WMRqQZUVtWXoxxbkVBVGjZsCLguKZ+cbpX+2WefsTHc4xJ1vEd8JCUVKkZjjCkOwkoYIjJVRKqISHVgFjBMRAZFN7SicdRRR9G1a1cAUlNTw5qnXbt24S3cEoYxpgQJt0uqqqruBq7FnU57NhDmXrP4K1euHAD9+vULq/7q1avDW/AJJ8DAgdCkSQEjM8aY4iPcmw+WEpHjgBuAPlGMJyZ8CSNclStXDq9i1arQq1cBIjLGmOIn3BbGs8AkYIWqzhCRRsCy6IVVtPKbMCpVqhR+5bVr4Z9/8hmRMcYUP2G1MFT1M+CzgPGVwHXRCqqoRa2FAdCjB6xbB/Pn5zMqY4wpXsI96F1HRMaLyBZv+EJE6kQ7uKKS34QxZ84cNmzYEF7ls86ChQvdVd/GGHMYC7dL6gNgAnC8N3ztleVKREZ4CSbkI1bFeVNElovIPBFpETDtdhFZ5g23hxlngeQ3YQDMnTs3vIqNGoGqnSlljDnshZswaqnqB6qa5g0jgVphzDcSuCyX6R2Axt7QHRgC4J2+2x84G2gJ9Peu/4iKgiSMXbt2AfB///d/PPvsszlXrFfP/bWHKRljDnPhJoxkEekiInHe0AVIzmsmVf0V2J5Llatxp+mqqk4D4r2zsS4FflDV7aq6A/iB3BNPoRQkYezevZthw4bRq1cv+vfvn3NFX8KwhykZYw5z4SaMO3Gn1G4CNgKdgTsisP7awLqA8SSvLKfybESku4gkikhiTldn56WgCaN79+55V6xbFz75BNq2LUBkxhhTfIR7a5A1qtpRVWup6tGqeg3F5CwpVR2qqgmqmlCrVji9ZNkVJGEcOnQovIplysANN2Q+ttUYYw5ThXni3mMRWP963LM1fOp4ZTmVR0VBEkY4t4X327gR/u//ICUl3+sxxpjiojAJQyKw/gnAbd7ZUucAu1R1I+4iwfYiUs072N3eK4uKgiSMjPzcsnz2bHfF92+/5Xs9xhhTXBQmYeT5E1tEPgb+Ak4SkSQRuUtEeohID6/Kd8BKYDkwDLgfQFW3A88BM7zhWa8sKkqXdo/2OO6448KeJyU/rYVWrdzfGTPyE5YxxhQruV7pLSJ7CJ0YBCif18JV9eY8pivwQA7TRgAj8lpHJMTFxQH5azUcOHAg/BVUq+bOlho1Cnr3BolE48wYY4pWri0MVa2sqlVCDJVVNdwbFxZ7tWrVonnz5gwfPjzsefbv35+/lRx/PCxZApMn5zM6Y4wpHkrMTr8wSpUqxezZs4PKjjnmGDZv3pzjPPlOGGPGwOuvw9KlcMopmddnGGPMYaIwxzBKpNNOO41nnnmGjRs30rdv3xzrffRR8BNqRST3M6caNYKuXaFnT6hfP1LhGmNMkbEWRhbz5s3zvz7xxBPzNW9aWpr/AHpIu3cHVoZS9vYbYw4f1sLIRZcuXfJVP88zpy6+GDp2hEcegXAv/DPGmGLCfuLmQvJ5NlNKSkruz8o46ij46qtCRmWMMbFhLYx8GD9+fK7Tw742IzERhg2LQETGGFN0LGGE6emnn+aaa67Jtc7BcB+S9MMP0L27exKfMcYcJixhhOn555/Ps07YLYwrrnB/f/65EBEZY0zRsoSRTz/++GOO08JOGE2bugv5XngBCnhLdmOMKWqWMPKpbdu2jB49OuS0sBPGUUfBG2+4i/iyXM9hjDHFlSWMAiiVw/UT+bohYefOcP319pwMY8xhw06rLQDfzQqzytcNCQE+/TQC0RhjTNGwFkYemjZtmq0spxbGzp0787+C9evhnXfyP58xxhQxSxh5mDlzJnv37g0q8yWMo44Kfvtuu+22/K/gP/+BBx6AhQsLHKMxxhQFSxh5KFOmDBUrVgwq83VJZS0vkHPPdX+bNYP09MIvzxhjosQSRgH4Whjly+f5DKm8XXedO802PR1+/bXwyzPGmCiJasIQkctEZImILBeRp0JMf01E5njDUhHZGTAtPWDahGjGmV++hBHqvlFjxozhk08+CX9hlSvD9OnuludbtkQqRGOMibionSUlInHAYOASIAmYISITVHWRr46qPhpQ/yHgjIBFHFDV5tGKrzB8XVLx8fH06dPHfxV4+fLl/Xe4vfHGG8NfYIUKsGIF5HD2lTHGFAfRPK22JbBcVVcCiMhY4GpgUQ71bwb6RzGeiPEd7C5dujTPPPMMW7ZsYceOHXxVmDvRxsW5bqn9+12rwxhjiplodknVBgLvrpfklWUjIvWBhkDgzZXKiUiiiEwTkRzv+ici3b16iVuL6DYbaWlpgDsgXrZsWYYOHcrJJ59MemEOWu/fDyedBD16wPDh7rYhGRkRitgYYwqvuFy4dxPwuaoG7nHrq+p6EWkE/Cwi81V1RdYZVXUoMBQgISEhl2ekRs4h7+FHZcqU8ZfFxcWRUZgdfIUK7gFL778P//sflC4N7dpBy5aFDdcYYyIimgljPRB434s6XlkoNwEPBBao6nrv70oRmYo7vpEtYcSCL2EEPo41p4v58qVfP5g/3yWLqVPtmIYxpliJZpfUDKCxiDSqFdJrAAAgAElEQVQUkTK4pJDtbCcRORmoBvwVUFZNRMp6r2sC55HzsY8i52tJVKhQwV+W0+1C8qVuXZg2DX77DX780d0GferUwi/XGGMiIGoJQ1XTgAeBScBi4FNVXSgiz4pIx4CqNwFjVTWwO+kUIFFE5gJTgJcCz66KtQ4dOvDggw8yePBgf1lEWhiB/vMf+O471021bVtkl22MMQUQ1WMYqvod8F2Wsn5ZxgeEmO9P4LRoxlYYpUuX5q233goqi3jCGDcOvv7aPZmvVi3Yvh2qVYvsOowxJh/sSu8IiUiXVKBjj4W774ZHHnHj+/dHdvnGGJNPljAiJOItDAAReO45+PBDWLkSXnsNtEhOBDPGmGwsYURIxFsYPpUqwW23uTOnHnsM6tSBXbuisy5jjMmFJYwIiUoLI9BZZ0HbtrBhA4wfH911GWNMCJYwIiRqLYzMFcAPP8Dxx8OLL8LBg9FdnzHGZGEJI0KytjCWLFkS+ZWIuGeBL18e+WUbY0weLGFESNaE0atXL7799lvi4+PZt29f5Fb03//ChAkwebI9pc8YU6QsYURI1i4pVaV3797s2rWLZcuWRW5FlSu7K8Cffx7at3cHwO0mhcaYImAJI0KytjAyMjL8SaRQNyXMybXXugPg8fHw8suuLDk58usxxhiPJYwIEZGg8YkTJ5Ls7cALddvznNx5J5x3nnv9yy8wejTUrAkzZ0Z+XcYYgyWMiAl1Wu26de5xIFFJGDVrwu+/w86dcPnl0LWrK//Pf6yLyhgTFZYwIqRDhw78/vvvNG7cONs03+3Qo6JqVXcxX48e8NBDsGYNROMMLWPMEc8SRoTExcVx3nnnkZqamm3agQMHorvya6+FIUNg4EC44AJo0AC+/daVr1kT3XUbY44YxeWJeyVGqNZESkpK0ay8bFl4+2244Qb47DNXdtJJ0L+/u7VIqIsLVd31HcYYkwdrYURYw4YNs5UVWcLwOf5497dOHTj5ZChfHkqVcomhTh3wXRfyzjtw443u9YEDcP318MorRRurMeawYQkjwsaPH8/zzz8fVBb1LqmsXnrJPUtjzRqoUSN42kMPueeHjxoFDzwAM2bA3LnumRuffw47drh6a9a4OunpkJICd90F339ftNthjClWREvQ7bITEhI0MTEx1mGwadMmjjvuOP/4XXfdxbBhw7KdeltkUlNh7VqoXRvKlYOffoJ27dy0jz+GsWPhq69cC+Pjj2H3bmjSBDZtgrfecknGZ88edwddY0yJICIzVTUhnLpRbWGIyGUiskRElovIUyGm3yEiW0VkjjfcHTDtdhFZ5g23RzPOSKtevXrQ+PDhw7M9oa9IlS4NJ5zgkgXAl1+6vxMnutbDV1/B00/Dp5+6mxpWr+6SxX33QdOmmct5/333N9SPjIwM12IZOdK9njDBLTuU9PSif67Hb7+5pxYaYwpOVaMyAHHACqARUAaYCzTJUucO4O0Q81YHVnp/q3mvq+W1zjPPPFOLCyBouPXWW2MdUqaUFNXvvlPNyHDD6tWq6elu2ptvqoLq889n1n/nHdWxY93rDRvc9FKlVNu3V33xRbe8K65w5Y8/rnrPPe51kyau/oknqrZurTpzplsPqD79dPS2b+dOF/9//6u6f7/qxo1unRddFL11GnOYAhI13P16uBXzOwCtgEkB472B3lnq5JQwbgbeCxh/D7g5r3UWp4TxzDPPBCWMzp07a2pqqj755JO6efPmWIeXs4MHVb/6yiWSUM4+231tatd2f1u0UK1Tx70Glzx6984cf/31zNeg+sILma87d3brGTJE9Z9/MteRlOSmjR6tmpzsxnOzb59bzqZNqqmpqmeckbmO555zy/eNz5vn5klLc0nw77/dNhtzhCouCaMz8H7AeNesycFLGBuBecDnQF2vvBfQN6DeM0CvHNbTHUgEEuvVqxeVN7Qgxo0bF5QwateurQ899JACeuONN8Y6vILbtk114kT3et481YULVZ98UvXYY1VXrHDlW7a4FkSLFu71jBluOqi+/77qVVdl7sA//9z9vfBCN+/Bg6qlS7uyr79W7dJFtWLF4BgyMlQHDlS9+27VVatUy5Z19cuWVb32Wvf61VfdvK++6pLOO++oxsWpNmzo1vHYY8GJbNcut+x9+1Q/+MDF7bN/f+brtDSXlGLp229VlyyJbQymxDicEkYNoKz3+l7gZ81nwggcilMLY9euXXrmmWdm65oCtGPHjrEOL/IOHQq/bkqKSwY//aQ6YUJwa2ThQve6WTO3Y774Yjfev79LFHv3qnbsmDnPzJmZrzt0UP3Pf1Rffjn0eqdPV/3+e9U9e1QbN1Y99VQ3X7durlVUqVLmskaMUH3kkczxv/92y7jhBtVHHw29/B9/VN2+3b1euNB1ze3d68bT01X79XPJKCND9eGHVefPD/89U3XvcevWLp6PPnLLee891x04ZUre8/u6HY90r76q2qOH++6YYpMw8uySylI/DtjlvT7su6R8QiWMTp06xTqs4iMjw3UNgWr16qpDh2YmAlXVZctUy5d3ZU88oXrKKZk78dtuczvRESNUO3VSXb8+vHUuXuySSkqK6oEDLjEFtjYaNHDHXsqVyyx76inXYvF1x+3bp/rzz67FkZHhjt2A6uWXu3WccIIb//e/3TqeftqNv/xycOvm7LNVZ81yO/NPP81MTI895lpdH32kumiR6htvZM5To4bq3LmZydU3rF6d+Z7eeqvqvfe68enTM+u88457T59/XvX000Mnv6lTVVeuVP34Y5f05s4N/T7u3q06eXJwa0zVxdyhg+rWrarLl7vtzyo93SW5rD80ApPaxImqLVuqjhuX68eZp40bVRcsUG3eXPWhh1THj898Py67zH2GgbZtyyxLSQluYRZEWpr7DAtr8GDVTz4p/HKyKC4Jo5R3sLphwEHvU7PUOS7gdSdgmve6OrDKO+BdzXtdPa91Hi4JA9Bff/011qEVL8nJbgd0zjmqN90UvONYssR9VYcPV61bV7Vv35yPsRRURoZreWQ1ZYrqddepXnKJi6FCBdeKePjh4J31gAGZr2+4IfP15MmutQQuqaWnu51XhQqZdRYuzGztZG01+QZfa2fgwMzY0tNVN292LSRQrVXL7dwC59u3T/X++zPHa9VyySSwTr9+bnl//KH67ruurG1b1dNOc6+POcYlBd+0J5909QOX8eKLrvXmG69UySXmu+922z9iROZnGniM65Zb3HufkqI6e7ZqlSqum/LXX4OXv2qVS6K7dgV/NzIyVCdNytypHzzoEoSq+75s2pT9vdyzR/W11zK7ST/91NV/++3MOgMHZrZEq1RxifOff1zy7NlT9fbb3QkiVaqo/vBDzt+rfftcNyi4HwY5OXDAJfPERPeZbt6s+sUX7gfSRRe5lqovtvvuU123LnPeN98sVFIrFgnDxcHlwFLc2VJ9vLJngY7e6xeBhV4ymQKcHDDvncByb+gWzvoOp4Rx2WWXabNmzbRWrVq60fcFNznbvDn/XTiRdN557t/Fd8LC9u3BLZBPPnFdUr6dILgut+++c69PPz37r+lNm9zOLSMjczlly7pjPjfeqDptmksUH3/s6uR28P/++90Of9y4zGW9/rrbEZ97rpueluZ2gvPmqY4a5X7933yzaz3t3+9+zfvmTU52O3vfCQO+5OEbNm9WLVMmc3zKFNWqVd3rMmUyk6/vGBW4Hb6qSy6By/r2W/f+BCbf++5zZ9c1buxOqtixI3ieUaPcTt03XrGi29aHHgqu98orqtWqZX4uo0Zlvmdpaa4lOHZscLLzJXHf8bCcBt9706eP+3x8x/DWrXPv54QJLqGAaqtWbtrw4a4F1ru3S1q7d7vPwLfMDz90CTk+PrPMd8bixImZZU2auHXu2OGOzQ0dWuCvdrFJGEU9FMeEUa9evRyThm+44447Yh2mycvUqW4nEmjDBnfW15o1mS2eP/90v5R93TDbt7tfo1m7bbL67jvX5fX114WPddo017LJj4wM1QcfVH9rJlC9eqonn+zKb7hBddgwV3/JErfD8nWFff65SwZZzwKcPz9zR+ezf79LoGeckXlKNmSebp2U5E6PPnjQJQLVzETs21l/+qk7m85XNmyYateumeNnnJH3GXArV7rE8eabLlnPn5/5WX7/vSsfP969/usvt9zzz3dn8O3f7xLOlVdmTvMNzz2X+YPi0kvd8lJTXWLz1SlVyrWcAo+dLV7sfiA0bqx6wQWZXZQ+kye7FuWgQcGtsEIcj7GEUYysXbtWP/nkE924cWOOCaNbt26xDtMYt+OcODF7n/4336iuXVu4Zb//vks4WZfts3ev66rLy4cfqj7wQPByFi925b4dfXq6O04TDdu3Z57IoJrZ3fftt8Fn//3wg+rSpa4rzpfwVFVHjnRJ7vrr3QkLqamu3g8/BNcLx5QpruUb2GoqgPwkDLs1SBFRVV588UX69OmTbVq3bt0YMWJEDKIyxhTKvHnQrBn8+CO0bQvjx7vyTp1iG1c+5OfWIHZ78yIiIjz99NO0adOGVq1aZZtmjDkMnX66u1eb74mbh1GiKAi7W20RO+ecczjrrLOCyixhGHMYC/F45pLKEkYMhLo5oTHGFHeWMGLgkUceyVbWvXt3VqxYEYNojDEmPJYwYuCyyy5DVYkLeGTqsGHDGDJkSAyjMsaY3FnCKEaOOso+DmNM8WV7KGOMMWGxhBFD5513XtD4uHHjuPnmm8np2hhVRUR48cUXiyI8Y4wJYhfuxdCuXbuIj4/PVr5w4UKaNGmSrTw1NZUyZcoA5JhUjDEmP4rNM71N7qpWrcr//d//ZStfunRpyPqHDh2KdkgmAlSVKVOmWFI3JY4ljBjr2rVrtrIdO3aErJuampqtbO3atQwePDjicZmCGzJkCG3atGHcuHGxDsWYiLKEEWNVqlTJVrZ9+3YmT57M77//zurVq+nSpQsHDx4M2cJo3LgxDz74IMnJyUURrgnDsmXLAJfMjSlJLGHEWNmyZbOVrV27lksvvZQLLriAO+64gzFjxvDXX39lSxiq6i87cOCAv3zBggWICLNmzcp3PI0aNeKFF17I93zRoqps2LAhpjF06tSJLl26hF3fuqJMSWUJoxiYMWMGf/zxh3/8nXfe8b/+5ZdfAJdE6tat6y8XkaDrNlJSUgB3IP20004D4OGHH2batGlhx6GqrFq1KuQddSOpe/funHzyyUFls2fPpm/fvtl2tq+88gq1a9dmxYoVTJ48mdWrV2db3pIlSxg1alTU4v3yyy8ZM2YMGzduzPe8aWlpvPHGG3b8yZQM4d4HvSADcBmwBPfUvKdCTH8MWATMA34C6gdMSwfmeMOEcNZXHJ+HEa6MjAzt0qVLng9bymmY7z2N7q+//so2TVX166+/1qSkJJ0wYYIezOGhMrt27QqaR1U1NTVVp0+fHtFtzbqO119/3V+2bdu2oLoXXXSRAnrLLbcooOXLl8+2vNKlSwctL7/69u2rM2bMyDPeq6++OqzlPfLIIwrooEGDdPDgwQroSy+9FLJuRkaGAtrP95jUKAP0uuuuK5J1HQmmTZum8+bNi3UYhUJxeIASEId7NGsjMp/p3SRLnYuBCt7r+4BPAqbtze86D+eE4fP8888XKGHMmDFDt2zZEnLazTffHDT+SMAT1TZt2qQzvad1rVq1KtvO/KmnnlJAZ8+eHRRncnKyLsz6BLocLFq0SBs0aKCbNm1S1ewJo1SpUv6ydYHPKlbV1q1bh0yAgQLnffbZZzUjzOd97969W+fMmaOAlipVKsd6vuW3adMmrOU+/PDD/oTx7LPPKqB9+vQJWffAgQMKaFxcXFjLLqyc3sNoGz16tA4fPrzI1xttsXo/Iyk/CSOaXVItgeWqulJVDwFjgasDK6jqFFXd741OA+pEMZ7DQrVq1Qo03759+zj66KNDTvv444+Dxl9//XWWL18OuIPmZ555JuAOtmc1c+ZMABITE2nRogXTp08HICEhgVNPPZXevXuzePFihg4dytixYxGRbMu5/vrrWb16NV9//XXI+NLS0vyvP/roo3A2N6ROnTrRr18/5s2bF1R+9dVXh7yF/LXXXkvz5s39MSxatIgxY8bkuPz9+/fnOC2UPXv2MGXKFABK5XAL7IMHDwLuh1tGRka+ln846dKlC3fddVehlrFo0SJ2796drXz9+vVBXbqxEPgdLtHCzSz5HYDOwPsB412Bt3Op/zbQN2A8DUjEJZJrcpmvu1cvsV69epFMvDGRmppaoBbGRx99lO95fN0heL+QfvjhBwVURHT9+vVBdc844wwF9Nxzz1VVDZpWrlw5BbROnToK6F9//RW0Tb56I0eODBpXVV2xYkW2uNLT0/3zXnzxxWG3MOrXr6+Azpo1K+T0rHxxh7t8QNfm8qjSO+64Qzt27OhvYQQOzz33XMh5Nm3a5K9TpUoVPXDggH4died6q+q//vUvbd68ua5Zs0b37NkTtC35MX/+fF26dKmqqh46dEgPHTqkqu67evvtt/u7Q3NTkPWGWkarVq2ylVevXj1mv/J92/XCCy/kWXfr1q16wQUX6IoVK3TXrl1ht4SjjWLSJRV2wgC6eImhbEBZbe9vI2A1cEJe6ywJXVKqGnInFjh89tlnBUoquQ3p6en6ySef+McDXwcO5cuX1759++a6rD///FP379+v06dP199//91f/uGHHwYlxJy2dcuWLf73ok2bNtmmX3vttXro0CHdu3evjhkzxl9+zDHHhJ0wApNl1qFXr16ampoa8vNo2LCh9uzZU1etWuWfPnv2bB0+fLi/TqiE8eKLLwatf8OGDbphwwZds2ZNUL1u3bopuC7GvXv3BsWRk9mzZ2t8fLxOnjzZXzZ9+vSg5bZq1SrbDwRV1fT0dL3ssst04sSJumXLFu3SpYvu2rVL09LSdNiwYXro0KGgeWrXrq21atVSVZdIAD3llFNUVTUxMVH//vtv3bFjh6ampvoTS+B7uGfPHj106JAuXbpUV65cmW1bUlNTtVOnTvr333+rquqZZ56pxx9/fFBXbVa+8rSA53zfd999euWVV+b53uVkzZo1euKJJ+rq1atVVXX16tW6c+fOoDojR470r/v+++/Xn376SR9//HFdvny5Tps2LVtC8H1H2rVrp4Cec845mpCQoOvXry9wnJFQXBJGK2BSwHhvoHeIeu2AxcDRuSxrJNA5r3WWlITx/vvv68CBA3X9+vU6c+bMbDufSZMmRTxhjB8/Xjt37uwfv+mmmwq8rD/++EObNGmSZ70dO3aELO/Ro4f/vQiVMACdM2eO3nvvvUFlVatWVcB/TMYncPquXbt05cqV+s477+Qa25gxY/TLL7/UTz/9NOT0888/X1VVf/3112zTQiWMgQMH+uN54okn/OX//PNPyOX7DpzfcMMNevDgQe3Vq5du3bo1aLsOHjyot912W9B8ycnJ2r59e23UqFG2ZfqOlwTudH0tyTJlymjXrl0V0BEjRuhrr72mgA4ZMsQ/zzXXXBM0/6JFixTQxo0b65QpU7KtT0R0+/btQesF/Ccy1KhRQ3v06KFfffWVPx7f+xEfHx/yBA5AV65cqV988UW2zzc5OTlb2YUXXqj9+vXTefPm6ZIlS4J+jAR6/PHH9dVXX/WP9+/fXwHt27evf3knnniiqqq+9tpr/h8nuQ1PPvlk0Dp836UqVaoE1Rs1apQ++eST/vd1wIABQbEEGjJkiI4YMcI/vmnTJp00aVLIuuGimCSMUsBKoCGZB71PzVLnDNyB8cZZyqvhtTaAmsAyshwwDzWUlIQRaO3atdm+iH/++WeeX9a2bdvmWSdaQ6idR36G+Ph4//bnlDB++umnbGVly5b1vz+BChKDiORZ5+WXXw57eb6EEdjiCnf43//+53+tqjp16lR94IEH9PPPP89W96STTspxOTt37vS/btCggS5YsCDkmXmBLdgBAwaEXJaqa9mAa3UV9jvz008/qapq3bp1w57Hd7afb3zFihWqqtlabYHDNddc4/9ebNy4UX/77begZai61uczzzwTtJ7A6fnZrkDDhg1TcMk5sM7jjz/uf/3qq68GzZuWlqaPP/64JiUlZYtTVfXEE09UCG5d5RfFIWG4OLgcWIpLCn28smeBjt7rH4HNZDl9FjgXmI9LMvOBu8JZX0lMGJs3b1bI/PUM6Lx58/yv9+3b53/96KOP+l9feumlhf4nLujw9ddfF2r+Ro0a+bc/p4TRq1evHOcfOXKk/vXXXzp+/HhVLVjCiPTw3//+NyKx7N+/3//6xRdfzNe8gS2bwg6rVq0K64dLfoYbbrghX/WXLVsW9J526NBBH3300Tx/LPn4WmfHHXecf9qUKVOynam4evVq/+tzzz03XzEOGjTI3zIZOHBgyDrHHntsjnH++OOPCmjHjh01OTnZP+2PP/7Q7du3+8fr169f4H0MxSVhFPVQEhOG70tx/vnn6/3336/gjjcEfqk++eQT/fTTT1VV9aijjlJAr7jiihy/xLn9yq1UqVJEdwKxHlSLR8IA9J577in0MnzdQLEe+vfvr++++25MY5g8ebLeeeed+Z5v+vTp+ssvv+R40kPWwXdqdGGH+Pj4fNXfvXu3fvHFFwruGqCsp8cHDoU5LRtLGCXLhAkTdNu2bZqRkeE/COr7omTlO1PpyiuvVMjepXDbbbcFXW+RdWjVqlVMdwKRHu6+++6oLPf2229XcAm2Z8+eRbY9eZ1wcLgN1apVK/C8sU5Y0R4uvPDCXLsYAwdLGJYwcgWhLzZbvny5Dho0SK+//noF1/+dnp6uU6dO1enTp+v+/fuDujWyDqGazT///LMC2rx5c507d66/vE+fPhH5x8h6Rk9OQ/ny5bOVherHz2nwJdFwh3//+985TluxYoW++uqrOnv2bP8B0uIynHzyyTFZ708//eQ/juQbLrnkklznGTBggP9ge05DTl1phUk2JW046qijCrMvsYRR0i1YsEA3bNiQ43TfQbtff/015PRZs2ZpxYoVg750/fv31/T0dG3ZsqW/bObMmTpjxgwF16Wiqv4Dc4G/rH2/uH1f3qxf6IYNG2qzZs1Cftl98vqnKFOmjM6cOVOvuuoqf9mXX34Z1j9UvXr1gk4RzTr4zly58MILFVyXXmpqqq5atcrfdXH00Uf762/fvt0f96BBg/zlLVq0yDF2QN9+++1sZzaBu04D0AoVKhR65zF//nxdt26dTp8+XU8//XStVatWtjo9evQIOe+yZcuylb3xxhthrVc1+PYygHbv3t3/esCAAf7vnO9A+b333qu7du3Sr7/+WmvXrh0073333af79+8P67txuA2NGzeO6PJEpMD7EixhmNTUVP+ZJzlp3769gvu1vGTJEn954MVyu3fvVlXVzz77zP/P+9577ymgd955p06cOFG//fZbVXW32ti+fbvu3LlTH3vsMR01apS++uqrGvi57Nu3T0ePHu1f/p133umfVqFCBe3YsaN+8MEHIf+5OnXqpKqqCxcuVHDdTRMmTAjrH8p3OmXTpk1DthxSU1N18+bNmpGRoenp6UHn0Pv6nsePH6/HH398tov3Dh48qA899JDOmjVLL7/88mzLbt++vTZv3lzBHT9avHhxtjpTp05VQM8+++yg8gULFugDDzwQcpteeeUV7dOnjzZo0CCoPOsPiYyMjGyJYNiwYf7pgQlMNfvO2Wfp0qU6ffp0/fHHHzUxMVHXrFkTdFzGJ9R2jR07VlXVf0+wQYMG6SWXXBL0vfOdzNGoUSOdOHFi0Db88ccfumrVKm3atGmBdqh9+vTR77//Plu5r0WUkJAQVD5kyJCQp0cDmpiYWKCE8MQTT/i/ew888IDu3Lkz6Iw033UtgI4bNy7f21hQljBMWHbu3KmJiYnZypOSkvxfwlA3Kty+fbtecMEFunz58gKtNyMjQ3fs2JHj9PT0dH9fve/K3unTpwddCDZ//nzNyMjQ5cuXB/3TjB07VtetW+cf79evX8h7GPmm//jjj/rll1/mGq8vYQSe55+TrVu3+ruELr30Ur3nnnt07dq1/us+tmzZEvIYku9U5IcfftjfBRi4E5g7d65+8803/vLWrVsHrfett97SW265Jds9vwK1bNlSL7zwQr3yyiuDtmX37t0hE0ajRo20YcOGeW5zs2bNgq45+P777/1Xhqtq0MWHvoPU7777bshlvfTSS7luw9q1a/Wtt97SAQMGaOXKlfX666/XlJSUkNfV+Fq61atX98/vOybQp08frVOnjn733XeamJio+/btC2oZL1u2TPv166eQ/TRYVXfDzGnTpunMmTM1OTk523Gs3r17+xPPpEmT/O+Hr/vSd28xXyK/+OKL/Z/RCSec4P9cAr/fV155pXbq1El79uypP/zwgzZp0sTfnXveeefl+TnlxBKGKbQFCxboyy+/HNMYduzYoQcOHMiznm+H5/un813R/OCDD+Y4z/fff6/t27cP60pqX8II7IbKy+bNm4OSbUZGhj/h+Y4hDR06VMeMGaNz587VjIwMHTt2rH+ePn36ZNupBibyrNeaFNaaNWt048aNqup+6c+bN093796tKSkpEV3PzJkzVUT8V1BHSkpKilauXFnvv/9+ffLJJ/Wkk07S0aNH68qVK4PugLx169agVk0g3wWFTz/9tKq67rXXXntNDx06pNu3b9e33norx+sdDh06FHQsJi0tTdesWZOtnu+aiyFDhqiq+14MHjzYXzdr61ZVgxJVKBs2bPDf+qUgLGGYI05ycnLQzi3UP15Bff/993rRRRcV6uKoSEhNTdUWLVroN998E9M4SirfSRF5tThzc+jQoVxbz9u2bdOePXuG9UPIp7BdTmEsP+yEEfoWmsYcZqpXrx40HvhwqcK69NJLufTSSyO2vIIqVaqU/+7BJvIqVqwIuDs/F1Tp0qWJj4/PcXqNGjV447V0RtwAAAeYSURBVI03Crz8WLOEYYwxQM2aNYHid6vyzz77jBo1asQ6DMAShjHGAPD8889Tvnx5brzxxliHEqRz586xDsHPEoYxxgBVq1Zl4MCBsQ6jWIvmE/eMMcaUIJYwjDHGhMUShjHGmLBYwjDGGBMWSxjGGGPCYgnDGGNMWCxhGGOMCYslDGOMMWERd++pkkFEtgJrCjh7TWBbBMM5HNg2Hxlsm0u+wmxvfVWtFU7FEpUwCkNEElU1IdZxFCXb5iODbXPJV1Tba11SxhhjwmIJwxhjTFgsYWQaGusAYsC2+chg21zyFcn22jEMY4wxYbEWhjHGmLBYwjDGGBOWIz5hiMhlIrJERJaLyFOxjidSRKSuiEwRkUUislBEHvbKq4vIDyKyzPtbzSsXEXnTex/miUiL2G5BwYlInIjMFpFvvPGGIjLd27ZPRKSMV17WG1/uTW8Qy7gLSkTiReRzEflHRBaLSKuS/jmLyKPe93qBiHwsIuVK2ucsIiNEZIuILAgoy/fnKiK3e/WXicjthYnpiE4YIhIHDAY6AE2Am0WkSWyjipg04N+q2gQ4B3jA27angJ9UtTHwkzcO7j1o7A3dgSFFH3LEPAwsDhh/GXhNVf8F7ADu8srvAnZ45a959Q5HbwDfq+rJQDPctpfYz1lEagM9gQRVbQrEATdR8j7nkcBlWcry9bmKSHWgP3A20BLo70syBaKqR+wAtAImBYz3BnrHOq4obetXwCXAEuA4r+w4YIn3+j3g5oD6/nqH0wDU8f6R2gDfAIK7ArZU1s8cmAS08l6X8upJrLchn9tbFViVNe6S/DkDtYF1QHXvc/sGuLQkfs5AA2BBQT9X4GbgvYDyoHr5HY7oFgaZXzyfJK+sRPGa4GcA04FjVHWjN2kTcIz3uqS8F68DTwAZ3ngNYKeqpnnjgdvl32Zv+i6v/uGkIbAV+MDrhntfRCpSgj9nVV0PvAqsBTbiPreZlOzP2Se/n2tEP+8jPWGUeCJSCfgCeERVdwdOU/eTo8ScVy0iVwJbVHVmrGMpQqWAFsAQVT0D2EdmNwVQIj/nasDVuGR5PFCR7F03JV4sPtcjPWGsB+oGjNfxykoEESmNSxZjVHWcV7xZRI7zph8HbPHKS8J7cR7QUURWA2Nx3VJvAPEiUsqrE7hd/m32plcFkosy4AhIApJUdbo3/jkugZTkz7kdsEpVt6pqKjAO99mX5M/ZJ7+fa0Q/7yM9YcwAGntnV5TBHTibEOOYIkJEBBgOLFbVQQGTJgC+MyVuxx3b8JXf5p1tcQ6wK6Dpe1hQ1d6qWkdVG+A+y59V9VZgCtDZq5Z1m33vRWev/mH1S1xVNwHrROQkr6gtsIgS/DnjuqLOEZEK3vfct80l9nMOkN/PdRLQXkSqeS2z9l5ZwcT6oE6sB+ByYCmwAugT63giuF3n45qr84A53nA5ru/2J2AZ8CNQ3asvuDPGVgDzcWegxHw7CrH9rYFvvNeNgL+B5cBnQFmvvJw3vtyb3ijWcRdwW5sDid5n/SVQraR/zsB/gH+ABcBHQNmS9jkDH+OO0aTiWpJ3FeRzBe70tn050K0wMdmtQYwxxoTlSO+SMsYYEyZLGMYYY8JiCcMYY0xYLGEYY4wJiyUMY4wxYbGEYY5oInKMiPxPRFaKyEwR+UtEOsUoltYicm7AeA8RuS0WsRgTSqm8qxhTMnkXfX0JfKiqt3hl9YGOUVxnKc2831FWrYG9wJ8AqvputOIwpiDsOgxzxBKRtkA/Vb0oxLQ44CXcTrwsMFhV3xOR1sAA3B1Pm+JuetdFVVVEzgQGAZW86Xeo6kYRmYq7cPJ83MVYS4G+QBncLSpuBcoD04B03M0EH8JdwbxXVV8VkebAu0AF3MVZd6rqDm/Z04GLgXjgLlX9LXLvkjGZrEvKHMlOBWblMO0u3O0VzgLOAu4RkYbetDOAR3DPUGkEnOfdt+stoLOqngmMAJ7///bumDWKIAzj+P/BGCQQzkI7CysbIViImMLPYMBaxDKIbVLGLxADEcFWK4NWWokgBKwExeYKm6BiZxFFUnggT4qZxTHkwkq46p5fs3tzc7tzxd17+87eO83xZm1ftr0OvAWuuhQLfAqs2P5MCQgbti8d8qX/BFi1vUD5J+9a89yM7St1TGtETEhSUhGVpIeUq4AR8AVYkNTVJhpQFqcZAe9sf6uv+UhZs+AH5Yrjdcl0cYJS1qGz1eyfA7Zq8bhZynoWR41rAJy2vV2bHlNKXXS6wpLv61giJiIBI6bZELjRPbB9R9IZSl2mr8Bd2/8Uaqspqd9N0x/K50jA0PbimHPtNfsPgPu2XzQpruPoxtONJWIikpKKafYGOCVpuWmbq9tXwHJNNSHpQl2YaJxPwFlJi7X/SUkXx/Qd8LfEdLvG8i9g/mBn2z+BXUnXatNNYPtgv4hJy6+RmFp1onoJ2JC0Qpls3gNWKSmf88CHejfVd2DpiGONavpqs6aQZiir/w0P6X4PeCZplxK0urmRl8BzSdcpk96tW8AjSXPADnD7/99xxPHkLqmIiOglKamIiOglASMiInpJwIiIiF4SMCIiopcEjIiI6CUBIyIieknAiIiIXvYBYmTlssntGhMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "# Plot loss (MSE) over time\n",
    "plt.plot(train_loss, 'k-', label='Train Loss')\n",
    "plt.plot(test_loss, 'r--', label='Test Loss')\n",
    "plt.title('Loss (MSE) per Generation')\n",
    "plt.legend(loc='upper right')\n",
    "plt.xlabel('Generation')\n",
    "plt.ylabel('Loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensorboard Graph\n",
    "\n",
    "\n",
    "What follows is the graph we have executed and all data about it. Note the \"save\" label.\n",
    "\n",
    "\n",
    "![graph_3](../images/graph_3.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving a Tensorflow model\n",
    "\n",
    "So, now we have our model saved.\n",
    "\n",
    "Tensorflow model has four main files:\n",
    "* a) Meta graph:\n",
    "This is a protocol buffer which saves the complete Tensorflow graph; i.e. all variables, operations, collections etc. This file has .meta extension.\n",
    "\n",
    "\n",
    "* b) y c) Checkpoint files:\n",
    "It is a binary file which contains all the values of the weights, biases, gradients and all the other variables saved. Tensorflow has changed from version 0.11. Instead of a single .ckpt file, we have now two files: .index and .data file that contains our training variables. \n",
    "\n",
    "\n",
    "* d) Along with this, Tensorflow also has a file named checkpoint which simply keeps a record of latest checkpoint files saved.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrain the model\n",
    "\n",
    "\n",
    "We can retrain the model as many times as we want to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting 2nd session...\n",
      "INFO:tensorflow:Restoring parameters from /tmp/model.ckpt\n",
      "Model restored from file: /tmp/model.ckpt\n",
      "Epoch: 0050 Lost= 0.045188859\n",
      "Epoch: 0100 Lost= 0.035137746\n",
      "Epoch: 0150 Lost= 0.040114976\n",
      "Epoch: 0200 Lost= 0.040839382\n",
      "Epoch: 0250 Lost= 0.029388864\n",
      "Epoch: 0300 Lost= 0.050860386\n",
      "Epoch: 0350 Lost= 0.023227667\n",
      "Epoch: 0400 Lost= 0.034531657\n",
      "Epoch: 0450 Lost= 0.036823772\n",
      "Epoch: 0500 Lost= 0.020957258\n",
      "Epoch: 0550 Lost= 0.023199901\n",
      "Epoch: 0600 Lost= 0.029416963\n",
      "Epoch: 0650 Lost= 0.028286777\n",
      "Epoch: 0700 Lost= 0.029708408\n",
      "Epoch: 0750 Lost= 0.038849130\n",
      "Epoch: 0800 Lost= 0.021901334\n",
      "Epoch: 0850 Lost= 0.019867409\n",
      "Epoch: 0900 Lost= 0.038035385\n",
      "Epoch: 0950 Lost= 0.046836123\n",
      "Epoch: 1000 Lost= 0.024480129\n",
      "Epoch: 1050 Lost= 0.025052661\n",
      "Epoch: 1100 Lost= 0.028433315\n",
      "Epoch: 1150 Lost= 0.022785973\n",
      "Epoch: 1200 Lost= 0.018632039\n",
      "Epoch: 1250 Lost= 0.024766553\n",
      "Epoch: 1300 Lost= 0.027888060\n",
      "Epoch: 1350 Lost= 0.030560365\n",
      "Epoch: 1400 Lost= 0.041359652\n",
      "Epoch: 1450 Lost= 0.015819877\n",
      "Epoch: 1500 Lost= 0.029382044\n",
      "Epoch: 1550 Lost= 0.034098670\n",
      "Epoch: 1600 Lost= 0.025412932\n",
      "Epoch: 1650 Lost= 0.036478702\n",
      "Epoch: 1700 Lost= 0.030148495\n",
      "Epoch: 1750 Lost= 0.016189585\n",
      "Epoch: 1800 Lost= 0.023110745\n",
      "Epoch: 1850 Lost= 0.029191718\n",
      "Epoch: 1900 Lost= 0.018225947\n",
      "Epoch: 1950 Lost= 0.023598077\n",
      "Epoch: 2000 Lost= 0.015231807\n",
      "Model saved in file: /tmp/model.ckpt\n",
      "Second Optimization Finished!\n"
     ]
    }
   ],
   "source": [
    "# Running a new session\n",
    "print(\"Starting 2nd session...\")\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables\n",
    "    sess.run(init)\n",
    "\n",
    "    # Restore model weights from previously saved model\n",
    "    saver.restore(sess, model_path)\n",
    "    print(\"Model restored from file: %s\" % model_path)\n",
    "\n",
    "    # Resume training\n",
    "    for epoch in range(epochs*2):\n",
    "        rand_index = np.random.choice(len(X_train), size=batch_size)\n",
    "        X_rand = X_train[rand_index]\n",
    "        y_rand = np.transpose([y_train[rand_index]])\n",
    "        sess.run(optimizer, feed_dict={X: X_rand, y: y_rand})\n",
    "\n",
    "        train_temp_loss = sess.run(loss, feed_dict={X: X_rand, y: y_rand})\n",
    "        train_loss.append(np.sqrt(train_temp_loss))\n",
    "    \n",
    "        test_temp_loss = sess.run(loss, feed_dict={X: X_test, y: np.transpose([y_test])})\n",
    "        test_loss.append(np.sqrt(test_temp_loss))\n",
    "        if (epoch+1) % display_step == 0:\n",
    "            print(\"Epoch:\", '%04d' % (epoch+1), \"Lost=\", \\\n",
    "                \"{:.9f}\".format(train_temp_loss))\n",
    "\n",
    "    # Close writer\n",
    "    writer.flush()\n",
    "    writer.close()\n",
    "    \n",
    "    # Save model weights to disk\n",
    "    save_path = saver.save(sess, model_path)\n",
    "    print(\"Model saved in file: %s\" % save_path)\n",
    "    print(\"Second Optimization Finished!\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can use the model to make some predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting prediction session...\n",
      "INFO:tensorflow:Restoring parameters from /tmp/model.ckpt\n",
      "Model restored from file: /tmp/model.ckpt\n",
      "[[0.19734718]\n",
      " [0.28260154]\n",
      " [1.7156498 ]]\n"
     ]
    }
   ],
   "source": [
    "# Running a new session for predictions\n",
    "print(\"Starting prediction session...\")\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables\n",
    "    sess.run(init)\n",
    "\n",
    "    # Restore model weights from previously saved model\n",
    "    saver.restore(sess, model_path)\n",
    "    print(\"Model restored from file: %s\" % model_path)\n",
    "\n",
    "    # We try to predict the petal width (cm) of three samples\n",
    "    #Caution!!! This data are not the right data (see below why)\n",
    "    feed_dict = {X: [[5.1, 3.5, 1.4],\n",
    "                     [4.8, 3.0, 1.4],\n",
    "                     [6.3, 3.4, 5.6]]\n",
    "                }\n",
    "    prediction = sess.run(y_hat, feed_dict)\n",
    "    print(prediction) # True value 0.2, 0.1, 2.4\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, not very good results. But it is worst that we could think!\n",
    "Data are not right because we have trained our model with a transformed data (standardization) and now we must\n",
    "use again transformed data to make predictions. Also wi will get back transformed data again. So, we must inverse the transformation to get the original kind of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First: transform our original data. The data we want to make the prediction about."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_pred = [[5.1, 3.5, 1.4],\n",
    "          [4.8, 3.0, 1.4],\n",
    "          [6.3, 3.4, 5.6]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[6.86549436, 4.28228483, 0.89182234],\n",
       "       [6.38114257, 3.47503186, 0.89182234],\n",
       "       [8.8029015 , 4.12083424, 7.67274733]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_pred_std = sc.transform(X_pred)\n",
    "X_pred_std"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second: we are ready to make the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting prediction session...\n",
      "INFO:tensorflow:Restoring parameters from /tmp/model.ckpt\n",
      "Model restored from file: /tmp/model.ckpt\n",
      "[[0.15292837]\n",
      " [0.20799588]\n",
      " [2.3737454 ]]\n"
     ]
    }
   ],
   "source": [
    "# Running a new session for predictions\n",
    "print(\"Starting prediction session...\")\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables\n",
    "    sess.run(init)\n",
    "\n",
    "    # Restore model weights from previously saved model\n",
    "    saver.restore(sess, model_path)\n",
    "    print(\"Model restored from file: %s\" % model_path)\n",
    "\n",
    "    # We try to predict the petal width (cm) of three samples\n",
    "    feed_dict_std = {X: [[6.86549436, 4.28228483, 0.89182234],\n",
    "       [6.38114257, 3.47503186, 0.89182234],\n",
    "       [8.8029015 , 4.12083424, 7.67274733]]}\n",
    "    prediction = sess.run(y_hat, feed_dict_std)\n",
    "    print(prediction) # True value 0.2, 0.1, 2.4\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Third: we reverse the transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.9423405],\n",
       "       [0.9764485],\n",
       "       [2.3178802]], dtype=float32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat_rev = sc.inverse_transform(prediction)\n",
    "y_hat_rev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad. We'll try to improve them with a deeper network. That is the goal of the next notebook."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "deeptrading",
   "language": "python",
   "name": "deeptrading"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}