papers/Identity Normalization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identity Normalization in Residual Networks\n",
    "\n",
    "This notebook contains the compact (residual) neural network, datasets and empirical results for the aforementioned paper."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset(s)\n",
    "\n",
    "The first dataset used for empirical benchmarking is the Fruits-360 dataset, which was formerly a Kaggle competition. It consists of images of fruit labeled by fruit type and the variety. \n",
    "\n",
    "    1. There are a total of 47 types of fruit (e.g., Apple, Orange, Pear, etc) and 81 varieties.\n",
    "    2. On average, there are 656 images per variety.\n",
    "    3. Each image is 128x128 RGB.\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Updates are available for some Cloud SDK components.  To install them,\n",
      "please run:\n",
      "  $ gcloud components update\n",
      "\n",
      "Copying gs://cloud-samples-data/air/fruits360/fruits360-combined.zip...\n",
      "\\ [1 files][230.9 MiB/230.9 MiB]                                                \n",
      "Operation completed over 1 objects/230.9 MiB.                                    \n",
      " arch-1.png\t     cascade.ipynb\t     fruits360-combined.zip   Training\n",
      " arch-2.png\t     CAS-CNN.ipynb\t     Save\n",
      " cascade-gap.ipynb  'CAS-CNN overview.jpg'   train_base_model.jpg\n"
     ]
    }
   ],
   "source": [
    "!gsutil cp gs://cloud-samples-data/air/fruits360/fruits360-combined.zip .\n",
    "!ls\n",
    "!unzip -qn fruits360-combined.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Install:\n",
    "    1. Tensorflow (1.13) machine learning framework\n",
    "    2. Tensorflow integration of Keras API for building and training models.\n",
    "    3. Numpy for general image/matrix manipulation.\n",
    "    4. OpenCV for reading and preprocessing of images.\n",
    "    5. Scikit-Learn for splitting datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.applications.resnet50 import ResNet50\n",
    "from tensorflow.keras.preprocessing import image\n",
    "from tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions\n",
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "from tensorflow.keras.layers import GlobalAveragePooling2D, Dense\n",
    "from tensorflow.keras import Sequential, Model, Input, optimizers\n",
    "from tensorflow.keras.layers import Conv2D, Flatten, MaxPooling2D, Dense, Dropout, BatchNormalization, ReLU\n",
    "from tensorflow.keras.models import load_model\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "import tensorflow.keras.layers as layers\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import numpy as np\n",
    "import cv2\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Steps\n",
    "\n",
    "*Benchmark Fruits-360*\n",
    "    1.  Create the Coarse Dataset for Training, Evaluation and Test\n",
    "    2.  Define training routines\n",
    "    3.  Run Benchmarks\n",
    "    \n",
    "*Benchmark Intel Image Classification*\n",
    "    1.  Create the Dataset for Training, Evaluation and Test\n",
    "    2.  Run Benchmarks\n",
    "    \n",
    "*Benchmark Columbia University COIL-100*\n",
    "    11. Create the Dataset for Training, Evaluation and Test\n",
    "    12. Run Benchmarks\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Coarse Fruits-360 Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## (Step 1) Make Coarse Category Dataset\n",
    "\n",
    "\n",
    "### Define Function to Make Coarse Train/Test Dataset from Fruits-360 dataset\n",
    "\n",
    "This makes the by Fruit Type dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def Fruits(root):\n",
    "    n_label = 0\n",
    "    images = []\n",
    "    labels = []\n",
    "    classes = {}\n",
    "    \n",
    "    os.chdir(root)\n",
    "    classes_ = os.scandir('./')\n",
    "    for class_ in classes_:\n",
    "        print(class_.name)\n",
    "        os.chdir(class_.name)\n",
    "        classes[class_.name] = n_label\n",
    "\n",
    "        # Finer Level Subdirectories per Coarse Level\n",
    "        subclasses = os.scandir('./')\n",
    "        for subclass in subclasses:\n",
    "            os.chdir(subclass.name)\n",
    "            files = os.listdir('./')\n",
    "            for file in files:\n",
    "                image = cv2.imread(file)\n",
    "                images.append(image)\n",
    "                labels.append(n_label)\n",
    "                \n",
    "            os.chdir('../')\n",
    "\n",
    "        os.chdir('../')\n",
    "        n_label += 1\n",
    "    os.chdir('../')\n",
    "    images = np.asarray(images)\n",
    "    # standardization of the pixel data\n",
    "    mean = np.mean(images)\n",
    "    std  = np.std(images)\n",
    "    images = ((images - mean) / std).astype(np.float32)\n",
    "    # convert to one-hot encoded labels\n",
    "    labels = to_categorical(labels, n_label)\n",
    "    print(\"Images\", images.shape, \"Labels\", labels.shape, \"Classes\", classes, \"Mean\", mean, \"Stddev\", std)\n",
    "\n",
    "    # Split the processed image dataset into training and test data\n",
    "    x_train, x_test, y_train, y_test = train_test_split(images, labels, test_size=0.20, shuffle=True)\n",
    "    return x_train, x_test, y_train, y_test, classes, mean, std"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate the preprocessed Coarse Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              total        used        free      shared  buff/cache   available\n",
      "Mem:          64146        4674       50986          96        8485       58881\n",
      "Swap:         65187           0       65187\n",
      "Avocado\n",
      "Rambutan\n",
      "Pineapple\n",
      "Kiwi\n",
      "Cantaloupe\n",
      "Apple\n",
      "Tamarillo\n",
      "Physalis\n",
      "Plum\n",
      "Pitahaya\n",
      "Guava\n",
      "Limes\n",
      "Grapefruit\n",
      "Peach\n",
      "Pomegranate\n",
      "Nectarine\n",
      "Apricot\n",
      "Banana\n",
      "Cherry\n",
      "Mulberry\n",
      "Raspberry\n",
      "Cactus Fruit\n",
      "Grape\n",
      "Mandarine\n",
      "Granadilla\n",
      "Carambula\n",
      "Passion Fruit\n",
      "Lychee\n",
      "Quince\n",
      "Maracuja\n",
      "Strawberry\n",
      "Tangelo\n",
      "Huckleberry\n",
      "Orange\n",
      "Dates\n",
      "Melon\n",
      "Pepino\n",
      "Clementine\n",
      "Papaya\n",
      "Mango\n",
      "Tomato\n",
      "Salak\n",
      "Kaki\n",
      "Pear\n",
      "Cocos\n",
      "Lemon\n",
      "Kumquats\n",
      "Images (51258, 100, 100, 3) Labels (51258, 47) Classes {'Avocado': 0, 'Rambutan': 1, 'Pineapple': 2, 'Kiwi': 3, 'Cantaloupe': 4, 'Apple': 5, 'Tamarillo': 6, 'Physalis': 7, 'Plum': 8, 'Pitahaya': 9, 'Guava': 10, 'Limes': 11, 'Grapefruit': 12, 'Peach': 13, 'Pomegranate': 14, 'Nectarine': 15, 'Apricot': 16, 'Banana': 17, 'Cherry': 18, 'Mulberry': 19, 'Raspberry': 20, 'Cactus Fruit': 21, 'Grape': 22, 'Mandarine': 23, 'Granadilla': 24, 'Carambula': 25, 'Passion Fruit': 26, 'Lychee': 27, 'Quince': 28, 'Maracuja': 29, 'Strawberry': 30, 'Tangelo': 31, 'Huckleberry': 32, 'Orange': 33, 'Dates': 34, 'Melon': 35, 'Pepino': 36, 'Clementine': 37, 'Papaya': 38, 'Mango': 39, 'Tomato': 40, 'Salak': 41, 'Kaki': 42, 'Pear': 43, 'Cocos': 44, 'Lemon': 45, 'Kumquats': 46} Mean 155.8832843426067 Stddev 89.50581300610455\n",
      "              total        used        free      shared  buff/cache   available\n",
      "Mem:          64146       12011       43191          96        8943       51545\n",
      "Swap:         65187           0       65187\n"
     ]
    }
   ],
   "source": [
    "!free -m\n",
    "x_train, x_test, y_train, y_test, fruits_classes, mean, std = Fruits('Training')\n",
    "!free -m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split Coarse Training Dataset (by Fruit) further into Train and Validation\n",
    "\n",
    "Next split out 10% of train to use for validation during training.\n",
    "\n",
    "    - Train: 80%\n",
    "        - Train: 90%\n",
    "        - Validation: 10%\n",
    "    - Test : 20%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train (36905, 100, 100, 3) (36905, 47)\n",
      "val   (4101, 100, 100, 3) (4101, 47)\n",
      "test  (10252, 100, 100, 3) (10252, 47)\n",
      "              total        used        free      shared  buff/cache   available\r\n",
      "Mem:          64146       12012       43197          88        8935       51551\r\n",
      "Swap:         65187           0       65187\r\n"
     ]
    }
   ],
   "source": [
    "# Split out 10% of Train to use for Validation\n",
    "pivot = int(len(x_train) * 0.9)\n",
    "x_val = x_train[pivot:]\n",
    "y_val = y_train[pivot:]\n",
    "x_train = x_train[:pivot]\n",
    "y_train = y_train[:pivot]\n",
    "\n",
    "print(\"train\", x_train.shape, y_train.shape)\n",
    "print(\"val  \", x_val.shape, y_val.shape)\n",
    "print(\"test \", x_test.shape, y_test.shape)\n",
    "!free -m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Step 2) Define Training Routines\n",
    "\n",
    "Define the routines we will use for training.\n",
    "\n",
    "### Make Feeder\n",
    "\n",
    "Prepare the Feeder mechanism for training the neural networkm using ImageDataGenerator. \n",
    "\n",
    "Add image augmentation for:\n",
    "\n",
    "    1. Horizontal Flip\n",
    "    2. Verticial  Flip\n",
    "    3. Random Rotation +/- 30 degrees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def Feeder():\n",
    "    datagen = ImageDataGenerator(horizontal_flip=True, vertical_flip=True, rotation_range=30, \n",
    "                                 width_shift_range=0.15, height_shift_range=0.15, shear_range=0.2)\n",
    "    return datagen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make Trainer\n",
    "\n",
    "Prepare a training session:\n",
    "\n",
    "    1. Epochs defaults to 10\n",
    "    2. Batch size defaults to 32\n",
    "    3. Train with validation data\n",
    "    4. Final evaluation with test data (holdout set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def Train(model, datagen, x_train, y_train, x_test, y_test, epochs=10, batch_size=32):\n",
    "    model.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size, shuffle=True),\n",
    "                    steps_per_epoch=len(x_train) / batch_size, epochs=epochs, verbose=1, validation_data=(x_test, y_test))\n",
    "    scores = model.evaluate(x_train, y_train, verbose=1)\n",
    "    print(\"Train\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Step 3) Build Base Model\n",
    "\n",
    "See paper for detailed description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from keras import Input, Model\n",
    "import keras.layers as layers\n",
    "import keras.optimizers as optimizers\n",
    "\n",
    "\n",
    "def ConvNetA(input_shape, nclasses):\n",
    "    ''' Compact Neural Network with Batch Normalization (Model 2)'''\n",
    "    def stem(inputs):\n",
    "        ''' The stem convolutional group '''\n",
    "        # Two 3x3 convolutional layers, representational equivalent to single 5x5,\n",
    "        # which reduces computational complexity (trainable weights) by 75%\n",
    "        x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(inputs)\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(x)\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        # Reduce the feature map sizes by 75%\n",
    "        x = layers.MaxPooling2D(2, strides=2)(x)\n",
    "        return x\n",
    "    \n",
    "    def conv_groups(x):\n",
    "        ''' Residual Groups (ResNet34 style) '''\n",
    "        # transition convolution for identity link, delay downsampling\n",
    "        shortcut = layers.Conv2D(128, (1,1), strides=1, padding='same')(x)\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        \n",
    "        # residual block - double filters (Replace two 3,3 with 3,3 and 1,1)\n",
    "        x = layers.Conv2D(128, (3,3), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        x = layers.Conv2D(128, (3,1), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        \n",
    "        # identity link\n",
    "        x = layers.add([shortcut, x])\n",
    "        x = layers.Dropout(0.50)(x)\n",
    "        \n",
    "        # transition convolution for identity link\n",
    "        shortcut = layers.Conv2D(256, (1,1), strides=1, padding='same')(x)\n",
    "        \n",
    "        # residual block - double filters\n",
    "        x = layers.Conv2D(256, (3,3), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        x = layers.Conv2D(256, (3,1), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        \n",
    "        # identity link\n",
    "        x = layers.add([shortcut, x])\n",
    "        \n",
    "        # pooling for final downsampling in convolutional layers\n",
    "        x = layers.MaxPooling2D(2, strides=2, name='encoder')(x)\n",
    "\n",
    "        return x\n",
    "    \n",
    "    def bottleneck(x):\n",
    "        ''' The bottleneck layer '''\n",
    "        # Use fast form of pooling: single value per feature map, \n",
    "        # which reduces the size substantially more than a Flatten().\n",
    "        x = layers.GlobalAveragePooling2D(name='bottleneck')(x)\n",
    "        return x\n",
    "    \n",
    "    def classifier(x, nclasses):\n",
    "        ''' The classifier layer '''\n",
    "        x = layers.Dense(nclasses, activation='softmax')(x)\n",
    "        return x\n",
    "    \n",
    "    inputs = Input(input_shape)\n",
    "    \n",
    "    x = stem(inputs)\n",
    "    x = conv_groups(x)\n",
    "    x = bottleneck(x)\n",
    "    outputs = classifier(x, nclasses)\n",
    "    \n",
    "    return Model(inputs, outputs)\n",
    "\n",
    "\n",
    "def ConvNetB(input_shape, nclasses):\n",
    "    ''' Compact Neural Network with Identity Normalization (Model 3)'''\n",
    "    def stem(inputs):\n",
    "        ''' The stem convolutional group '''\n",
    "        # Two 3x3 convolutional layers, representational equivalent to single 5x5,\n",
    "        # which reduces computational complexity (trainable weights) by 75%\n",
    "        x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(inputs)\n",
    "        x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(x)\n",
    "        # Reduce the feature map sizes by 75%\n",
    "        x = layers.MaxPooling2D(2, strides=2)(x)\n",
    "        return x\n",
    "    \n",
    "    def conv_groups(x):\n",
    "        ''' Residual Groups (ResNet34 style) '''\n",
    "        # transition convolution for identity link, delay downsampling\n",
    "        shortcut = layers.Conv2D(128, (1,1), strides=1, padding='same')(x)\n",
    "        \n",
    "        # residual block - double filters (Replace two 3,3 with 3,3 and 1,1)\n",
    "        x = layers.Conv2D(128, (3,3), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.Conv2D(128, (1,1), strides=1, padding='same', activation='relu')(x)\n",
    "        \n",
    "        # identity link\n",
    "        x = layers.add([shortcut, x])\n",
    "        x = layers.Dropout(0.50)(x)\n",
    "        \n",
    "        # transition convolution for identity link\n",
    "        shortcut = layers.Conv2D(256, (1,1), strides=1, padding='same')(x)\n",
    "        \n",
    "        # residual block - double filters\n",
    "        x = layers.Conv2D(256, (3,3), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.Conv2D(256, (1,1), strides=1, padding='same', activation='relu')(x)\n",
    "        \n",
    "        # identity link\n",
    "        x = layers.add([shortcut, x])\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        \n",
    "        # pooling for final downsampling in convolutional layers\n",
    "        x = layers.MaxPooling2D(2, strides=2, name='encoder')(x)\n",
    "\n",
    "        return x\n",
    "    \n",
    "    def bottleneck(x):\n",
    "        ''' The bottleneck layer '''\n",
    "        # Use fast form of pooling: single value per feature map, \n",
    "        # which reduces the size substantially more than a Flatten().\n",
    "        x = layers.GlobalAveragePooling2D(name='bottleneck')(x)\n",
    "        return x\n",
    "    \n",
    "    def classifier(x, nclasses):\n",
    "        ''' The classifier layer '''\n",
    "        x = layers.Dense(nclasses, activation='softmax')(x)\n",
    "        return x\n",
    "    \n",
    "    inputs = Input(input_shape)\n",
    "    \n",
    "    x = stem(inputs)\n",
    "    x = conv_groups(x)\n",
    "    x = bottleneck(x)\n",
    "    outputs = classifier(x, nclasses)\n",
    "    \n",
    "    return Model(inputs, outputs)\n",
    "    \n",
    "\n",
    "def ConvNetC(input_shape, nclasses):\n",
    "    ''' Compact Neural Network without Normalization (Model 1)'''\n",
    "    def stem(inputs):\n",
    "        ''' The stem convolutional group '''\n",
    "        # Two 3x3 convolutional layers, representational equivalent to single 5x5,\n",
    "        # which reduces computational complexity (trainable weights) by 75%\n",
    "        x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(inputs)\n",
    "        x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(x)\n",
    "        # Reduce the feature map sizes by 75%\n",
    "        x = layers.MaxPooling2D(2, strides=2)(x)\n",
    "        return x\n",
    "    \n",
    "    def conv_groups(x):\n",
    "        ''' Residual Groups (ResNet34 style) '''\n",
    "        # transition convolution for identity link, delay downsampling\n",
    "        shortcut = layers.Conv2D(128, (1,1), strides=1, padding='same')(x)\n",
    "        \n",
    "        # residual block - double filters (Replace two 3,3 with 3,3 and 1,1)\n",
    "        x = layers.Conv2D(128, (3,3), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.Conv2D(128, (1,1), strides=1, padding='same', activation='relu')(x)\n",
    "        \n",
    "        # identity link\n",
    "        x = layers.add([shortcut, x])\n",
    "        x = layers.Dropout(0.50)(x)\n",
    "        \n",
    "        # transition convolution for identity link\n",
    "        shortcut = layers.Conv2D(256, (1,1), strides=1, padding='same')(x)\n",
    "        \n",
    "        # residual block - double filters\n",
    "        x = layers.Conv2D(256, (3,3), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.Conv2D(256, (1,1), strides=1, padding='same', activation='relu')(x)\n",
    "        \n",
    "        # identity link\n",
    "        x = layers.add([shortcut, x])\n",
    "        \n",
    "        # pooling for final downsampling in convolutional layers\n",
    "        x = layers.MaxPooling2D(2, strides=2, name='encoder')(x)\n",
    "\n",
    "        return x\n",
    "    \n",
    "    def bottleneck(x):\n",
    "        ''' The bottleneck layer '''\n",
    "        # Use fast form of pooling: single value per feature map, \n",
    "        # which reduces the size substantially more than a Flatten().\n",
    "        x = layers.GlobalAveragePooling2D(name='bottleneck')(x)\n",
    "        return x\n",
    "    \n",
    "    def classifier(x, nclasses):\n",
    "        ''' The classifier layer '''\n",
    "        x = layers.Dense(nclasses, activation='softmax')(x)\n",
    "        return x\n",
    "    \n",
    "    inputs = Input(input_shape)\n",
    "    \n",
    "    x = stem(inputs)\n",
    "    x = conv_groups(x)\n",
    "    x = bottleneck(x)\n",
    "    outputs = classifier(x, nclasses)\n",
    "    \n",
    "    return Model(inputs, outputs)\n",
    "\n",
    "\n",
    "# Remove this (obsolete)\n",
    "def ConvNetD(input_shape, nclasses):\n",
    "    def stem(inputs):\n",
    "        ''' The stem convolutional group '''\n",
    "        # Two 3x3 convolutional layers, representational equivalent to single 5x5,\n",
    "        # which reduces computational complexity (trainable weights) by 75%\n",
    "        x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(inputs)\n",
    "        x = layers.Conv2D(64, 3, strides=2, padding='same', activation='relu')(x)\n",
    "        # Reduce the feature map sizes by 75%\n",
    "        x = layers.MaxPooling2D(2, strides=2)(x)\n",
    "        return x\n",
    "    \n",
    "    def conv_groups(x):\n",
    "        ''' Residual Groups (ResNet34 style) '''\n",
    "        # transition convolution for identity link, delay downsampling\n",
    "        shortcut = layers.Conv2D(128, (1,1), strides=1, padding='same')(x)\n",
    "        \n",
    "        # residual block - double filters (Replace two 3,3 with 3,3 and 1,1)\n",
    "        x = layers.Conv2D(128, (3,3), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.Conv2D(128, (1,1), strides=1, padding='same', activation='relu')(x)\n",
    "        \n",
    "        # identity link\n",
    "        x = layers.add([shortcut, x])\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        \n",
    "        x = layers.Dropout(0.50)(x)\n",
    "        \n",
    "        # transition convolution for identity link\n",
    "        shortcut = layers.Conv2D(256, (1,1), strides=1, padding='same')(x)\n",
    "        \n",
    "        # residual block - double filters\n",
    "        x = layers.Conv2D(256, (3,3), strides=1, padding='same', activation='relu')(x)\n",
    "        x = layers.Conv2D(256, (1,1), strides=1, padding='same', activation='relu')(x)\n",
    "        \n",
    "        # identity link\n",
    "        x = layers.add([shortcut, x])\n",
    "        x = layers.BatchNormalization()(x)\n",
    "        \n",
    "        # pooling for final downsampling in convolutional layers\n",
    "        x = layers.MaxPooling2D(2, strides=2, name='encoder')(x)\n",
    "\n",
    "        return x\n",
    "    \n",
    "    def bottleneck(x):\n",
    "        ''' The bottleneck layer '''\n",
    "        # Use fast form of pooling: single value per feature map, \n",
    "        # which reduces the size substantially more than a Flatten().\n",
    "        x = layers.GlobalAveragePooling2D(name='bottleneck')(x)\n",
    "        return x\n",
    "    \n",
    "    def classifier(x, nclasses):\n",
    "        ''' The classifier layer '''\n",
    "        x = layers.Dense(nclasses, activation='softmax')(x)\n",
    "        return x\n",
    "    \n",
    "    inputs = Input(input_shape)\n",
    "    \n",
    "    x = stem(inputs)\n",
    "    x = conv_groups(x)\n",
    "    x = bottleneck(x)\n",
    "    outputs = classifier(x, nclasses)\n",
    "    \n",
    "    return Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark Fruits-360 with Models 2 and 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 14.0790 - acc: 0.1258 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 267s 231ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 265s 230ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 260s 226ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 265s 230ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 262s 227ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 258s 223ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 257s 223ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 258s 223ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 261s 226ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 261s 226ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 260s 225ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "36905/36905 [==============================] - 55s 2ms/step\n",
      "Train [14.084607949674881, 0.12616176669827936]\n",
      "10252/10252 [==============================] - 15s 1ms/step\n",
      "Test [14.071103506054769, 0.12699960983804182]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 267s 231ms/step - loss: 15.5999 - acc: 0.0316 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 257s 223ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 263s 228ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 267s 231ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 273s 236ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 261s 226ms/step - loss: 15.6063 - acc: 0.0318 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 259s 225ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 15.6052 - acc: 0.0318 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 258s 223ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 262s 227ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 15.6063 - acc: 0.0318 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.6074 - acc: 0.0317 - val_loss: 15.5836 - val_acc: 0.0332\n",
      "36905/36905 [==============================] - 54s 1ms/step\n",
      "Train [15.607102876174032, 0.03170302127083051]\n",
      "10252/10252 [==============================] - 15s 1ms/step\n",
      "Test [15.596127998628887, 0.03238392508778775]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 15.7921 - acc: 0.0197 - val_loss: 15.7997 - val_acc: 0.0198\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.5085 - acc: 0.0378 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.4943 - acc: 0.0387 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "36905/36905 [==============================] - 54s 1ms/step\n",
      "Train [15.49616947440238, 0.038585557512532176]\n",
      "10252/10252 [==============================] - 15s 1ms/step\n",
      "Test [15.519090647256519, 0.03716348029652751]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.5956 - acc: 0.0315 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.5273 - acc: 0.0367 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.5289 - acc: 0.0366 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.5284 - acc: 0.0366 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.5961 - acc: 0.0323 - val_loss: 15.7054 - val_acc: 0.0256\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.6297 - acc: 0.0303 - val_loss: 15.7604 - val_acc: 0.0222\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.6288 - acc: 0.0304 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.6063 - acc: 0.0318 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.6065 - acc: 0.0317 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.5389 - acc: 0.0359 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.5332 - acc: 0.0363 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.5323 - acc: 0.0363 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.5328 - acc: 0.0363 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.4642 - acc: 0.0405 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 15.5158 - acc: 0.0374 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 15.5171 - acc: 0.0373 - val_loss: 15.5679 - val_acc: 0.0341\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 15.5711 - acc: 0.0339 - val_loss: 15.5718 - val_acc: 0.0339\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 15.5788 - acc: 0.0334 - val_loss: 15.6425 - val_acc: 0.0295\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 15.5550 - acc: 0.0349 - val_loss: 15.6425 - val_acc: 0.0295\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.5616 - acc: 0.0345 - val_loss: 15.6543 - val_acc: 0.0288\n",
      "36905/36905 [==============================] - 34s 932us/step\n",
      "Train [15.584828839975875, 0.033084947839248084]\n",
      "10252/10252 [==============================] - 10s 935us/step\n",
      "Test [15.577261702604142, 0.03355442840421381]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 14.0721 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0803 - acc: 0.1264 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "36905/36905 [==============================] - 35s 936us/step\n",
      "Train [14.084607949674881, 0.12616176669827936]\n",
      "10252/10252 [==============================] - 10s 934us/step\n",
      "Test [14.071103506054769, 0.12699960983804182]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 15.9006 - acc: 0.0127 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 15.9136 - acc: 0.0127 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 7/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.9136 - acc: 0.0127 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.9147 - acc: 0.0126 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "36905/36905 [==============================] - 35s 953us/step\n",
      "Train [15.914571864261584, 0.012627015309578648]\n",
      "10252/10252 [==============================] - 10s 938us/step\n",
      "Test [15.896416454635185, 0.013753413968006242]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 254s 221ms/step - loss: 0.9503 - acc: 0.7381 - val_loss: 1.7171 - val_acc: 0.6879\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.3882 - acc: 0.8832 - val_loss: 0.5733 - val_acc: 0.8439\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.3563 - acc: 0.8996 - val_loss: 3.5711 - val_acc: 0.6264\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.3298 - acc: 0.9176 - val_loss: 1.7497 - val_acc: 0.7569\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.4287 - acc: 0.9121 - val_loss: 1.0303 - val_acc: 0.8386\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.3676 - acc: 0.9284 - val_loss: 1.9853 - val_acc: 0.7918\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.3880 - acc: 0.9328 - val_loss: 1.9159 - val_acc: 0.8108\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.4678 - acc: 0.9314 - val_loss: 2.1202 - val_acc: 0.7942\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.4887 - acc: 0.9364 - val_loss: 4.0494 - val_acc: 0.6642\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.5774 - acc: 0.9348 - val_loss: 2.3066 - val_acc: 0.8100\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.4716 - acc: 0.9453 - val_loss: 1.6611 - val_acc: 0.8535\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.6963 - acc: 0.9322 - val_loss: 2.1155 - val_acc: 0.8291\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 0.6760 - acc: 0.9369 - val_loss: 2.5708 - val_acc: 0.8100\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.7571 - acc: 0.9329 - val_loss: 1.5085 - val_acc: 0.8866\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.7825 - acc: 0.9371 - val_loss: 3.0966 - val_acc: 0.7927\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.9559 - acc: 0.9267 - val_loss: 2.0495 - val_acc: 0.8527\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 1.0029 - acc: 0.9260 - val_loss: 3.2275 - val_acc: 0.7718\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 1.1998 - acc: 0.9159 - val_loss: 2.8947 - val_acc: 0.8074\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 1.0102 - acc: 0.9294 - val_loss: 4.5575 - val_acc: 0.6940\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 1.0638 - acc: 0.9265 - val_loss: 2.6590 - val_acc: 0.8215\n",
      "36905/36905 [==============================] - 55s 1ms/step\n",
      "Train [2.6995806546011702, 0.8183443977796939]\n",
      "10252/10252 [==============================] - 15s 1ms/step\n",
      "Test [2.600592338057454, 0.8252048380803746]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 14.6352 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6441 - acc: 0.0915 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6441 - acc: 0.0915 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6441 - acc: 0.0915 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.6452 - acc: 0.0914 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6463 - acc: 0.0913 - val_loss: 14.7464 - val_acc: 0.0851\n",
      "36905/36905 [==============================] - 55s 1ms/step\n",
      "Train [14.645389134017227, 0.09136973309849614]\n",
      "10252/10252 [==============================] - 15s 1ms/step\n",
      "Test [14.651241602025086, 0.0910066328550334]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 14.0698 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.0817 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.0824 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.0821 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 14.0788 - acc: 0.1265 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 14.0821 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.0841 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0819 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 14.0792 - acc: 0.1265 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.0834 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.0830 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 14.0817 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.0833 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.2570 - acc: 0.1155 - val_loss: 14.7111 - val_acc: 0.0873\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6238 - acc: 0.0927 - val_loss: 14.7111 - val_acc: 0.0873\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.6267 - acc: 0.0925 - val_loss: 14.7228 - val_acc: 0.0866\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 14.6328 - acc: 0.0922 - val_loss: 14.7150 - val_acc: 0.0871\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 14.6168 - acc: 0.0931 - val_loss: 14.7307 - val_acc: 0.0861\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 260s 225ms/step - loss: 14.6312 - acc: 0.0922 - val_loss: 14.7150 - val_acc: 0.0871\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 267s 231ms/step - loss: 14.5516 - acc: 0.0972 - val_loss: 14.3062 - val_acc: 0.1124\n",
      "36905/36905 [==============================] - 60s 2ms/step\n",
      "Train [14.266942192380498, 0.11483538815878608]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [14.256879575448812, 0.11539211861681668]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 1.6774 - acc: 0.5161 - val_loss: 1.5405 - val_acc: 0.5696\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.5127 - acc: 0.8299 - val_loss: 0.4888 - val_acc: 0.8298\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.2809 - acc: 0.9002 - val_loss: 0.2574 - val_acc: 0.9134\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.2073 - acc: 0.9250 - val_loss: 0.2168 - val_acc: 0.9283\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1689 - acc: 0.9382 - val_loss: 0.1653 - val_acc: 0.9461\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1702 - acc: 0.9481 - val_loss: 0.3338 - val_acc: 0.8988\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1264 - acc: 0.9520 - val_loss: 0.5129 - val_acc: 0.8937\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1266 - acc: 0.9536 - val_loss: 0.3050 - val_acc: 0.9151\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0993 - acc: 0.9620 - val_loss: 0.1085 - val_acc: 0.9607\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0966 - acc: 0.9631 - val_loss: 0.0714 - val_acc: 0.9734\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1011 - acc: 0.9625 - val_loss: 0.1205 - val_acc: 0.9603\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0802 - acc: 0.9684 - val_loss: 0.1139 - val_acc: 0.9549\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0732 - acc: 0.9713 - val_loss: 0.1313 - val_acc: 0.9537\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0879 - acc: 0.9662 - val_loss: 0.0491 - val_acc: 0.9785\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0706 - acc: 0.9717 - val_loss: 0.1454 - val_acc: 0.9544\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0631 - acc: 0.9738 - val_loss: 0.1066 - val_acc: 0.9571\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0805 - acc: 0.9702 - val_loss: 0.0590 - val_acc: 0.9781\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0610 - acc: 0.9745 - val_loss: 0.3909 - val_acc: 0.8898\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0678 - acc: 0.9727 - val_loss: 0.2092 - val_acc: 0.9534\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0630 - acc: 0.9747 - val_loss: 0.0528 - val_acc: 0.9802\n",
      "36905/36905 [==============================] - 35s 956us/step\n",
      "Train [0.06164983223660705, 0.9752066115702479]\n",
      "10252/10252 [==============================] - 10s 956us/step\n",
      "Test [0.05389369040160369, 0.9793211080764729]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 1.9404 - acc: 0.4596 - val_loss: 0.9022 - val_acc: 0.7345\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.6723 - acc: 0.7749 - val_loss: 0.9344 - val_acc: 0.7308\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.4222 - acc: 0.8520 - val_loss: 0.7835 - val_acc: 0.7349\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.2978 - acc: 0.8930 - val_loss: 0.3403 - val_acc: 0.8800\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.2411 - acc: 0.9158 - val_loss: 0.1257 - val_acc: 0.9583\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1995 - acc: 0.9279 - val_loss: 0.1908 - val_acc: 0.9300\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1682 - acc: 0.9394 - val_loss: 0.3429 - val_acc: 0.8942\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1422 - acc: 0.9487 - val_loss: 0.1770 - val_acc: 0.9483\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 134s 117ms/step - loss: 0.1387 - acc: 0.9479 - val_loss: 0.3440 - val_acc: 0.8839\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1260 - acc: 0.9537 - val_loss: 0.1616 - val_acc: 0.9432\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1117 - acc: 0.9576 - val_loss: 0.0824 - val_acc: 0.9698\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1087 - acc: 0.9590 - val_loss: 0.0718 - val_acc: 0.9688\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1088 - acc: 0.9596 - val_loss: 0.0603 - val_acc: 0.9761\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0834 - acc: 0.9670 - val_loss: 0.0728 - val_acc: 0.9705\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0914 - acc: 0.9644 - val_loss: 0.1105 - val_acc: 0.9629\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0892 - acc: 0.9662 - val_loss: 0.0589 - val_acc: 0.9766\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0792 - acc: 0.9700 - val_loss: 0.1015 - val_acc: 0.9661\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0883 - acc: 0.9685 - val_loss: 0.0511 - val_acc: 0.9783\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0761 - acc: 0.9704 - val_loss: 0.1067 - val_acc: 0.9627\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0699 - acc: 0.9728 - val_loss: 0.1163 - val_acc: 0.9688\n",
      "36905/36905 [==============================] - 35s 958us/step\n",
      "Train [0.10576896193150885, 0.9687847175179515]\n",
      "10252/10252 [==============================] - 10s 963us/step\n",
      "Test [0.11148275362918719, 0.9667381974248928]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 2.1120 - acc: 0.3937 - val_loss: 1.8150 - val_acc: 0.4287\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.8463 - acc: 0.7240 - val_loss: 1.1961 - val_acc: 0.6706\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.4245 - acc: 0.8503 - val_loss: 0.8228 - val_acc: 0.7488\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.2761 - acc: 0.9012 - val_loss: 1.0848 - val_acc: 0.7362\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.2145 - acc: 0.9228 - val_loss: 0.5495 - val_acc: 0.8583\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1744 - acc: 0.9367 - val_loss: 0.5351 - val_acc: 0.8503\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1557 - acc: 0.9431 - val_loss: 0.2186 - val_acc: 0.9278\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1402 - acc: 0.9492 - val_loss: 0.3803 - val_acc: 0.8903\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 134s 117ms/step - loss: 0.1272 - acc: 0.9534 - val_loss: 0.5889 - val_acc: 0.8393\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1139 - acc: 0.9571 - val_loss: 0.3068 - val_acc: 0.9188\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0980 - acc: 0.9621 - val_loss: 0.1875 - val_acc: 0.9356\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0960 - acc: 0.9636 - val_loss: 0.1496 - val_acc: 0.9488\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0900 - acc: 0.9656 - val_loss: 0.0693 - val_acc: 0.9707\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0823 - acc: 0.9676 - val_loss: 0.1937 - val_acc: 0.9466\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 134s 117ms/step - loss: 0.0889 - acc: 0.9666 - val_loss: 0.1393 - val_acc: 0.9564\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0732 - acc: 0.9706 - val_loss: 0.1142 - val_acc: 0.9603\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0813 - acc: 0.9695 - val_loss: 0.0925 - val_acc: 0.9678\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0667 - acc: 0.9727 - val_loss: 0.2132 - val_acc: 0.9561\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0779 - acc: 0.9709 - val_loss: 0.0899 - val_acc: 0.9639\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0734 - acc: 0.9719 - val_loss: 0.1013 - val_acc: 0.9624\n",
      "36905/36905 [==============================] - 36s 962us/step\n",
      "Train [0.10724767509769585, 0.9594092941335862]\n",
      "10252/10252 [==============================] - 10s 967us/step\n",
      "Test [0.11689155080242696, 0.9608856808427624]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.4763 - acc: 0.8567 - val_loss: 0.9055 - val_acc: 0.7757\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.1461 - acc: 0.9489 - val_loss: 0.0999 - val_acc: 0.9649\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.1060 - acc: 0.9617 - val_loss: 0.1753 - val_acc: 0.9459\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0938 - acc: 0.9647 - val_loss: 0.2359 - val_acc: 0.9178\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 257s 223ms/step - loss: 0.0748 - acc: 0.9709 - val_loss: 0.4447 - val_acc: 0.8959\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0707 - acc: 0.9718 - val_loss: 0.2861 - val_acc: 0.9161\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0599 - acc: 0.9752 - val_loss: 0.2507 - val_acc: 0.9339\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0612 - acc: 0.9749 - val_loss: 0.2077 - val_acc: 0.9429\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0570 - acc: 0.9773 - val_loss: 0.0979 - val_acc: 0.9654\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0489 - acc: 0.9778 - val_loss: 0.4645 - val_acc: 0.8951\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0526 - acc: 0.9775 - val_loss: 0.1368 - val_acc: 0.9617\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.0510 - acc: 0.9782 - val_loss: 0.0362 - val_acc: 0.9851\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0442 - acc: 0.9812 - val_loss: 0.0975 - val_acc: 0.9683\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0484 - acc: 0.9792 - val_loss: 0.0984 - val_acc: 0.9710\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0420 - acc: 0.9808 - val_loss: 0.1899 - val_acc: 0.9464\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0406 - acc: 0.9811 - val_loss: 0.0337 - val_acc: 0.9854\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0446 - acc: 0.9808 - val_loss: 0.0937 - val_acc: 0.9671\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0438 - acc: 0.9801 - val_loss: 0.0919 - val_acc: 0.9678\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0347 - acc: 0.9832 - val_loss: 0.0414 - val_acc: 0.9842\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0439 - acc: 0.9812 - val_loss: 0.0735 - val_acc: 0.9737\n",
      "36905/36905 [==============================] - 56s 2ms/step\n",
      "Train [0.06708064390503232, 0.973743395203902]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.0631280359463925, 0.9762973078423722]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.5638 - acc: 0.8260 - val_loss: 0.4972 - val_acc: 0.8500\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.1453 - acc: 0.9474 - val_loss: 0.1779 - val_acc: 0.9403\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.1064 - acc: 0.9609 - val_loss: 0.1746 - val_acc: 0.9332\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0924 - acc: 0.9642 - val_loss: 0.2117 - val_acc: 0.9166\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0768 - acc: 0.9692 - val_loss: 0.1303 - val_acc: 0.9505\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0669 - acc: 0.9733 - val_loss: 0.0946 - val_acc: 0.9629\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0635 - acc: 0.9733 - val_loss: 0.1103 - val_acc: 0.9588\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0601 - acc: 0.9754 - val_loss: 0.0778 - val_acc: 0.9673\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0589 - acc: 0.9763 - val_loss: 0.0583 - val_acc: 0.9812\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0505 - acc: 0.9781 - val_loss: 0.0436 - val_acc: 0.9800\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0510 - acc: 0.9788 - val_loss: 0.0587 - val_acc: 0.9771\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0521 - acc: 0.9772 - val_loss: 0.0729 - val_acc: 0.9724\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0449 - acc: 0.9795 - val_loss: 0.0638 - val_acc: 0.9690\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0461 - acc: 0.9791 - val_loss: 0.0504 - val_acc: 0.9793\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0446 - acc: 0.9811 - val_loss: 0.0840 - val_acc: 0.9703\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0429 - acc: 0.9808 - val_loss: 0.3391 - val_acc: 0.8927\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0453 - acc: 0.9799 - val_loss: 0.1437 - val_acc: 0.9529\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0437 - acc: 0.9802 - val_loss: 0.1712 - val_acc: 0.9659\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0451 - acc: 0.9801 - val_loss: 0.0335 - val_acc: 0.9863\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0389 - acc: 0.9818 - val_loss: 0.0750 - val_acc: 0.9739\n",
      "36905/36905 [==============================] - 56s 2ms/step\n",
      "Train [0.08598676381452469, 0.9703563202818046]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.08239571777957452, 0.972200546234881]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.7697 - acc: 0.8145 - val_loss: 0.5429 - val_acc: 0.8822\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.3466 - acc: 0.9361 - val_loss: 0.3475 - val_acc: 0.9405\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.3019 - acc: 0.9492 - val_loss: 0.3573 - val_acc: 0.9415\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.2868 - acc: 0.9548 - val_loss: 0.3905 - val_acc: 0.9276\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.2788 - acc: 0.9568 - val_loss: 0.3847 - val_acc: 0.9295\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.1317 - acc: 0.9679 - val_loss: 0.0599 - val_acc: 0.9754\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0602 - acc: 0.9753 - val_loss: 0.0880 - val_acc: 0.9690\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0619 - acc: 0.9737 - val_loss: 0.0459 - val_acc: 0.9783\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0548 - acc: 0.9764 - val_loss: 0.0441 - val_acc: 0.9810\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 250s 216ms/step - loss: 0.0518 - acc: 0.9774 - val_loss: 0.0810 - val_acc: 0.9732\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0474 - acc: 0.9793 - val_loss: 0.0480 - val_acc: 0.9802\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0485 - acc: 0.9786 - val_loss: 0.0730 - val_acc: 0.9712\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0500 - acc: 0.9780 - val_loss: 0.0446 - val_acc: 0.9820\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0411 - acc: 0.9810 - val_loss: 0.0442 - val_acc: 0.9800\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 0.0471 - acc: 0.9799 - val_loss: 1.6010 - val_acc: 0.7971\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 0.0443 - acc: 0.9811 - val_loss: 0.0376 - val_acc: 0.9844\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0431 - acc: 0.9812 - val_loss: 0.0836 - val_acc: 0.9671\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 257s 222ms/step - loss: 0.0420 - acc: 0.9808 - val_loss: 0.0729 - val_acc: 0.9703\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0410 - acc: 0.9810 - val_loss: 0.0276 - val_acc: 0.9885\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0412 - acc: 0.9813 - val_loss: 0.0288 - val_acc: 0.9861\n",
      "36905/36905 [==============================] - 58s 2ms/step\n",
      "Train [0.03156603098926729, 0.9837691369733098]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.030549241767617124, 0.9847834568864612]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.8767 - acc: 0.7357 - val_loss: 0.4681 - val_acc: 0.8500\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 140s 121ms/step - loss: 0.2512 - acc: 0.9168 - val_loss: 0.3161 - val_acc: 0.8925\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.1324 - acc: 0.9541 - val_loss: 0.1243 - val_acc: 0.9564\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0985 - acc: 0.9644 - val_loss: 0.2355 - val_acc: 0.9205\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 0.0814 - acc: 0.9695 - val_loss: 0.1196 - val_acc: 0.9578\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0719 - acc: 0.9717 - val_loss: 0.5850 - val_acc: 0.8703\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0649 - acc: 0.9733 - val_loss: 0.1062 - val_acc: 0.9610\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0530 - acc: 0.9774 - val_loss: 0.0404 - val_acc: 0.9837\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0573 - acc: 0.9760 - val_loss: 0.0780 - val_acc: 0.9673\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0526 - acc: 0.9777 - val_loss: 0.0396 - val_acc: 0.9790\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0459 - acc: 0.9803 - val_loss: 0.0459 - val_acc: 0.9785\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0481 - acc: 0.9787 - val_loss: 0.0352 - val_acc: 0.9824\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0491 - acc: 0.9787 - val_loss: 0.0703 - val_acc: 0.9722\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0344 - acc: 0.9830 - val_loss: 0.0453 - val_acc: 0.9785\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 0.0488 - acc: 0.9778 - val_loss: 0.0275 - val_acc: 0.9881\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0393 - acc: 0.9817 - val_loss: 0.0310 - val_acc: 0.9849\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.0362 - acc: 0.9827 - val_loss: 0.0417 - val_acc: 0.9817\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0401 - acc: 0.9824 - val_loss: 0.0284 - val_acc: 0.9861\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0380 - acc: 0.9826 - val_loss: 0.0388 - val_acc: 0.9810\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0378 - acc: 0.9813 - val_loss: 0.0615 - val_acc: 0.9751\n",
      "36905/36905 [==============================] - 37s 999us/step\n",
      "Train [0.056968428182961765, 0.974583389784582]\n",
      "10252/10252 [==============================] - 10s 1ms/step\n",
      "Test [0.05944807560344263, 0.9744440109246976]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.8122 - acc: 0.7501 - val_loss: 0.4444 - val_acc: 0.8544\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.2592 - acc: 0.9112 - val_loss: 0.2677 - val_acc: 0.9076\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1444 - acc: 0.9503 - val_loss: 0.1509 - val_acc: 0.9427\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.1009 - acc: 0.9631 - val_loss: 0.1262 - val_acc: 0.9534\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 0.0822 - acc: 0.9688 - val_loss: 0.0725 - val_acc: 0.9734\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0736 - acc: 0.9715 - val_loss: 0.1004 - val_acc: 0.9624\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 138s 119ms/step - loss: 0.0626 - acc: 0.9755 - val_loss: 0.0603 - val_acc: 0.9746\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 149s 129ms/step - loss: 0.0611 - acc: 0.9743 - val_loss: 0.0594 - val_acc: 0.9759\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 140s 121ms/step - loss: 0.0557 - acc: 0.9774 - val_loss: 0.0707 - val_acc: 0.9759\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0504 - acc: 0.9792 - val_loss: 0.0545 - val_acc: 0.9790\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0517 - acc: 0.9780 - val_loss: 0.0478 - val_acc: 0.9837\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0498 - acc: 0.9793 - val_loss: 0.0794 - val_acc: 0.9673\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0424 - acc: 0.9808 - val_loss: 0.0490 - val_acc: 0.9781\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0465 - acc: 0.9788 - val_loss: 0.0304 - val_acc: 0.9868\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 138s 119ms/step - loss: 0.0462 - acc: 0.9802 - val_loss: 0.0321 - val_acc: 0.9871\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0409 - acc: 0.9820 - val_loss: 0.0424 - val_acc: 0.9829\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0363 - acc: 0.9827 - val_loss: 0.0364 - val_acc: 0.9815\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0390 - acc: 0.9818 - val_loss: 0.0286 - val_acc: 0.9868\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0412 - acc: 0.9818 - val_loss: 0.0327 - val_acc: 0.9861\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 143s 124ms/step - loss: 0.0388 - acc: 0.9818 - val_loss: 0.0254 - val_acc: 0.9893\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [0.030183531340747763, 0.9859639615228288]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.02946667849259951, 0.9872220054623488]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 150s 130ms/step - loss: 0.8387 - acc: 0.7487 - val_loss: 0.6788 - val_acc: 0.7720\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.2460 - acc: 0.9204 - val_loss: 0.2399 - val_acc: 0.9190\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.1434 - acc: 0.9507 - val_loss: 0.2551 - val_acc: 0.9251\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 0.1056 - acc: 0.9619 - val_loss: 0.4496 - val_acc: 0.8788\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0830 - acc: 0.9685 - val_loss: 0.0977 - val_acc: 0.9617\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0758 - acc: 0.9703 - val_loss: 0.1230 - val_acc: 0.9571\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0660 - acc: 0.9733 - val_loss: 0.1850 - val_acc: 0.9456\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0635 - acc: 0.9743 - val_loss: 0.0527 - val_acc: 0.9802\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 143s 124ms/step - loss: 0.0599 - acc: 0.9762 - val_loss: 0.0521 - val_acc: 0.9781\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 140s 121ms/step - loss: 0.0513 - acc: 0.9786 - val_loss: 0.0582 - val_acc: 0.9729\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 141s 123ms/step - loss: 0.0520 - acc: 0.9780 - val_loss: 0.0429 - val_acc: 0.9817\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0458 - acc: 0.9807 - val_loss: 0.0704 - val_acc: 0.9737\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0470 - acc: 0.9790 - val_loss: 0.0366 - val_acc: 0.9829\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 0.0474 - acc: 0.9800 - val_loss: 0.0446 - val_acc: 0.9783\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0388 - acc: 0.9818 - val_loss: 0.0269 - val_acc: 0.9900\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 143s 124ms/step - loss: 0.0393 - acc: 0.9807 - val_loss: 0.0278 - val_acc: 0.9881\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0424 - acc: 0.9794 - val_loss: 0.0959 - val_acc: 0.9685\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 0.0464 - acc: 0.9804 - val_loss: 0.0431 - val_acc: 0.9802\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 148s 129ms/step - loss: 0.0313 - acc: 0.9842 - val_loss: 0.0235 - val_acc: 0.9854\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0391 - acc: 0.9817 - val_loss: 0.0990 - val_acc: 0.9646\n",
      "36905/36905 [==============================] - 36s 985us/step\n",
      "Train [0.0971076524795838, 0.9668608589622002]\n",
      "10252/10252 [==============================] - 10s 987us/step\n",
      "Test [0.09637107070394796, 0.9673234491063616]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 270s 234ms/step - loss: 0.7203 - acc: 0.8222 - val_loss: 0.1780 - val_acc: 0.9488\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.1779 - acc: 0.9497 - val_loss: 0.1785 - val_acc: 0.9415\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.1170 - acc: 0.9621 - val_loss: 0.0764 - val_acc: 0.9690\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0891 - acc: 0.9688 - val_loss: 0.0437 - val_acc: 0.9822\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0766 - acc: 0.9728 - val_loss: 0.0500 - val_acc: 0.9785\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0651 - acc: 0.9766 - val_loss: 0.0587 - val_acc: 0.9739\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0590 - acc: 0.9777 - val_loss: 0.0320 - val_acc: 0.9849\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0570 - acc: 0.9777 - val_loss: 0.1122 - val_acc: 0.9632\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0505 - acc: 0.9794 - val_loss: 0.0572 - val_acc: 0.9783\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 258s 223ms/step - loss: 0.0478 - acc: 0.9800 - val_loss: 0.0599 - val_acc: 0.9761\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 0.0463 - acc: 0.9797 - val_loss: 0.0322 - val_acc: 0.9834\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0423 - acc: 0.9818 - val_loss: 0.0380 - val_acc: 0.9854\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0414 - acc: 0.9820 - val_loss: 0.0540 - val_acc: 0.9783\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0392 - acc: 0.9828 - val_loss: 0.0280 - val_acc: 0.9890\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0379 - acc: 0.9826 - val_loss: 0.0351 - val_acc: 0.9846\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0377 - acc: 0.9828 - val_loss: 0.0617 - val_acc: 0.9756\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 257s 223ms/step - loss: 0.0357 - acc: 0.9837 - val_loss: 0.0453 - val_acc: 0.9781\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0332 - acc: 0.9840 - val_loss: 0.0268 - val_acc: 0.9883\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0344 - acc: 0.9832 - val_loss: 0.0240 - val_acc: 0.9895\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0347 - acc: 0.9837 - val_loss: 0.0232 - val_acc: 0.9895\n",
      "36905/36905 [==============================] - 57s 2ms/step\n",
      "Train [0.02768385387955215, 0.986316217314727]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.026907147270874593, 0.9877097151775264]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 0.4574 - acc: 0.8751 - val_loss: 0.1514 - val_acc: 0.9507\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 249s 215ms/step - loss: 0.1383 - acc: 0.9598 - val_loss: 0.1349 - val_acc: 0.9464\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0939 - acc: 0.9695 - val_loss: 0.0643 - val_acc: 0.9805\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0774 - acc: 0.9728 - val_loss: 0.1034 - val_acc: 0.9605\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0667 - acc: 0.9758 - val_loss: 0.0545 - val_acc: 0.9795\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.0598 - acc: 0.9772 - val_loss: 0.0666 - val_acc: 0.9705\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 0.0544 - acc: 0.9784 - val_loss: 0.0542 - val_acc: 0.9815\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 268s 233ms/step - loss: 0.0507 - acc: 0.9797 - val_loss: 0.0444 - val_acc: 0.9820\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0451 - acc: 0.9811 - val_loss: 0.0404 - val_acc: 0.9878\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0438 - acc: 0.9813 - val_loss: 0.0536 - val_acc: 0.9768\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 0.0446 - acc: 0.9813 - val_loss: 0.0398 - val_acc: 0.9795\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0400 - acc: 0.9821 - val_loss: 0.0314 - val_acc: 0.9837\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 257s 222ms/step - loss: 0.0383 - acc: 0.9828 - val_loss: 0.0411 - val_acc: 0.9817\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 260s 226ms/step - loss: 0.0393 - acc: 0.9820 - val_loss: 0.0853 - val_acc: 0.9720\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 0.0362 - acc: 0.9830 - val_loss: 0.0432 - val_acc: 0.9800\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 0.0349 - acc: 0.9833 - val_loss: 0.0565 - val_acc: 0.9739\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 0.0351 - acc: 0.9839 - val_loss: 0.1173 - val_acc: 0.9646\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0326 - acc: 0.9846 - val_loss: 0.0341 - val_acc: 0.9837\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 267s 232ms/step - loss: 0.0333 - acc: 0.9841 - val_loss: 0.0805 - val_acc: 0.9690\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0342 - acc: 0.9834 - val_loss: 0.0277 - val_acc: 0.9878\n",
      "36905/36905 [==============================] - 59s 2ms/step\n",
      "Train [0.032402126127159536, 0.9843110689608454]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.03621767774901334, 0.9845883730003901]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 0.4756 - acc: 0.8718 - val_loss: 0.1696 - val_acc: 0.9454\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.1361 - acc: 0.9592 - val_loss: 0.0813 - val_acc: 0.9734\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.0925 - acc: 0.9697 - val_loss: 0.0814 - val_acc: 0.9717\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 0.0737 - acc: 0.9750 - val_loss: 0.0548 - val_acc: 0.9815\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0638 - acc: 0.9772 - val_loss: 0.1606 - val_acc: 0.9573\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0563 - acc: 0.9788 - val_loss: 0.0593 - val_acc: 0.9761\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0540 - acc: 0.9783 - val_loss: 0.0499 - val_acc: 0.9810\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0497 - acc: 0.9802 - val_loss: 0.1129 - val_acc: 0.9583\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0453 - acc: 0.9807 - val_loss: 0.0296 - val_acc: 0.9856\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0418 - acc: 0.9822 - val_loss: 0.0731 - val_acc: 0.9715\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0421 - acc: 0.9817 - val_loss: 0.0551 - val_acc: 0.9805\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0399 - acc: 0.9821 - val_loss: 0.0822 - val_acc: 0.9703\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0399 - acc: 0.9822 - val_loss: 0.1412 - val_acc: 0.9639\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0343 - acc: 0.9841 - val_loss: 0.0820 - val_acc: 0.9688\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0358 - acc: 0.9829 - val_loss: 0.0845 - val_acc: 0.9695\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0337 - acc: 0.9837 - val_loss: 0.0397 - val_acc: 0.9854\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 257s 222ms/step - loss: 0.0348 - acc: 0.9836 - val_loss: 0.0266 - val_acc: 0.9888\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0358 - acc: 0.9827 - val_loss: 0.0447 - val_acc: 0.9820\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0317 - acc: 0.9841 - val_loss: 0.0384 - val_acc: 0.9839\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 271s 234ms/step - loss: 0.0336 - acc: 0.9835 - val_loss: 0.1240 - val_acc: 0.9573\n",
      "36905/36905 [==============================] - 63s 2ms/step\n",
      "Train [0.14695382364545748, 0.9512261211217992]\n",
      "10252/10252 [==============================] - 18s 2ms/step\n",
      "Test [0.14481789311063425, 0.9500585251658213]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 156s 135ms/step - loss: 0.9639 - acc: 0.7583 - val_loss: 0.3909 - val_acc: 0.8856\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.2549 - acc: 0.9411 - val_loss: 0.1682 - val_acc: 0.9483\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.1432 - acc: 0.9636 - val_loss: 0.0891 - val_acc: 0.9754\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0982 - acc: 0.9712 - val_loss: 0.0735 - val_acc: 0.9768\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 139s 121ms/step - loss: 0.0799 - acc: 0.9745 - val_loss: 0.0502 - val_acc: 0.9822\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 139s 121ms/step - loss: 0.0681 - acc: 0.9764 - val_loss: 0.0536 - val_acc: 0.9817\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0565 - acc: 0.9790 - val_loss: 0.0392 - val_acc: 0.9849\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 138s 119ms/step - loss: 0.0542 - acc: 0.9790 - val_loss: 0.0888 - val_acc: 0.9659\n",
      "Epoch 9/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0499 - acc: 0.9800 - val_loss: 0.1125 - val_acc: 0.9600\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 139s 121ms/step - loss: 0.0460 - acc: 0.9811 - val_loss: 0.0952 - val_acc: 0.9595\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 140s 121ms/step - loss: 0.0444 - acc: 0.9813 - val_loss: 0.0308 - val_acc: 0.9881\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0414 - acc: 0.9821 - val_loss: 0.0619 - val_acc: 0.9778\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0408 - acc: 0.9827 - val_loss: 0.0437 - val_acc: 0.9817\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 0.0366 - acc: 0.9834 - val_loss: 0.0403 - val_acc: 0.9837\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 138s 119ms/step - loss: 0.0390 - acc: 0.9825 - val_loss: 0.0779 - val_acc: 0.9729\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 139s 121ms/step - loss: 0.0347 - acc: 0.9839 - val_loss: 0.0268 - val_acc: 0.9890\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 140s 122ms/step - loss: 0.0346 - acc: 0.9841 - val_loss: 0.0307 - val_acc: 0.9881\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0322 - acc: 0.9844 - val_loss: 0.0266 - val_acc: 0.9883\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0330 - acc: 0.9839 - val_loss: 0.0342 - val_acc: 0.9866\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0326 - acc: 0.9844 - val_loss: 0.0302 - val_acc: 0.9871\n",
      "36905/36905 [==============================] - 38s 1ms/step\n",
      "Train [0.033145060165065535, 0.9856658989296844]\n",
      "10252/10252 [==============================] - 10s 1ms/step\n",
      "Test [0.03471149894353195, 0.9844908310573547]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.8748 - acc: 0.7866 - val_loss: 0.3036 - val_acc: 0.9259\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 0.2264 - acc: 0.9485 - val_loss: 0.1765 - val_acc: 0.9442\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.1292 - acc: 0.9669 - val_loss: 0.0878 - val_acc: 0.9763\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0884 - acc: 0.9744 - val_loss: 0.1169 - val_acc: 0.9615\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 139s 121ms/step - loss: 0.0716 - acc: 0.9764 - val_loss: 0.0417 - val_acc: 0.9837\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0627 - acc: 0.9778 - val_loss: 0.0427 - val_acc: 0.9856\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0554 - acc: 0.9789 - val_loss: 0.0522 - val_acc: 0.9768\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0487 - acc: 0.9810 - val_loss: 0.0641 - val_acc: 0.9712\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0450 - acc: 0.9817 - val_loss: 0.1064 - val_acc: 0.9585\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0420 - acc: 0.9827 - val_loss: 0.0340 - val_acc: 0.9834\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0431 - acc: 0.9806 - val_loss: 0.0397 - val_acc: 0.9829\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0392 - acc: 0.9824 - val_loss: 0.0230 - val_acc: 0.9876\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0354 - acc: 0.9839 - val_loss: 0.0321 - val_acc: 0.9856\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0399 - acc: 0.9825 - val_loss: 0.0478 - val_acc: 0.9812\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0331 - acc: 0.9843 - val_loss: 0.0282 - val_acc: 0.9844\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0329 - acc: 0.9843 - val_loss: 0.0238 - val_acc: 0.9871\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 140s 121ms/step - loss: 0.0338 - acc: 0.9840 - val_loss: 0.0301 - val_acc: 0.9846\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.0333 - acc: 0.9839 - val_loss: 0.0300 - val_acc: 0.9839\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.0319 - acc: 0.9844 - val_loss: 0.0234 - val_acc: 0.9893\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.0316 - acc: 0.9844 - val_loss: 0.0303 - val_acc: 0.9856\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [0.03332791694984901, 0.9827123695976155]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.032212947478248634, 0.9841006632852126]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 150s 130ms/step - loss: 0.8604 - acc: 0.7918 - val_loss: 0.2916 - val_acc: 0.9307\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.2230 - acc: 0.9485 - val_loss: 0.1567 - val_acc: 0.9459\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.1249 - acc: 0.9677 - val_loss: 0.1022 - val_acc: 0.9678\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0916 - acc: 0.9732 - val_loss: 0.0516 - val_acc: 0.9805\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0715 - acc: 0.9764 - val_loss: 0.0531 - val_acc: 0.9842\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 140s 121ms/step - loss: 0.0631 - acc: 0.9777 - val_loss: 0.0474 - val_acc: 0.9842\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.0558 - acc: 0.9791 - val_loss: 0.0352 - val_acc: 0.9871\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 143s 124ms/step - loss: 0.0489 - acc: 0.9817 - val_loss: 0.0426 - val_acc: 0.9842\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0470 - acc: 0.9813 - val_loss: 0.0429 - val_acc: 0.9800\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.0453 - acc: 0.9809 - val_loss: 0.1367 - val_acc: 0.9544\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.0406 - acc: 0.9828 - val_loss: 0.0593 - val_acc: 0.9751\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.0393 - acc: 0.9823 - val_loss: 0.0327 - val_acc: 0.9876\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0373 - acc: 0.9835 - val_loss: 0.0386 - val_acc: 0.9817\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 138s 119ms/step - loss: 0.0396 - acc: 0.9822 - val_loss: 0.1960 - val_acc: 0.9349\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 139s 121ms/step - loss: 0.0366 - acc: 0.9825 - val_loss: 0.0313 - val_acc: 0.9861\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0344 - acc: 0.9837 - val_loss: 0.0257 - val_acc: 0.9863\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0344 - acc: 0.9843 - val_loss: 0.0659 - val_acc: 0.9759\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0314 - acc: 0.9850 - val_loss: 0.0269 - val_acc: 0.9885\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0323 - acc: 0.9839 - val_loss: 0.0264 - val_acc: 0.9876\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.0303 - acc: 0.9851 - val_loss: 0.0339 - val_acc: 0.9832\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [0.03611897223570952, 0.9829833355913833]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.03764053886722507, 0.981271946937183]\n"
     ]
    }
   ],
   "source": [
    "import keras.optimizers\n",
    "for lr in [0.1, 0.01, 0.001, 0.0001]:\n",
    "    optimizer = optimizers.Adam(lr=lr)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH LAYER BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetA((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH IDENTITY BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetB((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark Fruits-360 with Model 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 130s 113ms/step - loss: 14.0752 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0863 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0852 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0863 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0867 - acc: 0.1260 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "36905/36905 [==============================] - 36s 976us/step\n",
      "Train [14.084607949674881, 0.12616176669827936]\n",
      "10252/10252 [==============================] - 10s 975us/step\n",
      "Test [14.071103506054769, 0.12699960983804182]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 130s 113ms/step - loss: 15.8958 - acc: 0.0131 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 15.9083 - acc: 0.0130 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 15.9095 - acc: 0.0129 - val_loss: 15.9452 - val_acc: 0.0107\n",
      "36905/36905 [==============================] - 37s 997us/step\n",
      "Train [15.909330917377003, 0.012952174502099987]\n",
      "10252/10252 [==============================] - 10s 986us/step\n",
      "Test [15.907421795223264, 0.013070620366757706]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 132s 114ms/step - loss: 15.7063 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.7159 - acc: 0.0250 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.7159 - acc: 0.0250 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 127s 110ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.7159 - acc: 0.0250 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 15.7159 - acc: 0.0250 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 15.7170 - acc: 0.0249 - val_loss: 15.7447 - val_acc: 0.0232\n",
      "36905/36905 [==============================] - 36s 971us/step\n",
      "Train [15.716726040604346, 0.02490177482725918]\n",
      "10252/10252 [==============================] - 10s 969us/step\n",
      "Test [15.649582486667997, 0.02906749902458057]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 130s 113ms/step - loss: 3.6239 - acc: 0.1229 - val_loss: 3.6197 - val_acc: 0.1231\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 3.6067 - acc: 0.1247 - val_loss: 3.6461 - val_acc: 0.1231\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 3.6064 - acc: 0.1240 - val_loss: 3.6151 - val_acc: 0.1231\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 3.6058 - acc: 0.1239 - val_loss: 3.6904 - val_acc: 0.1231\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 3.6081 - acc: 0.1227 - val_loss: 3.6074 - val_acc: 0.1231\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 3.6076 - acc: 0.1222 - val_loss: 3.6315 - val_acc: 0.1231\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 3.6087 - acc: 0.1233 - val_loss: 3.6292 - val_acc: 0.1231\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 3.6071 - acc: 0.1232 - val_loss: 3.6154 - val_acc: 0.1231\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 3.6228 - acc: 0.1206 - val_loss: 3.6137 - val_acc: 0.1231\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 3.6008 - acc: 0.1240 - val_loss: 3.6125 - val_acc: 0.1231\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 3.6057 - acc: 0.1234 - val_loss: 3.6520 - val_acc: 0.1231\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 3.6078 - acc: 0.1228 - val_loss: 3.6199 - val_acc: 0.1231\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 3.6087 - acc: 0.1233 - val_loss: 3.6168 - val_acc: 0.1231\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 3.6083 - acc: 0.1226 - val_loss: 3.6240 - val_acc: 0.1231\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 3.6084 - acc: 0.1228 - val_loss: 3.6100 - val_acc: 0.1231\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 3.6114 - acc: 0.1219 - val_loss: 3.6205 - val_acc: 0.1231\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 3.6052 - acc: 0.1223 - val_loss: 3.6295 - val_acc: 0.1231\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 3.6096 - acc: 0.1225 - val_loss: 3.6180 - val_acc: 0.1231\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 3.6068 - acc: 0.1231 - val_loss: 3.6164 - val_acc: 0.1231\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 3.6085 - acc: 0.1235 - val_loss: 3.6605 - val_acc: 0.0851\n",
      "36905/36905 [==============================] - 36s 967us/step\n",
      "Train [3.6486042865390877, 0.09136973309849614]\n",
      "10252/10252 [==============================] - 10s 967us/step\n",
      "Test [3.6464270270061454, 0.0910066328550334]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 130s 113ms/step - loss: 14.0742 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0814 - acc: 0.1264 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0803 - acc: 0.1264 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "36905/36905 [==============================] - 36s 981us/step\n",
      "Train [14.084607949674881, 0.12616176669827936]\n",
      "10252/10252 [==============================] - 10s 991us/step\n",
      "Test [14.071103506054769, 0.12699960983804182]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 131s 114ms/step - loss: 14.0746 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 8/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0825 - acc: 0.1263 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0859 - acc: 0.1261 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 14.0848 - acc: 0.1262 - val_loss: 14.1333 - val_acc: 0.1231\n",
      "36905/36905 [==============================] - 36s 979us/step\n",
      "Train [14.084607949674881, 0.12616176669827936]\n",
      "10252/10252 [==============================] - 10s 978us/step\n",
      "Test [14.071103506054769, 0.12699960983804182]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 131s 114ms/step - loss: 0.8849 - acc: 0.7182 - val_loss: 0.2885 - val_acc: 0.9061\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.3677 - acc: 0.8963 - val_loss: 0.4985 - val_acc: 0.9122\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.3414 - acc: 0.9288 - val_loss: 0.3806 - val_acc: 0.9334\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2782 - acc: 0.9478 - val_loss: 0.2581 - val_acc: 0.9532\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.3356 - acc: 0.9467 - val_loss: 0.2238 - val_acc: 0.9615\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.2546 - acc: 0.9556 - val_loss: 0.2839 - val_acc: 0.9464\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2449 - acc: 0.9593 - val_loss: 0.2280 - val_acc: 0.9654\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2845 - acc: 0.9532 - val_loss: 0.2792 - val_acc: 0.9522\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2145 - acc: 0.9680 - val_loss: 0.2057 - val_acc: 0.9683\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2282 - acc: 0.9642 - val_loss: 0.2092 - val_acc: 0.9700\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2332 - acc: 0.9652 - val_loss: 0.2021 - val_acc: 0.9749\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2317 - acc: 0.9630 - val_loss: 0.2536 - val_acc: 0.9566\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2305 - acc: 0.9638 - val_loss: 0.3658 - val_acc: 0.9432\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2256 - acc: 0.9647 - val_loss: 0.2721 - val_acc: 0.9473\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2048 - acc: 0.9717 - val_loss: 0.2088 - val_acc: 0.9722\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2172 - acc: 0.9671 - val_loss: 0.1964 - val_acc: 0.9751\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2089 - acc: 0.9710 - val_loss: 0.1927 - val_acc: 0.9729\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2256 - acc: 0.9660 - val_loss: 0.1946 - val_acc: 0.9781\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.1975 - acc: 0.9734 - val_loss: 0.3445 - val_acc: 0.9293\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 127s 110ms/step - loss: 0.2203 - acc: 0.9668 - val_loss: 0.1968 - val_acc: 0.9700\n",
      "36905/36905 [==============================] - 38s 1ms/step\n",
      "Train [0.2133545787838894, 0.9681885923316623]\n",
      "10252/10252 [==============================] - 10s 1ms/step\n",
      "Test [0.16501337597470345, 0.9728833398361295]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 1.1093 - acc: 0.6496 - val_loss: 0.4821 - val_acc: 0.8469\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2543 - acc: 0.9115 - val_loss: 0.1580 - val_acc: 0.9427\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.1536 - acc: 0.9439 - val_loss: 0.1206 - val_acc: 0.9546\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.1354 - acc: 0.9539 - val_loss: 1.1410 - val_acc: 0.7518\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.1087 - acc: 0.9608 - val_loss: 0.1052 - val_acc: 0.9595\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0925 - acc: 0.9648 - val_loss: 0.0601 - val_acc: 0.9771\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1059 - acc: 0.9612 - val_loss: 0.0858 - val_acc: 0.9649\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0713 - acc: 0.9715 - val_loss: 0.0888 - val_acc: 0.9654\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0781 - acc: 0.9692 - val_loss: 0.0438 - val_acc: 0.9783\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0754 - acc: 0.9717 - val_loss: 0.0564 - val_acc: 0.9810\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0615 - acc: 0.9748 - val_loss: 0.0409 - val_acc: 0.9844\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0863 - acc: 0.9699 - val_loss: 0.0404 - val_acc: 0.9798\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0673 - acc: 0.9729 - val_loss: 0.2408 - val_acc: 0.9208\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.0700 - acc: 0.9737 - val_loss: 0.0508 - val_acc: 0.9783\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0659 - acc: 0.9745 - val_loss: 0.0609 - val_acc: 0.9759\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0587 - acc: 0.9762 - val_loss: 0.0870 - val_acc: 0.9661\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0548 - acc: 0.9771 - val_loss: 0.1194 - val_acc: 0.9537\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0575 - acc: 0.9758 - val_loss: 0.0463 - val_acc: 0.9776\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0669 - acc: 0.9750 - val_loss: 0.1050 - val_acc: 0.9656\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.0571 - acc: 0.9766 - val_loss: 0.0428 - val_acc: 0.9790\n",
      "36905/36905 [==============================] - 36s 987us/step\n",
      "Train [0.04825874441077453, 0.9774827259178973]\n",
      "10252/10252 [==============================] - 10s 988us/step\n",
      "Test [0.046354854656666826, 0.9772727272727273]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 1.1524 - acc: 0.6339 - val_loss: 0.3288 - val_acc: 0.8891\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.3055 - acc: 0.8943 - val_loss: 0.2809 - val_acc: 0.9076\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.3324 - acc: 0.9228 - val_loss: 0.6230 - val_acc: 0.9017\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.3828 - acc: 0.9320 - val_loss: 0.3218 - val_acc: 0.9295\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2871 - acc: 0.9447 - val_loss: 0.3252 - val_acc: 0.9386\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2681 - acc: 0.9501 - val_loss: 0.4327 - val_acc: 0.9098\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2565 - acc: 0.9545 - val_loss: 0.2877 - val_acc: 0.9500\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.2571 - acc: 0.9541 - val_loss: 0.3254 - val_acc: 0.9466\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2460 - acc: 0.9579 - val_loss: 0.2735 - val_acc: 0.9595\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.2568 - acc: 0.9573 - val_loss: 0.2452 - val_acc: 0.9639\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2391 - acc: 0.9612 - val_loss: 0.2411 - val_acc: 0.9642\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2348 - acc: 0.9618 - val_loss: 0.2444 - val_acc: 0.9583\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2317 - acc: 0.9622 - val_loss: 0.3139 - val_acc: 0.9415\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2298 - acc: 0.9628 - val_loss: 0.2226 - val_acc: 0.9678\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.2345 - acc: 0.9619 - val_loss: 0.2885 - val_acc: 0.9505\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2311 - acc: 0.9625 - val_loss: 0.2225 - val_acc: 0.9673\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2255 - acc: 0.9651 - val_loss: 0.3151 - val_acc: 0.9459\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2108 - acc: 0.9690 - val_loss: 0.3080 - val_acc: 0.9485\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.2319 - acc: 0.9628 - val_loss: 0.2351 - val_acc: 0.9607\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.2187 - acc: 0.9673 - val_loss: 0.2089 - val_acc: 0.9690\n",
      "36905/36905 [==============================] - 36s 981us/step\n",
      "Train [0.21873937348805897, 0.9662647337759112]\n",
      "10252/10252 [==============================] - 10s 979us/step\n",
      "Test [0.1730658196513106, 0.9701521654311354]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 1.5994 - acc: 0.5261 - val_loss: 0.8544 - val_acc: 0.7276\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.6272 - acc: 0.8002 - val_loss: 0.5822 - val_acc: 0.8000\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.4261 - acc: 0.8564 - val_loss: 0.4626 - val_acc: 0.8495\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.3313 - acc: 0.8851 - val_loss: 0.4024 - val_acc: 0.8578\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.2734 - acc: 0.9049 - val_loss: 0.2749 - val_acc: 0.8969\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.2447 - acc: 0.9143 - val_loss: 0.3104 - val_acc: 0.8981\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.1972 - acc: 0.9301 - val_loss: 0.2121 - val_acc: 0.9193\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.1827 - acc: 0.9338 - val_loss: 0.1803 - val_acc: 0.9356\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1612 - acc: 0.9434 - val_loss: 0.2127 - val_acc: 0.9220\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1500 - acc: 0.9481 - val_loss: 0.1240 - val_acc: 0.9559\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 125s 108ms/step - loss: 0.1396 - acc: 0.9511 - val_loss: 0.0983 - val_acc: 0.9624\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1227 - acc: 0.9559 - val_loss: 0.1268 - val_acc: 0.9542\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.1126 - acc: 0.9593 - val_loss: 0.1648 - val_acc: 0.9439\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.1115 - acc: 0.9607 - val_loss: 0.0894 - val_acc: 0.9695\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.1005 - acc: 0.9628 - val_loss: 0.1006 - val_acc: 0.9629\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0919 - acc: 0.9666 - val_loss: 0.0985 - val_acc: 0.9622\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.0970 - acc: 0.9655 - val_loss: 0.1024 - val_acc: 0.9612\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0783 - acc: 0.9710 - val_loss: 0.1304 - val_acc: 0.9583\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0753 - acc: 0.9725 - val_loss: 0.0902 - val_acc: 0.9661\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.0751 - acc: 0.9720 - val_loss: 0.0572 - val_acc: 0.9756\n",
      "36905/36905 [==============================] - 37s 995us/step\n",
      "Train [0.06065219857881007, 0.975314997967755]\n",
      "10252/10252 [==============================] - 10s 998us/step\n",
      "Test [0.06277941602056572, 0.9743464689816621]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 1.0720 - acc: 0.6685 - val_loss: 0.6511 - val_acc: 0.7918\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.4375 - acc: 0.8550 - val_loss: 0.4494 - val_acc: 0.8420\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 127s 110ms/step - loss: 0.3068 - acc: 0.8942 - val_loss: 0.3784 - val_acc: 0.8710\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.2535 - acc: 0.9093 - val_loss: 0.2475 - val_acc: 0.9064\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.2055 - acc: 0.9251 - val_loss: 0.3489 - val_acc: 0.8864\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1874 - acc: 0.9322 - val_loss: 0.2741 - val_acc: 0.8956\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 126s 110ms/step - loss: 0.1618 - acc: 0.9423 - val_loss: 0.2860 - val_acc: 0.9005\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1500 - acc: 0.9450 - val_loss: 0.2010 - val_acc: 0.9293\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1362 - acc: 0.9522 - val_loss: 0.1440 - val_acc: 0.9407\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 127s 110ms/step - loss: 0.1235 - acc: 0.9558 - val_loss: 0.2187 - val_acc: 0.9266\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1187 - acc: 0.9579 - val_loss: 0.1633 - val_acc: 0.9405\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1079 - acc: 0.9617 - val_loss: 0.2671 - val_acc: 0.9329\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1052 - acc: 0.9618 - val_loss: 0.1865 - val_acc: 0.9327\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0927 - acc: 0.9651 - val_loss: 0.1051 - val_acc: 0.9629\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0881 - acc: 0.9672 - val_loss: 0.0784 - val_acc: 0.9666\n",
      "Epoch 16/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0934 - acc: 0.9653 - val_loss: 0.1198 - val_acc: 0.9554\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 126s 110ms/step - loss: 0.0789 - acc: 0.9704 - val_loss: 0.1240 - val_acc: 0.9622\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0804 - acc: 0.9709 - val_loss: 0.0668 - val_acc: 0.9712\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 125s 109ms/step - loss: 0.0757 - acc: 0.9719 - val_loss: 0.0907 - val_acc: 0.9663\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0731 - acc: 0.9716 - val_loss: 0.0786 - val_acc: 0.9703\n",
      "36905/36905 [==============================] - 36s 982us/step\n",
      "Train [0.09114295683952336, 0.9656144153908685]\n",
      "10252/10252 [==============================] - 10s 982us/step\n",
      "Test [0.09284801713881016, 0.9646898166444029]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.9475 - acc: 0.7070 - val_loss: 0.5586 - val_acc: 0.8183\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.3903 - acc: 0.8696 - val_loss: 0.3425 - val_acc: 0.8713\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.2872 - acc: 0.9015 - val_loss: 0.3367 - val_acc: 0.8815\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.2330 - acc: 0.9190 - val_loss: 0.2375 - val_acc: 0.9100\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 130s 113ms/step - loss: 0.1949 - acc: 0.9304 - val_loss: 0.1571 - val_acc: 0.9417\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1725 - acc: 0.9382 - val_loss: 0.1722 - val_acc: 0.9288\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1565 - acc: 0.9447 - val_loss: 0.3096 - val_acc: 0.9315\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1678 - acc: 0.9497 - val_loss: 0.1017 - val_acc: 0.9649\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1284 - acc: 0.9547 - val_loss: 0.1367 - val_acc: 0.9456\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.1149 - acc: 0.9590 - val_loss: 0.1095 - val_acc: 0.9639\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 127s 110ms/step - loss: 0.1080 - acc: 0.9614 - val_loss: 0.0809 - val_acc: 0.9705\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0999 - acc: 0.9643 - val_loss: 0.1091 - val_acc: 0.9581\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0923 - acc: 0.9676 - val_loss: 0.1048 - val_acc: 0.9598\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0892 - acc: 0.9664 - val_loss: 0.0615 - val_acc: 0.9781\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0915 - acc: 0.9685 - val_loss: 0.0928 - val_acc: 0.9649\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 126s 110ms/step - loss: 0.0764 - acc: 0.9721 - val_loss: 0.1484 - val_acc: 0.9420\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0725 - acc: 0.9723 - val_loss: 0.0654 - val_acc: 0.9715\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 127s 110ms/step - loss: 0.0778 - acc: 0.9714 - val_loss: 0.0512 - val_acc: 0.9781\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0691 - acc: 0.9740 - val_loss: 0.0723 - val_acc: 0.9700\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 126s 109ms/step - loss: 0.0659 - acc: 0.9755 - val_loss: 0.0597 - val_acc: 0.9722\n",
      "36905/36905 [==============================] - 37s 999us/step\n",
      "Train [0.0668689814559919, 0.9680802059341552]\n",
      "10252/10252 [==============================] - 10s 997us/step\n",
      "Test [0.0671869845596846, 0.9681037846273898]\n"
     ]
    }
   ],
   "source": [
    "import keras.optimizers\n",
    "for lr in [0.1, 0.01, 0.001, 0.0001]:\n",
    "    optimizer = optimizers.Adam(lr=lr)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITHOUT BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetC((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark Fruits-360 with modified Model 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 152s 132ms/step - loss: 2.2398 - acc: 0.5031 - val_loss: 5.2279 - val_acc: 0.4172\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 5.8521 - acc: 0.5582 - val_loss: 11.5262 - val_acc: 0.2770\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 7.0004 - acc: 0.5518 - val_loss: 5.9531 - val_acc: 0.6194\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 6.7228 - acc: 0.5753 - val_loss: 6.4827 - val_acc: 0.5935\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 6.7868 - acc: 0.5737 - val_loss: 8.7041 - val_acc: 0.4557\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 6.8281 - acc: 0.5722 - val_loss: 8.4538 - val_acc: 0.4728\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 7.1023 - acc: 0.5563 - val_loss: 6.9259 - val_acc: 0.5686\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 7.4284 - acc: 0.5365 - val_loss: 7.4789 - val_acc: 0.5343\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 7.7556 - acc: 0.5166 - val_loss: 8.3169 - val_acc: 0.4833\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 7.7731 - acc: 0.5161 - val_loss: 8.9772 - val_acc: 0.4426\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 7.7165 - acc: 0.5193 - val_loss: 8.3199 - val_acc: 0.4828\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 7.0247 - acc: 0.5627 - val_loss: 9.6518 - val_acc: 0.4001\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 7.7297 - acc: 0.5189 - val_loss: 5.9367 - val_acc: 0.6306\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 7.2482 - acc: 0.5491 - val_loss: 6.3203 - val_acc: 0.6069\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 7.1636 - acc: 0.5546 - val_loss: 7.0183 - val_acc: 0.5628\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 7.6157 - acc: 0.5261 - val_loss: 8.6999 - val_acc: 0.4596\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 7.4542 - acc: 0.5366 - val_loss: 7.8236 - val_acc: 0.5140\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 8.2760 - acc: 0.4856 - val_loss: 8.2297 - val_acc: 0.4887\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 7.8958 - acc: 0.5092 - val_loss: 7.7624 - val_acc: 0.5179\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 7.6162 - acc: 0.5266 - val_loss: 7.4444 - val_acc: 0.5374\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [7.349187420487259, 0.5435035902998212]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [7.269249289099983, 0.5484783456886461]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 152s 132ms/step - loss: 15.8905 - acc: 0.0134 - val_loss: 15.9255 - val_acc: 0.0119\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 15.8493 - acc: 0.0166 - val_loss: 15.3045 - val_acc: 0.0505\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 15.3726 - acc: 0.0463 - val_loss: 15.3045 - val_acc: 0.0505\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 15.3473 - acc: 0.0478 - val_loss: 15.3045 - val_acc: 0.0505\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 15.3468 - acc: 0.0478 - val_loss: 15.3045 - val_acc: 0.0505\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 15.3595 - acc: 0.0471 - val_loss: 15.3006 - val_acc: 0.0507\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 15.4138 - acc: 0.0437 - val_loss: 15.2967 - val_acc: 0.0510\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 15.3831 - acc: 0.0456 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.3503 - acc: 0.0476 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 15.3462 - acc: 0.0479 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 15.4183 - acc: 0.0434 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.4932 - acc: 0.0388 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.4943 - acc: 0.0387 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.4966 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 15.4954 - acc: 0.0386 - val_loss: 15.4775 - val_acc: 0.0397\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [15.49616947440238, 0.038585557512532176]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [15.519090647256519, 0.03716348029652751]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 160s 139ms/step - loss: 15.7966 - acc: 0.0191 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 144s 124ms/step - loss: 15.8117 - acc: 0.0190 - val_loss: 15.7683 - val_acc: 0.0217\n",
      "36905/36905 [==============================] - 39s 1ms/step\n",
      "Train [15.811499871084836, 0.01902181276270019]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [15.814662524047328, 0.018825595005852517]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 153s 132ms/step - loss: 0.8014 - acc: 0.7516 - val_loss: 0.4304 - val_acc: 0.8647\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.2801 - acc: 0.9023 - val_loss: 1.3304 - val_acc: 0.7015\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.1983 - acc: 0.9324 - val_loss: 0.3198 - val_acc: 0.8971\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.1647 - acc: 0.9409 - val_loss: 0.8819 - val_acc: 0.7725\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1364 - acc: 0.9512 - val_loss: 0.3852 - val_acc: 0.8869\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.1314 - acc: 0.9529 - val_loss: 0.1280 - val_acc: 0.9583\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.1147 - acc: 0.9582 - val_loss: 0.7183 - val_acc: 0.8666\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0982 - acc: 0.9642 - val_loss: 0.1689 - val_acc: 0.9427\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1026 - acc: 0.9639 - val_loss: 0.1992 - val_acc: 0.9415\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0956 - acc: 0.9644 - val_loss: 0.1132 - val_acc: 0.9559\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0846 - acc: 0.9683 - val_loss: 0.1981 - val_acc: 0.9368\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0874 - acc: 0.9673 - val_loss: 0.0503 - val_acc: 0.9776\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0768 - acc: 0.9698 - val_loss: 0.1936 - val_acc: 0.9542\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0757 - acc: 0.9712 - val_loss: 0.2806 - val_acc: 0.9212\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0686 - acc: 0.9729 - val_loss: 0.0377 - val_acc: 0.9851\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0752 - acc: 0.9724 - val_loss: 0.0593 - val_acc: 0.9768\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0629 - acc: 0.9757 - val_loss: 0.0556 - val_acc: 0.9768\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0677 - acc: 0.9735 - val_loss: 0.3066 - val_acc: 0.9256\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0851 - acc: 0.9714 - val_loss: 0.0277 - val_acc: 0.9846\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0585 - acc: 0.9761 - val_loss: 0.0977 - val_acc: 0.9629\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [0.096748575991985, 0.9622273404687711]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.09839793541861122, 0.9607881389229826]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 153s 133ms/step - loss: 1.2133 - acc: 0.6322 - val_loss: 0.5518 - val_acc: 0.8176\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.3370 - acc: 0.8835 - val_loss: 0.4530 - val_acc: 0.8571\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.2262 - acc: 0.9210 - val_loss: 0.1642 - val_acc: 0.9432\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.1730 - acc: 0.9381 - val_loss: 0.1840 - val_acc: 0.9344\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1521 - acc: 0.9452 - val_loss: 0.2064 - val_acc: 0.9303\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1309 - acc: 0.9511 - val_loss: 0.1083 - val_acc: 0.9578\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1147 - acc: 0.9572 - val_loss: 0.2659 - val_acc: 0.9151\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0987 - acc: 0.9623 - val_loss: 0.0904 - val_acc: 0.9734\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.1016 - acc: 0.9619 - val_loss: 0.2305 - val_acc: 0.9303\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0954 - acc: 0.9632 - val_loss: 0.5335 - val_acc: 0.8856\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0822 - acc: 0.9677 - val_loss: 0.0623 - val_acc: 0.9746\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0881 - acc: 0.9673 - val_loss: 0.0684 - val_acc: 0.9761\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0766 - acc: 0.9705 - val_loss: 0.1492 - val_acc: 0.9478\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0728 - acc: 0.9716 - val_loss: 0.0748 - val_acc: 0.9644\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0793 - acc: 0.9702 - val_loss: 0.1774 - val_acc: 0.9525\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0728 - acc: 0.9722 - val_loss: 0.0979 - val_acc: 0.9663\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0657 - acc: 0.9740 - val_loss: 0.1049 - val_acc: 0.9671\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0760 - acc: 0.9707 - val_loss: 0.0357 - val_acc: 0.9839\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 145s 125ms/step - loss: 0.0628 - acc: 0.9755 - val_loss: 0.3457 - val_acc: 0.9449\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0665 - acc: 0.9742 - val_loss: 0.0458 - val_acc: 0.9783\n",
      "36905/36905 [==============================] - 39s 1ms/step\n",
      "Train [0.045020103803087415, 0.9795149708711557]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.04470253674424564, 0.980491611392899]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 154s 134ms/step - loss: 1.1531 - acc: 0.6607 - val_loss: 0.7818 - val_acc: 0.7420\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.3595 - acc: 0.8774 - val_loss: 0.3858 - val_acc: 0.8593\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.2500 - acc: 0.9129 - val_loss: 0.5261 - val_acc: 0.8374\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1941 - acc: 0.9295 - val_loss: 0.1402 - val_acc: 0.9503\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1624 - acc: 0.9403 - val_loss: 0.1170 - val_acc: 0.9564\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1368 - acc: 0.9509 - val_loss: 0.3295 - val_acc: 0.8827\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1289 - acc: 0.9521 - val_loss: 0.1965 - val_acc: 0.9325\n",
      "Epoch 8/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.1198 - acc: 0.9553 - val_loss: 0.1202 - val_acc: 0.9617\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1018 - acc: 0.9625 - val_loss: 0.1419 - val_acc: 0.9593\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1041 - acc: 0.9608 - val_loss: 0.1534 - val_acc: 0.9500\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0973 - acc: 0.9633 - val_loss: 0.0736 - val_acc: 0.9742\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0839 - acc: 0.9677 - val_loss: 0.1091 - val_acc: 0.9637\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0903 - acc: 0.9656 - val_loss: 0.1152 - val_acc: 0.9588\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0855 - acc: 0.9682 - val_loss: 0.0337 - val_acc: 0.9861\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0767 - acc: 0.9693 - val_loss: 0.2800 - val_acc: 0.9127\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0728 - acc: 0.9715 - val_loss: 0.1137 - val_acc: 0.9637\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0754 - acc: 0.9708 - val_loss: 0.0326 - val_acc: 0.9844\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0744 - acc: 0.9710 - val_loss: 0.2903 - val_acc: 0.9266\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0696 - acc: 0.9736 - val_loss: 0.0756 - val_acc: 0.9727\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0635 - acc: 0.9743 - val_loss: 0.1072 - val_acc: 0.9649\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [0.12738671653487538, 0.9601138057173825]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.1338853989395168, 0.9602028872647697]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 154s 134ms/step - loss: 0.5350 - acc: 0.8510 - val_loss: 0.2087 - val_acc: 0.9200\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.1254 - acc: 0.9567 - val_loss: 0.0666 - val_acc: 0.9800\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0877 - acc: 0.9670 - val_loss: 0.0767 - val_acc: 0.9661\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0705 - acc: 0.9713 - val_loss: 0.0344 - val_acc: 0.9829\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0678 - acc: 0.9718 - val_loss: 0.0363 - val_acc: 0.9871\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0501 - acc: 0.9781 - val_loss: 0.0631 - val_acc: 0.9761\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0520 - acc: 0.9777 - val_loss: 0.0794 - val_acc: 0.9649\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0493 - acc: 0.9788 - val_loss: 0.0453 - val_acc: 0.9778\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0421 - acc: 0.9806 - val_loss: 0.0623 - val_acc: 0.9729\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0493 - acc: 0.9786 - val_loss: 0.1411 - val_acc: 0.9485\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0346 - acc: 0.9836 - val_loss: 0.1441 - val_acc: 0.9610\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0414 - acc: 0.9807 - val_loss: 0.0316 - val_acc: 0.9834\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0402 - acc: 0.9809 - val_loss: 0.0298 - val_acc: 0.9854\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0360 - acc: 0.9822 - val_loss: 0.0271 - val_acc: 0.9876\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0341 - acc: 0.9830 - val_loss: 0.0210 - val_acc: 0.9863\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0370 - acc: 0.9821 - val_loss: 0.0506 - val_acc: 0.9783\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0322 - acc: 0.9839 - val_loss: 0.0316 - val_acc: 0.9827\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0363 - acc: 0.9828 - val_loss: 0.0273 - val_acc: 0.9849\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0305 - acc: 0.9846 - val_loss: 0.0203 - val_acc: 0.9900\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0356 - acc: 0.9830 - val_loss: 0.0332 - val_acc: 0.9863\n",
      "36905/36905 [==============================] - 41s 1ms/step\n",
      "Train [0.037756674508193686, 0.9831730117870208]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.03878885454943498, 0.9832227857978931]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 155s 134ms/step - loss: 0.6215 - acc: 0.8121 - val_loss: 0.2705 - val_acc: 0.9132\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.1932 - acc: 0.9363 - val_loss: 0.3107 - val_acc: 0.9010\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.1151 - acc: 0.9609 - val_loss: 0.0554 - val_acc: 0.9790\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0898 - acc: 0.9672 - val_loss: 0.0512 - val_acc: 0.9812\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0813 - acc: 0.9689 - val_loss: 0.1069 - val_acc: 0.9522\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0641 - acc: 0.9733 - val_loss: 0.1173 - val_acc: 0.9590\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0647 - acc: 0.9748 - val_loss: 0.0586 - val_acc: 0.9805\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0567 - acc: 0.9766 - val_loss: 0.0280 - val_acc: 0.9868\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0497 - acc: 0.9786 - val_loss: 0.0516 - val_acc: 0.9761\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0484 - acc: 0.9792 - val_loss: 0.0356 - val_acc: 0.9849\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0493 - acc: 0.9781 - val_loss: 0.1042 - val_acc: 0.9607\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0460 - acc: 0.9800 - val_loss: 0.0358 - val_acc: 0.9824\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0423 - acc: 0.9812 - val_loss: 0.0279 - val_acc: 0.9883\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0405 - acc: 0.9820 - val_loss: 0.0656 - val_acc: 0.9768\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0438 - acc: 0.9804 - val_loss: 0.0306 - val_acc: 0.9859\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0369 - acc: 0.9826 - val_loss: 0.0426 - val_acc: 0.9783\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0379 - acc: 0.9821 - val_loss: 0.0307 - val_acc: 0.9861\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0379 - acc: 0.9821 - val_loss: 0.0295 - val_acc: 0.9866\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0353 - acc: 0.9832 - val_loss: 0.0373 - val_acc: 0.9829\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0372 - acc: 0.9822 - val_loss: 0.0424 - val_acc: 0.9802\n",
      "36905/36905 [==============================] - 41s 1ms/step\n",
      "Train [0.04935190539153067, 0.9784311068960846]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.049912643754362704, 0.9785407725321889]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 155s 134ms/step - loss: 0.6013 - acc: 0.8192 - val_loss: 0.2792 - val_acc: 0.9117\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.1818 - acc: 0.9416 - val_loss: 0.1352 - val_acc: 0.9566\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.1169 - acc: 0.9593 - val_loss: 0.2196 - val_acc: 0.9378\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0866 - acc: 0.9684 - val_loss: 0.0650 - val_acc: 0.9751\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0743 - acc: 0.9717 - val_loss: 0.0560 - val_acc: 0.9785\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0676 - acc: 0.9730 - val_loss: 0.0562 - val_acc: 0.9785\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0579 - acc: 0.9766 - val_loss: 0.0380 - val_acc: 0.9805\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0594 - acc: 0.9750 - val_loss: 0.0493 - val_acc: 0.9768\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0520 - acc: 0.9776 - val_loss: 0.2013 - val_acc: 0.9410\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0454 - acc: 0.9805 - val_loss: 0.0718 - val_acc: 0.9722\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0516 - acc: 0.9780 - val_loss: 0.0292 - val_acc: 0.9866\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0412 - acc: 0.9809 - val_loss: 0.3856 - val_acc: 0.9300\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0414 - acc: 0.9811 - val_loss: 0.0388 - val_acc: 0.9854\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0428 - acc: 0.9806 - val_loss: 0.0245 - val_acc: 0.9839\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0415 - acc: 0.9817 - val_loss: 0.1549 - val_acc: 0.9434\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0370 - acc: 0.9825 - val_loss: 0.0224 - val_acc: 0.9878\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0401 - acc: 0.9815 - val_loss: 0.0353 - val_acc: 0.9868\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0380 - acc: 0.9818 - val_loss: 0.0479 - val_acc: 0.9844\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 145s 126ms/step - loss: 0.0360 - acc: 0.9828 - val_loss: 0.0645 - val_acc: 0.9720\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0365 - acc: 0.9829 - val_loss: 0.0583 - val_acc: 0.9768\n",
      "36905/36905 [==============================] - 41s 1ms/step\n",
      "Train [0.062435409276677255, 0.97569435035903]\n",
      "10252/10252 [==============================] - 12s 1ms/step\n",
      "Test [0.06263859945151266, 0.9762973078423722]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 156s 135ms/step - loss: 1.2597 - acc: 0.7014 - val_loss: 0.4418 - val_acc: 0.8971\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.3396 - acc: 0.9268 - val_loss: 0.2708 - val_acc: 0.9176\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.1789 - acc: 0.9561 - val_loss: 0.1556 - val_acc: 0.9471\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.1204 - acc: 0.9675 - val_loss: 0.1585 - val_acc: 0.9539\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0897 - acc: 0.9729 - val_loss: 0.0733 - val_acc: 0.9742\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0713 - acc: 0.9759 - val_loss: 0.1833 - val_acc: 0.9434\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 153s 132ms/step - loss: 0.0641 - acc: 0.9776 - val_loss: 0.0368 - val_acc: 0.9876\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0573 - acc: 0.9784 - val_loss: 0.0299 - val_acc: 0.9863\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0505 - acc: 0.9806 - val_loss: 0.0317 - val_acc: 0.9878\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0458 - acc: 0.9816 - val_loss: 0.0319 - val_acc: 0.9878\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0458 - acc: 0.9810 - val_loss: 0.1500 - val_acc: 0.9517\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0433 - acc: 0.9825 - val_loss: 0.0368 - val_acc: 0.9781\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0423 - acc: 0.9820 - val_loss: 0.0635 - val_acc: 0.9710\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0380 - acc: 0.9832 - val_loss: 0.0264 - val_acc: 0.9893\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0371 - acc: 0.9830 - val_loss: 0.0495 - val_acc: 0.9785\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0368 - acc: 0.9833 - val_loss: 0.0380 - val_acc: 0.9859\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0386 - acc: 0.9828 - val_loss: 0.0253 - val_acc: 0.9888\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0343 - acc: 0.9835 - val_loss: 0.0610 - val_acc: 0.9763\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0344 - acc: 0.9837 - val_loss: 0.0248 - val_acc: 0.9885\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0327 - acc: 0.9845 - val_loss: 0.0257 - val_acc: 0.9876\n",
      "36905/36905 [==============================] - 41s 1ms/step\n",
      "Train [0.030321804340921105, 0.9851781601409023]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.030269034884964893, 0.9862465860319938]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 157s 136ms/step - loss: 0.7628 - acc: 0.8167 - val_loss: 0.3299 - val_acc: 0.9056\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.2232 - acc: 0.9483 - val_loss: 0.1963 - val_acc: 0.9322\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.1311 - acc: 0.9665 - val_loss: 0.1168 - val_acc: 0.9673\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0987 - acc: 0.9704 - val_loss: 0.0631 - val_acc: 0.9783\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0757 - acc: 0.9760 - val_loss: 0.0750 - val_acc: 0.9761\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0659 - acc: 0.9772 - val_loss: 0.0945 - val_acc: 0.9632\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0578 - acc: 0.9795 - val_loss: 0.0817 - val_acc: 0.9729\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0511 - acc: 0.9804 - val_loss: 0.0291 - val_acc: 0.9883\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0468 - acc: 0.9812 - val_loss: 0.0391 - val_acc: 0.9859\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0455 - acc: 0.9822 - val_loss: 0.0584 - val_acc: 0.9793\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 148s 128ms/step - loss: 0.0419 - acc: 0.9829 - val_loss: 0.0300 - val_acc: 0.9888\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0391 - acc: 0.9830 - val_loss: 0.0300 - val_acc: 0.9876\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0387 - acc: 0.9825 - val_loss: 0.0620 - val_acc: 0.9715\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0383 - acc: 0.9836 - val_loss: 0.0230 - val_acc: 0.9863\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0352 - acc: 0.9845 - val_loss: 0.0238 - val_acc: 0.9890\n",
      "Epoch 16/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0358 - acc: 0.9833 - val_loss: 0.0254 - val_acc: 0.9895\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 147s 128ms/step - loss: 0.0337 - acc: 0.9837 - val_loss: 0.0619 - val_acc: 0.9734\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0338 - acc: 0.9837 - val_loss: 0.0488 - val_acc: 0.9773\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0321 - acc: 0.9851 - val_loss: 0.0297 - val_acc: 0.9868\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 146s 127ms/step - loss: 0.0326 - acc: 0.9842 - val_loss: 0.0207 - val_acc: 0.9888\n",
      "36905/36905 [==============================] - 41s 1ms/step\n",
      "Train [0.023357738256325294, 0.9863975071128573]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.02381701098835489, 0.9877097151775264]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 157s 136ms/step - loss: 0.7896 - acc: 0.8102 - val_loss: 0.3030 - val_acc: 0.9161\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 148s 128ms/step - loss: 0.2368 - acc: 0.9453 - val_loss: 0.1485 - val_acc: 0.9522\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.1390 - acc: 0.9637 - val_loss: 0.0700 - val_acc: 0.9815\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0980 - acc: 0.9726 - val_loss: 0.0645 - val_acc: 0.9783\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0803 - acc: 0.9750 - val_loss: 0.0826 - val_acc: 0.9671\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0660 - acc: 0.9778 - val_loss: 0.0649 - val_acc: 0.9732\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 147s 128ms/step - loss: 0.0580 - acc: 0.9794 - val_loss: 0.0777 - val_acc: 0.9678\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0532 - acc: 0.9796 - val_loss: 0.0372 - val_acc: 0.9861\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 148s 128ms/step - loss: 0.0485 - acc: 0.9811 - val_loss: 0.0305 - val_acc: 0.9893\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0449 - acc: 0.9816 - val_loss: 0.1329 - val_acc: 0.9588\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 147s 128ms/step - loss: 0.0442 - acc: 0.9816 - val_loss: 0.1519 - val_acc: 0.9427\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 155s 134ms/step - loss: 0.0398 - acc: 0.9837 - val_loss: 0.0612 - val_acc: 0.9756\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 155s 135ms/step - loss: 0.0396 - acc: 0.9829 - val_loss: 0.0382 - val_acc: 0.9849\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 155s 134ms/step - loss: 0.0392 - acc: 0.9830 - val_loss: 0.0332 - val_acc: 0.9859\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 151s 131ms/step - loss: 0.0367 - acc: 0.9837 - val_loss: 0.0309 - val_acc: 0.9849\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0338 - acc: 0.9849 - val_loss: 0.0227 - val_acc: 0.9895\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0363 - acc: 0.9830 - val_loss: 0.0447 - val_acc: 0.9810\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0335 - acc: 0.9842 - val_loss: 0.0416 - val_acc: 0.9822\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0337 - acc: 0.9839 - val_loss: 0.0352 - val_acc: 0.9837\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 147s 127ms/step - loss: 0.0318 - acc: 0.9842 - val_loss: 0.0262 - val_acc: 0.9890\n",
      "36905/36905 [==============================] - 42s 1ms/step\n",
      "Train [0.029902044787441854, 0.9866142799078715]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.028730619956802473, 0.9872220054623488]\n"
     ]
    }
   ],
   "source": [
    "import keras.optimizers\n",
    "for lr in [0.1, 0.01, 0.001, 0.0001]:\n",
    "    optimizer = optimizers.Adam(lr=lr)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH IDENTITY BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetD((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "emnist\t\t     fruits360-Cascade\t\t mnist\r\n",
      "flowers\t\t     housing.zip\t\t mnist.py\r\n",
      "flowers-keras.ipynb  intel-image-classification  Untitled1.ipynb\r\n",
      "fruits360\t     keras.ipynb\t\t Untitled.ipynb\r\n"
     ]
    }
   ],
   "source": [
    "!ls ../"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset for Intel Image Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(12630, 150, 150, 3) (12630, 6)\n",
      "(1404, 150, 150, 3) (1404, 6)\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from keras.utils import to_categorical\n",
    "import random\n",
    "\n",
    "root_train = \"../intel-image-classification/seg_train\"\n",
    "root_test  = \"../intel-image-classification/seg_test\"\n",
    "\n",
    "def loadImages(root):\n",
    "    data = []\n",
    "    labels = []\n",
    "    \n",
    "    classes = {}\n",
    "    nclass = 0\n",
    "    subdirs = os.scandir(root)\n",
    "    for subdir in subdirs:\n",
    "        classes[subdir.name] = nclass\n",
    "        images = []\n",
    "        \n",
    "        files = os.scandir(subdir.path)\n",
    "        for file in files:\n",
    "            image = cv2.imread(file.path)\n",
    "            image = cv2.resize(image, (150, 150))\n",
    "            images.append(image)\n",
    "            \n",
    "        data.append(np.asarray(images))\n",
    "        labels.append(np.asarray([nclass for _ in range(len(images))]))\n",
    "        nclass += 1\n",
    "        \n",
    "    data = np.concatenate(np.asarray(data))\n",
    "    mean = np.mean(data)\n",
    "    std  = np.std(data)\n",
    "    data = (( data - mean ) / std).astype(np.float32)\n",
    "    labels = to_categorical(np.concatenate(labels))\n",
    "    return data, labels, classes\n",
    "        \n",
    "data, labels, classes = loadImages(root_train)\n",
    "n = data.shape[0]\n",
    "random.seed(101)\n",
    "random.shuffle(data)\n",
    "random.seed(101)\n",
    "random.shuffle(labels)\n",
    "\n",
    "pivot = int(n * .9)\n",
    "x_train = data[0:pivot]\n",
    "y_train = labels[0:pivot]\n",
    "x_val   = data[pivot:]\n",
    "y_val   = labels[pivot:]\n",
    "print(x_train.shape, y_train.shape)\n",
    "print(x_val.shape, y_val.shape)\n",
    "\n",
    "data = None\n",
    "labels = None\n",
    "\n",
    "x_test, y_test, _= loadImages(root_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks for Intel Image Classification for Models 1, 2 and 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 113s 287ms/step - loss: 7.6407 - acc: 0.5238 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 107s 270ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 104s 262ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 28s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 6s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 115s 290ms/step - loss: 7.6387 - acc: 0.5243 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6519 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 104s 264ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 105s 267ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 106s 269ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 104s 262ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 101s 255ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 101s 255ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 106s 269ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 28s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 117s 295ms/step - loss: 7.6467 - acc: 0.5233 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6514 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 105s 265ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 106s 269ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 17/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 27s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 6s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 230s 582ms/step - loss: 8.9678 - acc: 0.4420 - val_loss: 12.9037 - val_acc: 0.1994\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 220s 556ms/step - loss: 9.0188 - acc: 0.4405 - val_loss: 12.9037 - val_acc: 0.1994\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 8.9350 - acc: 0.4457 - val_loss: 12.9037 - val_acc: 0.1994\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 220s 557ms/step - loss: 8.9489 - acc: 0.4448 - val_loss: 12.9037 - val_acc: 0.1994\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 218s 553ms/step - loss: 8.9605 - acc: 0.4441 - val_loss: 12.9151 - val_acc: 0.1987\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 8.9648 - acc: 0.4438 - val_loss: 12.9151 - val_acc: 0.1987\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 8.9355 - acc: 0.4456 - val_loss: 12.9151 - val_acc: 0.1987\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 221s 560ms/step - loss: 8.9147 - acc: 0.4469 - val_loss: 12.9037 - val_acc: 0.1994\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 8.9313 - acc: 0.4459 - val_loss: 12.9037 - val_acc: 0.1994\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 220s 557ms/step - loss: 8.7988 - acc: 0.4541 - val_loss: 13.0185 - val_acc: 0.1923\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.6760 - acc: 0.5238 - val_loss: 13.0414 - val_acc: 0.1909\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 217s 548ms/step - loss: 7.6581 - acc: 0.5249 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 7.6739 - acc: 0.5239 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 7.6823 - acc: 0.5234 - val_loss: 13.0414 - val_acc: 0.1909\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 7.6876 - acc: 0.5230 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6742 - acc: 0.5239 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6746 - acc: 0.5238 - val_loss: 13.0414 - val_acc: 0.1909\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.6619 - acc: 0.5246 - val_loss: 13.0414 - val_acc: 0.1909\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.6630 - acc: 0.5246 - val_loss: 13.0414 - val_acc: 0.1909\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 226s 573ms/step - loss: 7.6585 - acc: 0.5248 - val_loss: 13.0414 - val_acc: 0.1909\n",
      "12630/12630 [==============================] - 49s 4ms/step\n",
      "Train [7.654500132973762, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [13.270565177281698, 0.17666666666666667]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 228s 578ms/step - loss: 12.2216 - acc: 0.2406 - val_loss: 12.2263 - val_acc: 0.2415\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 12.2319 - acc: 0.2411 - val_loss: 12.2149 - val_acc: 0.2422\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 12.2273 - acc: 0.2414 - val_loss: 12.2263 - val_acc: 0.2415\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 12.2359 - acc: 0.2409 - val_loss: 12.2149 - val_acc: 0.2422\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 218s 553ms/step - loss: 12.2699 - acc: 0.2388 - val_loss: 12.2263 - val_acc: 0.2415\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 12.2317 - acc: 0.2411 - val_loss: 12.2034 - val_acc: 0.2429\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 220s 557ms/step - loss: 12.2407 - acc: 0.2406 - val_loss: 12.2034 - val_acc: 0.2429\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 12.2502 - acc: 0.2400 - val_loss: 12.2034 - val_acc: 0.2429\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 12.2495 - acc: 0.2400 - val_loss: 12.2149 - val_acc: 0.2422\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 12.2871 - acc: 0.2377 - val_loss: 12.2149 - val_acc: 0.2422\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 221s 559ms/step - loss: 12.2623 - acc: 0.2392 - val_loss: 12.2263 - val_acc: 0.2415\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 221s 560ms/step - loss: 12.2407 - acc: 0.2406 - val_loss: 12.2034 - val_acc: 0.2429\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 220s 557ms/step - loss: 12.2731 - acc: 0.2386 - val_loss: 12.1919 - val_acc: 0.2436\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 12.2459 - acc: 0.2402 - val_loss: 12.2034 - val_acc: 0.2429\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 220s 558ms/step - loss: 12.2440 - acc: 0.2404 - val_loss: 12.2149 - val_acc: 0.2422\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 221s 559ms/step - loss: 12.2520 - acc: 0.2399 - val_loss: 12.2263 - val_acc: 0.2415\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 218s 553ms/step - loss: 12.2699 - acc: 0.2388 - val_loss: 12.2263 - val_acc: 0.2415\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 12.3839 - acc: 0.2317 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 12.4376 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 12.4370 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "12630/12630 [==============================] - 47s 4ms/step\n",
      "Train [12.43760555521043, 0.22834520982025355]\n",
      "3000/3000 [==============================] - 11s 4ms/step\n",
      "Test [13.141620468139648, 0.18466666666666667]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 233s 589ms/step - loss: 13.5573 - acc: 0.1568 - val_loss: 13.8450 - val_acc: 0.1410\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 222s 561ms/step - loss: 13.5909 - acc: 0.1568 - val_loss: 13.8450 - val_acc: 0.1410\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 231s 584ms/step - loss: 13.6050 - acc: 0.1559 - val_loss: 13.8565 - val_acc: 0.1403\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 13.6196 - acc: 0.1550 - val_loss: 13.8450 - val_acc: 0.1410\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 220s 556ms/step - loss: 13.6513 - acc: 0.1530 - val_loss: 13.8336 - val_acc: 0.1417\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 222s 561ms/step - loss: 13.6183 - acc: 0.1551 - val_loss: 13.8680 - val_acc: 0.1396\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 226s 572ms/step - loss: 13.6465 - acc: 0.1533 - val_loss: 13.8450 - val_acc: 0.1410\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 13.6482 - acc: 0.1532 - val_loss: 13.8450 - val_acc: 0.1410\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 13.5636 - acc: 0.1585 - val_loss: 13.8450 - val_acc: 0.1410\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 13.6151 - acc: 0.1553 - val_loss: 13.8565 - val_acc: 0.1403\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 11/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 13.5909 - acc: 0.1568 - val_loss: 13.8680 - val_acc: 0.1396\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 215s 544ms/step - loss: 13.5935 - acc: 0.1566 - val_loss: 13.8336 - val_acc: 0.1417\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 215s 544ms/step - loss: 13.5736 - acc: 0.1579 - val_loss: 13.8565 - val_acc: 0.1403\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 215s 545ms/step - loss: 13.6285 - acc: 0.1545 - val_loss: 13.8680 - val_acc: 0.1396\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 215s 546ms/step - loss: 13.5776 - acc: 0.1576 - val_loss: 13.8565 - val_acc: 0.1403\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 215s 545ms/step - loss: 13.6552 - acc: 0.1528 - val_loss: 13.8450 - val_acc: 0.1410\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 215s 544ms/step - loss: 13.6203 - acc: 0.1550 - val_loss: 13.8680 - val_acc: 0.1396\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 215s 545ms/step - loss: 13.5992 - acc: 0.1563 - val_loss: 13.8680 - val_acc: 0.1396\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 215s 544ms/step - loss: 13.5795 - acc: 0.1575 - val_loss: 13.8680 - val_acc: 0.1396\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 216s 546ms/step - loss: 13.6183 - acc: 0.1551 - val_loss: 13.8565 - val_acc: 0.1403\n",
      "12630/12630 [==============================] - 47s 4ms/step\n",
      "Train [13.907759535359846, 0.13713380840215433]\n",
      "3000/3000 [==============================] - 11s 4ms/step\n",
      "Test [13.872307382583617, 0.13933333333333334]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 127s 322ms/step - loss: 12.2575 - acc: 0.2366 - val_loss: 11.7327 - val_acc: 0.2721\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.3067 - acc: 0.2365 - val_loss: 11.7097 - val_acc: 0.2735\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 12.3863 - acc: 0.2315 - val_loss: 13.0529 - val_acc: 0.1902\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4434 - acc: 0.2280 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4439 - acc: 0.2280 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 114s 290ms/step - loss: 12.4409 - acc: 0.2281 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 12.4382 - acc: 0.2283 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4396 - acc: 0.2282 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4377 - acc: 0.2283 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4414 - acc: 0.2281 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4423 - acc: 0.2281 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4395 - acc: 0.2282 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4428 - acc: 0.2280 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 114s 290ms/step - loss: 12.4450 - acc: 0.2279 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4390 - acc: 0.2283 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4410 - acc: 0.2281 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4409 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 115s 290ms/step - loss: 12.4365 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4370 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 12.4388 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "12630/12630 [==============================] - 30s 2ms/step\n",
      "Train [12.43760555521043, 0.22834520982025355]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.146993169148763, 0.18433333333333332]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 128s 324ms/step - loss: 7.6494 - acc: 0.5225 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6559 - acc: 0.5250 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6519 - acc: 0.5253 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6512 - acc: 0.5253 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 114s 290ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.0759 - val_acc: 0.1887\n",
      "12630/12630 [==============================] - 30s 2ms/step\n",
      "Train [7.645566919165193, 0.5256532066649893]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.292056030273438, 0.17533333333333334]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 128s 324ms/step - loss: 7.6685 - acc: 0.5206 - val_loss: 13.1333 - val_acc: 0.1852\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6566 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 115s 290ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 114s 288ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 114s 289ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 114s 290ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 30s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 13.9292 - acc: 0.1336 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 13.9507 - acc: 0.1345 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 13.9536 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 13.9518 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 13.9512 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 13.9507 - acc: 0.1345 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 13.9524 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 13.9530 - acc: 0.1343 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 13.9512 - acc: 0.1344 - val_loss: 13.6613 - val_acc: 0.1524\n",
      "12630/12630 [==============================] - 27s 2ms/step\n",
      "Train [13.952425729596039, 0.13436262867135465]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.770226172129313, 0.14566666666666667]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 7.6399 - acc: 0.5240 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 106s 268ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6589 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 101s 256ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 20/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 28s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 7.6365 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 102s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 102s 258ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 101s 257ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 27s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 6s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 232s 586ms/step - loss: 13.3436 - acc: 0.1708 - val_loss: 13.0524 - val_acc: 0.1902\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 216s 546ms/step - loss: 12.2763 - acc: 0.2383 - val_loss: 13.1141 - val_acc: 0.1859\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 216s 546ms/step - loss: 10.1086 - acc: 0.3728 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 10.0584 - acc: 0.3760 - val_loss: 13.0235 - val_acc: 0.1916\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.9352 - acc: 0.5077 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 215s 545ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 216s 546ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 216s 546ms/step - loss: 7.6566 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 215s 545ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 219s 553ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 221s 559ms/step - loss: 7.6519 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 220s 557ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 217s 548ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 47s 4ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 11s 4ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 232s 588ms/step - loss: 7.7077 - acc: 0.5196 - val_loss: 13.1792 - val_acc: 0.1823\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6875 - acc: 0.5231 - val_loss: 13.1562 - val_acc: 0.1838\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.9478 - acc: 0.5069 - val_loss: 13.2940 - val_acc: 0.1752\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 216s 546ms/step - loss: 7.7098 - acc: 0.5217 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.7041 - acc: 0.5220 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6831 - acc: 0.5233 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6993 - acc: 0.5223 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.6995 - acc: 0.5223 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 217s 548ms/step - loss: 7.6983 - acc: 0.5224 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.7001 - acc: 0.5223 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.7076 - acc: 0.5218 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 217s 548ms/step - loss: 7.7055 - acc: 0.5219 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.7030 - acc: 0.5221 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.7058 - acc: 0.5219 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 217s 548ms/step - loss: 7.6957 - acc: 0.5225 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6991 - acc: 0.5223 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.7033 - acc: 0.5221 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.6787 - acc: 0.5236 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6592 - acc: 0.5248 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.0414 - val_acc: 0.1909\n",
      "12630/12630 [==============================] - 47s 4ms/step\n",
      "Train [7.632805161313792, 0.5264449723023605]\n",
      "3000/3000 [==============================] - 11s 4ms/step\n",
      "Test [13.249074419021607, 0.178]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 233s 591ms/step - loss: 7.6557 - acc: 0.5226 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 7.6514 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.6548 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 217s 548ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.6560 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 216s 548ms/step - loss: 7.6531 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6525 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 7.6542 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 216s 547ms/step - loss: 7.6537 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 7.6554 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 48s 4ms/step\n",
      "Train [7.658328677922699, 0.5248614410276179]\n",
      "3000/3000 [==============================] - 11s 4ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 131s 332ms/step - loss: 1.1706 - acc: 0.5635 - val_loss: 1.5077 - val_acc: 0.3412\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 115s 290ms/step - loss: 0.9338 - acc: 0.6343 - val_loss: 1.6754 - val_acc: 0.3476\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.8885 - acc: 0.6513 - val_loss: 1.5416 - val_acc: 0.4580\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.8589 - acc: 0.6641 - val_loss: 1.7082 - val_acc: 0.4387\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.8335 - acc: 0.6788 - val_loss: 1.7958 - val_acc: 0.4067\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 115s 290ms/step - loss: 0.8114 - acc: 0.6859 - val_loss: 1.7014 - val_acc: 0.3846\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7895 - acc: 0.6971 - val_loss: 1.4375 - val_acc: 0.5164\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7705 - acc: 0.7073 - val_loss: 1.5061 - val_acc: 0.4167\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7501 - acc: 0.7190 - val_loss: 1.2973 - val_acc: 0.5370\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7322 - acc: 0.7203 - val_loss: 1.5523 - val_acc: 0.4922\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.7113 - acc: 0.7317 - val_loss: 1.1443 - val_acc: 0.5719\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6801 - acc: 0.7424 - val_loss: 1.3248 - val_acc: 0.5356\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6583 - acc: 0.7466 - val_loss: 1.2081 - val_acc: 0.5840\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6526 - acc: 0.7534 - val_loss: 1.5083 - val_acc: 0.5050\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6266 - acc: 0.7651 - val_loss: 1.1548 - val_acc: 0.6026\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6198 - acc: 0.7691 - val_loss: 1.0573 - val_acc: 0.6104\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 115s 290ms/step - loss: 0.6013 - acc: 0.7694 - val_loss: 0.9596 - val_acc: 0.6239\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.5912 - acc: 0.7805 - val_loss: 1.1003 - val_acc: 0.5969\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.5772 - acc: 0.7820 - val_loss: 1.0215 - val_acc: 0.5997\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.5620 - acc: 0.7889 - val_loss: 0.8017 - val_acc: 0.6980\n",
      "12630/12630 [==============================] - 30s 2ms/step\n",
      "Train [0.5850288548454442, 0.7925574030464637]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [0.871991495291392, 0.6793333333333333]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 131s 333ms/step - loss: 1.3683 - acc: 0.5058 - val_loss: 2.3858 - val_acc: 0.2628\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 115s 290ms/step - loss: 1.0901 - acc: 0.5605 - val_loss: 1.7624 - val_acc: 0.2949\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.9863 - acc: 0.5950 - val_loss: 1.9778 - val_acc: 0.3825\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.9400 - acc: 0.6200 - val_loss: 1.7356 - val_acc: 0.3490\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.9007 - acc: 0.6431 - val_loss: 1.6817 - val_acc: 0.3682\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.8793 - acc: 0.6516 - val_loss: 1.8104 - val_acc: 0.4010\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.8667 - acc: 0.6646 - val_loss: 1.6219 - val_acc: 0.4145\n",
      "Epoch 8/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "395/394 [==============================] - 115s 291ms/step - loss: 0.8495 - acc: 0.6734 - val_loss: 1.5281 - val_acc: 0.4323\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.8341 - acc: 0.6800 - val_loss: 1.6177 - val_acc: 0.4074\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.8184 - acc: 0.6842 - val_loss: 1.4598 - val_acc: 0.4252\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.8057 - acc: 0.6894 - val_loss: 1.5124 - val_acc: 0.4145\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7871 - acc: 0.6991 - val_loss: 1.3254 - val_acc: 0.4822\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7776 - acc: 0.7033 - val_loss: 1.6119 - val_acc: 0.4003\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.7635 - acc: 0.7096 - val_loss: 1.3801 - val_acc: 0.5199\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7460 - acc: 0.7158 - val_loss: 1.5782 - val_acc: 0.4017\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7387 - acc: 0.7212 - val_loss: 1.7466 - val_acc: 0.5007\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7240 - acc: 0.7296 - val_loss: 1.3371 - val_acc: 0.5321\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7138 - acc: 0.7297 - val_loss: 1.3263 - val_acc: 0.5328\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6927 - acc: 0.7369 - val_loss: 1.1811 - val_acc: 0.5826\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.6860 - acc: 0.7387 - val_loss: 1.3072 - val_acc: 0.5670\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [0.6389566391871642, 0.7637371338367085]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [1.3925189368724824, 0.5413333332538605]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 132s 335ms/step - loss: 1.3822 - acc: 0.5032 - val_loss: 2.1829 - val_acc: 0.2016\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 1.1597 - acc: 0.5474 - val_loss: 1.4397 - val_acc: 0.3590\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 121s 306ms/step - loss: 0.9575 - acc: 0.6240 - val_loss: 1.4912 - val_acc: 0.4103\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.8961 - acc: 0.6533 - val_loss: 1.5314 - val_acc: 0.3668\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.8609 - acc: 0.6668 - val_loss: 1.5778 - val_acc: 0.4088\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.8315 - acc: 0.6828 - val_loss: 1.4363 - val_acc: 0.4751\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.8034 - acc: 0.6958 - val_loss: 1.5142 - val_acc: 0.4623\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7896 - acc: 0.7044 - val_loss: 1.6487 - val_acc: 0.4338\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.7584 - acc: 0.7138 - val_loss: 1.4518 - val_acc: 0.4330\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7464 - acc: 0.7200 - val_loss: 1.3067 - val_acc: 0.5021\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7363 - acc: 0.7228 - val_loss: 1.3388 - val_acc: 0.5007\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.7253 - acc: 0.7292 - val_loss: 1.0689 - val_acc: 0.5712\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 115s 290ms/step - loss: 0.7062 - acc: 0.7363 - val_loss: 1.2794 - val_acc: 0.5556\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6984 - acc: 0.7372 - val_loss: 1.1029 - val_acc: 0.5684\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.6815 - acc: 0.7410 - val_loss: 1.3725 - val_acc: 0.5577\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6574 - acc: 0.7540 - val_loss: 1.1404 - val_acc: 0.5719\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.6472 - acc: 0.7552 - val_loss: 1.2102 - val_acc: 0.5520\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.6341 - acc: 0.7652 - val_loss: 1.1352 - val_acc: 0.5769\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6158 - acc: 0.7692 - val_loss: 1.1709 - val_acc: 0.5826\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 115s 291ms/step - loss: 0.6031 - acc: 0.7707 - val_loss: 1.0634 - val_acc: 0.6004\n",
      "12630/12630 [==============================] - 30s 2ms/step\n",
      "Train [0.5893245458697292, 0.7870150434999171]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [1.1465798797607423, 0.5726666665077209]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.9304 - acc: 0.6385 - val_loss: 1.4740 - val_acc: 0.4644\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.7805 - acc: 0.7000 - val_loss: 1.1496 - val_acc: 0.5698\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.7174 - acc: 0.7201 - val_loss: 1.2420 - val_acc: 0.5712\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 0.6832 - acc: 0.7353 - val_loss: 1.0193 - val_acc: 0.5926\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 0.6370 - acc: 0.7579 - val_loss: 1.2761 - val_acc: 0.5962\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.6073 - acc: 0.7636 - val_loss: 1.0338 - val_acc: 0.6225\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.5663 - acc: 0.7856 - val_loss: 0.8653 - val_acc: 0.6823\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.5623 - acc: 0.7841 - val_loss: 0.9271 - val_acc: 0.6731\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.5299 - acc: 0.7989 - val_loss: 0.7001 - val_acc: 0.7486\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.5208 - acc: 0.8020 - val_loss: 0.7026 - val_acc: 0.7336\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.4991 - acc: 0.8127 - val_loss: 0.6985 - val_acc: 0.7301\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.4843 - acc: 0.8168 - val_loss: 0.6568 - val_acc: 0.7714\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 0.4686 - acc: 0.8194 - val_loss: 0.7163 - val_acc: 0.7415\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.4634 - acc: 0.8241 - val_loss: 0.6801 - val_acc: 0.7507\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 0.4558 - acc: 0.8295 - val_loss: 0.8027 - val_acc: 0.7037\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.4325 - acc: 0.8378 - val_loss: 0.8730 - val_acc: 0.7023\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.4275 - acc: 0.8365 - val_loss: 0.6555 - val_acc: 0.7685\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.4236 - acc: 0.8396 - val_loss: 0.5734 - val_acc: 0.8105\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.4146 - acc: 0.8430 - val_loss: 0.8198 - val_acc: 0.7123\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.4124 - acc: 0.8464 - val_loss: 0.6705 - val_acc: 0.7813\n",
      "12630/12630 [==============================] - 28s 2ms/step\n",
      "Train [0.39628937412422044, 0.8437846397843893]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [0.7092461784680685, 0.7523333331743877]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 120s 305ms/step - loss: 1.0371 - acc: 0.5929 - val_loss: 1.5631 - val_acc: 0.4053\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.8701 - acc: 0.6593 - val_loss: 1.4470 - val_acc: 0.4886\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.8214 - acc: 0.6783 - val_loss: 1.4608 - val_acc: 0.4672\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.7938 - acc: 0.6923 - val_loss: 1.3623 - val_acc: 0.5043\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.7626 - acc: 0.7065 - val_loss: 1.3619 - val_acc: 0.5313\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.7545 - acc: 0.7052 - val_loss: 1.3570 - val_acc: 0.5491\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.7343 - acc: 0.7179 - val_loss: 1.3152 - val_acc: 0.5214\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.7124 - acc: 0.7232 - val_loss: 1.3172 - val_acc: 0.5185\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.6939 - acc: 0.7307 - val_loss: 1.1347 - val_acc: 0.5769\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 0.6751 - acc: 0.7394 - val_loss: 1.1791 - val_acc: 0.5641\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 0.6614 - acc: 0.7392 - val_loss: 1.2757 - val_acc: 0.5456\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.6526 - acc: 0.7422 - val_loss: 1.0338 - val_acc: 0.6147\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.6344 - acc: 0.7513 - val_loss: 1.0594 - val_acc: 0.6218\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.6307 - acc: 0.7572 - val_loss: 0.9106 - val_acc: 0.6688\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.6112 - acc: 0.7680 - val_loss: 0.9101 - val_acc: 0.6595\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.6027 - acc: 0.7685 - val_loss: 0.8628 - val_acc: 0.6859\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.5888 - acc: 0.7714 - val_loss: 0.8736 - val_acc: 0.6709\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.5781 - acc: 0.7748 - val_loss: 0.8257 - val_acc: 0.6859\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 0.5711 - acc: 0.7802 - val_loss: 0.8134 - val_acc: 0.7058\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 0.5651 - acc: 0.7806 - val_loss: 0.9205 - val_acc: 0.6652\n",
      "12630/12630 [==============================] - 28s 2ms/step\n",
      "Train [0.5235380742631068, 0.7974663499415345]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [1.011319678624471, 0.6300000001589457]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 121s 305ms/step - loss: 1.3120 - acc: 0.5225 - val_loss: 2.5004 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2724 - acc: 0.5253 - val_loss: 2.5502 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 1.2729 - acc: 0.5252 - val_loss: 2.4025 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2721 - acc: 0.5251 - val_loss: 2.3600 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 1.2715 - acc: 0.5252 - val_loss: 2.4627 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2711 - acc: 0.5251 - val_loss: 2.4319 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 1.2718 - acc: 0.5250 - val_loss: 2.4460 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 1.2716 - acc: 0.5250 - val_loss: 2.4395 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2714 - acc: 0.5251 - val_loss: 2.3953 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 1.2712 - acc: 0.5251 - val_loss: 2.4321 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2711 - acc: 0.5252 - val_loss: 2.6394 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2711 - acc: 0.5250 - val_loss: 2.4883 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 1.2716 - acc: 0.5253 - val_loss: 2.3812 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 102s 259ms/step - loss: 1.2706 - acc: 0.5252 - val_loss: 2.4452 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2708 - acc: 0.5250 - val_loss: 2.4255 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 1.2714 - acc: 0.5250 - val_loss: 2.4181 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2705 - acc: 0.5253 - val_loss: 2.4295 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2716 - acc: 0.5250 - val_loss: 2.3719 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2706 - acc: 0.5252 - val_loss: 2.3903 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 103s 260ms/step - loss: 1.2711 - acc: 0.5252 - val_loss: 2.3909 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 28s 2ms/step\n",
      "Train [1.2702637098558536, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [2.5420491015116373, 0.175]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 237s 599ms/step - loss: 1.0583 - acc: 0.6332 - val_loss: 1.3563 - val_acc: 0.5157\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 0.8063 - acc: 0.6887 - val_loss: 1.1504 - val_acc: 0.5791\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 0.7438 - acc: 0.7138 - val_loss: 1.2733 - val_acc: 0.5819\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.7008 - acc: 0.7346 - val_loss: 0.9763 - val_acc: 0.6346\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.6666 - acc: 0.7479 - val_loss: 0.9064 - val_acc: 0.6517\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 0.6305 - acc: 0.7600 - val_loss: 0.8535 - val_acc: 0.6724\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 0.6035 - acc: 0.7717 - val_loss: 0.9590 - val_acc: 0.6774\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.5920 - acc: 0.7783 - val_loss: 0.8588 - val_acc: 0.6930\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.5772 - acc: 0.7796 - val_loss: 0.7728 - val_acc: 0.7165\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.5554 - acc: 0.7890 - val_loss: 0.7258 - val_acc: 0.7379\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.5365 - acc: 0.7990 - val_loss: 0.7787 - val_acc: 0.7293\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.5254 - acc: 0.8033 - val_loss: 0.7681 - val_acc: 0.7407\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.5051 - acc: 0.8072 - val_loss: 0.7183 - val_acc: 0.7429\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4865 - acc: 0.8163 - val_loss: 0.7824 - val_acc: 0.7279\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 218s 553ms/step - loss: 0.4787 - acc: 0.8202 - val_loss: 0.6981 - val_acc: 0.7585\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4660 - acc: 0.8265 - val_loss: 0.7001 - val_acc: 0.7429\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4560 - acc: 0.8323 - val_loss: 0.7783 - val_acc: 0.7329\n",
      "Epoch 18/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4436 - acc: 0.8309 - val_loss: 0.6599 - val_acc: 0.7821\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 219s 554ms/step - loss: 0.4355 - acc: 0.8360 - val_loss: 0.6806 - val_acc: 0.7628\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.4215 - acc: 0.8382 - val_loss: 0.7168 - val_acc: 0.7450\n",
      "12630/12630 [==============================] - 51s 4ms/step\n",
      "Train [0.3992283265928569, 0.8557403008615037]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.7266968308289846, 0.7390000001589457]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 249s 629ms/step - loss: 1.0286 - acc: 0.6399 - val_loss: 1.3295 - val_acc: 0.4879\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 222s 563ms/step - loss: 0.8042 - acc: 0.6905 - val_loss: 1.1832 - val_acc: 0.5691\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.7411 - acc: 0.7150 - val_loss: 1.0839 - val_acc: 0.5905\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 0.6941 - acc: 0.7344 - val_loss: 0.9339 - val_acc: 0.6681\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 0.6564 - acc: 0.7492 - val_loss: 0.9234 - val_acc: 0.6489\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.6242 - acc: 0.7618 - val_loss: 0.8885 - val_acc: 0.6517\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.5990 - acc: 0.7709 - val_loss: 0.9834 - val_acc: 0.6346\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.5721 - acc: 0.7823 - val_loss: 0.8171 - val_acc: 0.7001\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 0.5563 - acc: 0.7896 - val_loss: 0.9883 - val_acc: 0.6617\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.5381 - acc: 0.8010 - val_loss: 0.7816 - val_acc: 0.7179\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.5222 - acc: 0.8036 - val_loss: 0.7128 - val_acc: 0.7436\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 0.5085 - acc: 0.8080 - val_loss: 0.7556 - val_acc: 0.7358\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 0.4917 - acc: 0.8175 - val_loss: 0.7687 - val_acc: 0.7407\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 0.4724 - acc: 0.8241 - val_loss: 0.8972 - val_acc: 0.6745\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 218s 553ms/step - loss: 0.4644 - acc: 0.8258 - val_loss: 0.6176 - val_acc: 0.7792\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 217s 549ms/step - loss: 0.4588 - acc: 0.8276 - val_loss: 0.8560 - val_acc: 0.7308\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4473 - acc: 0.8388 - val_loss: 0.7340 - val_acc: 0.7365\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4311 - acc: 0.8383 - val_loss: 0.7377 - val_acc: 0.7543\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4260 - acc: 0.8411 - val_loss: 0.6013 - val_acc: 0.7870\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.4219 - acc: 0.8442 - val_loss: 0.6470 - val_acc: 0.7835\n",
      "12630/12630 [==============================] - 48s 4ms/step\n",
      "Train [0.40589928495817873, 0.8562153602722422]\n",
      "3000/3000 [==============================] - 11s 4ms/step\n",
      "Test [0.7588142329851786, 0.743666666507721]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 238s 601ms/step - loss: 1.0430 - acc: 0.6323 - val_loss: 1.1541 - val_acc: 0.5591\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.8043 - acc: 0.6886 - val_loss: 1.1036 - val_acc: 0.5791\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.7418 - acc: 0.7148 - val_loss: 1.1829 - val_acc: 0.5919\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 217s 551ms/step - loss: 0.6935 - acc: 0.7340 - val_loss: 1.1129 - val_acc: 0.6147\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 217s 551ms/step - loss: 0.6666 - acc: 0.7401 - val_loss: 1.0516 - val_acc: 0.6225\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.6312 - acc: 0.7619 - val_loss: 0.8388 - val_acc: 0.7037\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.6095 - acc: 0.7716 - val_loss: 0.7866 - val_acc: 0.7358\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.5825 - acc: 0.7814 - val_loss: 0.8789 - val_acc: 0.7030\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.5690 - acc: 0.7835 - val_loss: 0.7575 - val_acc: 0.7265\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.5452 - acc: 0.7948 - val_loss: 0.7351 - val_acc: 0.7365\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 0.5304 - acc: 0.8018 - val_loss: 0.7798 - val_acc: 0.7201\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.5066 - acc: 0.8085 - val_loss: 0.8905 - val_acc: 0.7130\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 218s 552ms/step - loss: 0.5004 - acc: 0.8108 - val_loss: 0.7225 - val_acc: 0.7443\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4811 - acc: 0.8193 - val_loss: 0.7943 - val_acc: 0.7172\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.4712 - acc: 0.8252 - val_loss: 0.7561 - val_acc: 0.7514\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4588 - acc: 0.8283 - val_loss: 0.7041 - val_acc: 0.7628\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 218s 551ms/step - loss: 0.4404 - acc: 0.8327 - val_loss: 0.6239 - val_acc: 0.7828\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 217s 550ms/step - loss: 0.4363 - acc: 0.8342 - val_loss: 0.7215 - val_acc: 0.7564\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 218s 553ms/step - loss: 0.4164 - acc: 0.8408 - val_loss: 0.6319 - val_acc: 0.8041\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 222s 561ms/step - loss: 0.4177 - acc: 0.8431 - val_loss: 0.7035 - val_acc: 0.7614\n",
      "12630/12630 [==============================] - 49s 4ms/step\n",
      "Train [0.5684077463589863, 0.8059382422708464]\n",
      "3000/3000 [==============================] - 11s 4ms/step\n",
      "Test [0.826595204859972, 0.7313333333333333]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 136s 344ms/step - loss: 1.0097 - acc: 0.6075 - val_loss: 1.3597 - val_acc: 0.4046\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.8402 - acc: 0.6753 - val_loss: 1.6483 - val_acc: 0.4309\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 116s 292ms/step - loss: 0.7933 - acc: 0.6917 - val_loss: 1.4287 - val_acc: 0.4829\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 115s 292ms/step - loss: 0.7617 - acc: 0.7070 - val_loss: 1.2254 - val_acc: 0.5377\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.7286 - acc: 0.7195 - val_loss: 1.1902 - val_acc: 0.5598\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.7069 - acc: 0.7310 - val_loss: 1.0606 - val_acc: 0.5954\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.6822 - acc: 0.7367 - val_loss: 1.3238 - val_acc: 0.5014\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.6580 - acc: 0.7475 - val_loss: 1.1191 - val_acc: 0.5962\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.6261 - acc: 0.7621 - val_loss: 1.0587 - val_acc: 0.6303\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.6097 - acc: 0.7698 - val_loss: 1.0773 - val_acc: 0.6026\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5917 - acc: 0.7772 - val_loss: 0.9208 - val_acc: 0.6417\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.5714 - acc: 0.7865 - val_loss: 0.8476 - val_acc: 0.6895\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 116s 295ms/step - loss: 0.5535 - acc: 0.7910 - val_loss: 0.8125 - val_acc: 0.6959\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.5430 - acc: 0.7949 - val_loss: 0.8035 - val_acc: 0.6895\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5227 - acc: 0.8040 - val_loss: 0.7148 - val_acc: 0.7450\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5221 - acc: 0.8025 - val_loss: 0.6915 - val_acc: 0.7379\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5046 - acc: 0.8075 - val_loss: 0.7661 - val_acc: 0.7094\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.4925 - acc: 0.8157 - val_loss: 0.7204 - val_acc: 0.7493\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.4765 - acc: 0.8199 - val_loss: 0.6756 - val_acc: 0.7464\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.4737 - acc: 0.8212 - val_loss: 0.7744 - val_acc: 0.7073\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [0.5031097126309114, 0.8106888361422674]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.8055630135536194, 0.688]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 137s 346ms/step - loss: 0.9890 - acc: 0.6120 - val_loss: 1.4735 - val_acc: 0.3939\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.8261 - acc: 0.6800 - val_loss: 1.5544 - val_acc: 0.4316\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.7855 - acc: 0.6978 - val_loss: 1.5336 - val_acc: 0.4366\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.7503 - acc: 0.7030 - val_loss: 1.1269 - val_acc: 0.5385\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.7117 - acc: 0.7225 - val_loss: 1.1612 - val_acc: 0.5962\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.6830 - acc: 0.7361 - val_loss: 1.0107 - val_acc: 0.6225\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.6507 - acc: 0.7479 - val_loss: 0.9718 - val_acc: 0.6375\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 116s 295ms/step - loss: 0.6340 - acc: 0.7617 - val_loss: 0.9848 - val_acc: 0.6147\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.6059 - acc: 0.7707 - val_loss: 1.0010 - val_acc: 0.5976\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5901 - acc: 0.7807 - val_loss: 0.9031 - val_acc: 0.6724\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 116s 292ms/step - loss: 0.5714 - acc: 0.7810 - val_loss: 0.9250 - val_acc: 0.6631\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5627 - acc: 0.7880 - val_loss: 0.8444 - val_acc: 0.6759\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5376 - acc: 0.7973 - val_loss: 0.8170 - val_acc: 0.6902\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.5320 - acc: 0.8027 - val_loss: 0.8657 - val_acc: 0.6595\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5099 - acc: 0.8093 - val_loss: 0.9457 - val_acc: 0.6738\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 116s 295ms/step - loss: 0.4969 - acc: 0.8137 - val_loss: 0.9240 - val_acc: 0.6745\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.4884 - acc: 0.8136 - val_loss: 0.8072 - val_acc: 0.7151\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.4875 - acc: 0.8189 - val_loss: 0.7546 - val_acc: 0.7507\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.4650 - acc: 0.8248 - val_loss: 0.8788 - val_acc: 0.6752\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.4621 - acc: 0.8290 - val_loss: 0.7734 - val_acc: 0.7179\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [0.4857637661181078, 0.8231195566489974]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [0.8054208159446716, 0.7026666668256124]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 136s 345ms/step - loss: 1.0082 - acc: 0.6117 - val_loss: 1.3840 - val_acc: 0.4964\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.8580 - acc: 0.6725 - val_loss: 1.3434 - val_acc: 0.4936\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 116s 295ms/step - loss: 0.7893 - acc: 0.6970 - val_loss: 1.2744 - val_acc: 0.5057\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.7435 - acc: 0.7144 - val_loss: 1.1563 - val_acc: 0.5264\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.7044 - acc: 0.7274 - val_loss: 0.9073 - val_acc: 0.6311\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.6578 - acc: 0.7508 - val_loss: 0.8396 - val_acc: 0.6660\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.6264 - acc: 0.7583 - val_loss: 0.8965 - val_acc: 0.6325\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.6054 - acc: 0.7681 - val_loss: 0.7834 - val_acc: 0.6973\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5821 - acc: 0.7757 - val_loss: 0.9296 - val_acc: 0.6353\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5664 - acc: 0.7849 - val_loss: 0.8387 - val_acc: 0.7009\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 116s 295ms/step - loss: 0.5405 - acc: 0.7926 - val_loss: 0.8162 - val_acc: 0.7130\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5307 - acc: 0.7989 - val_loss: 0.8620 - val_acc: 0.6581\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.5180 - acc: 0.8035 - val_loss: 0.7952 - val_acc: 0.6987\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.5124 - acc: 0.8035 - val_loss: 0.9042 - val_acc: 0.6745\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 116s 294ms/step - loss: 0.4878 - acc: 0.8188 - val_loss: 0.7289 - val_acc: 0.7251\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 116s 295ms/step - loss: 0.4843 - acc: 0.8203 - val_loss: 0.7130 - val_acc: 0.7365\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.4731 - acc: 0.8196 - val_loss: 0.6823 - val_acc: 0.7379\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.4638 - acc: 0.8255 - val_loss: 0.7877 - val_acc: 0.7051\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 116s 295ms/step - loss: 0.4590 - acc: 0.8321 - val_loss: 0.7822 - val_acc: 0.7165\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 116s 293ms/step - loss: 0.4489 - acc: 0.8316 - val_loss: 0.6976 - val_acc: 0.7472\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [0.43569284827578947, 0.8357878068469388]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [0.7501754279136658, 0.7406666668256124]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 124s 314ms/step - loss: 1.0600 - acc: 0.6031 - val_loss: 1.8507 - val_acc: 0.3789\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 0.8724 - acc: 0.6659 - val_loss: 1.7672 - val_acc: 0.4095\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.8122 - acc: 0.6855 - val_loss: 1.6023 - val_acc: 0.4487\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 0.7811 - acc: 0.7009 - val_loss: 1.5445 - val_acc: 0.4594\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 0.7584 - acc: 0.7067 - val_loss: 1.5214 - val_acc: 0.4444\n",
      "Epoch 6/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "395/394 [==============================] - 104s 263ms/step - loss: 0.7386 - acc: 0.7116 - val_loss: 1.5157 - val_acc: 0.4879\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 0.7296 - acc: 0.7154 - val_loss: 1.5333 - val_acc: 0.4459\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 104s 262ms/step - loss: 0.7100 - acc: 0.7230 - val_loss: 1.4136 - val_acc: 0.4808\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.7111 - acc: 0.7299 - val_loss: 1.3817 - val_acc: 0.4822\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 0.6864 - acc: 0.7351 - val_loss: 1.3002 - val_acc: 0.5128\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 0.6823 - acc: 0.7384 - val_loss: 1.3112 - val_acc: 0.5477\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 0.6713 - acc: 0.7392 - val_loss: 1.2400 - val_acc: 0.4964\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.6594 - acc: 0.7448 - val_loss: 1.1514 - val_acc: 0.5741\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 0.6396 - acc: 0.7548 - val_loss: 1.1850 - val_acc: 0.5719\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.6343 - acc: 0.7552 - val_loss: 1.0278 - val_acc: 0.6147\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.6256 - acc: 0.7581 - val_loss: 1.0839 - val_acc: 0.5997\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 103s 261ms/step - loss: 0.6237 - acc: 0.7577 - val_loss: 1.0230 - val_acc: 0.5734\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.6068 - acc: 0.7674 - val_loss: 1.1277 - val_acc: 0.5556\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 0.6003 - acc: 0.7636 - val_loss: 1.0312 - val_acc: 0.5791\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 104s 262ms/step - loss: 0.6014 - acc: 0.7714 - val_loss: 1.0346 - val_acc: 0.6268\n",
      "12630/12630 [==============================] - 29s 2ms/step\n",
      "Train [0.6349761711163645, 0.774346793386923]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [1.1038854475021362, 0.616]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 124s 315ms/step - loss: 0.9542 - acc: 0.6342 - val_loss: 1.6464 - val_acc: 0.4900\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 106s 267ms/step - loss: 0.7993 - acc: 0.6892 - val_loss: 1.5557 - val_acc: 0.4786\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 107s 272ms/step - loss: 0.7687 - acc: 0.6989 - val_loss: 1.4729 - val_acc: 0.5150\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 108s 273ms/step - loss: 0.7399 - acc: 0.7123 - val_loss: 1.3825 - val_acc: 0.4950\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 107s 271ms/step - loss: 0.7211 - acc: 0.7191 - val_loss: 1.2790 - val_acc: 0.5513\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 107s 270ms/step - loss: 0.7044 - acc: 0.7255 - val_loss: 1.2501 - val_acc: 0.5463\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 107s 270ms/step - loss: 0.6820 - acc: 0.7378 - val_loss: 1.2233 - val_acc: 0.5171\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 106s 268ms/step - loss: 0.6686 - acc: 0.7425 - val_loss: 1.2606 - val_acc: 0.5185\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 107s 271ms/step - loss: 0.6621 - acc: 0.7430 - val_loss: 1.1838 - val_acc: 0.5641\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 106s 269ms/step - loss: 0.6413 - acc: 0.7507 - val_loss: 1.0719 - val_acc: 0.5605\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 107s 270ms/step - loss: 0.6360 - acc: 0.7555 - val_loss: 1.0039 - val_acc: 0.6011\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 106s 268ms/step - loss: 0.6160 - acc: 0.7617 - val_loss: 0.9396 - val_acc: 0.6254\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 106s 268ms/step - loss: 0.6177 - acc: 0.7601 - val_loss: 0.9860 - val_acc: 0.6289\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 106s 268ms/step - loss: 0.6020 - acc: 0.7688 - val_loss: 1.0628 - val_acc: 0.6083\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 106s 269ms/step - loss: 0.5883 - acc: 0.7738 - val_loss: 0.8895 - val_acc: 0.6588\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 107s 271ms/step - loss: 0.5842 - acc: 0.7782 - val_loss: 0.9023 - val_acc: 0.6588\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 106s 269ms/step - loss: 0.5719 - acc: 0.7845 - val_loss: 0.9833 - val_acc: 0.6560\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 106s 268ms/step - loss: 0.5665 - acc: 0.7863 - val_loss: 0.8881 - val_acc: 0.6702\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 106s 267ms/step - loss: 0.5598 - acc: 0.7885 - val_loss: 0.9285 - val_acc: 0.6425\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 107s 271ms/step - loss: 0.5472 - acc: 0.7927 - val_loss: 0.8601 - val_acc: 0.6759\n",
      "12630/12630 [==============================] - 29s 2ms/step\n",
      "Train [0.5715021999790285, 0.7882026920126056]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [0.9138294084866841, 0.662]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 128s 324ms/step - loss: 0.9600 - acc: 0.6292 - val_loss: 1.6473 - val_acc: 0.4209\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 106s 268ms/step - loss: 0.8124 - acc: 0.6848 - val_loss: 1.3618 - val_acc: 0.5036\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 106s 267ms/step - loss: 0.7597 - acc: 0.7064 - val_loss: 1.3567 - val_acc: 0.5157\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 103s 262ms/step - loss: 0.7307 - acc: 0.7141 - val_loss: 1.3840 - val_acc: 0.5484\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 104s 262ms/step - loss: 0.7143 - acc: 0.7201 - val_loss: 1.2354 - val_acc: 0.5513\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.6957 - acc: 0.7252 - val_loss: 1.2429 - val_acc: 0.5634\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 104s 264ms/step - loss: 0.6755 - acc: 0.7379 - val_loss: 1.2205 - val_acc: 0.5613\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 105s 266ms/step - loss: 0.6612 - acc: 0.7439 - val_loss: 1.1129 - val_acc: 0.5691\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 105s 265ms/step - loss: 0.6498 - acc: 0.7444 - val_loss: 1.0506 - val_acc: 0.5897\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 104s 264ms/step - loss: 0.6366 - acc: 0.7539 - val_loss: 1.0132 - val_acc: 0.6026\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.6302 - acc: 0.7581 - val_loss: 1.0240 - val_acc: 0.5926\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.6096 - acc: 0.7645 - val_loss: 0.9462 - val_acc: 0.6396\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.6119 - acc: 0.7660 - val_loss: 0.9828 - val_acc: 0.6147\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 105s 265ms/step - loss: 0.5912 - acc: 0.7718 - val_loss: 0.9141 - val_acc: 0.6446\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 104s 263ms/step - loss: 0.5818 - acc: 0.7728 - val_loss: 1.0237 - val_acc: 0.6168\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 105s 265ms/step - loss: 0.5769 - acc: 0.7789 - val_loss: 0.9678 - val_acc: 0.6396\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 105s 266ms/step - loss: 0.5701 - acc: 0.7792 - val_loss: 0.9348 - val_acc: 0.6382\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 108s 274ms/step - loss: 0.5702 - acc: 0.7788 - val_loss: 0.8786 - val_acc: 0.6560\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 108s 273ms/step - loss: 0.5524 - acc: 0.7910 - val_loss: 0.9450 - val_acc: 0.6467\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 107s 271ms/step - loss: 0.5430 - acc: 0.7982 - val_loss: 0.8741 - val_acc: 0.6560\n",
      "12630/12630 [==============================] - 30s 2ms/step\n",
      "Train [0.5698398003951954, 0.7844813935452761]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.9110103793144226, 0.6579999998410543]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "395/394 [==============================] - 259s 656ms/step - loss: 0.8759 - acc: 0.6763 - val_loss: 1.1489 - val_acc: 0.5520\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 224s 566ms/step - loss: 0.7005 - acc: 0.7323 - val_loss: 0.8484 - val_acc: 0.6788\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.6416 - acc: 0.7593 - val_loss: 0.7948 - val_acc: 0.6966\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 231s 584ms/step - loss: 0.6142 - acc: 0.7703 - val_loss: 0.8289 - val_acc: 0.6923\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 226s 572ms/step - loss: 0.5771 - acc: 0.7771 - val_loss: 0.9196 - val_acc: 0.6425\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 225s 571ms/step - loss: 0.5634 - acc: 0.7885 - val_loss: 0.7324 - val_acc: 0.7187\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 219s 554ms/step - loss: 0.5401 - acc: 0.7947 - val_loss: 0.7435 - val_acc: 0.7179\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 219s 555ms/step - loss: 0.5300 - acc: 0.7993 - val_loss: 0.6914 - val_acc: 0.7350\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 219s 555ms/step - loss: 0.5136 - acc: 0.8084 - val_loss: 0.6992 - val_acc: 0.7322\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 221s 559ms/step - loss: 0.5062 - acc: 0.8097 - val_loss: 0.7758 - val_acc: 0.7101\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.4975 - acc: 0.8098 - val_loss: 0.6760 - val_acc: 0.7479\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 223s 563ms/step - loss: 0.4811 - acc: 0.8192 - val_loss: 0.7654 - val_acc: 0.7215\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 223s 565ms/step - loss: 0.4719 - acc: 0.8192 - val_loss: 0.6966 - val_acc: 0.7393\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.4695 - acc: 0.8252 - val_loss: 0.8279 - val_acc: 0.7066\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 226s 573ms/step - loss: 0.4639 - acc: 0.8248 - val_loss: 0.7649 - val_acc: 0.7265\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 226s 572ms/step - loss: 0.4524 - acc: 0.8294 - val_loss: 0.6783 - val_acc: 0.7436\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 226s 573ms/step - loss: 0.4540 - acc: 0.8303 - val_loss: 0.6747 - val_acc: 0.7500\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 228s 576ms/step - loss: 0.4392 - acc: 0.8336 - val_loss: 0.6305 - val_acc: 0.7785\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 227s 574ms/step - loss: 0.4413 - acc: 0.8337 - val_loss: 0.6212 - val_acc: 0.7707\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 224s 566ms/step - loss: 0.4216 - acc: 0.8441 - val_loss: 0.6602 - val_acc: 0.7585\n",
      "12630/12630 [==============================] - 49s 4ms/step\n",
      "Train [0.5728233767848872, 0.7882026919937285]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.730690250813961, 0.7419999998410542]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 245s 620ms/step - loss: 0.8687 - acc: 0.6829 - val_loss: 1.2447 - val_acc: 0.5157\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 223s 563ms/step - loss: 0.6990 - acc: 0.7329 - val_loss: 1.1377 - val_acc: 0.5819\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 222s 561ms/step - loss: 0.6457 - acc: 0.7550 - val_loss: 0.9797 - val_acc: 0.6211\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 223s 563ms/step - loss: 0.5995 - acc: 0.7736 - val_loss: 0.9846 - val_acc: 0.6254\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 224s 567ms/step - loss: 0.5828 - acc: 0.7806 - val_loss: 0.8516 - val_acc: 0.6880\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 228s 576ms/step - loss: 0.5572 - acc: 0.7888 - val_loss: 0.7714 - val_acc: 0.7066\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 222s 563ms/step - loss: 0.5498 - acc: 0.7919 - val_loss: 0.7668 - val_acc: 0.7144\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 222s 563ms/step - loss: 0.5169 - acc: 0.8060 - val_loss: 0.9069 - val_acc: 0.6588\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 224s 567ms/step - loss: 0.5158 - acc: 0.8043 - val_loss: 0.8375 - val_acc: 0.6845\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 222s 563ms/step - loss: 0.4992 - acc: 0.8120 - val_loss: 0.7585 - val_acc: 0.7187\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 220s 557ms/step - loss: 0.4966 - acc: 0.8138 - val_loss: 0.7045 - val_acc: 0.7301\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 220s 558ms/step - loss: 0.4769 - acc: 0.8194 - val_loss: 0.7084 - val_acc: 0.7343\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 221s 558ms/step - loss: 0.4723 - acc: 0.8196 - val_loss: 0.7295 - val_acc: 0.7208\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 221s 560ms/step - loss: 0.4594 - acc: 0.8270 - val_loss: 0.6398 - val_acc: 0.7685\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 231s 585ms/step - loss: 0.4505 - acc: 0.8328 - val_loss: 0.6036 - val_acc: 0.7806\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 228s 577ms/step - loss: 0.4368 - acc: 0.8349 - val_loss: 0.6528 - val_acc: 0.7621\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 225s 570ms/step - loss: 0.4294 - acc: 0.8392 - val_loss: 0.6067 - val_acc: 0.7714\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 225s 569ms/step - loss: 0.4270 - acc: 0.8417 - val_loss: 0.6240 - val_acc: 0.7806\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 223s 566ms/step - loss: 0.4133 - acc: 0.8440 - val_loss: 0.5842 - val_acc: 0.7991\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 223s 565ms/step - loss: 0.4158 - acc: 0.8410 - val_loss: 0.6512 - val_acc: 0.7714\n",
      "12630/12630 [==============================] - 50s 4ms/step\n",
      "Train [0.44233245259698006, 0.8398258115786554]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.7519666048685709, 0.7346666666666667]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 248s 628ms/step - loss: 0.9119 - acc: 0.6799 - val_loss: 1.2843 - val_acc: 0.5171\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 224s 567ms/step - loss: 0.7037 - acc: 0.7354 - val_loss: 0.9677 - val_acc: 0.6246\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.6425 - acc: 0.7545 - val_loss: 0.9666 - val_acc: 0.6154\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 221s 559ms/step - loss: 0.6114 - acc: 0.7688 - val_loss: 0.9818 - val_acc: 0.6489\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.5791 - acc: 0.7828 - val_loss: 0.8722 - val_acc: 0.6460\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.5610 - acc: 0.7918 - val_loss: 0.9168 - val_acc: 0.6403\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 222s 562ms/step - loss: 0.5404 - acc: 0.7947 - val_loss: 0.7882 - val_acc: 0.6994\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 222s 561ms/step - loss: 0.5271 - acc: 0.8004 - val_loss: 0.8366 - val_acc: 0.6873\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.5067 - acc: 0.8075 - val_loss: 0.7783 - val_acc: 0.7101\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 225s 569ms/step - loss: 0.4960 - acc: 0.8159 - val_loss: 0.8168 - val_acc: 0.7080\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 223s 563ms/step - loss: 0.4865 - acc: 0.8207 - val_loss: 0.8273 - val_acc: 0.6952\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 224s 567ms/step - loss: 0.4777 - acc: 0.8201 - val_loss: 0.7396 - val_acc: 0.7151\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 223s 565ms/step - loss: 0.4722 - acc: 0.8212 - val_loss: 0.7771 - val_acc: 0.7151\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 226s 572ms/step - loss: 0.4606 - acc: 0.8256 - val_loss: 0.7886 - val_acc: 0.7108\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 225s 571ms/step - loss: 0.4514 - acc: 0.8309 - val_loss: 0.7935 - val_acc: 0.7308\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.4430 - acc: 0.8346 - val_loss: 0.7584 - val_acc: 0.7101\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 223s 564ms/step - loss: 0.4303 - acc: 0.8394 - val_loss: 0.7636 - val_acc: 0.7301\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 224s 567ms/step - loss: 0.4357 - acc: 0.8343 - val_loss: 0.7213 - val_acc: 0.7365\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 220s 556ms/step - loss: 0.4261 - acc: 0.8414 - val_loss: 0.7131 - val_acc: 0.7350\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 221s 558ms/step - loss: 0.4153 - acc: 0.8433 - val_loss: 0.6931 - val_acc: 0.7343\n",
      "12630/12630 [==============================] - 50s 4ms/step\n",
      "Train [0.5281654605956089, 0.8047505938431052]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.782392004330953, 0.715]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 142s 361ms/step - loss: 0.8912 - acc: 0.6565 - val_loss: 1.1390 - val_acc: 0.5499\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 118s 299ms/step - loss: 0.7686 - acc: 0.7003 - val_loss: 1.0695 - val_acc: 0.5755\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 119s 302ms/step - loss: 0.7130 - acc: 0.7273 - val_loss: 1.0244 - val_acc: 0.6211\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 118s 299ms/step - loss: 0.6745 - acc: 0.7430 - val_loss: 0.9215 - val_acc: 0.6517\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 118s 299ms/step - loss: 0.6520 - acc: 0.7538 - val_loss: 0.9205 - val_acc: 0.6581\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 118s 298ms/step - loss: 0.6210 - acc: 0.7638 - val_loss: 0.8018 - val_acc: 0.7073\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 118s 300ms/step - loss: 0.6019 - acc: 0.7707 - val_loss: 0.9513 - val_acc: 0.6083\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 119s 300ms/step - loss: 0.5830 - acc: 0.7794 - val_loss: 0.7365 - val_acc: 0.7358\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 119s 300ms/step - loss: 0.5655 - acc: 0.7872 - val_loss: 0.9581 - val_acc: 0.6652\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.5550 - acc: 0.7929 - val_loss: 0.9907 - val_acc: 0.6517\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 119s 300ms/step - loss: 0.5466 - acc: 0.7947 - val_loss: 0.7765 - val_acc: 0.7066\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 120s 305ms/step - loss: 0.5330 - acc: 0.7994 - val_loss: 0.9044 - val_acc: 0.6717\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 121s 305ms/step - loss: 0.5262 - acc: 0.8006 - val_loss: 0.8081 - val_acc: 0.7044\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 129s 327ms/step - loss: 0.5119 - acc: 0.8088 - val_loss: 0.6788 - val_acc: 0.7443\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 121s 305ms/step - loss: 0.5035 - acc: 0.8076 - val_loss: 0.8453 - val_acc: 0.7066\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 119s 302ms/step - loss: 0.5021 - acc: 0.8096 - val_loss: 0.7658 - val_acc: 0.7329\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.4874 - acc: 0.8127 - val_loss: 0.7627 - val_acc: 0.7144\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 120s 305ms/step - loss: 0.4879 - acc: 0.8157 - val_loss: 0.8183 - val_acc: 0.6973\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.4783 - acc: 0.8186 - val_loss: 0.6402 - val_acc: 0.7585\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.4683 - acc: 0.8212 - val_loss: 0.7071 - val_acc: 0.7422\n",
      "12630/12630 [==============================] - 32s 3ms/step\n",
      "Train [0.5615977612781977, 0.8011084718545656]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.7241855516433716, 0.7370000001589457]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 145s 367ms/step - loss: 0.8679 - acc: 0.6666 - val_loss: 1.0669 - val_acc: 0.5862\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.7384 - acc: 0.7149 - val_loss: 1.0735 - val_acc: 0.5570\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.6859 - acc: 0.7396 - val_loss: 0.9173 - val_acc: 0.6125\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 119s 300ms/step - loss: 0.6512 - acc: 0.7510 - val_loss: 0.8929 - val_acc: 0.6660\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 121s 306ms/step - loss: 0.6170 - acc: 0.7658 - val_loss: 0.9329 - val_acc: 0.6660\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 125s 316ms/step - loss: 0.5962 - acc: 0.7794 - val_loss: 1.0448 - val_acc: 0.6147\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.5724 - acc: 0.7859 - val_loss: 0.8340 - val_acc: 0.6895\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 118s 300ms/step - loss: 0.5577 - acc: 0.7903 - val_loss: 0.8300 - val_acc: 0.6973\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 118s 299ms/step - loss: 0.5507 - acc: 0.7949 - val_loss: 0.7903 - val_acc: 0.6973\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 118s 300ms/step - loss: 0.5357 - acc: 0.7998 - val_loss: 0.7228 - val_acc: 0.7443\n",
      "Epoch 11/20\n",
      "395/394 [==============================] - 118s 300ms/step - loss: 0.5291 - acc: 0.8000 - val_loss: 0.7580 - val_acc: 0.7308\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.5126 - acc: 0.8090 - val_loss: 0.8057 - val_acc: 0.6916\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 118s 300ms/step - loss: 0.5093 - acc: 0.8075 - val_loss: 0.7219 - val_acc: 0.7322\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 119s 301ms/step - loss: 0.5012 - acc: 0.8113 - val_loss: 0.7599 - val_acc: 0.7265\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.4921 - acc: 0.8161 - val_loss: 0.6982 - val_acc: 0.7479\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 121s 305ms/step - loss: 0.4885 - acc: 0.8141 - val_loss: 0.6826 - val_acc: 0.7486\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 125s 317ms/step - loss: 0.4806 - acc: 0.8213 - val_loss: 0.6772 - val_acc: 0.7578\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 119s 302ms/step - loss: 0.4778 - acc: 0.8219 - val_loss: 0.8540 - val_acc: 0.6937\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 119s 302ms/step - loss: 0.4698 - acc: 0.8238 - val_loss: 0.7379 - val_acc: 0.7308\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 122s 310ms/step - loss: 0.4604 - acc: 0.8248 - val_loss: 0.6322 - val_acc: 0.7707\n",
      "12630/12630 [==============================] - 32s 3ms/step\n",
      "Train [0.569099187426216, 0.798178939052923]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.6588128871122996, 0.7696666666666667]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "395/394 [==============================] - 143s 361ms/step - loss: 0.8916 - acc: 0.6537 - val_loss: 1.0606 - val_acc: 0.5627\n",
      "Epoch 2/20\n",
      "395/394 [==============================] - 119s 302ms/step - loss: 0.7565 - acc: 0.7096 - val_loss: 1.0680 - val_acc: 0.6011\n",
      "Epoch 3/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.7068 - acc: 0.7314 - val_loss: 0.9249 - val_acc: 0.6432\n",
      "Epoch 4/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.6661 - acc: 0.7447 - val_loss: 0.9922 - val_acc: 0.6026\n",
      "Epoch 5/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.6304 - acc: 0.7623 - val_loss: 0.9425 - val_acc: 0.6125\n",
      "Epoch 6/20\n",
      "395/394 [==============================] - 119s 302ms/step - loss: 0.6051 - acc: 0.7745 - val_loss: 0.9631 - val_acc: 0.6353\n",
      "Epoch 7/20\n",
      "395/394 [==============================] - 120s 305ms/step - loss: 0.5858 - acc: 0.7808 - val_loss: 0.8534 - val_acc: 0.6838\n",
      "Epoch 8/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.5719 - acc: 0.7840 - val_loss: 0.8901 - val_acc: 0.6510\n",
      "Epoch 9/20\n",
      "395/394 [==============================] - 120s 303ms/step - loss: 0.5534 - acc: 0.7932 - val_loss: 0.8359 - val_acc: 0.6852\n",
      "Epoch 10/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.5393 - acc: 0.7964 - val_loss: 0.8693 - val_acc: 0.6588\n",
      "Epoch 11/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "395/394 [==============================] - 120s 304ms/step - loss: 0.5260 - acc: 0.8031 - val_loss: 0.7756 - val_acc: 0.7187\n",
      "Epoch 12/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.5237 - acc: 0.8070 - val_loss: 0.7059 - val_acc: 0.7372\n",
      "Epoch 13/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.5126 - acc: 0.8060 - val_loss: 0.7515 - val_acc: 0.7130\n",
      "Epoch 14/20\n",
      "395/394 [==============================] - 120s 305ms/step - loss: 0.5009 - acc: 0.8134 - val_loss: 0.8458 - val_acc: 0.6702\n",
      "Epoch 15/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.4949 - acc: 0.8139 - val_loss: 0.8784 - val_acc: 0.6752\n",
      "Epoch 16/20\n",
      "395/394 [==============================] - 121s 306ms/step - loss: 0.4884 - acc: 0.8174 - val_loss: 0.8304 - val_acc: 0.6994\n",
      "Epoch 17/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.4813 - acc: 0.8193 - val_loss: 0.8522 - val_acc: 0.7201\n",
      "Epoch 18/20\n",
      "395/394 [==============================] - 120s 304ms/step - loss: 0.4744 - acc: 0.8207 - val_loss: 0.6281 - val_acc: 0.7735\n",
      "Epoch 19/20\n",
      "395/394 [==============================] - 120s 305ms/step - loss: 0.4746 - acc: 0.8252 - val_loss: 0.9103 - val_acc: 0.6766\n",
      "Epoch 20/20\n",
      "395/394 [==============================] - 121s 306ms/step - loss: 0.4686 - acc: 0.8251 - val_loss: 0.6459 - val_acc: 0.7692\n",
      "12630/12630 [==============================] - 32s 3ms/step\n",
      "Train [0.6297786386354708, 0.7756927949043841]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.6851620855331421, 0.7553333333333333]\n"
     ]
    }
   ],
   "source": [
    "import keras.optimizers\n",
    "for lr in [0.1, 0.01, 0.001, 0.0001]:\n",
    "    optimizer = optimizers.Adam(lr=lr)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITHOUT BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetC((150, 150, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH LAYER BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetA((150, 150, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH IDENTITY BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetB((150, 150, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks for Fruits-360 for He-normal for Models 1, 2 and 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 14.0882 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0858 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 121s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 121s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 32s 857us/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 9s 850us/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 14.0917 - acc: 0.1256 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 118s 102ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 118s 102ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 118s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 118s 102ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 32s 864us/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 9s 852us/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.4968 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 118s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4948 - acc: 0.0387 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 118s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 118s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 118s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 118s 102ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 118s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 118s 102ms/step - loss: 15.4948 - acc: 0.0387 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 118s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 118s 102ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 118s 102ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 118s 102ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 118s 103ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "36905/36905 [==============================] - 32s 854us/step\n",
      "Train [15.496606202817452, 0.0385584609131554]\n",
      "10252/10252 [==============================] - 9s 853us/step\n",
      "Test [15.495507764946517, 0.038626609447874036]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 14.0813 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 257s 222ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 257s 222ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 257s 223ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 262s 227ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 265s 230ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 257s 223ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 259s 224ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 260s 226ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 260s 225ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 262s 227ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1608 - val_acc: 0.1214\n",
      "36905/36905 [==============================] - 58s 2ms/step\n",
      "Train [14.089412142768202, 0.1258637041053367]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [14.0475206378843, 0.12846273897776045]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 14.0802 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 263s 228ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 263s 228ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 263s 228ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 261s 226ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 258s 223ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 258s 223ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 54s 1ms/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 15s 1ms/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.9066 - acc: 0.0126 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 260s 225ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.9127 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9127 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 15.9127 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 15.9127 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.9138 - acc: 0.0127 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "36905/36905 [==============================] - 55s 1ms/step\n",
      "Train [15.91369837518146, 0.012681208508332204]\n",
      "10252/10252 [==============================] - 15s 1ms/step\n",
      "Test [15.901133027897187, 0.013460788138899726]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 15.5659 - acc: 0.0332 - val_loss: 15.4735 - val_acc: 0.0400\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.6754 - acc: 0.0275 - val_loss: 15.4185 - val_acc: 0.0434\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.4199 - acc: 0.0433 - val_loss: 15.4185 - val_acc: 0.0434\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 15.4938 - acc: 0.0387 - val_loss: 15.5082 - val_acc: 0.0378\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.5168 - acc: 0.0373 - val_loss: 15.4185 - val_acc: 0.0434\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.4037 - acc: 0.0443 - val_loss: 15.4185 - val_acc: 0.0434\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.4055 - acc: 0.0442 - val_loss: 15.4185 - val_acc: 0.0434\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.4086 - acc: 0.0440 - val_loss: 15.4185 - val_acc: 0.0434\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.4075 - acc: 0.0441 - val_loss: 15.4185 - val_acc: 0.0434\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.7069 - acc: 0.0255 - val_loss: 15.7015 - val_acc: 0.0258\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.7296 - acc: 0.0241 - val_loss: 15.6976 - val_acc: 0.0261\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.7361 - acc: 0.0237 - val_loss: 15.6740 - val_acc: 0.0276\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.7205 - acc: 0.0247 - val_loss: 15.6740 - val_acc: 0.0276\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.7240 - acc: 0.0245 - val_loss: 15.6740 - val_acc: 0.0276\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.7213 - acc: 0.0246 - val_loss: 15.6740 - val_acc: 0.0276\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.7202 - acc: 0.0247 - val_loss: 15.6740 - val_acc: 0.0276\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.7167 - acc: 0.0249 - val_loss: 15.6740 - val_acc: 0.0276\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.7248 - acc: 0.0244 - val_loss: 15.6740 - val_acc: 0.0276\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.7462 - acc: 0.0231 - val_loss: 15.8037 - val_acc: 0.0195\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 15.7264 - acc: 0.0243 - val_loss: 15.6740 - val_acc: 0.0276\n",
      "36905/36905 [==============================] - 34s 929us/step\n",
      "Train [15.703623674633283, 0.025714672808562527]\n",
      "10252/10252 [==============================] - 10s 929us/step\n",
      "Test [15.731336411302367, 0.023995317986734297]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 14.1896 - acc: 0.1187 - val_loss: 14.2905 - val_acc: 0.1134\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0920 - acc: 0.1257 - val_loss: 14.2942 - val_acc: 0.1131\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0924 - acc: 0.1257 - val_loss: 14.2866 - val_acc: 0.1136\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0907 - acc: 0.1258 - val_loss: 14.2942 - val_acc: 0.1131\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 14.0889 - acc: 0.1259 - val_loss: 14.2944 - val_acc: 0.1131\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0922 - acc: 0.1257 - val_loss: 14.2866 - val_acc: 0.1136\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.2866 - val_acc: 0.1136\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0896 - acc: 0.1259 - val_loss: 14.2984 - val_acc: 0.1129\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.2866 - val_acc: 0.1136\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0889 - acc: 0.1259 - val_loss: 14.2984 - val_acc: 0.1129\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0913 - acc: 0.1257 - val_loss: 14.2944 - val_acc: 0.1131\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 14.0907 - acc: 0.1258 - val_loss: 14.2984 - val_acc: 0.1129\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0896 - acc: 0.1259 - val_loss: 14.2931 - val_acc: 0.1131\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 14.0890 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 35s 945us/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 10s 933us/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  2\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 15.8999 - acc: 0.0126 - val_loss: 15.9609 - val_acc: 0.0098\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.9155 - acc: 0.0126 - val_loss: 15.9570 - val_acc: 0.0100\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 15.1679 - acc: 0.0589 - val_loss: 14.4910 - val_acc: 0.1010\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.4870 - val_acc: 0.1012\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4962 - acc: 0.1006 - val_loss: 14.4752 - val_acc: 0.1019\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4887 - acc: 0.1011 - val_loss: 14.4792 - val_acc: 0.1017\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4643 - acc: 0.1026 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4477 - acc: 0.1036 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.4437 - acc: 0.1039 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4455 - acc: 0.1038 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4460 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4464 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.4448 - acc: 0.1038 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 14.4457 - acc: 0.1038 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4433 - acc: 0.1039 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4473 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 14.4462 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4479 - acc: 0.1036 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 14.4446 - acc: 0.1038 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 14.4464 - acc: 0.1037 - val_loss: 14.4438 - val_acc: 0.1039\n",
      "36905/36905 [==============================] - 35s 935us/step\n",
      "Train [14.444486209232137, 0.103834168812016]\n",
      "10252/10252 [==============================] - 10s 934us/step\n",
      "Test [14.464150959376568, 0.1026141240791655]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 14.0897 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0858 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 121s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 32s 876us/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 9s 881us/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 14.0895 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 121s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0858 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 32s 873us/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 9s 873us/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 15.8965 - acc: 0.0137 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8979 - acc: 0.0137 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 15.8979 - acc: 0.0137 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 15.8990 - acc: 0.0136 - val_loss: 15.9412 - val_acc: 0.0110\n",
      "36905/36905 [==============================] - 32s 881us/step\n",
      "Train [15.898849021127079, 0.013602492887142664]\n",
      "10252/10252 [==============================] - 9s 877us/step\n",
      "Test [15.946726575383348, 0.010632071790870074]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 15.2697 - acc: 0.0518 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 15.2999 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 15.2999 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.2999 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 257s 222ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 250s 216ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 15.3010 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "36905/36905 [==============================] - 55s 1ms/step\n",
      "Train [15.300507428435804, 0.050724834033328815]\n",
      "10252/10252 [==============================] - 15s 1ms/step\n",
      "Test [15.265967945079527, 0.05286773312524386]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 14.0846 - acc: 0.1254 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 250s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 57s 2ms/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 270s 234ms/step - loss: 14.0825 - acc: 0.1256 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 265s 230ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 250s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 250s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 14.0858 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0869 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0880 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 14.0902 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 56s 2ms/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 15s 1ms/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 1.9199 - acc: 0.4422 - val_loss: 1.8997 - val_acc: 0.4535\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.8038 - acc: 0.7320 - val_loss: 0.6294 - val_acc: 0.7776\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.4170 - acc: 0.8547 - val_loss: 0.3288 - val_acc: 0.8961\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.2390 - acc: 0.9152 - val_loss: 0.3117 - val_acc: 0.9108\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1913 - acc: 0.9334 - val_loss: 0.1063 - val_acc: 0.9595\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.1580 - acc: 0.9427 - val_loss: 0.1363 - val_acc: 0.9476\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1344 - acc: 0.9506 - val_loss: 0.0759 - val_acc: 0.9703\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1221 - acc: 0.9532 - val_loss: 0.0973 - val_acc: 0.9627\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1145 - acc: 0.9574 - val_loss: 0.4933 - val_acc: 0.8830\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1091 - acc: 0.9587 - val_loss: 0.0663 - val_acc: 0.9715\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1002 - acc: 0.9627 - val_loss: 0.0737 - val_acc: 0.9722\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.0839 - acc: 0.9677 - val_loss: 0.0527 - val_acc: 0.9776\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1013 - acc: 0.9633 - val_loss: 0.0606 - val_acc: 0.9727\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0797 - acc: 0.9700 - val_loss: 0.0449 - val_acc: 0.9820\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0838 - acc: 0.9685 - val_loss: 0.0358 - val_acc: 0.9856\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0782 - acc: 0.9699 - val_loss: 0.0402 - val_acc: 0.9827\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0831 - acc: 0.9676 - val_loss: 0.0788 - val_acc: 0.9693\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.0653 - acc: 0.9731 - val_loss: 0.0768 - val_acc: 0.9698\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.0693 - acc: 0.9724 - val_loss: 0.0880 - val_acc: 0.9681\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0682 - acc: 0.9735 - val_loss: 0.0523 - val_acc: 0.9807\n",
      "36905/36905 [==============================] - 35s 956us/step\n",
      "Train [0.05727835456773378, 0.9794065844736486]\n",
      "10252/10252 [==============================] - 10s 958us/step\n",
      "Test [0.056855838296159526, 0.9790284822473664]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 2.3493 - acc: 0.3352 - val_loss: 2.1540 - val_acc: 0.3428\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.9881 - acc: 0.6806 - val_loss: 0.9791 - val_acc: 0.6506\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.5087 - acc: 0.8273 - val_loss: 0.7333 - val_acc: 0.7642\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.3050 - acc: 0.8946 - val_loss: 1.8633 - val_acc: 0.6116\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.2272 - acc: 0.9194 - val_loss: 0.8002 - val_acc: 0.7810\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.1846 - acc: 0.9326 - val_loss: 0.1916 - val_acc: 0.9320\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1625 - acc: 0.9406 - val_loss: 0.6188 - val_acc: 0.8386\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1469 - acc: 0.9458 - val_loss: 0.2057 - val_acc: 0.9225\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.1168 - acc: 0.9559 - val_loss: 0.3395 - val_acc: 0.8810\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1171 - acc: 0.9564 - val_loss: 0.5165 - val_acc: 0.8791\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1115 - acc: 0.9588 - val_loss: 0.7416 - val_acc: 0.8152\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0978 - acc: 0.9627 - val_loss: 0.2154 - val_acc: 0.9422\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0975 - acc: 0.9633 - val_loss: 0.0795 - val_acc: 0.9654\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0840 - acc: 0.9672 - val_loss: 0.1175 - val_acc: 0.9585\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0881 - acc: 0.9671 - val_loss: 0.1147 - val_acc: 0.9554\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0760 - acc: 0.9706 - val_loss: 0.1327 - val_acc: 0.9539\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0729 - acc: 0.9704 - val_loss: 0.0865 - val_acc: 0.9673\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0683 - acc: 0.9730 - val_loss: 0.1372 - val_acc: 0.9490\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0744 - acc: 0.9723 - val_loss: 0.1391 - val_acc: 0.9585\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.0764 - acc: 0.9718 - val_loss: 0.1033 - val_acc: 0.9639\n",
      "36905/36905 [==============================] - 36s 966us/step\n",
      "Train [0.10425853950058939, 0.9625254030619157]\n",
      "10252/10252 [==============================] - 10s 956us/step\n",
      "Test [0.10350305934427899, 0.9616660163870464]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 2.2574 - acc: 0.3590 - val_loss: 2.1157 - val_acc: 0.3382\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.8713 - acc: 0.7208 - val_loss: 1.4675 - val_acc: 0.6503\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.4465 - acc: 0.8465 - val_loss: 1.2112 - val_acc: 0.7223\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.3159 - acc: 0.8872 - val_loss: 1.0377 - val_acc: 0.7362\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.2403 - acc: 0.9147 - val_loss: 0.3690 - val_acc: 0.8717\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1968 - acc: 0.9282 - val_loss: 0.2987 - val_acc: 0.8883\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.1740 - acc: 0.9374 - val_loss: 0.1507 - val_acc: 0.9420\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.1460 - acc: 0.9479 - val_loss: 0.1686 - val_acc: 0.9320\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1404 - acc: 0.9500 - val_loss: 0.1192 - val_acc: 0.9568\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.1188 - acc: 0.9553 - val_loss: 0.1214 - val_acc: 0.9573\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.1054 - acc: 0.9613 - val_loss: 0.2014 - val_acc: 0.9327\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.1048 - acc: 0.9611 - val_loss: 0.1441 - val_acc: 0.9459\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.1008 - acc: 0.9628 - val_loss: 0.2024 - val_acc: 0.9320\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0958 - acc: 0.9651 - val_loss: 0.0710 - val_acc: 0.9756\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.0755 - acc: 0.9702 - val_loss: 0.0933 - val_acc: 0.9590\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0921 - acc: 0.9654 - val_loss: 0.1627 - val_acc: 0.9393\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 132s 115ms/step - loss: 0.0750 - acc: 0.9701 - val_loss: 0.0697 - val_acc: 0.9754\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0783 - acc: 0.9696 - val_loss: 0.1160 - val_acc: 0.9683\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0776 - acc: 0.9704 - val_loss: 0.0677 - val_acc: 0.9749\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0801 - acc: 0.9714 - val_loss: 0.0480 - val_acc: 0.9812\n",
      "36905/36905 [==============================] - 36s 965us/step\n",
      "Train [0.0553755560595119, 0.978620783091722]\n",
      "10252/10252 [==============================] - 10s 960us/step\n",
      "Test [0.05520571416201305, 0.9783456886461178]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 123s 107ms/step - loss: 0.9568 - acc: 0.7241 - val_loss: 0.3357 - val_acc: 0.8876\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.2422 - acc: 0.9194 - val_loss: 0.1874 - val_acc: 0.9337\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 121s 104ms/step - loss: 0.1716 - acc: 0.9426 - val_loss: 0.2232 - val_acc: 0.9295\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.1043 - acc: 0.9618 - val_loss: 0.0811 - val_acc: 0.9712\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.0878 - acc: 0.9676 - val_loss: 0.1988 - val_acc: 0.9364\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 124s 108ms/step - loss: 0.0872 - acc: 0.9686 - val_loss: 0.1256 - val_acc: 0.9507\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0837 - acc: 0.9680 - val_loss: 0.0481 - val_acc: 0.9798\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 121s 104ms/step - loss: 0.0755 - acc: 0.9726 - val_loss: 0.1972 - val_acc: 0.9588\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 0.0645 - acc: 0.9751 - val_loss: 0.1094 - val_acc: 0.9532\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.0611 - acc: 0.9754 - val_loss: 0.1881 - val_acc: 0.9451\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.0808 - acc: 0.9750 - val_loss: 0.0422 - val_acc: 0.9846\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.0522 - acc: 0.9780 - val_loss: 0.0713 - val_acc: 0.9729\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.0563 - acc: 0.9768 - val_loss: 0.0782 - val_acc: 0.9698\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.0441 - acc: 0.9808 - val_loss: 0.1115 - val_acc: 0.9581\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.0536 - acc: 0.9777 - val_loss: 0.0443 - val_acc: 0.9812\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.0471 - acc: 0.9806 - val_loss: 0.0521 - val_acc: 0.9776\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 0.0468 - acc: 0.9795 - val_loss: 0.0389 - val_acc: 0.9822\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.0454 - acc: 0.9796 - val_loss: 0.0715 - val_acc: 0.9795\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.1314 - acc: 0.9728 - val_loss: 0.2689 - val_acc: 0.9676\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 0.2490 - acc: 0.9694 - val_loss: 0.4210 - val_acc: 0.9281\n",
      "36905/36905 [==============================] - 33s 903us/step\n",
      "Train [0.3634710467340395, 0.9360520254708035]\n",
      "10252/10252 [==============================] - 9s 902us/step\n",
      "Test [0.35617262881277667, 0.934159188451034]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 15.4873 - acc: 0.0388 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 119s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 121s 104ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.4948 - acc: 0.0387 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 119s 103ms/step - loss: 15.4970 - acc: 0.0385 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.4959 - acc: 0.0386 - val_loss: 15.5325 - val_acc: 0.0363\n",
      "36905/36905 [==============================] - 33s 892us/step\n",
      "Train [15.496606202817452, 0.0385584609131554]\n",
      "10252/10252 [==============================] - 9s 894us/step\n",
      "Test [15.495507764946517, 0.038626609447874036]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 123s 107ms/step - loss: 15.9099 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 15.9105 - acc: 0.0129 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 121s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 120s 104ms/step - loss: 15.9116 - acc: 0.0128 - val_loss: 15.8823 - val_acc: 0.0146\n",
      "36905/36905 [==============================] - 33s 902us/step\n",
      "Train [15.911514644625386, 0.012816691505216096]\n",
      "10252/10252 [==============================] - 9s 899us/step\n",
      "Test [15.924715897183933, 0.011997658993367149]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 252s 219ms/step - loss: 0.5729 - acc: 0.8254 - val_loss: 0.3746 - val_acc: 0.8713\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.1477 - acc: 0.9472 - val_loss: 0.1536 - val_acc: 0.9483\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 263s 228ms/step - loss: 0.0989 - acc: 0.9641 - val_loss: 0.2343 - val_acc: 0.9193\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 0.0886 - acc: 0.9662 - val_loss: 0.2560 - val_acc: 0.9225\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 0.0735 - acc: 0.9713 - val_loss: 0.1270 - val_acc: 0.9556\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 0.0661 - acc: 0.9725 - val_loss: 0.0723 - val_acc: 0.9720\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0612 - acc: 0.9748 - val_loss: 0.1077 - val_acc: 0.9666\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 0.0572 - acc: 0.9759 - val_loss: 0.1486 - val_acc: 0.9456\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0552 - acc: 0.9769 - val_loss: 0.1292 - val_acc: 0.9559\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0479 - acc: 0.9791 - val_loss: 0.1308 - val_acc: 0.9578\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0508 - acc: 0.9783 - val_loss: 0.0920 - val_acc: 0.9605\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0500 - acc: 0.9782 - val_loss: 0.2368 - val_acc: 0.9295\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0495 - acc: 0.9784 - val_loss: 0.2213 - val_acc: 0.9507\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 0.0435 - acc: 0.9800 - val_loss: 0.0453 - val_acc: 0.9805\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0405 - acc: 0.9810 - val_loss: 0.0548 - val_acc: 0.9768\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0453 - acc: 0.9806 - val_loss: 0.0701 - val_acc: 0.9766\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0409 - acc: 0.9816 - val_loss: 0.0384 - val_acc: 0.9837\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 258s 223ms/step - loss: 0.0392 - acc: 0.9814 - val_loss: 0.0260 - val_acc: 0.9878\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 258s 223ms/step - loss: 0.0381 - acc: 0.9828 - val_loss: 0.0395 - val_acc: 0.9837\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 0.0394 - acc: 0.9823 - val_loss: 0.0436 - val_acc: 0.9837\n",
      "36905/36905 [==============================] - 61s 2ms/step\n",
      "Train [0.043537130206427466, 0.9832001083863975]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.04902634205987281, 0.9813694888802185]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 0.6206 - acc: 0.8219 - val_loss: 0.2838 - val_acc: 0.9042\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 249s 215ms/step - loss: 0.1500 - acc: 0.9460 - val_loss: 0.3022 - val_acc: 0.9027\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.1089 - acc: 0.9595 - val_loss: 0.2824 - val_acc: 0.9217\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0915 - acc: 0.9656 - val_loss: 0.1652 - val_acc: 0.9347\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0775 - acc: 0.9701 - val_loss: 0.1239 - val_acc: 0.9607\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0660 - acc: 0.9734 - val_loss: 0.1998 - val_acc: 0.9388\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0655 - acc: 0.9740 - val_loss: 0.2894 - val_acc: 0.9354\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0596 - acc: 0.9753 - val_loss: 0.1059 - val_acc: 0.9600\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0554 - acc: 0.9764 - val_loss: 0.0751 - val_acc: 0.9739\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0521 - acc: 0.9777 - val_loss: 0.1130 - val_acc: 0.9585\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0506 - acc: 0.9783 - val_loss: 0.0499 - val_acc: 0.9761\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 259s 225ms/step - loss: 0.0498 - acc: 0.9780 - val_loss: 0.1541 - val_acc: 0.9624\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0439 - acc: 0.9808 - val_loss: 0.0531 - val_acc: 0.9790\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0493 - acc: 0.9788 - val_loss: 0.0496 - val_acc: 0.9781\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0393 - acc: 0.9820 - val_loss: 0.0461 - val_acc: 0.9822\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 0.0464 - acc: 0.9799 - val_loss: 0.0281 - val_acc: 0.9861\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0413 - acc: 0.9810 - val_loss: 0.0456 - val_acc: 0.9805\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0420 - acc: 0.9807 - val_loss: 0.0249 - val_acc: 0.9881\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 247s 214ms/step - loss: 0.0415 - acc: 0.9809 - val_loss: 0.0672 - val_acc: 0.9727\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0387 - acc: 0.9810 - val_loss: 0.0585 - val_acc: 0.9800\n",
      "36905/36905 [==============================] - 56s 2ms/step\n",
      "Train [0.05713966680129657, 0.9767782143341011]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.059964513120774166, 0.9758095981271947]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 260s 226ms/step - loss: 0.7748 - acc: 0.8125 - val_loss: 0.5075 - val_acc: 0.9027\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.3477 - acc: 0.9366 - val_loss: 0.4571 - val_acc: 0.9078\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.3111 - acc: 0.9477 - val_loss: 0.4126 - val_acc: 0.9232\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.2434 - acc: 0.9575 - val_loss: 0.1540 - val_acc: 0.9451\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 0.0711 - acc: 0.9707 - val_loss: 0.1953 - val_acc: 0.9429\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 250s 216ms/step - loss: 0.0656 - acc: 0.9738 - val_loss: 0.1754 - val_acc: 0.9449\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 0.0604 - acc: 0.9746 - val_loss: 0.1525 - val_acc: 0.9503\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0520 - acc: 0.9782 - val_loss: 0.0604 - val_acc: 0.9781\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0607 - acc: 0.9754 - val_loss: 0.0520 - val_acc: 0.9768\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0521 - acc: 0.9778 - val_loss: 0.1686 - val_acc: 0.9415\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 262s 227ms/step - loss: 0.0494 - acc: 0.9791 - val_loss: 0.0696 - val_acc: 0.9754\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.0466 - acc: 0.9793 - val_loss: 0.0320 - val_acc: 0.9868\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 253s 219ms/step - loss: 0.0404 - acc: 0.9813 - val_loss: 0.0375 - val_acc: 0.9795\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0507 - acc: 0.9786 - val_loss: 0.1325 - val_acc: 0.9578\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0430 - acc: 0.9815 - val_loss: 0.0490 - val_acc: 0.9783\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0436 - acc: 0.9808 - val_loss: 0.0361 - val_acc: 0.9854\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 0.0380 - acc: 0.9823 - val_loss: 0.1049 - val_acc: 0.9556\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0443 - acc: 0.9809 - val_loss: 0.0742 - val_acc: 0.9742\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 258s 224ms/step - loss: 0.0381 - acc: 0.9816 - val_loss: 0.0294 - val_acc: 0.9861\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 252s 218ms/step - loss: 0.0402 - acc: 0.9813 - val_loss: 0.0271 - val_acc: 0.9871\n",
      "36905/36905 [==============================] - 58s 2ms/step\n",
      "Train [0.03338253934027419, 0.9838233301720634]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [0.03530925782023911, 0.9840031213421772]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.7613 - acc: 0.7691 - val_loss: 0.6182 - val_acc: 0.7891\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1992 - acc: 0.9346 - val_loss: 0.2384 - val_acc: 0.9127\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.1183 - acc: 0.9577 - val_loss: 0.1343 - val_acc: 0.9522\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0877 - acc: 0.9680 - val_loss: 0.0837 - val_acc: 0.9659\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0777 - acc: 0.9699 - val_loss: 0.1162 - val_acc: 0.9603\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0680 - acc: 0.9728 - val_loss: 0.2536 - val_acc: 0.9244\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.0609 - acc: 0.9752 - val_loss: 0.1124 - val_acc: 0.9559\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0565 - acc: 0.9762 - val_loss: 0.0671 - val_acc: 0.9703\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0540 - acc: 0.9776 - val_loss: 0.0836 - val_acc: 0.9629\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0482 - acc: 0.9790 - val_loss: 0.0318 - val_acc: 0.9866\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0467 - acc: 0.9800 - val_loss: 0.1275 - val_acc: 0.9615\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0447 - acc: 0.9798 - val_loss: 0.0568 - val_acc: 0.9815\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.0418 - acc: 0.9813 - val_loss: 0.0788 - val_acc: 0.9690\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 133s 116ms/step - loss: 0.0398 - acc: 0.9823 - val_loss: 0.0494 - val_acc: 0.9773\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0369 - acc: 0.9828 - val_loss: 0.0632 - val_acc: 0.9717\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0437 - acc: 0.9797 - val_loss: 0.0321 - val_acc: 0.9854\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0340 - acc: 0.9836 - val_loss: 0.1225 - val_acc: 0.9654\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0392 - acc: 0.9819 - val_loss: 0.0390 - val_acc: 0.9815\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0371 - acc: 0.9822 - val_loss: 0.0397 - val_acc: 0.9829\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0321 - acc: 0.9840 - val_loss: 0.0393 - val_acc: 0.9812\n",
      "36905/36905 [==============================] - 36s 970us/step\n",
      "Train [0.04248631164946254, 0.9807614144424874]\n",
      "10252/10252 [==============================] - 10s 968us/step\n",
      "Test [0.04644123991980235, 0.9777604369879048]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 138s 119ms/step - loss: 0.7606 - acc: 0.7737 - val_loss: 0.4787 - val_acc: 0.8213\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.2078 - acc: 0.9309 - val_loss: 0.2040 - val_acc: 0.9237\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1160 - acc: 0.9578 - val_loss: 0.1264 - val_acc: 0.9527\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0860 - acc: 0.9679 - val_loss: 0.2064 - val_acc: 0.9295\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0775 - acc: 0.9692 - val_loss: 0.0674 - val_acc: 0.9734\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0653 - acc: 0.9732 - val_loss: 0.1005 - val_acc: 0.9615\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0549 - acc: 0.9774 - val_loss: 0.1249 - val_acc: 0.9495\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0596 - acc: 0.9757 - val_loss: 0.1648 - val_acc: 0.9398\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.0504 - acc: 0.9782 - val_loss: 0.1213 - val_acc: 0.9568\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0508 - acc: 0.9782 - val_loss: 0.5209 - val_acc: 0.8927\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 133s 115ms/step - loss: 0.0460 - acc: 0.9792 - val_loss: 0.0692 - val_acc: 0.9690\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0486 - acc: 0.9796 - val_loss: 0.0539 - val_acc: 0.9781\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0419 - acc: 0.9807 - val_loss: 0.0572 - val_acc: 0.9759\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0370 - acc: 0.9819 - val_loss: 0.0242 - val_acc: 0.9888\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 0.0399 - acc: 0.9812 - val_loss: 0.0235 - val_acc: 0.9883\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0450 - acc: 0.9809 - val_loss: 0.0236 - val_acc: 0.9883\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 138s 119ms/step - loss: 0.0373 - acc: 0.9822 - val_loss: 0.0381 - val_acc: 0.9842\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0385 - acc: 0.9820 - val_loss: 0.1361 - val_acc: 0.9559\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 138s 119ms/step - loss: 0.0342 - acc: 0.9836 - val_loss: 0.0507 - val_acc: 0.9778\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 138s 119ms/step - loss: 0.0372 - acc: 0.9822 - val_loss: 0.0258 - val_acc: 0.9883\n",
      "36905/36905 [==============================] - 37s 1ms/step\n",
      "Train [0.028072355634550787, 0.9854762227340469]\n",
      "10252/10252 [==============================] - 10s 1ms/step\n",
      "Test [0.030031298064435972, 0.9851736246586033]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 141s 122ms/step - loss: 0.7918 - acc: 0.7646 - val_loss: 0.4178 - val_acc: 0.8825\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.2013 - acc: 0.9338 - val_loss: 0.2226 - val_acc: 0.9139\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1207 - acc: 0.9574 - val_loss: 0.1222 - val_acc: 0.9568\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0953 - acc: 0.9638 - val_loss: 0.0994 - val_acc: 0.9634\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0812 - acc: 0.9680 - val_loss: 0.1206 - val_acc: 0.9537\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0657 - acc: 0.9742 - val_loss: 0.1153 - val_acc: 0.9551\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0644 - acc: 0.9733 - val_loss: 0.0789 - val_acc: 0.9710\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0525 - acc: 0.9778 - val_loss: 0.0755 - val_acc: 0.9698\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0544 - acc: 0.9774 - val_loss: 0.2097 - val_acc: 0.9276\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0523 - acc: 0.9787 - val_loss: 0.0424 - val_acc: 0.9854\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 137s 118ms/step - loss: 0.0520 - acc: 0.9775 - val_loss: 0.0337 - val_acc: 0.9868\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0425 - acc: 0.9808 - val_loss: 0.0392 - val_acc: 0.9815\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0438 - acc: 0.9803 - val_loss: 0.0281 - val_acc: 0.9873\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0433 - acc: 0.9805 - val_loss: 0.1767 - val_acc: 0.9464\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 139s 121ms/step - loss: 0.0414 - acc: 0.9811 - val_loss: 0.0470 - val_acc: 0.9812\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0396 - acc: 0.9814 - val_loss: 0.0297 - val_acc: 0.9849\n",
      "Epoch 17/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0411 - acc: 0.9820 - val_loss: 0.0530 - val_acc: 0.9771\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0372 - acc: 0.9828 - val_loss: 0.0482 - val_acc: 0.9837\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0386 - acc: 0.9819 - val_loss: 0.0249 - val_acc: 0.9883\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 137s 119ms/step - loss: 0.0397 - acc: 0.9810 - val_loss: 0.0216 - val_acc: 0.9898\n",
      "36905/36905 [==============================] - 36s 976us/step\n",
      "Train [0.025248373208029726, 0.986803956103509]\n",
      "10252/10252 [==============================] - 10s 984us/step\n",
      "Test [0.02779397345039938, 0.9865392118611003]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 128s 111ms/step - loss: 1.4890 - acc: 0.5784 - val_loss: 0.9808 - val_acc: 0.7042\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 0.5421 - acc: 0.8278 - val_loss: 0.5849 - val_acc: 0.7983\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 124s 107ms/step - loss: 0.3569 - acc: 0.8807 - val_loss: 0.3768 - val_acc: 0.8593\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.2746 - acc: 0.9061 - val_loss: 0.2737 - val_acc: 0.9015\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.2107 - acc: 0.9291 - val_loss: 0.2998 - val_acc: 0.9032\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.1783 - acc: 0.9376 - val_loss: 0.1972 - val_acc: 0.9288\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.1558 - acc: 0.9452 - val_loss: 0.2736 - val_acc: 0.9149\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.1343 - acc: 0.9528 - val_loss: 0.1266 - val_acc: 0.9522\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.1195 - acc: 0.9589 - val_loss: 0.1337 - val_acc: 0.9539\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 123s 107ms/step - loss: 0.1049 - acc: 0.9623 - val_loss: 0.1081 - val_acc: 0.9564\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.0969 - acc: 0.9648 - val_loss: 0.0814 - val_acc: 0.9712\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0880 - acc: 0.9681 - val_loss: 0.1009 - val_acc: 0.9532\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0783 - acc: 0.9709 - val_loss: 0.1017 - val_acc: 0.9622\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0774 - acc: 0.9711 - val_loss: 0.0870 - val_acc: 0.9656\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0696 - acc: 0.9740 - val_loss: 0.0817 - val_acc: 0.9707\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0706 - acc: 0.9728 - val_loss: 0.0598 - val_acc: 0.9759\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 123s 106ms/step - loss: 0.0609 - acc: 0.9763 - val_loss: 0.0437 - val_acc: 0.9846\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0634 - acc: 0.9752 - val_loss: 0.0641 - val_acc: 0.9754\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0601 - acc: 0.9764 - val_loss: 0.0675 - val_acc: 0.9773\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0586 - acc: 0.9768 - val_loss: 0.0558 - val_acc: 0.9798\n",
      "36905/36905 [==============================] - 34s 919us/step\n",
      "Train [0.05746929755956394, 0.9774285327191438]\n",
      "10252/10252 [==============================] - 9s 919us/step\n",
      "Test [0.06046378158448592, 0.9758095981271947]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 127s 110ms/step - loss: 1.0286 - acc: 0.7062 - val_loss: 0.7832 - val_acc: 0.7579\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.3580 - acc: 0.8835 - val_loss: 0.6104 - val_acc: 0.8015\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.2500 - acc: 0.9151 - val_loss: 0.4304 - val_acc: 0.8613\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 123s 106ms/step - loss: 0.1961 - acc: 0.9327 - val_loss: 0.3594 - val_acc: 0.8888\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.1574 - acc: 0.9452 - val_loss: 0.3102 - val_acc: 0.8883\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.1322 - acc: 0.9524 - val_loss: 0.2102 - val_acc: 0.9286\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.1176 - acc: 0.9575 - val_loss: 0.2435 - val_acc: 0.9195\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.1009 - acc: 0.9633 - val_loss: 0.1912 - val_acc: 0.9354\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.1202 - acc: 0.9650 - val_loss: 0.1268 - val_acc: 0.9527\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0843 - acc: 0.9692 - val_loss: 0.1508 - val_acc: 0.9576\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.0784 - acc: 0.9699 - val_loss: 0.0998 - val_acc: 0.9676\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0715 - acc: 0.9730 - val_loss: 0.1524 - val_acc: 0.9407\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0681 - acc: 0.9745 - val_loss: 0.1536 - val_acc: 0.9471\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0630 - acc: 0.9760 - val_loss: 0.1189 - val_acc: 0.9576\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0651 - acc: 0.9743 - val_loss: 0.0722 - val_acc: 0.9734\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0536 - acc: 0.9792 - val_loss: 0.0773 - val_acc: 0.9778\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0564 - acc: 0.9783 - val_loss: 0.0836 - val_acc: 0.9666\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0577 - acc: 0.9780 - val_loss: 0.0822 - val_acc: 0.9690\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.0505 - acc: 0.9797 - val_loss: 0.0484 - val_acc: 0.9820\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0493 - acc: 0.9803 - val_loss: 0.0563 - val_acc: 0.9773\n",
      "36905/36905 [==============================] - 35s 937us/step\n",
      "Train [0.06287341669086777, 0.9717924400487739]\n",
      "10252/10252 [==============================] - 10s 930us/step\n",
      "Test [0.06658183533801518, 0.9700546234880999]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 127s 110ms/step - loss: 1.0245 - acc: 0.6986 - val_loss: 0.6759 - val_acc: 0.7713\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.3745 - acc: 0.8787 - val_loss: 0.4133 - val_acc: 0.8659\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.2623 - acc: 0.9130 - val_loss: 0.3949 - val_acc: 0.8732\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.2073 - acc: 0.9296 - val_loss: 0.4061 - val_acc: 0.9090\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.1657 - acc: 0.9417 - val_loss: 0.2498 - val_acc: 0.9161\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.1454 - acc: 0.9498 - val_loss: 0.2705 - val_acc: 0.9156\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.1210 - acc: 0.9558 - val_loss: 0.2251 - val_acc: 0.9264\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.1023 - acc: 0.9627 - val_loss: 0.1874 - val_acc: 0.9395\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0952 - acc: 0.9657 - val_loss: 0.1602 - val_acc: 0.9398\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 122s 106ms/step - loss: 0.0904 - acc: 0.9669 - val_loss: 0.1493 - val_acc: 0.9481\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0801 - acc: 0.9708 - val_loss: 0.1171 - val_acc: 0.9532\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0733 - acc: 0.9728 - val_loss: 0.0881 - val_acc: 0.9605\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 123s 106ms/step - loss: 0.0740 - acc: 0.9719 - val_loss: 0.1961 - val_acc: 0.9368\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0631 - acc: 0.9748 - val_loss: 0.0732 - val_acc: 0.9717\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0601 - acc: 0.9772 - val_loss: 0.0648 - val_acc: 0.9744\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0630 - acc: 0.9754 - val_loss: 0.0436 - val_acc: 0.9817\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0570 - acc: 0.9781 - val_loss: 0.0507 - val_acc: 0.9812\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0530 - acc: 0.9795 - val_loss: 0.0527 - val_acc: 0.9798\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 121s 105ms/step - loss: 0.0545 - acc: 0.9792 - val_loss: 0.0449 - val_acc: 0.9790\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 122s 105ms/step - loss: 0.0478 - acc: 0.9807 - val_loss: 0.0444 - val_acc: 0.9851\n",
      "36905/36905 [==============================] - 35s 939us/step\n",
      "Train [0.04730630727793833, 0.9837149437745563]\n",
      "10252/10252 [==============================] - 10s 934us/step\n",
      "Test [0.05183099982387034, 0.9833203277409286]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 256s 222ms/step - loss: 0.4568 - acc: 0.8774 - val_loss: 0.1594 - val_acc: 0.9529\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.1312 - acc: 0.9600 - val_loss: 0.0732 - val_acc: 0.9761\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0908 - acc: 0.9701 - val_loss: 0.0657 - val_acc: 0.9788\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0744 - acc: 0.9739 - val_loss: 0.0386 - val_acc: 0.9866\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0608 - acc: 0.9771 - val_loss: 0.0445 - val_acc: 0.9846\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0583 - acc: 0.9771 - val_loss: 0.0532 - val_acc: 0.9783\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0485 - acc: 0.9807 - val_loss: 0.0438 - val_acc: 0.9861\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0489 - acc: 0.9801 - val_loss: 0.0494 - val_acc: 0.9822\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0434 - acc: 0.9813 - val_loss: 0.0315 - val_acc: 0.9863\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 251s 217ms/step - loss: 0.0426 - acc: 0.9815 - val_loss: 0.0372 - val_acc: 0.9849\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 253s 220ms/step - loss: 0.0402 - acc: 0.9821 - val_loss: 0.0427 - val_acc: 0.9802\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.0380 - acc: 0.9827 - val_loss: 0.0756 - val_acc: 0.9693\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 251s 218ms/step - loss: 0.0384 - acc: 0.9827 - val_loss: 0.0385 - val_acc: 0.9854\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0359 - acc: 0.9835 - val_loss: 0.0467 - val_acc: 0.9798\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 249s 216ms/step - loss: 0.0367 - acc: 0.9820 - val_loss: 0.0946 - val_acc: 0.9681\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 249s 215ms/step - loss: 0.0345 - acc: 0.9832 - val_loss: 0.0337 - val_acc: 0.9873\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0345 - acc: 0.9838 - val_loss: 0.0559 - val_acc: 0.9788\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0334 - acc: 0.9845 - val_loss: 0.0483 - val_acc: 0.9798\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 247s 214ms/step - loss: 0.0321 - acc: 0.9844 - val_loss: 0.0294 - val_acc: 0.9856\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0332 - acc: 0.9842 - val_loss: 0.0322 - val_acc: 0.9837\n",
      "36905/36905 [==============================] - 56s 2ms/step\n",
      "Train [0.037401676071269464, 0.9816014090231676]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.04037061664100044, 0.981076863051112]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.4794 - acc: 0.8684 - val_loss: 0.1404 - val_acc: 0.9571\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 254s 220ms/step - loss: 0.1430 - acc: 0.9569 - val_loss: 0.0673 - val_acc: 0.9798\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 247s 214ms/step - loss: 0.0985 - acc: 0.9676 - val_loss: 0.1169 - val_acc: 0.9576\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0795 - acc: 0.9718 - val_loss: 0.0822 - val_acc: 0.9693\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0710 - acc: 0.9737 - val_loss: 0.0897 - val_acc: 0.9610\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0636 - acc: 0.9759 - val_loss: 0.1265 - val_acc: 0.9612\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0541 - acc: 0.9782 - val_loss: 0.0325 - val_acc: 0.9876\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0539 - acc: 0.9783 - val_loss: 0.0855 - val_acc: 0.9676\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0468 - acc: 0.9804 - val_loss: 0.0323 - val_acc: 0.9846\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 247s 214ms/step - loss: 0.0437 - acc: 0.9818 - val_loss: 0.0301 - val_acc: 0.9873\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0436 - acc: 0.9809 - val_loss: 0.0296 - val_acc: 0.9876\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0391 - acc: 0.9823 - val_loss: 0.1771 - val_acc: 0.9525\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 247s 214ms/step - loss: 0.0388 - acc: 0.9830 - val_loss: 0.0636 - val_acc: 0.9707\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0401 - acc: 0.9822 - val_loss: 0.0377 - val_acc: 0.9863\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0357 - acc: 0.9834 - val_loss: 0.0586 - val_acc: 0.9766\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0363 - acc: 0.9833 - val_loss: 0.0373 - val_acc: 0.9807\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0348 - acc: 0.9838 - val_loss: 0.0372 - val_acc: 0.9849\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0330 - acc: 0.9836 - val_loss: 0.0649 - val_acc: 0.9759\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0357 - acc: 0.9827 - val_loss: 0.0402 - val_acc: 0.9832\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0331 - acc: 0.9839 - val_loss: 0.0287 - val_acc: 0.9873\n",
      "36905/36905 [==============================] - 57s 2ms/step\n",
      "Train [0.03390332641672871, 0.9855575125321772]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.034269971722088156, 0.9850760827155677]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 255s 221ms/step - loss: 0.4576 - acc: 0.8765 - val_loss: 0.1741 - val_acc: 0.9383\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.1306 - acc: 0.9600 - val_loss: 0.0656 - val_acc: 0.9783\n",
      "Epoch 3/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1154/1153 [==============================] - 250s 217ms/step - loss: 0.0911 - acc: 0.9687 - val_loss: 0.0822 - val_acc: 0.9681\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 249s 215ms/step - loss: 0.0728 - acc: 0.9748 - val_loss: 0.1202 - val_acc: 0.9612\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0619 - acc: 0.9768 - val_loss: 0.0374 - val_acc: 0.9827\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0546 - acc: 0.9795 - val_loss: 0.0612 - val_acc: 0.9754\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0520 - acc: 0.9795 - val_loss: 0.0256 - val_acc: 0.9881\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0469 - acc: 0.9806 - val_loss: 0.0368 - val_acc: 0.9839\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0444 - acc: 0.9814 - val_loss: 0.0568 - val_acc: 0.9776\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 250s 216ms/step - loss: 0.0414 - acc: 0.9823 - val_loss: 0.0502 - val_acc: 0.9810\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0387 - acc: 0.9836 - val_loss: 0.3367 - val_acc: 0.9054\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0407 - acc: 0.9825 - val_loss: 0.0512 - val_acc: 0.9795\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0392 - acc: 0.9821 - val_loss: 0.0850 - val_acc: 0.9668\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 249s 215ms/step - loss: 0.0375 - acc: 0.9834 - val_loss: 0.0326 - val_acc: 0.9844\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0355 - acc: 0.9833 - val_loss: 0.0248 - val_acc: 0.9888\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0360 - acc: 0.9838 - val_loss: 0.0258 - val_acc: 0.9881\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0348 - acc: 0.9837 - val_loss: 0.0428 - val_acc: 0.9805\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0311 - acc: 0.9852 - val_loss: 0.0641 - val_acc: 0.9768\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0332 - acc: 0.9835 - val_loss: 0.0372 - val_acc: 0.9817\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 248s 215ms/step - loss: 0.0296 - acc: 0.9848 - val_loss: 0.0340 - val_acc: 0.9854\n",
      "36905/36905 [==============================] - 57s 2ms/step\n",
      "Train [0.03947699583603541, 0.983633653976426]\n",
      "10252/10252 [==============================] - 16s 2ms/step\n",
      "Test [0.042380694470343594, 0.9825399921966446]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.9747 - acc: 0.7574 - val_loss: 0.4449 - val_acc: 0.8873\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 134s 117ms/step - loss: 0.3225 - acc: 0.9235 - val_loss: 0.1789 - val_acc: 0.9559\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1840 - acc: 0.9527 - val_loss: 0.1112 - val_acc: 0.9717\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1242 - acc: 0.9655 - val_loss: 0.1124 - val_acc: 0.9607\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 134s 117ms/step - loss: 0.0959 - acc: 0.9709 - val_loss: 0.0749 - val_acc: 0.9771\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0769 - acc: 0.9750 - val_loss: 0.0515 - val_acc: 0.9815\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0718 - acc: 0.9748 - val_loss: 0.0554 - val_acc: 0.9810\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0593 - acc: 0.9787 - val_loss: 0.0931 - val_acc: 0.9688\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0563 - acc: 0.9787 - val_loss: 0.0563 - val_acc: 0.9802\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0481 - acc: 0.9810 - val_loss: 0.0446 - val_acc: 0.9827\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0471 - acc: 0.9812 - val_loss: 0.0342 - val_acc: 0.9829\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0447 - acc: 0.9816 - val_loss: 0.0632 - val_acc: 0.9754\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0430 - acc: 0.9818 - val_loss: 0.0479 - val_acc: 0.9802\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0396 - acc: 0.9831 - val_loss: 0.0256 - val_acc: 0.9878\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0390 - acc: 0.9822 - val_loss: 0.0500 - val_acc: 0.9790\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0365 - acc: 0.9835 - val_loss: 0.0833 - val_acc: 0.9685\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0365 - acc: 0.9833 - val_loss: 0.0409 - val_acc: 0.9815\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 134s 116ms/step - loss: 0.0383 - acc: 0.9828 - val_loss: 0.0351 - val_acc: 0.9820\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0336 - acc: 0.9841 - val_loss: 0.0334 - val_acc: 0.9842\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0330 - acc: 0.9845 - val_loss: 0.0430 - val_acc: 0.9834\n",
      "36905/36905 [==============================] - 37s 1ms/step\n",
      "Train [0.052338644452902816, 0.977889174908549]\n",
      "10252/10252 [==============================] - 10s 1ms/step\n",
      "Test [0.05513681921431791, 0.9771751853296917]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.9278 - acc: 0.7736 - val_loss: 0.4273 - val_acc: 0.8693\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.3104 - acc: 0.9233 - val_loss: 0.2105 - val_acc: 0.9288\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1875 - acc: 0.9499 - val_loss: 0.1356 - val_acc: 0.9568\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1275 - acc: 0.9640 - val_loss: 0.0887 - val_acc: 0.9666\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1001 - acc: 0.9694 - val_loss: 0.1529 - val_acc: 0.9400\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0792 - acc: 0.9744 - val_loss: 0.1048 - val_acc: 0.9627\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0676 - acc: 0.9764 - val_loss: 0.0749 - val_acc: 0.9737\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0608 - acc: 0.9781 - val_loss: 0.0889 - val_acc: 0.9688\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0541 - acc: 0.9794 - val_loss: 0.1006 - val_acc: 0.9644\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0497 - acc: 0.9797 - val_loss: 0.0359 - val_acc: 0.9863\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0466 - acc: 0.9811 - val_loss: 0.0601 - val_acc: 0.9754\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0445 - acc: 0.9814 - val_loss: 0.0511 - val_acc: 0.9795\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0425 - acc: 0.9819 - val_loss: 0.0656 - val_acc: 0.9754\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0402 - acc: 0.9827 - val_loss: 0.1280 - val_acc: 0.9610\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0409 - acc: 0.9825 - val_loss: 0.0350 - val_acc: 0.9832\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0357 - acc: 0.9838 - val_loss: 0.0413 - val_acc: 0.9829\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0368 - acc: 0.9832 - val_loss: 0.0342 - val_acc: 0.9832\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 136s 117ms/step - loss: 0.0350 - acc: 0.9841 - val_loss: 0.0411 - val_acc: 0.9805\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0346 - acc: 0.9838 - val_loss: 0.0313 - val_acc: 0.9863\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0316 - acc: 0.9855 - val_loss: 0.0568 - val_acc: 0.9732\n",
      "36905/36905 [==============================] - 38s 1ms/step\n",
      "Train [0.06148814120326257, 0.9724427584338166]\n",
      "10252/10252 [==============================] - 10s 1ms/step\n",
      "Test [0.06926287411534239, 0.9707374170893485]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 1.0335 - acc: 0.7454 - val_loss: 0.5160 - val_acc: 0.8591\n",
      "Epoch 2/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.3584 - acc: 0.9138 - val_loss: 0.2688 - val_acc: 0.9154\n",
      "Epoch 3/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.2026 - acc: 0.9489 - val_loss: 0.1510 - val_acc: 0.9559\n",
      "Epoch 4/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1382 - acc: 0.9631 - val_loss: 0.1325 - val_acc: 0.9588\n",
      "Epoch 5/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.1046 - acc: 0.9688 - val_loss: 0.0612 - val_acc: 0.9842\n",
      "Epoch 6/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0839 - acc: 0.9733 - val_loss: 0.1057 - val_acc: 0.9600\n",
      "Epoch 7/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0702 - acc: 0.9766 - val_loss: 0.0808 - val_acc: 0.9668\n",
      "Epoch 8/20\n",
      "1154/1153 [==============================] - 136s 118ms/step - loss: 0.0621 - acc: 0.9780 - val_loss: 0.0554 - val_acc: 0.9820\n",
      "Epoch 9/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0574 - acc: 0.9785 - val_loss: 0.0607 - val_acc: 0.9793\n",
      "Epoch 10/20\n",
      "1154/1153 [==============================] - 135s 117ms/step - loss: 0.0509 - acc: 0.9804 - val_loss: 0.0326 - val_acc: 0.9876\n",
      "Epoch 11/20\n",
      "1154/1153 [==============================] - 139s 121ms/step - loss: 0.0491 - acc: 0.9810 - val_loss: 0.0329 - val_acc: 0.9876\n",
      "Epoch 12/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0469 - acc: 0.9810 - val_loss: 0.0429 - val_acc: 0.9837\n",
      "Epoch 13/20\n",
      "1154/1153 [==============================] - 140s 122ms/step - loss: 0.0401 - acc: 0.9833 - val_loss: 0.0611 - val_acc: 0.9766\n",
      "Epoch 14/20\n",
      "1154/1153 [==============================] - 139s 120ms/step - loss: 0.0428 - acc: 0.9815 - val_loss: 0.0270 - val_acc: 0.9888\n",
      "Epoch 15/20\n",
      "1154/1153 [==============================] - 146s 126ms/step - loss: 0.0389 - acc: 0.9829 - val_loss: 0.0303 - val_acc: 0.9861\n",
      "Epoch 16/20\n",
      "1154/1153 [==============================] - 144s 125ms/step - loss: 0.0385 - acc: 0.9832 - val_loss: 0.0368 - val_acc: 0.9842\n",
      "Epoch 17/20\n",
      "1154/1153 [==============================] - 143s 124ms/step - loss: 0.0387 - acc: 0.9826 - val_loss: 0.0389 - val_acc: 0.9817\n",
      "Epoch 18/20\n",
      "1154/1153 [==============================] - 142s 123ms/step - loss: 0.0364 - acc: 0.9830 - val_loss: 0.0739 - val_acc: 0.9617\n",
      "Epoch 19/20\n",
      "1154/1153 [==============================] - 140s 122ms/step - loss: 0.0371 - acc: 0.9832 - val_loss: 0.0413 - val_acc: 0.9795\n",
      "Epoch 20/20\n",
      "1154/1153 [==============================] - 138s 120ms/step - loss: 0.0344 - acc: 0.9845 - val_loss: 0.0326 - val_acc: 0.9851\n",
      "36905/36905 [==============================] - 38s 1ms/step\n",
      "Train [0.03732276504744254, 0.9835252675789189]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.03936799737421041, 0.9821498244245025]\n"
     ]
    }
   ],
   "source": [
    "# He-Normal\n",
    "import keras.optimizers\n",
    "for lr in [0.1, 0.01, 0.001, 0.0001]:\n",
    "    optimizer = optimizers.Adam(lr=lr)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITHOUT BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetC((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH LAYER BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetA((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH IDENTITY BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetB((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, 20)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks for Fruits-360 and batch=128 for Models 1, 2 and 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 127s 438ms/step - loss: 14.0551 - acc: 0.1254 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 118s 409ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 118s 410ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 120s 416ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 119s 410ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 118s 409ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 118s 410ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 118s 410ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 118s 410ms/step - loss: 14.0928 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 118s 410ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 119s 410ms/step - loss: 14.0854 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 118s 410ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 118s 410ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 36s 971us/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 10s 971us/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 126s 436ms/step - loss: 15.8195 - acc: 0.0159 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 119s 410ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 119s 410ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 119s 410ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 119s 410ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 122s 424ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 122s 421ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 15.9159 - acc: 0.0125 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 15.9150 - acc: 0.0126 - val_loss: 15.8941 - val_acc: 0.0139\n",
      "36905/36905 [==============================] - 36s 986us/step\n",
      "Train [15.915445359130164, 0.012572822110825091]\n",
      "10252/10252 [==============================] - 10s 987us/step\n",
      "Test [15.905849601159186, 0.01316816230979321]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 126s 436ms/step - loss: 14.0567 - acc: 0.1254 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 118s 410ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 119s 410ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 119s 410ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0937 - acc: 0.1256 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "289/288 [==============================] - 118s 409ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 118s 410ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 118s 409ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 36s 964us/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 10s 961us/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 264s 912ms/step - loss: 14.0715 - acc: 0.1251 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 257s 889ms/step - loss: 14.0836 - acc: 0.1262 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 256s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 258s 893ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 271s 938ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 269s 930ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 257s 888ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 60s 2ms/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 15.8453 - acc: 0.0147 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 15.9045 - acc: 0.0133 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 255s 882ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 256s 884ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 256s 884ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 256s 884ms/step - loss: 15.9054 - acc: 0.0132 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 15.9063 - acc: 0.0131 - val_loss: 15.9216 - val_acc: 0.0122\n",
      "36905/36905 [==============================] - 60s 2ms/step\n",
      "Train [15.905836950306517, 0.013168947297114212]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [15.929432473422677, 0.011705033164260631]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 265s 919ms/step - loss: 14.0601 - acc: 0.1255 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 256s 884ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 14.0854 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 255s 884ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 256s 884ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 255s 882ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 255s 883ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 60s 2ms/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 142s 491ms/step - loss: 15.6113 - acc: 0.0270 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 134s 464ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 133s 461ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 133s 461ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 133s 461ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 133s 460ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 133s 461ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 135s 466ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 133s 462ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 133s 461ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 133s 462ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 133s 462ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 135s 469ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 139s 480ms/step - loss: 15.6901 - acc: 0.0266 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 133s 462ms/step - loss: 15.6910 - acc: 0.0265 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 134s 464ms/step - loss: 15.6920 - acc: 0.0264 - val_loss: 15.7290 - val_acc: 0.0241\n",
      "36905/36905 [==============================] - 39s 1ms/step\n",
      "Train [15.690958044519594, 0.026500474190489093]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [15.748630510286294, 0.02292235661334374]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 143s 493ms/step - loss: 15.6350 - acc: 0.0264 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 134s 465ms/step - loss: 15.7054 - acc: 0.0256 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 134s 465ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 134s 464ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 134s 464ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.7054 - acc: 0.0256 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 135s 466ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 134s 465ms/step - loss: 15.7072 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.7054 - acc: 0.0256 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.7063 - acc: 0.0255 - val_loss: 15.6936 - val_acc: 0.0263\n",
      "36905/36905 [==============================] - 39s 1ms/step\n",
      "Train [15.706244144612834, 0.02555209321250374]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [15.7077535483412, 0.02545844713517385]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 144s 497ms/step - loss: 15.8189 - acc: 0.0142 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 2/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "289/288 [==============================] - 134s 463ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 134s 464ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 133s 462ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 133s 461ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 134s 464ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 133s 461ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 133s 462ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 135s 467ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 133s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 133s 461ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 134s 465ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 134s 464ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 134s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 134s 463ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 133s 462ms/step - loss: 15.9141 - acc: 0.0127 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 133s 462ms/step - loss: 15.9151 - acc: 0.0126 - val_loss: 15.9177 - val_acc: 0.0124\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [15.914571866742351, 0.012627015309578648]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [15.899560848716828, 0.013558330081935232]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 128s 444ms/step - loss: 2.8501 - acc: 0.2195 - val_loss: 2.5441 - val_acc: 0.2712\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 120s 416ms/step - loss: 2.0876 - acc: 0.3627 - val_loss: 2.1350 - val_acc: 0.3621\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 1.6936 - acc: 0.4706 - val_loss: 1.7213 - val_acc: 0.4972\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 1.2284 - acc: 0.6010 - val_loss: 1.2510 - val_acc: 0.5972\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 0.9321 - acc: 0.6848 - val_loss: 1.0403 - val_acc: 0.6759\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 0.7561 - acc: 0.7410 - val_loss: 0.9868 - val_acc: 0.7081\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 0.6209 - acc: 0.7865 - val_loss: 0.6283 - val_acc: 0.8035\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 0.5134 - acc: 0.8324 - val_loss: 0.4076 - val_acc: 0.8564\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 0.3858 - acc: 0.8657 - val_loss: 0.6528 - val_acc: 0.7983\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.3753 - acc: 0.8738 - val_loss: 0.5700 - val_acc: 0.8291\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 0.3316 - acc: 0.8867 - val_loss: 0.4519 - val_acc: 0.8517\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 0.3279 - acc: 0.8884 - val_loss: 0.3600 - val_acc: 0.8973\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.2667 - acc: 0.9116 - val_loss: 0.5456 - val_acc: 0.8598\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 0.3212 - acc: 0.8986 - val_loss: 0.3456 - val_acc: 0.9056\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 0.2792 - acc: 0.9116 - val_loss: 0.4655 - val_acc: 0.8722\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 0.3001 - acc: 0.9050 - val_loss: 0.2645 - val_acc: 0.9125\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.2848 - acc: 0.9090 - val_loss: 0.9283 - val_acc: 0.8186\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 0.4045 - acc: 0.8810 - val_loss: 0.5268 - val_acc: 0.8803\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 0.3128 - acc: 0.9022 - val_loss: 0.2811 - val_acc: 0.9105\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 0.2715 - acc: 0.9165 - val_loss: 0.2738 - val_acc: 0.9249\n",
      "36905/36905 [==============================] - 38s 1ms/step\n",
      "Train [0.25548148900512724, 0.9294404552260997]\n",
      "10252/10252 [==============================] - 10s 1ms/step\n",
      "Test [0.2628769279572818, 0.9264533749279732]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 129s 446ms/step - loss: 14.0546 - acc: 0.1255 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 14.0854 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 18/20\n",
      "289/288 [==============================] - 119s 411ms/step - loss: 14.0854 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 38s 1ms/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 10s 1ms/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 129s 446ms/step - loss: 14.0560 - acc: 0.1254 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0845 - acc: 0.1262 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 120s 417ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 120s 416ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 120s 416ms/step - loss: 14.0937 - acc: 0.1256 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0864 - acc: 0.1261 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 120s 417ms/step - loss: 14.0910 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0900 - acc: 0.1258 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0882 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0891 - acc: 0.1259 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 14.0919 - acc: 0.1257 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 119s 412ms/step - loss: 14.0873 - acc: 0.1260 - val_loss: 14.1569 - val_acc: 0.1217\n",
      "36905/36905 [==============================] - 38s 1ms/step\n",
      "Train [14.088975397814679, 0.12589080070471348]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [14.045948446796967, 0.12856028092079594]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 270s 934ms/step - loss: 15.0428 - acc: 0.0624 - val_loss: 15.1080 - val_acc: 0.0627\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 257s 888ms/step - loss: 15.1267 - acc: 0.0615 - val_loss: 15.1010 - val_acc: 0.0629\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 258s 892ms/step - loss: 15.2548 - acc: 0.0535 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 257s 891ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 259s 897ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 257s 888ms/step - loss: 15.3006 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.2987 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 257s 891ms/step - loss: 15.3024 - acc: 0.0506 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.3024 - acc: 0.0506 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 15.2997 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 258s 894ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 259s 896ms/step - loss: 15.3006 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 257s 891ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.3006 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 258s 891ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.3006 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 15.3015 - acc: 0.0507 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 257s 889ms/step - loss: 15.3024 - acc: 0.0506 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.2997 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 257s 889ms/step - loss: 15.2987 - acc: 0.0508 - val_loss: 15.2927 - val_acc: 0.0512\n",
      "36905/36905 [==============================] - 61s 2ms/step\n",
      "Train [15.300507428435804, 0.050724834033328815]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [15.265967945079527, 0.05286773312524386]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 268s 926ms/step - loss: 15.5353 - acc: 0.0316 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 257s 889ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.6012 - acc: 0.0321 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 257s 889ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 257s 888ms/step - loss: 15.6012 - acc: 0.0321 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 15.6039 - acc: 0.0319 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 15.6039 - acc: 0.0319 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 256s 885ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 256s 886ms/step - loss: 15.6039 - acc: 0.0319 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 262s 906ms/step - loss: 15.6039 - acc: 0.0319 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 256s 887ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 257s 889ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 272s 939ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 266s 919ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 259s 896ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 260s 899ms/step - loss: 15.6021 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 260s 898ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 262s 908ms/step - loss: 15.6030 - acc: 0.0320 - val_loss: 15.6465 - val_acc: 0.0293\n",
      "36905/36905 [==============================] - 60s 2ms/step\n",
      "Train [15.602735414234964, 0.03197398726459829]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [15.583550470302312, 0.033164260634978765]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 268s 929ms/step - loss: 14.0349 - acc: 0.1241 - val_loss: 14.0547 - val_acc: 0.1280\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 259s 896ms/step - loss: 14.0466 - acc: 0.1285 - val_loss: 14.3377 - val_acc: 0.1105\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 258s 893ms/step - loss: 14.0280 - acc: 0.1297 - val_loss: 14.0468 - val_acc: 0.1285\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 14.4475 - acc: 0.1037 - val_loss: 14.6246 - val_acc: 0.0927\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 269s 929ms/step - loss: 14.3871 - acc: 0.1074 - val_loss: 14.3062 - val_acc: 0.1124\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 262s 908ms/step - loss: 14.0753 - acc: 0.1267 - val_loss: 14.0743 - val_acc: 0.1268\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 262s 907ms/step - loss: 14.0255 - acc: 0.1298 - val_loss: 14.0743 - val_acc: 0.1268\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 263s 912ms/step - loss: 14.0248 - acc: 0.1299 - val_loss: 14.0743 - val_acc: 0.1268\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 262s 905ms/step - loss: 14.0242 - acc: 0.1299 - val_loss: 14.0743 - val_acc: 0.1268\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 259s 898ms/step - loss: 14.0223 - acc: 0.1300 - val_loss: 14.0743 - val_acc: 0.1268\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 258s 893ms/step - loss: 14.0228 - acc: 0.1300 - val_loss: 14.0743 - val_acc: 0.1268\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 262s 907ms/step - loss: 14.0242 - acc: 0.1299 - val_loss: 14.0743 - val_acc: 0.1268\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 268s 928ms/step - loss: 14.0192 - acc: 0.1302 - val_loss: 14.0743 - val_acc: 0.1268\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 263s 910ms/step - loss: 14.0640 - acc: 0.1274 - val_loss: 14.4084 - val_acc: 0.1061\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 260s 901ms/step - loss: 14.2735 - acc: 0.1144 - val_loss: 14.2984 - val_acc: 0.1129\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 260s 899ms/step - loss: 14.2824 - acc: 0.1139 - val_loss: 14.3102 - val_acc: 0.1122\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 260s 901ms/step - loss: 14.2863 - acc: 0.1137 - val_loss: 14.2984 - val_acc: 0.1129\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 261s 904ms/step - loss: 14.2874 - acc: 0.1136 - val_loss: 14.2984 - val_acc: 0.1129\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 14.2799 - acc: 0.1140 - val_loss: 14.2984 - val_acc: 0.1129\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 260s 900ms/step - loss: 14.2935 - acc: 0.1132 - val_loss: 14.2984 - val_acc: 0.1129\n",
      "36905/36905 [==============================] - 62s 2ms/step\n",
      "Train [14.287694584032883, 0.11356184798827938]\n",
      "10252/10252 [==============================] - 18s 2ms/step\n",
      "Test [14.22517814442742, 0.11744049941474834]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 149s 515ms/step - loss: 2.4441 - acc: 0.3350 - val_loss: 2.4572 - val_acc: 0.3014\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 1.4326 - acc: 0.5512 - val_loss: 2.8915 - val_acc: 0.2919\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.8629 - acc: 0.7229 - val_loss: 1.7861 - val_acc: 0.5396\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 140s 484ms/step - loss: 0.4775 - acc: 0.8399 - val_loss: 0.7298 - val_acc: 0.7267\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 139s 481ms/step - loss: 0.3332 - acc: 0.8851 - val_loss: 0.2990 - val_acc: 0.8969\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 136s 470ms/step - loss: 0.2682 - acc: 0.9052 - val_loss: 0.2939 - val_acc: 0.8956\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.2177 - acc: 0.9212 - val_loss: 0.2319 - val_acc: 0.9181\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.1925 - acc: 0.9292 - val_loss: 0.3139 - val_acc: 0.8773\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.1588 - acc: 0.9406 - val_loss: 0.1346 - val_acc: 0.9522\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 139s 481ms/step - loss: 0.1250 - acc: 0.9536 - val_loss: 0.0876 - val_acc: 0.9651\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 138s 478ms/step - loss: 0.1134 - acc: 0.9566 - val_loss: 0.1614 - val_acc: 0.9446\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 136s 469ms/step - loss: 0.1040 - acc: 0.9601 - val_loss: 0.0575 - val_acc: 0.9773\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0994 - acc: 0.9614 - val_loss: 0.0585 - val_acc: 0.9759\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 141s 489ms/step - loss: 0.0799 - acc: 0.9681 - val_loss: 0.1157 - val_acc: 0.9520\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0958 - acc: 0.9623 - val_loss: 0.0673 - val_acc: 0.9720\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0766 - acc: 0.9693 - val_loss: 0.0961 - val_acc: 0.9598\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 138s 479ms/step - loss: 0.0860 - acc: 0.9662 - val_loss: 0.0480 - val_acc: 0.9790\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0714 - acc: 0.9706 - val_loss: 0.0586 - val_acc: 0.9761\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.0642 - acc: 0.9735 - val_loss: 0.0785 - val_acc: 0.9705\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0708 - acc: 0.9704 - val_loss: 0.1335 - val_acc: 0.9542\n",
      "36905/36905 [==============================] - 41s 1ms/step\n",
      "Train [0.1484264876194641, 0.9499254843517139]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.14785967666984268, 0.9505462348809989]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 148s 513ms/step - loss: 2.7549 - acc: 0.2617 - val_loss: 6.8384 - val_acc: 0.1146\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 1.2391 - acc: 0.6166 - val_loss: 2.6804 - val_acc: 0.3511\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.5120 - acc: 0.8284 - val_loss: 0.9131 - val_acc: 0.6886\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 135s 469ms/step - loss: 0.2456 - acc: 0.9151 - val_loss: 0.5648 - val_acc: 0.8100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5/20\n",
      "289/288 [==============================] - 135s 469ms/step - loss: 0.1594 - acc: 0.9412 - val_loss: 0.7071 - val_acc: 0.7932\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 136s 469ms/step - loss: 0.1213 - acc: 0.9545 - val_loss: 0.3262 - val_acc: 0.9083\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.0972 - acc: 0.9625 - val_loss: 0.1742 - val_acc: 0.9337\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 136s 471ms/step - loss: 0.0918 - acc: 0.9642 - val_loss: 0.1632 - val_acc: 0.9473\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.0789 - acc: 0.9689 - val_loss: 0.1679 - val_acc: 0.9573\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 136s 471ms/step - loss: 0.0753 - acc: 0.9688 - val_loss: 0.1988 - val_acc: 0.9359\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 140s 483ms/step - loss: 0.0587 - acc: 0.9755 - val_loss: 0.2343 - val_acc: 0.9154\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 139s 479ms/step - loss: 0.0695 - acc: 0.9723 - val_loss: 0.2209 - val_acc: 0.9290\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 143s 494ms/step - loss: 0.0585 - acc: 0.9750 - val_loss: 0.1237 - val_acc: 0.9583\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 139s 481ms/step - loss: 0.0576 - acc: 0.9753 - val_loss: 0.1435 - val_acc: 0.9537\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 135s 467ms/step - loss: 0.0656 - acc: 0.9735 - val_loss: 0.3422 - val_acc: 0.9000\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0665 - acc: 0.9728 - val_loss: 0.2762 - val_acc: 0.9329\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.0453 - acc: 0.9800 - val_loss: 0.0970 - val_acc: 0.9637\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0440 - acc: 0.9801 - val_loss: 0.0938 - val_acc: 0.9663\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0613 - acc: 0.9746 - val_loss: 0.0854 - val_acc: 0.9717\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0460 - acc: 0.9793 - val_loss: 0.0968 - val_acc: 0.9673\n",
      "36905/36905 [==============================] - 41s 1ms/step\n",
      "Train [0.09448430042036556, 0.9669150521609537]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.08829999325408472, 0.9682013265704252]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 149s 515ms/step - loss: 2.2746 - acc: 0.3583 - val_loss: 2.6015 - val_acc: 0.3165\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 138s 479ms/step - loss: 1.2490 - acc: 0.6016 - val_loss: 3.3811 - val_acc: 0.2668\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 138s 477ms/step - loss: 0.6729 - acc: 0.7824 - val_loss: 1.0579 - val_acc: 0.6747\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.3709 - acc: 0.8730 - val_loss: 0.9445 - val_acc: 0.6935\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.2531 - acc: 0.9100 - val_loss: 0.3675 - val_acc: 0.8644\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 136s 471ms/step - loss: 0.2121 - acc: 0.9226 - val_loss: 0.3477 - val_acc: 0.8959\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 136s 471ms/step - loss: 0.1682 - acc: 0.9369 - val_loss: 0.2405 - val_acc: 0.9166\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.1407 - acc: 0.9463 - val_loss: 0.2947 - val_acc: 0.8976\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 136s 470ms/step - loss: 0.1341 - acc: 0.9500 - val_loss: 0.2003 - val_acc: 0.9164\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 135s 469ms/step - loss: 0.1096 - acc: 0.9580 - val_loss: 0.2352 - val_acc: 0.9198\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 136s 470ms/step - loss: 0.0907 - acc: 0.9647 - val_loss: 0.1274 - val_acc: 0.9466\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 136s 471ms/step - loss: 0.1022 - acc: 0.9615 - val_loss: 0.0769 - val_acc: 0.9715\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0788 - acc: 0.9682 - val_loss: 0.1876 - val_acc: 0.9337\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0764 - acc: 0.9690 - val_loss: 0.0756 - val_acc: 0.9668\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0733 - acc: 0.9709 - val_loss: 0.2005 - val_acc: 0.9344\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0802 - acc: 0.9682 - val_loss: 0.1186 - val_acc: 0.9495\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.0661 - acc: 0.9730 - val_loss: 0.0696 - val_acc: 0.9720\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 136s 471ms/step - loss: 0.0672 - acc: 0.9727 - val_loss: 0.0705 - val_acc: 0.9712\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0576 - acc: 0.9754 - val_loss: 0.1678 - val_acc: 0.9405\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 137s 476ms/step - loss: 0.0569 - acc: 0.9750 - val_loss: 0.1242 - val_acc: 0.9581\n",
      "36905/36905 [==============================] - 41s 1ms/step\n",
      "Train [0.13395559806962162, 0.9526893374897604]\n",
      "10252/10252 [==============================] - 12s 1ms/step\n",
      "Test [0.13970160247283536, 0.9524970737417089]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 135s 467ms/step - loss: 1.1378 - acc: 0.6529 - val_loss: 0.5344 - val_acc: 0.8213\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 123s 425ms/step - loss: 0.4003 - acc: 0.8630 - val_loss: 0.2975 - val_acc: 0.9008\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 122s 420ms/step - loss: 0.4034 - acc: 0.9031 - val_loss: 0.3501 - val_acc: 0.9110\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 123s 425ms/step - loss: 0.3356 - acc: 0.9258 - val_loss: 0.2950 - val_acc: 0.9234\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.3205 - acc: 0.9316 - val_loss: 0.2229 - val_acc: 0.9605\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 123s 424ms/step - loss: 0.2879 - acc: 0.9438 - val_loss: 0.2201 - val_acc: 0.9571\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 122s 421ms/step - loss: 0.2524 - acc: 0.9541 - val_loss: 0.3792 - val_acc: 0.9208\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 120s 417ms/step - loss: 0.2413 - acc: 0.9572 - val_loss: 0.2052 - val_acc: 0.9612\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 122s 424ms/step - loss: 0.2385 - acc: 0.9580 - val_loss: 0.2059 - val_acc: 0.9620\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 121s 419ms/step - loss: 0.2248 - acc: 0.9624 - val_loss: 0.2051 - val_acc: 0.9659\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 122s 420ms/step - loss: 0.2352 - acc: 0.9607 - val_loss: 0.2029 - val_acc: 0.9593\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 122s 421ms/step - loss: 0.2078 - acc: 0.9669 - val_loss: 0.2631 - val_acc: 0.9478\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 121s 420ms/step - loss: 0.2331 - acc: 0.9597 - val_loss: 0.1841 - val_acc: 0.9698\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 122s 421ms/step - loss: 0.2026 - acc: 0.9695 - val_loss: 0.1720 - val_acc: 0.9712\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 121s 418ms/step - loss: 0.2174 - acc: 0.9645 - val_loss: 0.2116 - val_acc: 0.9576\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 121s 419ms/step - loss: 0.2034 - acc: 0.9688 - val_loss: 0.1837 - val_acc: 0.9676\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 123s 425ms/step - loss: 0.1929 - acc: 0.9726 - val_loss: 0.1635 - val_acc: 0.9778\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 123s 427ms/step - loss: 0.1925 - acc: 0.9723 - val_loss: 0.2198 - val_acc: 0.9634\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.2257 - acc: 0.9630 - val_loss: 0.1812 - val_acc: 0.9703\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 122s 424ms/step - loss: 0.2042 - acc: 0.9685 - val_loss: 0.1975 - val_acc: 0.9627\n",
      "36905/36905 [==============================] - 39s 1ms/step\n",
      "Train [0.22142698323888363, 0.9601409023183743]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.24158749335064694, 0.9573741708934842]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 135s 466ms/step - loss: 2.0040 - acc: 0.4007 - val_loss: 1.1909 - val_acc: 0.6364\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.7553 - acc: 0.7564 - val_loss: 0.5528 - val_acc: 0.8196\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 123s 427ms/step - loss: 0.3886 - acc: 0.8709 - val_loss: 0.3562 - val_acc: 0.8695\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 123s 426ms/step - loss: 0.2893 - acc: 0.9000 - val_loss: 0.2637 - val_acc: 0.8971\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 121s 417ms/step - loss: 0.2057 - acc: 0.9278 - val_loss: 0.1766 - val_acc: 0.9307\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.1751 - acc: 0.9388 - val_loss: 0.1631 - val_acc: 0.9373\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 120s 417ms/step - loss: 0.1455 - acc: 0.9474 - val_loss: 0.2890 - val_acc: 0.9015\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 120s 416ms/step - loss: 0.1563 - acc: 0.9431 - val_loss: 0.1310 - val_acc: 0.9561\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 120s 416ms/step - loss: 0.1242 - acc: 0.9549 - val_loss: 0.1021 - val_acc: 0.9624\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 121s 417ms/step - loss: 0.1164 - acc: 0.9558 - val_loss: 0.0853 - val_acc: 0.9659\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 121s 419ms/step - loss: 0.0963 - acc: 0.9636 - val_loss: 0.0942 - val_acc: 0.9607\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 0.0972 - acc: 0.9635 - val_loss: 0.0916 - val_acc: 0.9637\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.1044 - acc: 0.9609 - val_loss: 0.0648 - val_acc: 0.9749\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 0.0890 - acc: 0.9641 - val_loss: 0.1359 - val_acc: 0.9549\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0989 - acc: 0.9623 - val_loss: 0.0646 - val_acc: 0.9744\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0794 - acc: 0.9686 - val_loss: 0.0867 - val_acc: 0.9632\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 119s 413ms/step - loss: 0.0707 - acc: 0.9714 - val_loss: 0.0767 - val_acc: 0.9642\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 120s 416ms/step - loss: 0.0846 - acc: 0.9674 - val_loss: 0.0827 - val_acc: 0.9676\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0691 - acc: 0.9723 - val_loss: 0.0743 - val_acc: 0.9698\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 120s 416ms/step - loss: 0.0703 - acc: 0.9710 - val_loss: 0.0438 - val_acc: 0.9842\n",
      "36905/36905 [==============================] - 38s 1ms/step\n",
      "Train [0.04528895893800777, 0.9815201192250372]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.04896112149428697, 0.9809793211080765]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 132s 457ms/step - loss: 1.2629 - acc: 0.6099 - val_loss: 0.5084 - val_acc: 0.8281\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.3143 - acc: 0.8893 - val_loss: 0.2544 - val_acc: 0.9022\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 0.1839 - acc: 0.9316 - val_loss: 0.1640 - val_acc: 0.9368\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 0.1511 - acc: 0.9474 - val_loss: 0.2026 - val_acc: 0.9361\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 124s 429ms/step - loss: 0.1191 - acc: 0.9558 - val_loss: 0.1321 - val_acc: 0.9624\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 123s 426ms/step - loss: 0.0974 - acc: 0.9640 - val_loss: 0.1659 - val_acc: 0.9485\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0794 - acc: 0.9700 - val_loss: 0.1745 - val_acc: 0.9361\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0793 - acc: 0.9690 - val_loss: 0.0622 - val_acc: 0.9722\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 121s 419ms/step - loss: 0.0666 - acc: 0.9732 - val_loss: 0.0658 - val_acc: 0.9712\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 0.0594 - acc: 0.9760 - val_loss: 0.0970 - val_acc: 0.9681\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0596 - acc: 0.9753 - val_loss: 0.0589 - val_acc: 0.9761\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0844 - acc: 0.9689 - val_loss: 0.0475 - val_acc: 0.9807\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 121s 417ms/step - loss: 0.0622 - acc: 0.9734 - val_loss: 0.0395 - val_acc: 0.9832\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 0.0530 - acc: 0.9771 - val_loss: 0.0991 - val_acc: 0.9624\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 120s 416ms/step - loss: 0.0801 - acc: 0.9688 - val_loss: 0.0629 - val_acc: 0.9703\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0427 - acc: 0.9809 - val_loss: 0.0317 - val_acc: 0.9839\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 120s 415ms/step - loss: 0.0470 - acc: 0.9796 - val_loss: 0.0368 - val_acc: 0.9829\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0598 - acc: 0.9772 - val_loss: 0.0463 - val_acc: 0.9822\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 120s 414ms/step - loss: 0.0468 - acc: 0.9788 - val_loss: 0.0602 - val_acc: 0.9749\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 120s 417ms/step - loss: 0.0451 - acc: 0.9790 - val_loss: 0.0337 - val_acc: 0.9859\n",
      "36905/36905 [==============================] - 38s 1ms/step\n",
      "Train [0.03802521532587745, 0.9823872104050941]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.03702103360221764, 0.9829301599687866]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 271s 939ms/step - loss: 0.7861 - acc: 0.8196 - val_loss: 0.6219 - val_acc: 0.8478\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 258s 893ms/step - loss: 0.3308 - acc: 0.9444 - val_loss: 0.5858 - val_acc: 0.8593\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 258s 892ms/step - loss: 0.2907 - acc: 0.9562 - val_loss: 0.4259 - val_acc: 0.9232\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 259s 895ms/step - loss: 0.2711 - acc: 0.9626 - val_loss: 0.6015 - val_acc: 0.8613\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 258s 894ms/step - loss: 0.2627 - acc: 0.9645 - val_loss: 0.3589 - val_acc: 0.9366\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 258s 894ms/step - loss: 0.2550 - acc: 0.9662 - val_loss: 0.5318 - val_acc: 0.8930\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 259s 895ms/step - loss: 0.2567 - acc: 0.9663 - val_loss: 0.3452 - val_acc: 0.9503\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 258s 892ms/step - loss: 0.2467 - acc: 0.9689 - val_loss: 0.3245 - val_acc: 0.9505\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 259s 896ms/step - loss: 0.2485 - acc: 0.9690 - val_loss: 0.3011 - val_acc: 0.9600\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 258s 892ms/step - loss: 0.2574 - acc: 0.9657 - val_loss: 0.3187 - val_acc: 0.9551\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 259s 895ms/step - loss: 0.2445 - acc: 0.9687 - val_loss: 0.3238 - val_acc: 0.9505\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 258s 893ms/step - loss: 0.2411 - acc: 0.9708 - val_loss: 0.3086 - val_acc: 0.9561\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2393 - acc: 0.9714 - val_loss: 0.2761 - val_acc: 0.9688\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 262s 908ms/step - loss: 0.2448 - acc: 0.9694 - val_loss: 0.5743 - val_acc: 0.9122\n",
      "Epoch 15/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "289/288 [==============================] - 262s 906ms/step - loss: 0.2403 - acc: 0.9705 - val_loss: 0.5273 - val_acc: 0.9117\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 263s 908ms/step - loss: 0.2432 - acc: 0.9696 - val_loss: 0.3447 - val_acc: 0.9476\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 263s 910ms/step - loss: 0.2491 - acc: 0.9680 - val_loss: 0.3757 - val_acc: 0.9410\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2419 - acc: 0.9705 - val_loss: 0.3250 - val_acc: 0.9600\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2399 - acc: 0.9712 - val_loss: 0.3070 - val_acc: 0.9583\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 260s 901ms/step - loss: 0.2442 - acc: 0.9699 - val_loss: 0.5338 - val_acc: 0.9083\n",
      "36905/36905 [==============================] - 62s 2ms/step\n",
      "Train [0.5028287088435132, 0.9087386532990109]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [0.489476295015831, 0.9056769410614106]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 277s 957ms/step - loss: 0.7708 - acc: 0.8207 - val_loss: 0.8798 - val_acc: 0.8105\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 261s 904ms/step - loss: 0.3316 - acc: 0.9395 - val_loss: 0.6596 - val_acc: 0.8503\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2839 - acc: 0.9552 - val_loss: 0.5607 - val_acc: 0.8883\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 262s 906ms/step - loss: 0.2618 - acc: 0.9621 - val_loss: 0.6417 - val_acc: 0.8632\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2625 - acc: 0.9623 - val_loss: 0.4876 - val_acc: 0.9193\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 261s 903ms/step - loss: 0.2562 - acc: 0.9646 - val_loss: 0.3469 - val_acc: 0.9505\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 261s 905ms/step - loss: 0.2448 - acc: 0.9682 - val_loss: 0.3667 - val_acc: 0.9442\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2376 - acc: 0.9694 - val_loss: 0.3856 - val_acc: 0.9381\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2433 - acc: 0.9679 - val_loss: 0.4609 - val_acc: 0.9190\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2400 - acc: 0.9684 - val_loss: 0.4548 - val_acc: 0.9300\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 261s 904ms/step - loss: 0.2364 - acc: 0.9695 - val_loss: 0.7233 - val_acc: 0.8520\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 261s 903ms/step - loss: 0.2435 - acc: 0.9685 - val_loss: 0.3509 - val_acc: 0.9498\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 262s 905ms/step - loss: 0.2328 - acc: 0.9711 - val_loss: 0.3660 - val_acc: 0.9405\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 262s 905ms/step - loss: 0.2350 - acc: 0.9702 - val_loss: 0.3203 - val_acc: 0.9622\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2420 - acc: 0.9677 - val_loss: 0.3700 - val_acc: 0.9476\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 261s 904ms/step - loss: 0.2360 - acc: 0.9703 - val_loss: 0.3430 - val_acc: 0.9537\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 261s 903ms/step - loss: 0.2286 - acc: 0.9723 - val_loss: 0.3471 - val_acc: 0.9544\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 261s 905ms/step - loss: 0.2297 - acc: 0.9717 - val_loss: 0.3328 - val_acc: 0.9598\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2387 - acc: 0.9695 - val_loss: 0.6300 - val_acc: 0.8776\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 261s 904ms/step - loss: 0.2326 - acc: 0.9700 - val_loss: 0.3496 - val_acc: 0.9527\n",
      "36905/36905 [==============================] - 62s 2ms/step\n",
      "Train [0.2794274739650567, 0.9538002980625931]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [0.2939411495967397, 0.954252828716348]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 275s 951ms/step - loss: 0.7618 - acc: 0.8229 - val_loss: 0.7256 - val_acc: 0.8208\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 260s 899ms/step - loss: 0.3192 - acc: 0.9477 - val_loss: 0.6438 - val_acc: 0.8630\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 260s 900ms/step - loss: 0.2907 - acc: 0.9555 - val_loss: 0.6087 - val_acc: 0.8620\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 261s 903ms/step - loss: 0.2797 - acc: 0.9582 - val_loss: 0.5089 - val_acc: 0.9003\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 260s 900ms/step - loss: 0.2621 - acc: 0.9637 - val_loss: 0.3707 - val_acc: 0.9422\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 260s 899ms/step - loss: 0.2535 - acc: 0.9666 - val_loss: 0.4096 - val_acc: 0.9337\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 260s 900ms/step - loss: 0.2569 - acc: 0.9658 - val_loss: 0.4503 - val_acc: 0.9222\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 261s 903ms/step - loss: 0.2478 - acc: 0.9682 - val_loss: 0.4178 - val_acc: 0.9254\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 260s 899ms/step - loss: 0.2465 - acc: 0.9691 - val_loss: 0.4066 - val_acc: 0.9371\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2456 - acc: 0.9704 - val_loss: 0.3385 - val_acc: 0.9500\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 260s 900ms/step - loss: 0.2494 - acc: 0.9682 - val_loss: 0.3384 - val_acc: 0.9546\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 260s 900ms/step - loss: 0.2432 - acc: 0.9694 - val_loss: 0.4233 - val_acc: 0.9366\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 260s 898ms/step - loss: 0.2489 - acc: 0.9682 - val_loss: 0.6954 - val_acc: 0.8851\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 261s 902ms/step - loss: 0.2389 - acc: 0.9707 - val_loss: 0.3956 - val_acc: 0.9442\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 261s 901ms/step - loss: 0.2504 - acc: 0.9672 - val_loss: 0.4762 - val_acc: 0.9225\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 260s 899ms/step - loss: 0.2369 - acc: 0.9722 - val_loss: 0.3667 - val_acc: 0.9466\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 260s 899ms/step - loss: 0.2421 - acc: 0.9700 - val_loss: 0.3915 - val_acc: 0.9476\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 260s 901ms/step - loss: 0.2418 - acc: 0.9701 - val_loss: 0.3084 - val_acc: 0.9620\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 260s 901ms/step - loss: 0.2376 - acc: 0.9714 - val_loss: 0.3389 - val_acc: 0.9520\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 260s 900ms/step - loss: 0.2382 - acc: 0.9711 - val_loss: 0.3674 - val_acc: 0.9507\n",
      "36905/36905 [==============================] - 62s 2ms/step\n",
      "Train [0.32953533401434615, 0.9489500067741499]\n",
      "10252/10252 [==============================] - 17s 2ms/step\n",
      "Test [0.3213469386287636, 0.9484003121342177]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 151s 523ms/step - loss: 0.9013 - acc: 0.7400 - val_loss: 1.0631 - val_acc: 0.6235\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.2307 - acc: 0.9289 - val_loss: 0.6762 - val_acc: 0.7681\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.1197 - acc: 0.9617 - val_loss: 0.1651 - val_acc: 0.9459\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0820 - acc: 0.9720 - val_loss: 0.0846 - val_acc: 0.9700\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0723 - acc: 0.9742 - val_loss: 0.0619 - val_acc: 0.9759\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0487 - acc: 0.9808 - val_loss: 0.0362 - val_acc: 0.9863\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0511 - acc: 0.9801 - val_loss: 0.1169 - val_acc: 0.9583\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 138s 477ms/step - loss: 0.0436 - acc: 0.9819 - val_loss: 0.0365 - val_acc: 0.9846\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 142s 492ms/step - loss: 0.0446 - acc: 0.9816 - val_loss: 0.0568 - val_acc: 0.9817\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0435 - acc: 0.9820 - val_loss: 0.0341 - val_acc: 0.9842\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0380 - acc: 0.9832 - val_loss: 0.0287 - val_acc: 0.9873\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0327 - acc: 0.9845 - val_loss: 0.0985 - val_acc: 0.9607\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0356 - acc: 0.9836 - val_loss: 0.0232 - val_acc: 0.9888\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0401 - acc: 0.9817 - val_loss: 0.0267 - val_acc: 0.9876\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0398 - acc: 0.9811 - val_loss: 0.4282 - val_acc: 0.8812\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0336 - acc: 0.9841 - val_loss: 0.0514 - val_acc: 0.9776\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0343 - acc: 0.9845 - val_loss: 0.0270 - val_acc: 0.9868\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0353 - acc: 0.9839 - val_loss: 0.0248 - val_acc: 0.9871\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0298 - acc: 0.9853 - val_loss: 0.0489 - val_acc: 0.9790\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0285 - acc: 0.9853 - val_loss: 0.0728 - val_acc: 0.9703\n",
      "36905/36905 [==============================] - 42s 1ms/step\n",
      "Train [0.08382979166683766, 0.9655331255927381]\n",
      "10252/10252 [==============================] - 12s 1ms/step\n",
      "Test [0.0888062823697626, 0.963129145532579]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 153s 528ms/step - loss: 0.8787 - acc: 0.7420 - val_loss: 0.6389 - val_acc: 0.7820\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.2430 - acc: 0.9250 - val_loss: 0.2887 - val_acc: 0.9008\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.1244 - acc: 0.9602 - val_loss: 0.1386 - val_acc: 0.9534\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 138s 477ms/step - loss: 0.0841 - acc: 0.9713 - val_loss: 0.2323 - val_acc: 0.9205\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0622 - acc: 0.9776 - val_loss: 0.0852 - val_acc: 0.9690\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0548 - acc: 0.9786 - val_loss: 0.0579 - val_acc: 0.9781\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 138s 478ms/step - loss: 0.0513 - acc: 0.9798 - val_loss: 0.0723 - val_acc: 0.9744\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 143s 495ms/step - loss: 0.0500 - acc: 0.9797 - val_loss: 0.0452 - val_acc: 0.9844\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0415 - acc: 0.9823 - val_loss: 0.0419 - val_acc: 0.9810\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 137s 476ms/step - loss: 0.0385 - acc: 0.9824 - val_loss: 0.0323 - val_acc: 0.9883\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 138s 477ms/step - loss: 0.0402 - acc: 0.9823 - val_loss: 0.0496 - val_acc: 0.9807\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0420 - acc: 0.9817 - val_loss: 0.0879 - val_acc: 0.9632\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0368 - acc: 0.9835 - val_loss: 0.0298 - val_acc: 0.9878\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0368 - acc: 0.9829 - val_loss: 0.1418 - val_acc: 0.9551\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0362 - acc: 0.9832 - val_loss: 0.0298 - val_acc: 0.9866\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 138s 477ms/step - loss: 0.0380 - acc: 0.9827 - val_loss: 0.2206 - val_acc: 0.9388\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0350 - acc: 0.9835 - val_loss: 0.0258 - val_acc: 0.9863\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0281 - acc: 0.9853 - val_loss: 0.1620 - val_acc: 0.9539\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0292 - acc: 0.9853 - val_loss: 0.0267 - val_acc: 0.9873\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0259 - acc: 0.9858 - val_loss: 0.0305 - val_acc: 0.9856\n",
      "36905/36905 [==============================] - 43s 1ms/step\n",
      "Train [0.03074883210176622, 0.9844465519577293]\n",
      "10252/10252 [==============================] - 12s 1ms/step\n",
      "Test [0.03469495375915343, 0.9845883730003901]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 152s 526ms/step - loss: 0.8650 - acc: 0.7483 - val_loss: 0.5190 - val_acc: 0.8218\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.2183 - acc: 0.9329 - val_loss: 0.3149 - val_acc: 0.8881\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.1130 - acc: 0.9643 - val_loss: 0.1290 - val_acc: 0.9566\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0785 - acc: 0.9727 - val_loss: 0.0851 - val_acc: 0.9703\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0610 - acc: 0.9772 - val_loss: 0.0669 - val_acc: 0.9766\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0574 - acc: 0.9776 - val_loss: 0.0650 - val_acc: 0.9763\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 137s 476ms/step - loss: 0.0499 - acc: 0.9804 - val_loss: 0.0873 - val_acc: 0.9668\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0437 - acc: 0.9821 - val_loss: 0.0574 - val_acc: 0.9781\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0430 - acc: 0.9815 - val_loss: 0.0583 - val_acc: 0.9742\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0413 - acc: 0.9820 - val_loss: 0.0486 - val_acc: 0.9805\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0385 - acc: 0.9827 - val_loss: 0.0263 - val_acc: 0.9900\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0345 - acc: 0.9841 - val_loss: 0.0370 - val_acc: 0.9854\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 138s 477ms/step - loss: 0.0356 - acc: 0.9835 - val_loss: 0.0353 - val_acc: 0.9873\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0340 - acc: 0.9840 - val_loss: 0.0255 - val_acc: 0.9876\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0328 - acc: 0.9842 - val_loss: 0.1709 - val_acc: 0.9442\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0376 - acc: 0.9827 - val_loss: 0.0227 - val_acc: 0.9893\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0354 - acc: 0.9832 - val_loss: 0.0548 - val_acc: 0.9778\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0292 - acc: 0.9851 - val_loss: 0.0297 - val_acc: 0.9851\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0353 - acc: 0.9831 - val_loss: 0.0411 - val_acc: 0.9834\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 137s 476ms/step - loss: 0.0327 - acc: 0.9838 - val_loss: 0.0205 - val_acc: 0.9885\n",
      "36905/36905 [==============================] - 42s 1ms/step\n",
      "Train [0.023361892411662595, 0.9868310527028857]\n",
      "10252/10252 [==============================] - 12s 1ms/step\n",
      "Test [0.025455840583888407, 0.9865392118611003]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 2.1843 - acc: 0.3876 - val_loss: 1.6337 - val_acc: 0.5518\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 122s 421ms/step - loss: 0.9317 - acc: 0.7113 - val_loss: 1.3100 - val_acc: 0.6352\n",
      "Epoch 3/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "289/288 [==============================] - 122s 422ms/step - loss: 0.6621 - acc: 0.7912 - val_loss: 1.1527 - val_acc: 0.6742\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.5334 - acc: 0.8255 - val_loss: 1.0636 - val_acc: 0.7223\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.4495 - acc: 0.8518 - val_loss: 0.7962 - val_acc: 0.7674\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.3963 - acc: 0.8663 - val_loss: 0.8706 - val_acc: 0.7564\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 122s 424ms/step - loss: 0.3477 - acc: 0.8832 - val_loss: 0.9464 - val_acc: 0.7825\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.3225 - acc: 0.8886 - val_loss: 0.6644 - val_acc: 0.8093\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.2912 - acc: 0.8990 - val_loss: 0.6034 - val_acc: 0.8244\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.2616 - acc: 0.9072 - val_loss: 0.6684 - val_acc: 0.8257\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.2496 - acc: 0.9119 - val_loss: 0.5538 - val_acc: 0.8378\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.2289 - acc: 0.9190 - val_loss: 0.5557 - val_acc: 0.8371\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 122s 424ms/step - loss: 0.2115 - acc: 0.9258 - val_loss: 0.5228 - val_acc: 0.8503\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 123s 425ms/step - loss: 0.1978 - acc: 0.9292 - val_loss: 0.4619 - val_acc: 0.8578\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.1902 - acc: 0.9328 - val_loss: 0.4661 - val_acc: 0.8644\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.1844 - acc: 0.9348 - val_loss: 0.4842 - val_acc: 0.8569\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.1728 - acc: 0.9374 - val_loss: 0.4244 - val_acc: 0.8625\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1564 - acc: 0.9434 - val_loss: 0.4072 - val_acc: 0.8717\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1555 - acc: 0.9439 - val_loss: 0.3381 - val_acc: 0.8893\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1591 - acc: 0.9413 - val_loss: 0.3548 - val_acc: 0.8873\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [0.35923127795354715, 0.8859233166237637]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.3736951472502968, 0.883047210323685]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 139s 481ms/step - loss: 1.3868 - acc: 0.5861 - val_loss: 0.9899 - val_acc: 0.6945\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 122s 421ms/step - loss: 0.5551 - acc: 0.8239 - val_loss: 0.6168 - val_acc: 0.7959\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.4191 - acc: 0.8605 - val_loss: 0.5201 - val_acc: 0.8274\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.3353 - acc: 0.8864 - val_loss: 0.5152 - val_acc: 0.8483\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 123s 424ms/step - loss: 0.2910 - acc: 0.9000 - val_loss: 0.3705 - val_acc: 0.8727\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 123s 424ms/step - loss: 0.2545 - acc: 0.9109 - val_loss: 0.4142 - val_acc: 0.8647\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.2329 - acc: 0.9169 - val_loss: 0.3752 - val_acc: 0.8810\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 123s 425ms/step - loss: 0.2172 - acc: 0.9247 - val_loss: 0.3436 - val_acc: 0.8859\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1891 - acc: 0.9322 - val_loss: 0.3057 - val_acc: 0.8898\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.1835 - acc: 0.9350 - val_loss: 0.3067 - val_acc: 0.8993\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1693 - acc: 0.9381 - val_loss: 0.3129 - val_acc: 0.9037\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1522 - acc: 0.9469 - val_loss: 0.2398 - val_acc: 0.9203\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1522 - acc: 0.9472 - val_loss: 0.2142 - val_acc: 0.9198\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1382 - acc: 0.9496 - val_loss: 0.1897 - val_acc: 0.9295\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 122s 424ms/step - loss: 0.1253 - acc: 0.9556 - val_loss: 0.2090 - val_acc: 0.9293\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1295 - acc: 0.9530 - val_loss: 0.1533 - val_acc: 0.9459\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 122s 421ms/step - loss: 0.1209 - acc: 0.9567 - val_loss: 0.2128 - val_acc: 0.9327\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 124s 431ms/step - loss: 0.1127 - acc: 0.9584 - val_loss: 0.1569 - val_acc: 0.9422\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 130s 449ms/step - loss: 0.1052 - acc: 0.9617 - val_loss: 0.1498 - val_acc: 0.9454\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 130s 452ms/step - loss: 0.1027 - acc: 0.9627 - val_loss: 0.1427 - val_acc: 0.9449\n",
      "36905/36905 [==============================] - 42s 1ms/step\n",
      "Train [0.14523277494982198, 0.9452377726595312]\n",
      "10252/10252 [==============================] - 12s 1ms/step\n",
      "Test [0.15631921636176863, 0.9413772922356614]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 145s 502ms/step - loss: 1.3891 - acc: 0.5834 - val_loss: 0.7585 - val_acc: 0.7671\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 129s 446ms/step - loss: 0.5647 - acc: 0.8195 - val_loss: 0.5155 - val_acc: 0.8249\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.4137 - acc: 0.8637 - val_loss: 0.5247 - val_acc: 0.8347\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.3294 - acc: 0.8895 - val_loss: 0.3555 - val_acc: 0.8681\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.2824 - acc: 0.9016 - val_loss: 0.3884 - val_acc: 0.8639\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.2481 - acc: 0.9158 - val_loss: 0.2655 - val_acc: 0.9012\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 122s 424ms/step - loss: 0.2187 - acc: 0.9244 - val_loss: 0.2806 - val_acc: 0.9034\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1944 - acc: 0.9325 - val_loss: 0.2483 - val_acc: 0.9069\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 123s 425ms/step - loss: 0.1720 - acc: 0.9384 - val_loss: 0.2596 - val_acc: 0.9095\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1635 - acc: 0.9430 - val_loss: 0.2105 - val_acc: 0.9205\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1578 - acc: 0.9439 - val_loss: 0.1906 - val_acc: 0.9208\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1422 - acc: 0.9491 - val_loss: 0.2112 - val_acc: 0.9222\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1361 - acc: 0.9521 - val_loss: 0.2290 - val_acc: 0.9183\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 123s 425ms/step - loss: 0.1197 - acc: 0.9564 - val_loss: 0.2030 - val_acc: 0.9259\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.1160 - acc: 0.9579 - val_loss: 0.1898 - val_acc: 0.9364\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.1083 - acc: 0.9605 - val_loss: 0.1559 - val_acc: 0.9376\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 123s 426ms/step - loss: 0.1042 - acc: 0.9627 - val_loss: 0.1692 - val_acc: 0.9383\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.1066 - acc: 0.9615 - val_loss: 0.1225 - val_acc: 0.9510\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 122s 423ms/step - loss: 0.0924 - acc: 0.9673 - val_loss: 0.1100 - val_acc: 0.9527\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 122s 422ms/step - loss: 0.0963 - acc: 0.9662 - val_loss: 0.1418 - val_acc: 0.9398\n",
      "36905/36905 [==============================] - 40s 1ms/step\n",
      "Train [0.1437880659883731, 0.9406042541661022]\n",
      "10252/10252 [==============================] - 11s 1ms/step\n",
      "Test [0.1461454103684465, 0.940694498611157]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 278s 963ms/step - loss: 0.5340 - acc: 0.8725 - val_loss: 0.2883 - val_acc: 0.9064\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.1254 - acc: 0.9664 - val_loss: 0.1673 - val_acc: 0.9544\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 262s 907ms/step - loss: 0.0796 - acc: 0.9768 - val_loss: 0.1577 - val_acc: 0.9490\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 262s 907ms/step - loss: 0.0647 - acc: 0.9790 - val_loss: 0.0599 - val_acc: 0.9788\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 0.0573 - acc: 0.9796 - val_loss: 0.0615 - val_acc: 0.9798\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 262s 908ms/step - loss: 0.0516 - acc: 0.9809 - val_loss: 0.0882 - val_acc: 0.9666\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 263s 908ms/step - loss: 0.0458 - acc: 0.9826 - val_loss: 0.0497 - val_acc: 0.9817\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 263s 909ms/step - loss: 0.0431 - acc: 0.9823 - val_loss: 0.0288 - val_acc: 0.9890\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0392 - acc: 0.9839 - val_loss: 0.0483 - val_acc: 0.9800\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 262s 907ms/step - loss: 0.0367 - acc: 0.9843 - val_loss: 0.0367 - val_acc: 0.9849\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 263s 909ms/step - loss: 0.0360 - acc: 0.9844 - val_loss: 0.0579 - val_acc: 0.9778\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 263s 910ms/step - loss: 0.0360 - acc: 0.9840 - val_loss: 0.0438 - val_acc: 0.9839\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 263s 908ms/step - loss: 0.0328 - acc: 0.9850 - val_loss: 0.0926 - val_acc: 0.9642\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0359 - acc: 0.9835 - val_loss: 0.0366 - val_acc: 0.9863\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0326 - acc: 0.9845 - val_loss: 0.0276 - val_acc: 0.9873\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 263s 909ms/step - loss: 0.0322 - acc: 0.9848 - val_loss: 0.0257 - val_acc: 0.9854\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 262s 907ms/step - loss: 0.0315 - acc: 0.9846 - val_loss: 0.0511 - val_acc: 0.9812\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 263s 909ms/step - loss: 0.0297 - acc: 0.9850 - val_loss: 0.1280 - val_acc: 0.9590\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0294 - acc: 0.9847 - val_loss: 0.0312 - val_acc: 0.9854\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 262s 908ms/step - loss: 0.0313 - acc: 0.9840 - val_loss: 0.0596 - val_acc: 0.9790\n",
      "36905/36905 [==============================] - 63s 2ms/step\n",
      "Train [0.07376093757600642, 0.9736892020051483]\n",
      "10252/10252 [==============================] - 18s 2ms/step\n",
      "Test [0.07460235170983466, 0.972200546234881]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 280s 968ms/step - loss: 0.5126 - acc: 0.8741 - val_loss: 0.1822 - val_acc: 0.9476\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.1205 - acc: 0.9676 - val_loss: 0.0796 - val_acc: 0.9739\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 0.0841 - acc: 0.9743 - val_loss: 0.0700 - val_acc: 0.9717\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 264s 912ms/step - loss: 0.0679 - acc: 0.9775 - val_loss: 0.0572 - val_acc: 0.9812\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 264s 912ms/step - loss: 0.0568 - acc: 0.9810 - val_loss: 0.0578 - val_acc: 0.9795\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 263s 910ms/step - loss: 0.0517 - acc: 0.9809 - val_loss: 0.0514 - val_acc: 0.9810\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 264s 912ms/step - loss: 0.0492 - acc: 0.9808 - val_loss: 0.0398 - val_acc: 0.9871\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0419 - acc: 0.9828 - val_loss: 0.1209 - val_acc: 0.9634\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 0.0421 - acc: 0.9824 - val_loss: 0.0542 - val_acc: 0.9802\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 264s 912ms/step - loss: 0.0392 - acc: 0.9842 - val_loss: 0.0983 - val_acc: 0.9673\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 263s 910ms/step - loss: 0.0357 - acc: 0.9846 - val_loss: 0.0366 - val_acc: 0.9842\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 264s 914ms/step - loss: 0.0370 - acc: 0.9834 - val_loss: 0.0326 - val_acc: 0.9868\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 263s 910ms/step - loss: 0.0348 - acc: 0.9846 - val_loss: 0.0586 - val_acc: 0.9751\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 0.0349 - acc: 0.9843 - val_loss: 0.0471 - val_acc: 0.9807\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0327 - acc: 0.9847 - val_loss: 0.1111 - val_acc: 0.9656\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 270s 935ms/step - loss: 0.0329 - acc: 0.9848 - val_loss: 0.0273 - val_acc: 0.9868\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 263s 910ms/step - loss: 0.0311 - acc: 0.9853 - val_loss: 0.0219 - val_acc: 0.9893\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 264s 914ms/step - loss: 0.0311 - acc: 0.9853 - val_loss: 0.0268 - val_acc: 0.9883\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 264s 914ms/step - loss: 0.0306 - acc: 0.9851 - val_loss: 0.0389 - val_acc: 0.9851\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0308 - acc: 0.9850 - val_loss: 0.0543 - val_acc: 0.9749\n",
      "36905/36905 [==============================] - 64s 2ms/step\n",
      "Train [0.07325525553042796, 0.9698956780923994]\n",
      "10252/10252 [==============================] - 18s 2ms/step\n",
      "Test [0.07374121123903776, 0.972200546234881]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 281s 971ms/step - loss: 0.5285 - acc: 0.8700 - val_loss: 0.2285 - val_acc: 0.9259\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 264s 914ms/step - loss: 0.1142 - acc: 0.9688 - val_loss: 0.0753 - val_acc: 0.9720\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 264s 912ms/step - loss: 0.0781 - acc: 0.9759 - val_loss: 0.0896 - val_acc: 0.9715\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0632 - acc: 0.9787 - val_loss: 0.0885 - val_acc: 0.9678\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0554 - acc: 0.9800 - val_loss: 0.0496 - val_acc: 0.9834\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 264s 914ms/step - loss: 0.0475 - acc: 0.9823 - val_loss: 0.0359 - val_acc: 0.9881\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 264s 914ms/step - loss: 0.0458 - acc: 0.9817 - val_loss: 0.0357 - val_acc: 0.9866\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 0.0424 - acc: 0.9835 - val_loss: 0.0919 - val_acc: 0.9607\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 264s 914ms/step - loss: 0.0386 - acc: 0.9840 - val_loss: 0.0335 - val_acc: 0.9885\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 263s 911ms/step - loss: 0.0395 - acc: 0.9838 - val_loss: 0.0780 - val_acc: 0.9688\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 0.0355 - acc: 0.9840 - val_loss: 0.0380 - val_acc: 0.9859\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 264s 915ms/step - loss: 0.0354 - acc: 0.9839 - val_loss: 0.0292 - val_acc: 0.9888\n",
      "Epoch 13/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "289/288 [==============================] - 264s 912ms/step - loss: 0.0357 - acc: 0.9836 - val_loss: 0.1084 - val_acc: 0.9678\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 265s 916ms/step - loss: 0.0330 - acc: 0.9846 - val_loss: 0.0231 - val_acc: 0.9890\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 0.0333 - acc: 0.9838 - val_loss: 0.0301 - val_acc: 0.9861\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 264s 915ms/step - loss: 0.0322 - acc: 0.9839 - val_loss: 0.0427 - val_acc: 0.9812\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 264s 913ms/step - loss: 0.0314 - acc: 0.9841 - val_loss: 0.0232 - val_acc: 0.9883\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 264s 912ms/step - loss: 0.0329 - acc: 0.9845 - val_loss: 0.0702 - val_acc: 0.9744\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 264s 915ms/step - loss: 0.0297 - acc: 0.9849 - val_loss: 0.0567 - val_acc: 0.9790\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 264s 915ms/step - loss: 0.0309 - acc: 0.9853 - val_loss: 0.0215 - val_acc: 0.9900\n",
      "36905/36905 [==============================] - 64s 2ms/step\n",
      "Train [0.024803246493714882, 0.9862891207153502]\n",
      "10252/10252 [==============================] - 18s 2ms/step\n",
      "Test [0.02436515665762438, 0.9878072571205618]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 155s 538ms/step - loss: 1.0952 - acc: 0.7378 - val_loss: 0.5095 - val_acc: 0.8808\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.3292 - acc: 0.9339 - val_loss: 0.2091 - val_acc: 0.9598\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.1827 - acc: 0.9624 - val_loss: 0.1849 - val_acc: 0.9427\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.1300 - acc: 0.9707 - val_loss: 0.0760 - val_acc: 0.9810\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 137s 472ms/step - loss: 0.0992 - acc: 0.9750 - val_loss: 0.0829 - val_acc: 0.9732\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 137s 472ms/step - loss: 0.0827 - acc: 0.9774 - val_loss: 0.0822 - val_acc: 0.9751\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 137s 472ms/step - loss: 0.0722 - acc: 0.9786 - val_loss: 0.0662 - val_acc: 0.9781\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 137s 476ms/step - loss: 0.0631 - acc: 0.9810 - val_loss: 0.0499 - val_acc: 0.9849\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.0569 - acc: 0.9822 - val_loss: 0.0989 - val_acc: 0.9678\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0538 - acc: 0.9812 - val_loss: 0.0460 - val_acc: 0.9837\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0498 - acc: 0.9814 - val_loss: 0.0503 - val_acc: 0.9822\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0451 - acc: 0.9829 - val_loss: 0.0406 - val_acc: 0.9837\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0442 - acc: 0.9832 - val_loss: 0.0791 - val_acc: 0.9727\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 137s 472ms/step - loss: 0.0431 - acc: 0.9833 - val_loss: 0.0317 - val_acc: 0.9893\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.0391 - acc: 0.9846 - val_loss: 0.0286 - val_acc: 0.9893\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 137s 472ms/step - loss: 0.0399 - acc: 0.9839 - val_loss: 0.0444 - val_acc: 0.9817\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0386 - acc: 0.9844 - val_loss: 0.0373 - val_acc: 0.9839\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 136s 472ms/step - loss: 0.0368 - acc: 0.9846 - val_loss: 0.0328 - val_acc: 0.9854\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0354 - acc: 0.9848 - val_loss: 0.0273 - val_acc: 0.9888\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 137s 473ms/step - loss: 0.0326 - acc: 0.9852 - val_loss: 0.0277 - val_acc: 0.9863\n",
      "36905/36905 [==============================] - 43s 1ms/step\n",
      "Train [0.030842189119780247, 0.9853678363365398]\n",
      "10252/10252 [==============================] - 12s 1ms/step\n",
      "Test [0.03282986144724924, 0.9841006632852126]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 156s 540ms/step - loss: 1.1461 - acc: 0.7265 - val_loss: 0.5627 - val_acc: 0.8530\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 138s 477ms/step - loss: 0.3433 - acc: 0.9320 - val_loss: 0.2511 - val_acc: 0.9473\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 139s 482ms/step - loss: 0.1958 - acc: 0.9578 - val_loss: 0.1396 - val_acc: 0.9637\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 138s 478ms/step - loss: 0.1353 - acc: 0.9697 - val_loss: 0.1188 - val_acc: 0.9710\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 138s 478ms/step - loss: 0.1049 - acc: 0.9735 - val_loss: 0.0795 - val_acc: 0.9807\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 138s 479ms/step - loss: 0.0861 - acc: 0.9765 - val_loss: 0.0743 - val_acc: 0.9737\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 140s 483ms/step - loss: 0.0741 - acc: 0.9792 - val_loss: 0.0787 - val_acc: 0.9722\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 140s 484ms/step - loss: 0.0666 - acc: 0.9790 - val_loss: 0.0469 - val_acc: 0.9844\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 145s 503ms/step - loss: 0.0582 - acc: 0.9806 - val_loss: 0.0478 - val_acc: 0.9859\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 145s 500ms/step - loss: 0.0540 - acc: 0.9823 - val_loss: 0.0481 - val_acc: 0.9829\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 142s 490ms/step - loss: 0.0495 - acc: 0.9822 - val_loss: 0.0424 - val_acc: 0.9861\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 141s 487ms/step - loss: 0.0476 - acc: 0.9826 - val_loss: 0.0465 - val_acc: 0.9842\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 142s 491ms/step - loss: 0.0449 - acc: 0.9831 - val_loss: 0.0594 - val_acc: 0.9773\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 139s 479ms/step - loss: 0.0419 - acc: 0.9829 - val_loss: 0.0460 - val_acc: 0.9824\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 138s 478ms/step - loss: 0.0423 - acc: 0.9830 - val_loss: 0.0498 - val_acc: 0.9790\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 141s 488ms/step - loss: 0.0384 - acc: 0.9843 - val_loss: 0.0293 - val_acc: 0.9890\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 140s 483ms/step - loss: 0.0399 - acc: 0.9835 - val_loss: 0.0374 - val_acc: 0.9849\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 139s 480ms/step - loss: 0.0341 - acc: 0.9851 - val_loss: 0.0467 - val_acc: 0.9802\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 138s 479ms/step - loss: 0.0353 - acc: 0.9848 - val_loss: 0.0408 - val_acc: 0.9822\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 142s 492ms/step - loss: 0.0341 - acc: 0.9848 - val_loss: 0.0277 - val_acc: 0.9871\n",
      "36905/36905 [==============================] - 43s 1ms/step\n",
      "Train [0.03047756422775404, 0.9848800975477577]\n",
      "10252/10252 [==============================] - 13s 1ms/step\n",
      "Test [0.03316706830769978, 0.9838080374561061]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "289/288 [==============================] - 157s 544ms/step - loss: 1.1733 - acc: 0.7179 - val_loss: 0.6089 - val_acc: 0.8493\n",
      "Epoch 2/20\n",
      "289/288 [==============================] - 139s 482ms/step - loss: 0.3473 - acc: 0.9295 - val_loss: 0.2319 - val_acc: 0.9444\n",
      "Epoch 3/20\n",
      "289/288 [==============================] - 136s 471ms/step - loss: 0.1923 - acc: 0.9609 - val_loss: 0.1924 - val_acc: 0.9525\n",
      "Epoch 4/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.1310 - acc: 0.9717 - val_loss: 0.0906 - val_acc: 0.9763\n",
      "Epoch 5/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.1022 - acc: 0.9753 - val_loss: 0.1667 - val_acc: 0.9451\n",
      "Epoch 6/20\n",
      "289/288 [==============================] - 138s 476ms/step - loss: 0.0845 - acc: 0.9779 - val_loss: 0.1784 - val_acc: 0.9400\n",
      "Epoch 7/20\n",
      "289/288 [==============================] - 137s 474ms/step - loss: 0.0712 - acc: 0.9796 - val_loss: 0.0640 - val_acc: 0.9817\n",
      "Epoch 8/20\n",
      "289/288 [==============================] - 139s 481ms/step - loss: 0.0646 - acc: 0.9803 - val_loss: 0.1056 - val_acc: 0.9642\n",
      "Epoch 9/20\n",
      "289/288 [==============================] - 137s 475ms/step - loss: 0.0578 - acc: 0.9815 - val_loss: 0.0404 - val_acc: 0.9842\n",
      "Epoch 10/20\n",
      "289/288 [==============================] - 138s 479ms/step - loss: 0.0544 - acc: 0.9820 - val_loss: 0.0354 - val_acc: 0.9885\n",
      "Epoch 11/20\n",
      "289/288 [==============================] - 139s 481ms/step - loss: 0.0493 - acc: 0.9820 - val_loss: 0.0352 - val_acc: 0.9854\n",
      "Epoch 12/20\n",
      "289/288 [==============================] - 139s 482ms/step - loss: 0.0439 - acc: 0.9838 - val_loss: 0.0394 - val_acc: 0.9866\n",
      "Epoch 13/20\n",
      "289/288 [==============================] - 140s 484ms/step - loss: 0.0434 - acc: 0.9840 - val_loss: 0.0436 - val_acc: 0.9844\n",
      "Epoch 14/20\n",
      "289/288 [==============================] - 139s 480ms/step - loss: 0.0404 - acc: 0.9840 - val_loss: 0.0465 - val_acc: 0.9798\n",
      "Epoch 15/20\n",
      "289/288 [==============================] - 139s 483ms/step - loss: 0.0409 - acc: 0.9832 - val_loss: 0.0558 - val_acc: 0.9790\n",
      "Epoch 16/20\n",
      "289/288 [==============================] - 139s 480ms/step - loss: 0.0366 - acc: 0.9844 - val_loss: 0.0304 - val_acc: 0.9863\n",
      "Epoch 17/20\n",
      "289/288 [==============================] - 139s 483ms/step - loss: 0.0373 - acc: 0.9832 - val_loss: 0.0245 - val_acc: 0.9883\n",
      "Epoch 18/20\n",
      "289/288 [==============================] - 139s 483ms/step - loss: 0.0336 - acc: 0.9855 - val_loss: 0.0305 - val_acc: 0.9876\n",
      "Epoch 19/20\n",
      "289/288 [==============================] - 139s 483ms/step - loss: 0.0340 - acc: 0.9851 - val_loss: 0.0258 - val_acc: 0.9873\n",
      "Epoch 20/20\n",
      "289/288 [==============================] - 139s 482ms/step - loss: 0.0345 - acc: 0.9853 - val_loss: 0.0644 - val_acc: 0.9751\n",
      "36905/36905 [==============================] - 44s 1ms/step\n",
      "Train [0.07644755111459899, 0.9702479338842975]\n",
      "10252/10252 [==============================] - 12s 1ms/step\n",
      "Test [0.0775058480837919, 0.9696644557159578]\n"
     ]
    }
   ],
   "source": [
    "# batch 128 / fruits-360\n",
    "import keras.optimizers\n",
    "for lr in [0.1, 0.01, 0.001, 0.0001]:\n",
    "    optimizer = optimizers.Adam(lr=lr)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITHOUT BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetC((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH LAYER BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetA((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH IDENTITY BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetB((100, 100, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks for Intel Image Classification and batch=128 for Models 1, 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 124s 1s/step - loss: 7.5996 - acc: 0.5197 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6620 - acc: 0.5246 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6601 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6478 - acc: 0.5255 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 29s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 125s 1s/step - loss: 7.5857 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 108s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 108s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 107s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6607 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6607 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6491 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 29s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 124s 1s/step - loss: 7.5969 - acc: 0.5212 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6497 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 107s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 108s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 107s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 108s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 107s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 28s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 245s 2s/step - loss: 7.6422 - acc: 0.5179 - val_loss: 13.0299 - val_acc: 0.1916\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 7.6591 - acc: 0.5248 - val_loss: 12.9840 - val_acc: 0.1944\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 7.6489 - acc: 0.5254 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 230s 2s/step - loss: 7.6496 - acc: 0.5254 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 7.6545 - acc: 0.5251 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 7.6602 - acc: 0.5247 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 7.6578 - acc: 0.5249 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 7.6532 - acc: 0.5252 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 7.6603 - acc: 0.5247 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 7.6450 - acc: 0.5257 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 7.6495 - acc: 0.5254 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 7.6585 - acc: 0.5249 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6508 - acc: 0.5253 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 7.6490 - acc: 0.5254 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 7.6373 - acc: 0.5262 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 7.6656 - acc: 0.5244 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 7.6621 - acc: 0.5246 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 7.6591 - acc: 0.5248 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 7.6469 - acc: 0.5256 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 7.6445 - acc: 0.5257 - val_loss: 12.9496 - val_acc: 0.1966\n",
      "12630/12630 [==============================] - 50s 4ms/step\n",
      "Train [7.67491893798514, 0.5238321456754387]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [13.120129594167073, 0.186]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 248s 3s/step - loss: 7.8922 - acc: 0.5031 - val_loss: 13.3743 - val_acc: 0.1702\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 7.8962 - acc: 0.5101 - val_loss: 13.3973 - val_acc: 0.1688\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 7.9007 - acc: 0.5098 - val_loss: 13.3973 - val_acc: 0.1688\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 7.9241 - acc: 0.5084 - val_loss: 13.4088 - val_acc: 0.1681\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 7.9292 - acc: 0.5081 - val_loss: 13.4088 - val_acc: 0.1681\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 7.8812 - acc: 0.5110 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 237s 2s/step - loss: 7.6620 - acc: 0.5246 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 233s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 230s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 231s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 49s 4ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 246s 2s/step - loss: 7.7291 - acc: 0.5145 - val_loss: 14.7175 - val_acc: 0.0869\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.7055 - acc: 0.5219 - val_loss: 14.7175 - val_acc: 0.0869\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.7080 - acc: 0.5218 - val_loss: 14.7405 - val_acc: 0.0855\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.7073 - acc: 0.5218 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.7061 - acc: 0.5219 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6992 - acc: 0.5223 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.7099 - acc: 0.5217 - val_loss: 14.7749 - val_acc: 0.0833\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6960 - acc: 0.5225 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.7081 - acc: 0.5218 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.7003 - acc: 0.5223 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.7038 - acc: 0.5220 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.7142 - acc: 0.5214 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6915 - acc: 0.5228 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 7.7099 - acc: 0.5217 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 232s 2s/step - loss: 7.7061 - acc: 0.5219 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 7.7099 - acc: 0.5217 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 7.7150 - acc: 0.5213 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.7113 - acc: 0.5216 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.7105 - acc: 0.5216 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.7137 - acc: 0.5214 - val_loss: 14.7634 - val_acc: 0.0840\n",
      "12630/12630 [==============================] - 49s 4ms/step\n",
      "Train [13.689533458809388, 0.15067300079648493]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [14.581503573099772, 0.09533333337306976]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 140s 1s/step - loss: 12.1834 - acc: 0.2336 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 122s 1s/step - loss: 12.4374 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 124s 1s/step - loss: 12.4343 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 125s 1s/step - loss: 12.4343 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 122s 1s/step - loss: 12.4392 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 12.4392 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 12.4417 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4411 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 12.4411 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4411 - acc: 0.2281 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 12.4362 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4343 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4331 - acc: 0.2286 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4398 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4405 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4380 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4337 - acc: 0.2286 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4386 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4398 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4386 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [12.43760555521043, 0.22834520982025355]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.146993169148763, 0.18433333333333332]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 138s 1s/step - loss: 12.2523 - acc: 0.2327 - val_loss: 13.2533 - val_acc: 0.1774\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4331 - acc: 0.2286 - val_loss: 13.1035 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4442 - acc: 0.2279 - val_loss: 13.1335 - val_acc: 0.1852\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4362 - acc: 0.2284 - val_loss: 13.1139 - val_acc: 0.1859\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4387 - acc: 0.2283 - val_loss: 13.1032 - val_acc: 0.1852\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 12.4436 - acc: 0.2280 - val_loss: 13.1057 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4386 - acc: 0.2283 - val_loss: 13.1052 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4405 - acc: 0.2282 - val_loss: 13.1083 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4380 - acc: 0.2283 - val_loss: 13.1063 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 12.4411 - acc: 0.2281 - val_loss: 13.1087 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4365 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4349 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4374 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 12.4362 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4374 - acc: 0.2284 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4392 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4386 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4349 - acc: 0.2285 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4380 - acc: 0.2283 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 12.4405 - acc: 0.2282 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [12.43760555521043, 0.22834520982025355]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.146993169148763, 0.18433333333333332]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 139s 1s/step - loss: 7.5721 - acc: 0.5219 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99/98 [==============================] - 117s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6620 - acc: 0.5246 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 119s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 120s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 32s 3ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 124s 1s/step - loss: 7.6091 - acc: 0.5214 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6614 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6491 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6595 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6534 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6595 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6503 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 29s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 125s 1s/step - loss: 7.5993 - acc: 0.5219 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6497 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6484 - acc: 0.5255 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6589 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 29s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 125s 1s/step - loss: 7.5926 - acc: 0.5223 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6614 - acc: 0.5247 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6497 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6595 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6460 - acc: 0.5256 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 7.6583 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6595 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 7.6564 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 29s 2ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 248s 3s/step - loss: 7.7401 - acc: 0.5106 - val_loss: 13.1447 - val_acc: 0.1845\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.8929 - acc: 0.5102 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.0873 - val_acc: 0.1880\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6478 - acc: 0.5255 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 7.6515 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6533 - acc: 0.5252 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6478 - acc: 0.5255 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6546 - acc: 0.5251 - val_loss: 13.0988 - val_acc: 0.1873\n",
      "12630/12630 [==============================] - 49s 4ms/step\n",
      "Train [7.6545001488305795, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [13.28131062825521, 0.176]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 248s 3s/step - loss: 7.6275 - acc: 0.5156 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6491 - acc: 0.5254 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6509 - acc: 0.5253 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6540 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6577 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6589 - acc: 0.5248 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6558 - acc: 0.5250 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 7.6528 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6521 - acc: 0.5252 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6552 - acc: 0.5251 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6626 - acc: 0.5246 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 7.6571 - acc: 0.5249 - val_loss: 13.1103 - val_acc: 0.1866\n",
      "12630/12630 [==============================] - 49s 4ms/step\n",
      "Train [7.6545001464143025, 0.5250989707188293]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [13.297428731282553, 0.175]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 248s 3s/step - loss: 11.7056 - acc: 0.2639 - val_loss: 12.6511 - val_acc: 0.2151\n",
      "Epoch 2/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99/98 [==============================] - 224s 2s/step - loss: 12.1386 - acc: 0.2469 - val_loss: 12.0082 - val_acc: 0.2550\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 12.0820 - acc: 0.2504 - val_loss: 11.9508 - val_acc: 0.2585\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 12.0858 - acc: 0.2502 - val_loss: 11.8475 - val_acc: 0.2650\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 12.1825 - acc: 0.2442 - val_loss: 10.7798 - val_acc: 0.3312\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 12.4703 - acc: 0.2263 - val_loss: 12.8118 - val_acc: 0.2051\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.3467 - acc: 0.1719 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.3537 - acc: 0.1715 - val_loss: 12.6855 - val_acc: 0.2130\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 13.3912 - acc: 0.1692 - val_loss: 12.7544 - val_acc: 0.2087\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.3505 - acc: 0.1717 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 13.3576 - acc: 0.1713 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.3640 - acc: 0.1709 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 13.3583 - acc: 0.1712 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.3486 - acc: 0.1718 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.3525 - acc: 0.1716 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.3792 - acc: 0.1699 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.3850 - acc: 0.1696 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.3569 - acc: 0.1713 - val_loss: 12.7200 - val_acc: 0.2108\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 13.7239 - acc: 0.1485 - val_loss: 12.4904 - val_acc: 0.2251\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 14.3018 - acc: 0.1127 - val_loss: 12.7544 - val_acc: 0.2087\n",
      "12630/12630 [==============================] - 49s 4ms/step\n",
      "Train [14.237012767565222, 0.11670625495089487]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [12.985812191645305, 0.19433333333333333]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 142s 1s/step - loss: 1.7127 - acc: 0.4897 - val_loss: 2.2035 - val_acc: 0.2265\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 1.2252 - acc: 0.5340 - val_loss: 2.1088 - val_acc: 0.2151\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 1.2013 - acc: 0.5368 - val_loss: 2.0699 - val_acc: 0.2315\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 1.1845 - acc: 0.5443 - val_loss: 2.2084 - val_acc: 0.2699\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 1.1402 - acc: 0.5547 - val_loss: 2.2809 - val_acc: 0.2692\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 1.1004 - acc: 0.5675 - val_loss: 2.1671 - val_acc: 0.2714\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 1.0462 - acc: 0.5888 - val_loss: 3.1006 - val_acc: 0.2315\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 1.0010 - acc: 0.6080 - val_loss: 2.6991 - val_acc: 0.2699\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 122s 1s/step - loss: 0.9755 - acc: 0.6249 - val_loss: 2.2805 - val_acc: 0.2735\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.9489 - acc: 0.6275 - val_loss: 2.0343 - val_acc: 0.2322\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.9517 - acc: 0.6337 - val_loss: 2.6488 - val_acc: 0.2123\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.9287 - acc: 0.6379 - val_loss: 1.8857 - val_acc: 0.2657\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.9072 - acc: 0.6521 - val_loss: 1.7632 - val_acc: 0.3084\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.9004 - acc: 0.6491 - val_loss: 2.3002 - val_acc: 0.2422\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8861 - acc: 0.6544 - val_loss: 1.6369 - val_acc: 0.3618\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8700 - acc: 0.6679 - val_loss: 1.9113 - val_acc: 0.3226\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8628 - acc: 0.6678 - val_loss: 1.7248 - val_acc: 0.3604\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8493 - acc: 0.6735 - val_loss: 1.6637 - val_acc: 0.3191\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8425 - acc: 0.6742 - val_loss: 1.6034 - val_acc: 0.3654\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8400 - acc: 0.6788 - val_loss: 1.5541 - val_acc: 0.3732\n",
      "12630/12630 [==============================] - 32s 3ms/step\n",
      "Train [0.8120686146539068, 0.66896278703864]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [1.588128349939982, 0.3586666666766008]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 142s 1s/step - loss: 1.3406 - acc: 0.5375 - val_loss: 2.9544 - val_acc: 0.2813\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 1.0607 - acc: 0.5867 - val_loss: 1.6042 - val_acc: 0.4238\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.9743 - acc: 0.6236 - val_loss: 1.6579 - val_acc: 0.3846\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.9274 - acc: 0.6410 - val_loss: 1.6218 - val_acc: 0.4345\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.9069 - acc: 0.6464 - val_loss: 1.8635 - val_acc: 0.3611\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8873 - acc: 0.6576 - val_loss: 1.6164 - val_acc: 0.3868\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8787 - acc: 0.6610 - val_loss: 1.5354 - val_acc: 0.3789\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8609 - acc: 0.6675 - val_loss: 1.5519 - val_acc: 0.4224\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8458 - acc: 0.6769 - val_loss: 1.3993 - val_acc: 0.4537\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.8379 - acc: 0.6771 - val_loss: 1.5473 - val_acc: 0.4452\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.8309 - acc: 0.6747 - val_loss: 1.3825 - val_acc: 0.5114\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.8107 - acc: 0.6845 - val_loss: 1.5555 - val_acc: 0.4345\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7943 - acc: 0.6934 - val_loss: 1.4422 - val_acc: 0.5157\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7734 - acc: 0.6999 - val_loss: 1.3748 - val_acc: 0.5164\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7557 - acc: 0.7134 - val_loss: 1.2869 - val_acc: 0.5349\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7461 - acc: 0.7111 - val_loss: 1.4126 - val_acc: 0.5370\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7414 - acc: 0.7151 - val_loss: 1.4510 - val_acc: 0.4537\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7252 - acc: 0.7226 - val_loss: 1.3465 - val_acc: 0.4943\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.7103 - acc: 0.7304 - val_loss: 1.3481 - val_acc: 0.5434\n",
      "Epoch 20/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99/98 [==============================] - 117s 1s/step - loss: 0.7028 - acc: 0.7330 - val_loss: 1.4286 - val_acc: 0.4245\n",
      "12630/12630 [==============================] - 32s 3ms/step\n",
      "Train [0.6752317362896337, 0.7327790973824541]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [1.505130288441976, 0.4126666666666667]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 143s 1s/step - loss: 1.4547 - acc: 0.5077 - val_loss: 2.3463 - val_acc: 0.2229\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 1.0604 - acc: 0.5863 - val_loss: 1.5680 - val_acc: 0.2835\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 120s 1s/step - loss: 0.9743 - acc: 0.6083 - val_loss: 1.6445 - val_acc: 0.2991\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 123s 1s/step - loss: 0.9287 - acc: 0.6282 - val_loss: 1.6452 - val_acc: 0.2984\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 123s 1s/step - loss: 0.9164 - acc: 0.6387 - val_loss: 1.5867 - val_acc: 0.2863\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 123s 1s/step - loss: 0.8899 - acc: 0.6594 - val_loss: 1.4384 - val_acc: 0.3362\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 123s 1s/step - loss: 0.8746 - acc: 0.6582 - val_loss: 1.5995 - val_acc: 0.4003\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 119s 1s/step - loss: 0.8540 - acc: 0.6699 - val_loss: 1.3878 - val_acc: 0.4060\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8338 - acc: 0.6749 - val_loss: 1.7436 - val_acc: 0.2635\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8097 - acc: 0.6893 - val_loss: 1.3442 - val_acc: 0.5321\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8015 - acc: 0.6927 - val_loss: 1.2552 - val_acc: 0.4950\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8005 - acc: 0.6985 - val_loss: 1.3478 - val_acc: 0.4274\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7748 - acc: 0.7056 - val_loss: 1.4046 - val_acc: 0.4900\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.7584 - acc: 0.7140 - val_loss: 1.3256 - val_acc: 0.4601\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7537 - acc: 0.7107 - val_loss: 1.5474 - val_acc: 0.3839\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.7466 - acc: 0.7165 - val_loss: 1.4250 - val_acc: 0.3853\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7433 - acc: 0.7209 - val_loss: 1.4772 - val_acc: 0.4566\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7319 - acc: 0.7249 - val_loss: 1.3544 - val_acc: 0.4444\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7202 - acc: 0.7285 - val_loss: 1.1873 - val_acc: 0.5420\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7118 - acc: 0.7312 - val_loss: 1.5743 - val_acc: 0.4858\n",
      "12630/12630 [==============================] - 32s 3ms/step\n",
      "Train [0.7085567077000936, 0.7330958036609991]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [1.715334046681722, 0.456]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 130s 1s/step - loss: 1.0462 - acc: 0.5984 - val_loss: 1.5623 - val_acc: 0.4295\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.8309 - acc: 0.6781 - val_loss: 1.4620 - val_acc: 0.5171\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7545 - acc: 0.7055 - val_loss: 1.4901 - val_acc: 0.5071\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7329 - acc: 0.7173 - val_loss: 1.2848 - val_acc: 0.5328\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6913 - acc: 0.7357 - val_loss: 1.1669 - val_acc: 0.5406\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6615 - acc: 0.7502 - val_loss: 0.9473 - val_acc: 0.6432\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6290 - acc: 0.7600 - val_loss: 1.0190 - val_acc: 0.6161\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6055 - acc: 0.7724 - val_loss: 0.9085 - val_acc: 0.6460\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.5738 - acc: 0.7835 - val_loss: 0.9904 - val_acc: 0.6517\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5788 - acc: 0.7870 - val_loss: 0.8643 - val_acc: 0.6781\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5447 - acc: 0.7910 - val_loss: 0.9610 - val_acc: 0.6396\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5307 - acc: 0.8003 - val_loss: 0.8689 - val_acc: 0.6652\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5250 - acc: 0.7996 - val_loss: 0.7489 - val_acc: 0.7244\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.5059 - acc: 0.8086 - val_loss: 0.8196 - val_acc: 0.6902\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.4907 - acc: 0.8127 - val_loss: 0.7900 - val_acc: 0.7158\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.4724 - acc: 0.8224 - val_loss: 0.7776 - val_acc: 0.7101\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.4626 - acc: 0.8241 - val_loss: 0.7048 - val_acc: 0.7521\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.4660 - acc: 0.8245 - val_loss: 0.7348 - val_acc: 0.7101\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.4434 - acc: 0.8367 - val_loss: 0.8286 - val_acc: 0.6966\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.4608 - acc: 0.8256 - val_loss: 0.7597 - val_acc: 0.7144\n",
      "12630/12630 [==============================] - 30s 2ms/step\n",
      "Train [0.40664635419090495, 0.8442596991668121]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [0.8190337677001953, 0.7086666666666667]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 129s 1s/step - loss: 1.0630 - acc: 0.5787 - val_loss: 1.6120 - val_acc: 0.4466\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.8373 - acc: 0.6718 - val_loss: 1.4096 - val_acc: 0.5164\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7919 - acc: 0.6878 - val_loss: 1.3512 - val_acc: 0.5470\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7697 - acc: 0.7048 - val_loss: 1.2101 - val_acc: 0.5356\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7426 - acc: 0.7059 - val_loss: 1.2044 - val_acc: 0.4979\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7232 - acc: 0.7136 - val_loss: 1.1961 - val_acc: 0.5477\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6917 - acc: 0.7298 - val_loss: 1.0626 - val_acc: 0.5648\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6753 - acc: 0.7393 - val_loss: 1.2316 - val_acc: 0.5584\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6667 - acc: 0.7450 - val_loss: 1.0487 - val_acc: 0.5833\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6363 - acc: 0.7547 - val_loss: 0.9930 - val_acc: 0.6047\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6154 - acc: 0.7647 - val_loss: 0.9986 - val_acc: 0.6496\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6123 - acc: 0.7678 - val_loss: 0.8423 - val_acc: 0.6823\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6003 - acc: 0.7699 - val_loss: 0.9444 - val_acc: 0.6368\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5926 - acc: 0.7716 - val_loss: 0.8361 - val_acc: 0.6952\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.5786 - acc: 0.7757 - val_loss: 0.9761 - val_acc: 0.6439\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5746 - acc: 0.7821 - val_loss: 0.9928 - val_acc: 0.6197\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5773 - acc: 0.7793 - val_loss: 0.9787 - val_acc: 0.6339\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5748 - acc: 0.7798 - val_loss: 0.8577 - val_acc: 0.6781\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5502 - acc: 0.7883 - val_loss: 0.9624 - val_acc: 0.6182\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5507 - acc: 0.7882 - val_loss: 0.8377 - val_acc: 0.6788\n",
      "12630/12630 [==============================] - 30s 2ms/step\n",
      "Train [0.5387609428861079, 0.797228820278639]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [0.8758555947939555, 0.6773333333333333]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 129s 1s/step - loss: 1.1583 - acc: 0.5518 - val_loss: 1.6737 - val_acc: 0.3675\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.9117 - acc: 0.6245 - val_loss: 1.5415 - val_acc: 0.4558\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.8389 - acc: 0.6672 - val_loss: 1.3968 - val_acc: 0.5207\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7799 - acc: 0.6941 - val_loss: 1.3080 - val_acc: 0.5085\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7598 - acc: 0.7053 - val_loss: 1.2637 - val_acc: 0.5142\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7484 - acc: 0.7108 - val_loss: 1.2404 - val_acc: 0.5306\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7179 - acc: 0.7221 - val_loss: 1.1808 - val_acc: 0.5449\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7145 - acc: 0.7237 - val_loss: 1.2382 - val_acc: 0.5605\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7064 - acc: 0.7261 - val_loss: 1.1802 - val_acc: 0.5684\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6913 - acc: 0.7317 - val_loss: 1.1269 - val_acc: 0.5641\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6808 - acc: 0.7354 - val_loss: 1.1554 - val_acc: 0.5734\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.6773 - acc: 0.7379 - val_loss: 1.0861 - val_acc: 0.5755\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6589 - acc: 0.7426 - val_loss: 1.1861 - val_acc: 0.5264\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6531 - acc: 0.7438 - val_loss: 1.0275 - val_acc: 0.6011\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6498 - acc: 0.7480 - val_loss: 0.9979 - val_acc: 0.6303\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6386 - acc: 0.7505 - val_loss: 1.1945 - val_acc: 0.5855\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6272 - acc: 0.7562 - val_loss: 1.3633 - val_acc: 0.5741\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6280 - acc: 0.7601 - val_loss: 1.0396 - val_acc: 0.6268\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.6142 - acc: 0.7629 - val_loss: 0.9356 - val_acc: 0.6503\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.5976 - acc: 0.7683 - val_loss: 0.9311 - val_acc: 0.6410\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [0.6188057999807224, 0.7503562944896242]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [0.9690122103691101, 0.628666666507721]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 250s 3s/step - loss: 1.1734 - acc: 0.6219 - val_loss: 1.5780 - val_acc: 0.4281\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.8141 - acc: 0.6857 - val_loss: 1.0876 - val_acc: 0.5691\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.7536 - acc: 0.7116 - val_loss: 1.0972 - val_acc: 0.5506\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.7076 - acc: 0.7289 - val_loss: 1.1124 - val_acc: 0.5805\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.6718 - acc: 0.7405 - val_loss: 1.0435 - val_acc: 0.5997\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.6639 - acc: 0.7463 - val_loss: 0.9811 - val_acc: 0.6339\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.6236 - acc: 0.7601 - val_loss: 1.0094 - val_acc: 0.6246\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.5992 - acc: 0.7697 - val_loss: 0.8678 - val_acc: 0.6624\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5799 - acc: 0.7806 - val_loss: 1.1417 - val_acc: 0.5734\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5634 - acc: 0.7914 - val_loss: 0.8993 - val_acc: 0.6724\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.5513 - acc: 0.7924 - val_loss: 0.8652 - val_acc: 0.6973\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5318 - acc: 0.8001 - val_loss: 0.9326 - val_acc: 0.6930\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5226 - acc: 0.8020 - val_loss: 0.8819 - val_acc: 0.6959\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.5079 - acc: 0.8084 - val_loss: 0.7765 - val_acc: 0.7009\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4974 - acc: 0.8137 - val_loss: 0.9883 - val_acc: 0.6403\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4827 - acc: 0.8213 - val_loss: 0.9015 - val_acc: 0.6774\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4760 - acc: 0.8189 - val_loss: 0.7429 - val_acc: 0.7265\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4570 - acc: 0.8317 - val_loss: 0.6759 - val_acc: 0.7714\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4459 - acc: 0.8329 - val_loss: 0.7609 - val_acc: 0.7322\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4483 - acc: 0.8311 - val_loss: 0.6461 - val_acc: 0.7650\n",
      "12630/12630 [==============================] - 50s 4ms/step\n",
      "Train [0.5555722406858126, 0.8115597783244987]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.7249271276046833, 0.7586666666666667]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 251s 3s/step - loss: 1.1642 - acc: 0.6266 - val_loss: 1.7533 - val_acc: 0.4594\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.8068 - acc: 0.6888 - val_loss: 1.2014 - val_acc: 0.5278\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.7492 - acc: 0.7139 - val_loss: 1.0561 - val_acc: 0.6161\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.7134 - acc: 0.7299 - val_loss: 0.9389 - val_acc: 0.6553\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.6627 - acc: 0.7471 - val_loss: 0.8741 - val_acc: 0.6838\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.6404 - acc: 0.7580 - val_loss: 1.1431 - val_acc: 0.6083\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.6099 - acc: 0.7692 - val_loss: 0.9413 - val_acc: 0.6774\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5948 - acc: 0.7733 - val_loss: 0.9414 - val_acc: 0.6446\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5830 - acc: 0.7784 - val_loss: 0.7636 - val_acc: 0.7265\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5508 - acc: 0.7934 - val_loss: 1.0496 - val_acc: 0.6439\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5453 - acc: 0.7933 - val_loss: 0.7805 - val_acc: 0.7386\n",
      "Epoch 12/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99/98 [==============================] - 224s 2s/step - loss: 0.5265 - acc: 0.8032 - val_loss: 0.8623 - val_acc: 0.7066\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5072 - acc: 0.8107 - val_loss: 0.7796 - val_acc: 0.7130\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4934 - acc: 0.8168 - val_loss: 0.7337 - val_acc: 0.7429\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4760 - acc: 0.8225 - val_loss: 0.7224 - val_acc: 0.7429\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4726 - acc: 0.8237 - val_loss: 0.7523 - val_acc: 0.7422\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4733 - acc: 0.8244 - val_loss: 0.7686 - val_acc: 0.7358\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4567 - acc: 0.8319 - val_loss: 0.6487 - val_acc: 0.7771\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4519 - acc: 0.8318 - val_loss: 0.7781 - val_acc: 0.7472\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 0.4339 - acc: 0.8364 - val_loss: 0.6335 - val_acc: 0.7742\n",
      "12630/12630 [==============================] - 50s 4ms/step\n",
      "Train [0.7008114476668297, 0.7468725256946289]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.6931772590875626, 0.7576666668256123]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 251s 3s/step - loss: 1.1316 - acc: 0.6300 - val_loss: 3.1268 - val_acc: 0.4473\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.8494 - acc: 0.6883 - val_loss: 2.4280 - val_acc: 0.4131\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.7735 - acc: 0.7086 - val_loss: 1.4691 - val_acc: 0.4772\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.7208 - acc: 0.7269 - val_loss: 1.0650 - val_acc: 0.5990\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.6816 - acc: 0.7389 - val_loss: 0.9873 - val_acc: 0.6168\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.6439 - acc: 0.7553 - val_loss: 1.1018 - val_acc: 0.5876\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.6283 - acc: 0.7567 - val_loss: 0.8502 - val_acc: 0.6823\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.5976 - acc: 0.7749 - val_loss: 0.8891 - val_acc: 0.6852\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.5722 - acc: 0.7804 - val_loss: 1.3691 - val_acc: 0.5833\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.5582 - acc: 0.7913 - val_loss: 0.9514 - val_acc: 0.6681\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.5419 - acc: 0.7979 - val_loss: 0.8215 - val_acc: 0.7016\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.5329 - acc: 0.8005 - val_loss: 0.7510 - val_acc: 0.7187\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.5120 - acc: 0.8055 - val_loss: 0.8084 - val_acc: 0.7066\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4997 - acc: 0.8105 - val_loss: 0.8183 - val_acc: 0.7080\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4885 - acc: 0.8192 - val_loss: 0.9172 - val_acc: 0.6859\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4718 - acc: 0.8267 - val_loss: 0.7720 - val_acc: 0.7001\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 223s 2s/step - loss: 0.4677 - acc: 0.8207 - val_loss: 0.7106 - val_acc: 0.7521\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4576 - acc: 0.8307 - val_loss: 0.6662 - val_acc: 0.7692\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4408 - acc: 0.8340 - val_loss: 0.7381 - val_acc: 0.7521\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4340 - acc: 0.8389 - val_loss: 0.8577 - val_acc: 0.6887\n",
      "12630/12630 [==============================] - 50s 4ms/step\n",
      "Train [0.5946154295973427, 0.7811559777928079]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.844678606847922, 0.7013333333333334]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 145s 1s/step - loss: 1.0495 - acc: 0.5856 - val_loss: 1.4522 - val_acc: 0.4060\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8721 - acc: 0.6617 - val_loss: 1.5966 - val_acc: 0.3120\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8173 - acc: 0.6826 - val_loss: 1.5515 - val_acc: 0.3241\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7823 - acc: 0.6964 - val_loss: 1.5291 - val_acc: 0.3234\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7642 - acc: 0.7067 - val_loss: 1.3286 - val_acc: 0.4900\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.7369 - acc: 0.7126 - val_loss: 1.1693 - val_acc: 0.5235\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7033 - acc: 0.7267 - val_loss: 1.1356 - val_acc: 0.5007\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6791 - acc: 0.7358 - val_loss: 1.0796 - val_acc: 0.4907\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6581 - acc: 0.7463 - val_loss: 1.0053 - val_acc: 0.6211\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6353 - acc: 0.7588 - val_loss: 0.8665 - val_acc: 0.6574\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6136 - acc: 0.7644 - val_loss: 0.9253 - val_acc: 0.6318\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6029 - acc: 0.7709 - val_loss: 0.7867 - val_acc: 0.7165\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5879 - acc: 0.7755 - val_loss: 1.0400 - val_acc: 0.5997\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5850 - acc: 0.7735 - val_loss: 0.8766 - val_acc: 0.6524\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5567 - acc: 0.7919 - val_loss: 0.9404 - val_acc: 0.6432\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5408 - acc: 0.7990 - val_loss: 1.1317 - val_acc: 0.5712\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5422 - acc: 0.7981 - val_loss: 0.8883 - val_acc: 0.6660\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5235 - acc: 0.8010 - val_loss: 0.9386 - val_acc: 0.6617\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5190 - acc: 0.8065 - val_loss: 0.8663 - val_acc: 0.6887\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.4999 - acc: 0.8120 - val_loss: 0.8207 - val_acc: 0.7001\n",
      "12630/12630 [==============================] - 33s 3ms/step\n",
      "Train [0.6086510470955502, 0.7819477434301791]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.8197077368895213, 0.7053333333333334]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 146s 1s/step - loss: 0.9929 - acc: 0.6163 - val_loss: 1.4157 - val_acc: 0.4274\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8311 - acc: 0.6773 - val_loss: 1.7672 - val_acc: 0.3575\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7782 - acc: 0.7007 - val_loss: 1.3159 - val_acc: 0.4637\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7457 - acc: 0.7142 - val_loss: 1.1074 - val_acc: 0.5007\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.7094 - acc: 0.7243 - val_loss: 1.1287 - val_acc: 0.5890\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6794 - acc: 0.7412 - val_loss: 1.1441 - val_acc: 0.6033\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6498 - acc: 0.7553 - val_loss: 0.9793 - val_acc: 0.6075\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6270 - acc: 0.7615 - val_loss: 1.2132 - val_acc: 0.5463\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.6159 - acc: 0.7681 - val_loss: 1.3033 - val_acc: 0.5078\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5998 - acc: 0.7741 - val_loss: 0.9650 - val_acc: 0.6189\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5708 - acc: 0.7853 - val_loss: 1.0065 - val_acc: 0.5947\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5577 - acc: 0.7884 - val_loss: 0.9249 - val_acc: 0.6517\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 120s 1s/step - loss: 0.5469 - acc: 0.7927 - val_loss: 0.8226 - val_acc: 0.6788\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 119s 1s/step - loss: 0.5395 - acc: 0.7964 - val_loss: 0.7809 - val_acc: 0.7151\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 120s 1s/step - loss: 0.5126 - acc: 0.8084 - val_loss: 0.8304 - val_acc: 0.6916\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 119s 1s/step - loss: 0.5092 - acc: 0.8076 - val_loss: 0.7929 - val_acc: 0.7108\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5033 - acc: 0.8114 - val_loss: 0.8233 - val_acc: 0.6816\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.4786 - acc: 0.8220 - val_loss: 0.9679 - val_acc: 0.6510\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.4826 - acc: 0.8173 - val_loss: 0.8616 - val_acc: 0.6845\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 119s 1s/step - loss: 0.4612 - acc: 0.8254 - val_loss: 0.7935 - val_acc: 0.7258\n",
      "12630/12630 [==============================] - 33s 3ms/step\n",
      "Train [0.5532708516407844, 0.813460015873067]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.8390145451227824, 0.7186666665077209]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 148s 1s/step - loss: 1.0393 - acc: 0.5989 - val_loss: 1.5903 - val_acc: 0.4103\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.8606 - acc: 0.6610 - val_loss: 1.5634 - val_acc: 0.3447\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.8361 - acc: 0.6736 - val_loss: 1.4568 - val_acc: 0.4053\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7855 - acc: 0.6984 - val_loss: 1.3948 - val_acc: 0.4274\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.7628 - acc: 0.7019 - val_loss: 1.1761 - val_acc: 0.5335\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 128s 1s/step - loss: 0.7373 - acc: 0.7166 - val_loss: 1.1906 - val_acc: 0.4964\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 126s 1s/step - loss: 0.7153 - acc: 0.7239 - val_loss: 1.1625 - val_acc: 0.5271\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 125s 1s/step - loss: 0.6928 - acc: 0.7316 - val_loss: 1.2233 - val_acc: 0.4729\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 121s 1s/step - loss: 0.6759 - acc: 0.7411 - val_loss: 1.0995 - val_acc: 0.5805\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 129s 1s/step - loss: 0.6436 - acc: 0.7503 - val_loss: 1.0166 - val_acc: 0.6510\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 120s 1s/step - loss: 0.6318 - acc: 0.7636 - val_loss: 0.9250 - val_acc: 0.6702\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 119s 1s/step - loss: 0.6134 - acc: 0.7680 - val_loss: 1.0817 - val_acc: 0.5933\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 119s 1s/step - loss: 0.5942 - acc: 0.7735 - val_loss: 1.0454 - val_acc: 0.5926\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5847 - acc: 0.7838 - val_loss: 0.9208 - val_acc: 0.6560\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5712 - acc: 0.7841 - val_loss: 0.9436 - val_acc: 0.6752\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5653 - acc: 0.7898 - val_loss: 0.9726 - val_acc: 0.6375\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5521 - acc: 0.7923 - val_loss: 0.8698 - val_acc: 0.6695\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 119s 1s/step - loss: 0.5378 - acc: 0.7944 - val_loss: 0.7861 - val_acc: 0.7023\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5289 - acc: 0.7989 - val_loss: 0.8431 - val_acc: 0.6880\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5211 - acc: 0.8048 - val_loss: 0.8742 - val_acc: 0.6652\n",
      "12630/12630 [==============================] - 33s 3ms/step\n",
      "Train [0.580911651208867, 0.7825019794235321]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.8959825271765391, 0.6623333331743876]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 134s 1s/step - loss: 1.1739 - acc: 0.5553 - val_loss: 2.3157 - val_acc: 0.2443\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.9739 - acc: 0.6313 - val_loss: 1.7724 - val_acc: 0.4003\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.9028 - acc: 0.6556 - val_loss: 1.7768 - val_acc: 0.3583\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.8640 - acc: 0.6665 - val_loss: 1.7011 - val_acc: 0.4658\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.8225 - acc: 0.6765 - val_loss: 1.6462 - val_acc: 0.4409\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.7960 - acc: 0.6969 - val_loss: 1.5718 - val_acc: 0.4338\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7816 - acc: 0.6988 - val_loss: 1.5578 - val_acc: 0.4138\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.7692 - acc: 0.7013 - val_loss: 1.5981 - val_acc: 0.4103\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.7665 - acc: 0.7001 - val_loss: 1.5215 - val_acc: 0.4672\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7576 - acc: 0.7053 - val_loss: 1.4802 - val_acc: 0.4601\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7471 - acc: 0.7098 - val_loss: 1.4856 - val_acc: 0.4850\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.7376 - acc: 0.7123 - val_loss: 1.4914 - val_acc: 0.4815\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.7258 - acc: 0.7189 - val_loss: 1.4166 - val_acc: 0.5007\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.7277 - acc: 0.7198 - val_loss: 1.4488 - val_acc: 0.4665\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.7156 - acc: 0.7255 - val_loss: 1.3524 - val_acc: 0.5235\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.7115 - acc: 0.7227 - val_loss: 1.4116 - val_acc: 0.5121\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.7043 - acc: 0.7273 - val_loss: 1.3260 - val_acc: 0.5007\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.6906 - acc: 0.7356 - val_loss: 1.3262 - val_acc: 0.5299\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.6903 - acc: 0.7347 - val_loss: 1.2771 - val_acc: 0.5014\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6880 - acc: 0.7372 - val_loss: 1.4567 - val_acc: 0.4801\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [0.6643383336274848, 0.7445764053368135]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [1.5440834786891937, 0.459]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 135s 1s/step - loss: 1.0726 - acc: 0.5793 - val_loss: 1.5854 - val_acc: 0.4501\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.8734 - acc: 0.6641 - val_loss: 1.6189 - val_acc: 0.4530\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.8117 - acc: 0.6842 - val_loss: 1.6053 - val_acc: 0.4801\n",
      "Epoch 4/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99/98 [==============================] - 104s 1s/step - loss: 0.7788 - acc: 0.6961 - val_loss: 1.5639 - val_acc: 0.4872\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.7703 - acc: 0.6994 - val_loss: 1.4789 - val_acc: 0.4886\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7450 - acc: 0.7082 - val_loss: 1.4195 - val_acc: 0.5071\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7348 - acc: 0.7169 - val_loss: 1.4245 - val_acc: 0.4929\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 103s 1s/step - loss: 0.7227 - acc: 0.7229 - val_loss: 1.4359 - val_acc: 0.5299\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7133 - acc: 0.7246 - val_loss: 1.4415 - val_acc: 0.5121\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7036 - acc: 0.7258 - val_loss: 1.3493 - val_acc: 0.5321\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.6899 - acc: 0.7292 - val_loss: 1.3595 - val_acc: 0.5591\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.6900 - acc: 0.7327 - val_loss: 1.2986 - val_acc: 0.5491\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.6754 - acc: 0.7382 - val_loss: 1.2525 - val_acc: 0.5613\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6799 - acc: 0.7362 - val_loss: 1.1819 - val_acc: 0.5719\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6616 - acc: 0.7441 - val_loss: 1.2102 - val_acc: 0.5434\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6569 - acc: 0.7470 - val_loss: 1.1499 - val_acc: 0.5862\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.6472 - acc: 0.7498 - val_loss: 1.1872 - val_acc: 0.5890\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6514 - acc: 0.7449 - val_loss: 1.1624 - val_acc: 0.5719\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.6423 - acc: 0.7516 - val_loss: 1.0728 - val_acc: 0.5855\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.6298 - acc: 0.7601 - val_loss: 1.1401 - val_acc: 0.5848\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [0.6331271374027406, 0.7542359461127437]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [1.2465165349642435, 0.5689999998410543]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 136s 1s/step - loss: 1.0742 - acc: 0.5858 - val_loss: 1.8686 - val_acc: 0.3746\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.8827 - acc: 0.6648 - val_loss: 1.6140 - val_acc: 0.4338\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.8183 - acc: 0.6836 - val_loss: 1.6507 - val_acc: 0.4701\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.7844 - acc: 0.6949 - val_loss: 1.6971 - val_acc: 0.4566\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7641 - acc: 0.7041 - val_loss: 1.5929 - val_acc: 0.4694\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.7484 - acc: 0.7104 - val_loss: 1.6526 - val_acc: 0.4523\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.7301 - acc: 0.7140 - val_loss: 1.4168 - val_acc: 0.4843\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 107s 1s/step - loss: 0.7109 - acc: 0.7278 - val_loss: 1.4444 - val_acc: 0.4858\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.7076 - acc: 0.7263 - val_loss: 1.3446 - val_acc: 0.5527\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.6880 - acc: 0.7344 - val_loss: 1.3971 - val_acc: 0.5470\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 106s 1s/step - loss: 0.6816 - acc: 0.7373 - val_loss: 1.4089 - val_acc: 0.5071\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6640 - acc: 0.7429 - val_loss: 1.3349 - val_acc: 0.5249\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.6580 - acc: 0.7451 - val_loss: 1.2874 - val_acc: 0.5491\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6555 - acc: 0.7482 - val_loss: 1.2519 - val_acc: 0.5256\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 104s 1s/step - loss: 0.6359 - acc: 0.7540 - val_loss: 1.1140 - val_acc: 0.5926\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6442 - acc: 0.7514 - val_loss: 1.1421 - val_acc: 0.5705\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6350 - acc: 0.7577 - val_loss: 1.1163 - val_acc: 0.5897\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6248 - acc: 0.7585 - val_loss: 1.1273 - val_acc: 0.5741\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6151 - acc: 0.7591 - val_loss: 1.0876 - val_acc: 0.6097\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 105s 1s/step - loss: 0.6128 - acc: 0.7604 - val_loss: 1.0153 - val_acc: 0.6282\n",
      "12630/12630 [==============================] - 31s 2ms/step\n",
      "Train [0.5980758410734689, 0.7773555028089341]\n",
      "3000/3000 [==============================] - 7s 2ms/step\n",
      "Test [1.107325613975525, 0.5970000000794728]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 260s 3s/step - loss: 0.8970 - acc: 0.6674 - val_loss: 1.2381 - val_acc: 0.5328\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 231s 2s/step - loss: 0.6948 - acc: 0.7334 - val_loss: 1.0578 - val_acc: 0.5655\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 0.6243 - acc: 0.7598 - val_loss: 0.9614 - val_acc: 0.6011\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 0.5988 - acc: 0.7714 - val_loss: 1.0276 - val_acc: 0.5862\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 0.5793 - acc: 0.7806 - val_loss: 0.8666 - val_acc: 0.6624\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.5574 - acc: 0.7903 - val_loss: 0.9012 - val_acc: 0.6360\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.5316 - acc: 0.7965 - val_loss: 0.9445 - val_acc: 0.6446\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.5192 - acc: 0.8040 - val_loss: 0.8125 - val_acc: 0.7030\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 0.5055 - acc: 0.8085 - val_loss: 0.8513 - val_acc: 0.6795\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.4897 - acc: 0.8105 - val_loss: 0.8347 - val_acc: 0.6859\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4783 - acc: 0.8214 - val_loss: 0.9002 - val_acc: 0.6731\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.4833 - acc: 0.8194 - val_loss: 0.8203 - val_acc: 0.7037\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 0.4691 - acc: 0.8251 - val_loss: 0.7799 - val_acc: 0.7158\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 233s 2s/step - loss: 0.4694 - acc: 0.8259 - val_loss: 0.8129 - val_acc: 0.7101\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 0.4558 - acc: 0.8282 - val_loss: 0.7410 - val_acc: 0.7158\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 0.4517 - acc: 0.8307 - val_loss: 0.7851 - val_acc: 0.7208\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.4380 - acc: 0.8392 - val_loss: 0.7014 - val_acc: 0.7543\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4400 - acc: 0.8347 - val_loss: 0.8363 - val_acc: 0.6994\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.4333 - acc: 0.8348 - val_loss: 0.7662 - val_acc: 0.7336\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.4307 - acc: 0.8365 - val_loss: 0.6790 - val_acc: 0.7528\n",
      "12630/12630 [==============================] - 50s 4ms/step\n",
      "Train [0.5322682105541606, 0.7993665874806644]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.7504655334949494, 0.7296666668256124]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "99/98 [==============================] - 257s 3s/step - loss: 0.9276 - acc: 0.6697 - val_loss: 1.4045 - val_acc: 0.5534\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.6969 - acc: 0.7330 - val_loss: 1.2625 - val_acc: 0.5670\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.6396 - acc: 0.7540 - val_loss: 1.2456 - val_acc: 0.5869\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.6082 - acc: 0.7654 - val_loss: 0.9752 - val_acc: 0.6410\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.5809 - acc: 0.7798 - val_loss: 0.9338 - val_acc: 0.6538\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.5613 - acc: 0.7933 - val_loss: 0.9617 - val_acc: 0.6581\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.5342 - acc: 0.8004 - val_loss: 0.7944 - val_acc: 0.7073\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.5291 - acc: 0.8027 - val_loss: 0.8638 - val_acc: 0.6759\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.5225 - acc: 0.8050 - val_loss: 0.8252 - val_acc: 0.7130\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4952 - acc: 0.8155 - val_loss: 0.8220 - val_acc: 0.7023\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.4988 - acc: 0.8142 - val_loss: 0.7956 - val_acc: 0.7194\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.4848 - acc: 0.8190 - val_loss: 0.8102 - val_acc: 0.7066\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 0.4730 - acc: 0.8217 - val_loss: 0.7346 - val_acc: 0.7386\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 0.4716 - acc: 0.8232 - val_loss: 0.7879 - val_acc: 0.7244\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 226s 2s/step - loss: 0.4603 - acc: 0.8320 - val_loss: 0.7749 - val_acc: 0.7222\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 227s 2s/step - loss: 0.4582 - acc: 0.8294 - val_loss: 0.7817 - val_acc: 0.7258\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 0.4502 - acc: 0.8317 - val_loss: 0.7330 - val_acc: 0.7564\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4414 - acc: 0.8323 - val_loss: 0.6544 - val_acc: 0.7714\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4412 - acc: 0.8318 - val_loss: 0.7904 - val_acc: 0.7457\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4290 - acc: 0.8385 - val_loss: 0.7444 - val_acc: 0.7365\n",
      "12630/12630 [==============================] - 50s 4ms/step\n",
      "Train [0.6002772404481188, 0.765241488481644]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.7889941775798798, 0.7229999998410542]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 258s 3s/step - loss: 0.9451 - acc: 0.6642 - val_loss: 1.4310 - val_acc: 0.5591\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 228s 2s/step - loss: 0.6885 - acc: 0.7356 - val_loss: 1.2251 - val_acc: 0.5356\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 231s 2s/step - loss: 0.6332 - acc: 0.7574 - val_loss: 0.9921 - val_acc: 0.6325\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 234s 2s/step - loss: 0.6053 - acc: 0.7710 - val_loss: 1.0664 - val_acc: 0.6047\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 231s 2s/step - loss: 0.5884 - acc: 0.7770 - val_loss: 0.9449 - val_acc: 0.6660\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 0.5549 - acc: 0.7909 - val_loss: 0.8598 - val_acc: 0.6745\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 0.5388 - acc: 0.7981 - val_loss: 0.8622 - val_acc: 0.6852\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 230s 2s/step - loss: 0.5210 - acc: 0.7984 - val_loss: 0.9187 - val_acc: 0.6709\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 232s 2s/step - loss: 0.5124 - acc: 0.8071 - val_loss: 0.9533 - val_acc: 0.6660\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 231s 2s/step - loss: 0.5055 - acc: 0.8107 - val_loss: 0.8037 - val_acc: 0.7080\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 231s 2s/step - loss: 0.4930 - acc: 0.8152 - val_loss: 0.8365 - val_acc: 0.6887\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 231s 2s/step - loss: 0.4803 - acc: 0.8205 - val_loss: 0.8238 - val_acc: 0.6916\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 232s 2s/step - loss: 0.4706 - acc: 0.8241 - val_loss: 0.7326 - val_acc: 0.7365\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 231s 2s/step - loss: 0.4631 - acc: 0.8263 - val_loss: 0.8135 - val_acc: 0.6887\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 0.4613 - acc: 0.8251 - val_loss: 0.7806 - val_acc: 0.7194\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 229s 2s/step - loss: 0.4462 - acc: 0.8314 - val_loss: 0.7205 - val_acc: 0.7336\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.4467 - acc: 0.8315 - val_loss: 0.7068 - val_acc: 0.7571\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4307 - acc: 0.8382 - val_loss: 0.7207 - val_acc: 0.7336\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 224s 2s/step - loss: 0.4309 - acc: 0.8375 - val_loss: 0.8559 - val_acc: 0.6973\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 225s 2s/step - loss: 0.4280 - acc: 0.8426 - val_loss: 0.6648 - val_acc: 0.7664\n",
      "12630/12630 [==============================] - 52s 4ms/step\n",
      "Train [0.45442247520055645, 0.8273159144798726]\n",
      "3000/3000 [==============================] - 12s 4ms/step\n",
      "Test [0.7099273521105448, 0.7346666666666667]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 154s 2s/step - loss: 0.9950 - acc: 0.6235 - val_loss: 1.4849 - val_acc: 0.4380\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 125s 1s/step - loss: 0.8038 - acc: 0.6909 - val_loss: 1.1397 - val_acc: 0.5577\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 121s 1s/step - loss: 0.7514 - acc: 0.7113 - val_loss: 1.0213 - val_acc: 0.5819\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.7088 - acc: 0.7237 - val_loss: 0.9551 - val_acc: 0.6567\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6836 - acc: 0.7370 - val_loss: 0.8461 - val_acc: 0.6702\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.6500 - acc: 0.7530 - val_loss: 0.8421 - val_acc: 0.6774\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.6265 - acc: 0.7574 - val_loss: 0.8183 - val_acc: 0.6852\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.6173 - acc: 0.7620 - val_loss: 0.9099 - val_acc: 0.6560\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5996 - acc: 0.7702 - val_loss: 0.8838 - val_acc: 0.6595\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5859 - acc: 0.7780 - val_loss: 0.8719 - val_acc: 0.6738\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5758 - acc: 0.7845 - val_loss: 1.1211 - val_acc: 0.5890\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5668 - acc: 0.7872 - val_loss: 0.8542 - val_acc: 0.6959\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5539 - acc: 0.7920 - val_loss: 0.8072 - val_acc: 0.6944\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5465 - acc: 0.7934 - val_loss: 0.7132 - val_acc: 0.7251\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5321 - acc: 0.8008 - val_loss: 0.8629 - val_acc: 0.6738\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5282 - acc: 0.8009 - val_loss: 0.6996 - val_acc: 0.7472\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5173 - acc: 0.8083 - val_loss: 0.9501 - val_acc: 0.6588\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5152 - acc: 0.8075 - val_loss: 0.7693 - val_acc: 0.7087\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5104 - acc: 0.8075 - val_loss: 0.8237 - val_acc: 0.7030\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.4993 - acc: 0.8182 - val_loss: 0.7840 - val_acc: 0.7137\n",
      "12630/12630 [==============================] - 34s 3ms/step\n",
      "Train [0.7320851718916934, 0.7484560569693716]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.8218554126024247, 0.703333333492279]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 150s 2s/step - loss: 0.9572 - acc: 0.6347 - val_loss: 1.2765 - val_acc: 0.4879\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.7839 - acc: 0.6932 - val_loss: 1.1761 - val_acc: 0.5164\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7246 - acc: 0.7179 - val_loss: 1.0859 - val_acc: 0.5933\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6879 - acc: 0.7350 - val_loss: 0.9811 - val_acc: 0.6154\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6558 - acc: 0.7498 - val_loss: 1.0039 - val_acc: 0.6239\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6358 - acc: 0.7569 - val_loss: 0.9657 - val_acc: 0.6182\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6150 - acc: 0.7644 - val_loss: 0.9274 - val_acc: 0.6261\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5955 - acc: 0.7764 - val_loss: 0.8021 - val_acc: 0.6887\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5774 - acc: 0.7802 - val_loss: 0.8613 - val_acc: 0.6788\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5665 - acc: 0.7866 - val_loss: 0.8399 - val_acc: 0.6916\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5619 - acc: 0.7870 - val_loss: 0.8530 - val_acc: 0.6795\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5451 - acc: 0.7942 - val_loss: 0.9316 - val_acc: 0.6745\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5325 - acc: 0.8024 - val_loss: 0.7966 - val_acc: 0.7123\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5217 - acc: 0.8042 - val_loss: 0.8644 - val_acc: 0.7073\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5211 - acc: 0.8051 - val_loss: 0.7485 - val_acc: 0.7222\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5038 - acc: 0.8110 - val_loss: 0.7946 - val_acc: 0.7215\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5155 - acc: 0.8067 - val_loss: 0.7756 - val_acc: 0.7251\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5014 - acc: 0.8133 - val_loss: 0.7163 - val_acc: 0.7514\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5032 - acc: 0.8090 - val_loss: 0.7089 - val_acc: 0.7386\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.4944 - acc: 0.8152 - val_loss: 0.7745 - val_acc: 0.7229\n",
      "12630/12630 [==============================] - 34s 3ms/step\n",
      "Train [0.7302485177654741, 0.7467933491592075]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.8082740044593811, 0.710333333492279]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "99/98 [==============================] - 150s 2s/step - loss: 0.9862 - acc: 0.6292 - val_loss: 1.3843 - val_acc: 0.4829\n",
      "Epoch 2/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.8052 - acc: 0.6948 - val_loss: 1.1249 - val_acc: 0.5463\n",
      "Epoch 3/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7484 - acc: 0.7140 - val_loss: 1.0762 - val_acc: 0.5648\n",
      "Epoch 4/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.7113 - acc: 0.7268 - val_loss: 0.9444 - val_acc: 0.6246\n",
      "Epoch 5/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.6808 - acc: 0.7400 - val_loss: 0.9150 - val_acc: 0.6197\n",
      "Epoch 6/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6472 - acc: 0.7542 - val_loss: 0.8813 - val_acc: 0.6517\n",
      "Epoch 7/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6298 - acc: 0.7598 - val_loss: 0.8399 - val_acc: 0.6923\n",
      "Epoch 8/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.6049 - acc: 0.7718 - val_loss: 0.8242 - val_acc: 0.6916\n",
      "Epoch 9/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5874 - acc: 0.7817 - val_loss: 0.7765 - val_acc: 0.6916\n",
      "Epoch 10/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5699 - acc: 0.7855 - val_loss: 0.8531 - val_acc: 0.6788\n",
      "Epoch 11/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5552 - acc: 0.7929 - val_loss: 0.8201 - val_acc: 0.6766\n",
      "Epoch 12/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.5552 - acc: 0.7889 - val_loss: 0.7530 - val_acc: 0.7201\n",
      "Epoch 13/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5413 - acc: 0.7981 - val_loss: 0.8523 - val_acc: 0.7023\n",
      "Epoch 14/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5287 - acc: 0.8007 - val_loss: 0.8428 - val_acc: 0.6830\n",
      "Epoch 15/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5188 - acc: 0.8060 - val_loss: 0.7956 - val_acc: 0.7165\n",
      "Epoch 16/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5127 - acc: 0.8079 - val_loss: 0.7231 - val_acc: 0.7479\n",
      "Epoch 17/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5046 - acc: 0.8104 - val_loss: 0.6772 - val_acc: 0.7486\n",
      "Epoch 18/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.5019 - acc: 0.8106 - val_loss: 0.9452 - val_acc: 0.6695\n",
      "Epoch 19/20\n",
      "99/98 [==============================] - 117s 1s/step - loss: 0.4940 - acc: 0.8139 - val_loss: 0.7068 - val_acc: 0.7322\n",
      "Epoch 20/20\n",
      "99/98 [==============================] - 118s 1s/step - loss: 0.4856 - acc: 0.8191 - val_loss: 0.7049 - val_acc: 0.7350\n",
      "12630/12630 [==============================] - 34s 3ms/step\n",
      "Train [0.5855426279448934, 0.7850356294159274]\n",
      "3000/3000 [==============================] - 8s 3ms/step\n",
      "Test [0.7265497652689615, 0.730333333492279]\n"
     ]
    }
   ],
   "source": [
    "# batch 128 / intel\n",
    "import keras.optimizers\n",
    "for lr in [0.1, 0.01, 0.001, 0.0001]:\n",
    "    optimizer = optimizers.Adam(lr=lr)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITHOUT BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetC((150, 150, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH LAYER BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetA((150, 150, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH IDENTITY BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetB((150, 150, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=128)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset for Columbia University COIL-100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7200\n",
      "(5184, 128, 128, 3) (5184, 101)\n",
      "(1440, 128, 128, 3) (1440, 101)\n",
      "(576, 128, 128, 3) (576, 101)\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from keras.utils import to_categorical\n",
    "import random\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "root = '/usr/local/google/home/aferlitsch/Desktop/Datasets/coil-100'\n",
    "def loadImages(root):\n",
    "    nfiles=0\n",
    "    images = []\n",
    "    labels = []\n",
    "    \n",
    "    classes = {}\n",
    "    nclass = 0\n",
    "    files = os.scandir(root)\n",
    "    for file in files:\n",
    "        pair = file.name.split('_')\n",
    "        label = pair[0][3:]\n",
    "        image = cv2.imread(file.path)\n",
    "        if image is None:\n",
    "            continue\n",
    "        image = cv2.resize(image, (128, 128))\n",
    "        images.append(image)\n",
    "        labels.append(int(label))\n",
    "        nfiles += 1\n",
    "        \n",
    "        \n",
    "    return nfiles, np.asarray(images).astype(np.float32), np.asarray(labels)\n",
    "        \n",
    "nfiles, images, labels = loadImages(root)\n",
    "print(nfiles)\n",
    "\n",
    "labels = to_categorical(labels)\n",
    "mean = np.mean(images)\n",
    "std  = np.mean(images)\n",
    "images = (images - mean) / std\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(images, labels, test_size=0.20, random_state=42)\n",
    "\n",
    "pivot = int(len(x_train) * 0.9)\n",
    "x_val = x_train[pivot:]\n",
    "y_val = y_train[pivot:]\n",
    "x_train = x_train[:pivot]\n",
    "y_train = y_train[:pivot]\n",
    "print(x_train.shape, y_train.shape)\n",
    "print(x_test.shape, y_test.shape)\n",
    "print(x_val.shape, y_val.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks for COIL-100 for Models 1, 2 and 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 64s 397ms/step - loss: 15.8853 - acc: 0.0102 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 31s 189ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 31s 190ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 31s 190ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 32s 199ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 32s 200ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "5184/5184 [==============================] - 10s 2ms/step\n",
      "Train [15.950198491414389, 0.010416666666666666]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [15.972584745619033, 0.009027777777777777]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 65s 401ms/step - loss: 15.8888 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.9502 - val_acc: 0.0104\n",
      "5184/5184 [==============================] - 10s 2ms/step\n",
      "Train [15.956416895360123, 0.010030864197530864]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [15.96139161851671, 0.009722222222222222]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 67s 415ms/step - loss: 15.8949 - acc: 0.0095 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 31s 188ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 31s 188ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 31s 190ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 31s 188ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 31s 189ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9626 - acc: 0.0096 - val_loss: 15.9782 - val_acc: 0.0087\n",
      "5184/5184 [==============================] - 10s 2ms/step\n",
      "Train [15.962635311079614, 0.009645061728395061]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [15.927812237209745, 0.011805555555555555]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 100s 619ms/step - loss: 15.8993 - acc: 0.0098 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 66s 405ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 70s 430ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 71s 438ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 398ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 66s 406ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 398ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 64s 397ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 65s 398ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [15.953307693387256, 0.010223765432098766]\n",
      "1440/1440 [==============================] - 4s 3ms/step\n",
      "Test [15.994970999823677, 0.007638888888888889]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 101s 622ms/step - loss: 15.9123 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 64s 397ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 63s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [15.965744501278724, 0.00945216049382716]\n",
      "1440/1440 [==============================] - 4s 3ms/step\n",
      "Test [15.972584745619033, 0.009027777777777777]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 100s 620ms/step - loss: 15.9176 - acc: 0.0085 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 64s 397ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10/20\n",
      "162/162 [==============================] - 63s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 63s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 63s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [15.943980087468654, 0.010802469135802469]\n",
      "1440/1440 [==============================] - 4s 3ms/step\n",
      "Test [15.939005364312067, 0.011111111111111112]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 71s 440ms/step - loss: 15.8757 - acc: 0.0096 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [15.956416901247001, 0.010030864197530864]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [15.983777872721355, 0.008333333333333333]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 72s 443ms/step - loss: 15.8406 - acc: 0.0096 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.9689 - acc: 0.0093 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [15.968853703251591, 0.009259259259259259]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [15.96139161851671, 0.009722222222222222]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 73s 452ms/step - loss: 15.7327 - acc: 0.0170 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 15.8091 - acc: 0.0191 - val_loss: 15.8103 - val_acc: 0.0191\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 213ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.7823 - acc: 0.0208 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.7947 - acc: 0.0201 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 34s 211ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 15.7823 - acc: 0.0208 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 34s 213ms/step - loss: 15.8010 - acc: 0.0197 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 34s 212ms/step - loss: 15.7854 - acc: 0.0206 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 36s 220ms/step - loss: 15.7916 - acc: 0.0203 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 36s 220ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 15.7885 - acc: 0.0204 - val_loss: 15.8383 - val_acc: 0.0174\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [15.78851999471217, 0.02044753086419753]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [15.838267220391167, 0.017361111111111112]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 70s 430ms/step - loss: 4.7549 - acc: 0.0062 - val_loss: 4.6327 - val_acc: 0.0156\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 4.6863 - acc: 0.0112 - val_loss: 4.9424 - val_acc: 0.0104\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 4.6584 - acc: 0.0096 - val_loss: 4.6459 - val_acc: 0.0174\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 4.6349 - acc: 0.0089 - val_loss: 4.6508 - val_acc: 0.0104\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 4.6364 - acc: 0.0110 - val_loss: 4.6436 - val_acc: 0.0035\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 4.6340 - acc: 0.0112 - val_loss: 4.6594 - val_acc: 0.0069\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 4.6397 - acc: 0.0083 - val_loss: 4.6564 - val_acc: 0.0052\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 4.6403 - acc: 0.0089 - val_loss: 4.6444 - val_acc: 0.0035\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 4.6394 - acc: 0.0096 - val_loss: 4.6509 - val_acc: 0.0174\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 4.6486 - acc: 0.0091 - val_loss: 4.6439 - val_acc: 0.0122\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 4.6517 - acc: 0.0077 - val_loss: 4.6469 - val_acc: 0.0069\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 4.6550 - acc: 0.0075 - val_loss: 4.6550 - val_acc: 0.0122\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 4.6582 - acc: 0.0123 - val_loss: 4.7055 - val_acc: 0.0087\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 4.6622 - acc: 0.0102 - val_loss: 4.7084 - val_acc: 0.0122\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 4.6571 - acc: 0.0112 - val_loss: 4.6758 - val_acc: 0.0087\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 32s 194ms/step - loss: 4.6590 - acc: 0.0083 - val_loss: 4.6790 - val_acc: 0.0069\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 4.6543 - acc: 0.0104 - val_loss: 4.6549 - val_acc: 0.0122\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 4.6616 - acc: 0.0081 - val_loss: 4.6662 - val_acc: 0.0087\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 4.6591 - acc: 0.0079 - val_loss: 4.6996 - val_acc: 0.0139\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 4.6606 - acc: 0.0106 - val_loss: 4.6694 - val_acc: 0.0104\n",
      "5184/5184 [==============================] - 10s 2ms/step\n",
      "Train [4.645883292327692, 0.010609567901234568]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [4.6915514945983885, 0.007638888888888889]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 71s 435ms/step - loss: 15.8885 - acc: 0.0098 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9502 - acc: 0.0104 - val_loss: 16.0621 - val_acc: 0.0035\n",
      "5184/5184 [==============================] - 10s 2ms/step\n",
      "Train [15.950198491414389, 0.010416666666666666]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [15.939005364312067, 0.011111111111111112]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 70s 432ms/step - loss: 15.9041 - acc: 0.0091 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 31s 191ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 31s 192ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.9222 - val_acc: 0.0122\n",
      "5184/5184 [==============================] - 10s 2ms/step\n",
      "Train [15.965744501278724, 0.00945216049382716]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [15.939005364312067, 0.011111111111111112]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 105s 647ms/step - loss: 15.8480 - acc: 0.0118 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9409 - acc: 0.0110 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 15.9409 - acc: 0.0110 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9409 - acc: 0.0110 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 63s 392ms/step - loss: 15.9440 - acc: 0.0108 - val_loss: 15.8942 - val_acc: 0.0139\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 63s 391ms/step - loss: 15.9350 - acc: 0.0112 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 63s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9657 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [15.965744501278724, 0.00945216049382716]\n",
      "1440/1440 [==============================] - 4s 3ms/step\n",
      "Test [15.96139161851671, 0.009722222222222222]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 105s 648ms/step - loss: 15.8843 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9533 - acc: 0.0102 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 13/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "162/162 [==============================] - 64s 394ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9564 - acc: 0.0100 - val_loss: 16.1181 - val_acc: 0.0000e+00\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [16.11809539794922, 0.0]\n",
      "1440/1440 [==============================] - 4s 3ms/step\n",
      "Test [16.11809539794922, 0.0]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 107s 659ms/step - loss: 15.8724 - acc: 0.0095 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 63s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 64s 397ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 64s 392ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 393ms/step - loss: 15.9782 - acc: 0.0087 - val_loss: 15.8663 - val_acc: 0.0156\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [15.978181309170193, 0.008680555555555556]\n",
      "1440/1440 [==============================] - 4s 3ms/step\n",
      "Test [15.916619110107423, 0.0125]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 77s 475ms/step - loss: 4.5899 - acc: 0.0320 - val_loss: 5.5103 - val_acc: 0.0191\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 3.2889 - acc: 0.1466 - val_loss: 6.2844 - val_acc: 0.0417\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 2.0867 - acc: 0.3970 - val_loss: 2.9553 - val_acc: 0.2639\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 1.2784 - acc: 0.6028 - val_loss: 2.4643 - val_acc: 0.3767\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.8780 - acc: 0.7193 - val_loss: 1.5277 - val_acc: 0.5469\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.6306 - acc: 0.7895 - val_loss: 1.4191 - val_acc: 0.5816\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.4844 - acc: 0.8428 - val_loss: 1.1301 - val_acc: 0.6910\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.3894 - acc: 0.8762 - val_loss: 0.6764 - val_acc: 0.7917\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.3483 - acc: 0.8872 - val_loss: 0.3167 - val_acc: 0.9028\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.3109 - acc: 0.8941 - val_loss: 0.3449 - val_acc: 0.8854\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.2494 - acc: 0.9186 - val_loss: 0.4206 - val_acc: 0.8698\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.2405 - acc: 0.9201 - val_loss: 0.3879 - val_acc: 0.8576\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.2335 - acc: 0.9236 - val_loss: 2.5840 - val_acc: 0.5920\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.2043 - acc: 0.9286 - val_loss: 0.8731 - val_acc: 0.7760\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.2118 - acc: 0.9300 - val_loss: 0.6394 - val_acc: 0.8299\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1780 - acc: 0.9416 - val_loss: 0.1722 - val_acc: 0.9375\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1656 - acc: 0.9443 - val_loss: 0.2076 - val_acc: 0.9149\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1405 - acc: 0.9506 - val_loss: 0.5695 - val_acc: 0.8142\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1976 - acc: 0.9446 - val_loss: 0.3746 - val_acc: 0.8941\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1211 - acc: 0.9616 - val_loss: 0.1460 - val_acc: 0.9462\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.13383117897068092, 0.9517746913580247]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.14524975799851947, 0.9486111111111111]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 77s 478ms/step - loss: 4.5643 - acc: 0.0311 - val_loss: 4.2675 - val_acc: 0.0295\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 3.8859 - acc: 0.0507 - val_loss: 6.3649 - val_acc: 0.0139\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 3.2209 - acc: 0.1453 - val_loss: 8.4231 - val_acc: 0.0243\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 1.8852 - acc: 0.4427 - val_loss: 7.6808 - val_acc: 0.0590\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 1.1987 - acc: 0.6244 - val_loss: 3.9300 - val_acc: 0.2101\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.8645 - acc: 0.7215 - val_loss: 1.6467 - val_acc: 0.4965\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.7408 - acc: 0.7504 - val_loss: 1.6376 - val_acc: 0.5434\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.5696 - acc: 0.8142 - val_loss: 1.0668 - val_acc: 0.6806\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.4548 - acc: 0.8463 - val_loss: 0.6345 - val_acc: 0.7969\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.3976 - acc: 0.8659 - val_loss: 1.1461 - val_acc: 0.6597\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 0.3972 - acc: 0.8634 - val_loss: 1.0711 - val_acc: 0.6840\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.3125 - acc: 0.8920 - val_loss: 0.4984 - val_acc: 0.8264\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.2891 - acc: 0.8999 - val_loss: 0.5686 - val_acc: 0.8160\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.2732 - acc: 0.9047 - val_loss: 1.1999 - val_acc: 0.6719\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.2191 - acc: 0.9238 - val_loss: 1.0723 - val_acc: 0.7257\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.2512 - acc: 0.9178 - val_loss: 0.3470 - val_acc: 0.8715\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.1817 - acc: 0.9402 - val_loss: 0.3187 - val_acc: 0.9115\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.2064 - acc: 0.9288 - val_loss: 0.5682 - val_acc: 0.8351\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.2007 - acc: 0.9296 - val_loss: 0.3243 - val_acc: 0.8854\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1710 - acc: 0.9423 - val_loss: 0.3146 - val_acc: 0.9080\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.25727763135032156, 0.9158950617283951]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.2728603238860766, 0.9159722222222222]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 78s 480ms/step - loss: 4.2186 - acc: 0.0573 - val_loss: 4.3222 - val_acc: 0.0781\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 2.6343 - acc: 0.2541 - val_loss: 2.9154 - val_acc: 0.2483\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 1.6695 - acc: 0.4913 - val_loss: 4.2645 - val_acc: 0.1354\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 1.1518 - acc: 0.6304 - val_loss: 2.9560 - val_acc: 0.3646\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.8225 - acc: 0.7313 - val_loss: 2.0629 - val_acc: 0.4861\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.6499 - acc: 0.7853 - val_loss: 1.4058 - val_acc: 0.6250\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.5382 - acc: 0.8167 - val_loss: 1.2335 - val_acc: 0.6389\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.4494 - acc: 0.8493 - val_loss: 0.7447 - val_acc: 0.7934\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.3613 - acc: 0.8742 - val_loss: 0.9001 - val_acc: 0.7431\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.3543 - acc: 0.8848 - val_loss: 0.7705 - val_acc: 0.7708\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.2985 - acc: 0.9020 - val_loss: 1.0668 - val_acc: 0.7083\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.2738 - acc: 0.9049 - val_loss: 0.5472 - val_acc: 0.8542\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.2088 - acc: 0.9319 - val_loss: 0.3461 - val_acc: 0.9097\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.2007 - acc: 0.9331 - val_loss: 0.5305 - val_acc: 0.8333\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.2184 - acc: 0.9277 - val_loss: 0.7515 - val_acc: 0.8177\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1469 - acc: 0.9489 - val_loss: 0.5715 - val_acc: 0.8611\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1700 - acc: 0.9427 - val_loss: 0.4194 - val_acc: 0.8524\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.1843 - acc: 0.9360 - val_loss: 0.1910 - val_acc: 0.9497\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1612 - acc: 0.9471 - val_loss: 0.3083 - val_acc: 0.9062\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 214ms/step - loss: 0.1266 - acc: 0.9585 - val_loss: 0.1263 - val_acc: 0.9618\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.09070968778982161, 0.9691358024691358]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.1218191758212116, 0.9583333333333334]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 75s 465ms/step - loss: 2.9783 - acc: 0.2240 - val_loss: 1.6811 - val_acc: 0.5486\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 1.1782 - acc: 0.6416 - val_loss: 0.8436 - val_acc: 0.7465\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.8110 - acc: 0.7606 - val_loss: 0.3997 - val_acc: 0.8646\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.4500 - acc: 0.8582 - val_loss: 0.5812 - val_acc: 0.8281\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 0.3281 - acc: 0.8916 - val_loss: 0.2024 - val_acc: 0.9340\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 0.2830 - acc: 0.9051 - val_loss: 0.2100 - val_acc: 0.9219\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.2986 - acc: 0.8964 - val_loss: 0.2683 - val_acc: 0.9219\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 0.2341 - acc: 0.9252 - val_loss: 0.0970 - val_acc: 0.9618\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.2495 - acc: 0.9196 - val_loss: 0.2116 - val_acc: 0.9340\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 0.2186 - acc: 0.9313 - val_loss: 0.2574 - val_acc: 0.9253\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1800 - acc: 0.9433 - val_loss: 0.1122 - val_acc: 0.9722\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 0.0962 - acc: 0.9691 - val_loss: 0.0712 - val_acc: 0.9722\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1414 - acc: 0.9537 - val_loss: 0.1391 - val_acc: 0.9618\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1544 - acc: 0.9562 - val_loss: 0.1303 - val_acc: 0.9635\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.1030 - acc: 0.9670 - val_loss: 0.0622 - val_acc: 0.9809\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 0.1238 - acc: 0.9614 - val_loss: 0.0830 - val_acc: 0.9826\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 0.1000 - acc: 0.9668 - val_loss: 0.0923 - val_acc: 0.9774\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1156 - acc: 0.9620 - val_loss: 0.1023 - val_acc: 0.9757\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.0504 - acc: 0.9826 - val_loss: 0.0762 - val_acc: 0.9792\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.0631 - acc: 0.9792 - val_loss: 0.1331 - val_acc: 0.9653\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.0769740307698584, 0.9741512345679012]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.10651456906149785, 0.9736111111111111]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 75s 463ms/step - loss: 2.9416 - acc: 0.2348 - val_loss: 1.7759 - val_acc: 0.4444\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 1.0031 - acc: 0.6796 - val_loss: 1.0497 - val_acc: 0.6858\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.5857 - acc: 0.8183 - val_loss: 0.8771 - val_acc: 0.7170\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.4063 - acc: 0.8661 - val_loss: 0.5529 - val_acc: 0.8403\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.3008 - acc: 0.8976 - val_loss: 0.6478 - val_acc: 0.8003\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.2235 - acc: 0.9223 - val_loss: 0.3530 - val_acc: 0.8854\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2257 - acc: 0.9238 - val_loss: 0.4712 - val_acc: 0.8507\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 0.1694 - acc: 0.9431 - val_loss: 0.4507 - val_acc: 0.8750\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1562 - acc: 0.9460 - val_loss: 0.5652 - val_acc: 0.8594\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.2274 - acc: 0.9369 - val_loss: 0.3655 - val_acc: 0.8837\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1078 - acc: 0.9664 - val_loss: 0.5740 - val_acc: 0.8455\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1343 - acc: 0.9545 - val_loss: 0.3320 - val_acc: 0.9028\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.0866 - acc: 0.9724 - val_loss: 0.5038 - val_acc: 0.8733\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1119 - acc: 0.9660 - val_loss: 0.4949 - val_acc: 0.8698\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1287 - acc: 0.9595 - val_loss: 0.2177 - val_acc: 0.9358\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 32s 194ms/step - loss: 0.0700 - acc: 0.9782 - val_loss: 0.2409 - val_acc: 0.9271\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.0644 - acc: 0.9780 - val_loss: 0.2705 - val_acc: 0.9201\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1002 - acc: 0.9670 - val_loss: 0.3193 - val_acc: 0.9167\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 0.1384 - acc: 0.9742 - val_loss: 0.5062 - val_acc: 0.8993\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1297 - acc: 0.9745 - val_loss: 0.5887 - val_acc: 0.8507\n",
      "5184/5184 [==============================] - 10s 2ms/step\n",
      "Train [0.6889986780782541, 0.8364197530864198]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.63609446088473, 0.8430555555555556]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 77s 473ms/step - loss: 3.8304 - acc: 0.0966 - val_loss: 3.1568 - val_acc: 0.1615\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 2.1824 - acc: 0.3505 - val_loss: 1.9629 - val_acc: 0.4271\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 1.3380 - acc: 0.5754 - val_loss: 1.3287 - val_acc: 0.5885\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.9542 - acc: 0.6935 - val_loss: 1.2495 - val_acc: 0.6389\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 32s 194ms/step - loss: 0.6837 - acc: 0.7757 - val_loss: 0.9372 - val_acc: 0.7222\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.5062 - acc: 0.8341 - val_loss: 0.7072 - val_acc: 0.7830\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 31s 193ms/step - loss: 0.4037 - acc: 0.8611 - val_loss: 0.7967 - val_acc: 0.7934\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.4537 - acc: 0.8466 - val_loss: 0.5081 - val_acc: 0.8299\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.3265 - acc: 0.8908 - val_loss: 0.7178 - val_acc: 0.8038\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.3079 - acc: 0.8972 - val_loss: 0.5176 - val_acc: 0.8524\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.2543 - acc: 0.9157 - val_loss: 0.5769 - val_acc: 0.8420\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.3260 - acc: 0.8968 - val_loss: 1.0205 - val_acc: 0.7622\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.2414 - acc: 0.9194 - val_loss: 0.3720 - val_acc: 0.8802\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 32s 198ms/step - loss: 0.2010 - acc: 0.9342 - val_loss: 0.3479 - val_acc: 0.9132\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.1831 - acc: 0.9390 - val_loss: 0.2737 - val_acc: 0.9288\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.2762 - acc: 0.9136 - val_loss: 0.5303 - val_acc: 0.8559\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 31s 194ms/step - loss: 0.1493 - acc: 0.9502 - val_loss: 0.2745 - val_acc: 0.9132\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 0.1462 - acc: 0.9531 - val_loss: 0.4896 - val_acc: 0.8924\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 32s 200ms/step - loss: 0.1632 - acc: 0.9460 - val_loss: 0.4078 - val_acc: 0.9080\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 32s 200ms/step - loss: 0.3505 - acc: 0.9350 - val_loss: 0.3751 - val_acc: 0.9028\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.39961953293302177, 0.9048996913580247]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.46106614329748685, 0.9]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 111s 683ms/step - loss: 2.0268 - acc: 0.4855 - val_loss: 1.9876 - val_acc: 0.4826\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 65s 404ms/step - loss: 0.5895 - acc: 0.8243 - val_loss: 1.2691 - val_acc: 0.6302\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 66s 404ms/step - loss: 0.3872 - acc: 0.8816 - val_loss: 0.4091 - val_acc: 0.8681\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 65s 402ms/step - loss: 0.2677 - acc: 0.9198 - val_loss: 0.2229 - val_acc: 0.9288\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 65s 401ms/step - loss: 0.2239 - acc: 0.9311 - val_loss: 0.4950 - val_acc: 0.8472\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 397ms/step - loss: 0.1843 - acc: 0.9412 - val_loss: 0.5600 - val_acc: 0.8611\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.1436 - acc: 0.9558 - val_loss: 0.1971 - val_acc: 0.9375\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1383 - acc: 0.9556 - val_loss: 0.6409 - val_acc: 0.8559\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.1247 - acc: 0.9612 - val_loss: 0.2773 - val_acc: 0.9132\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 66s 405ms/step - loss: 0.1137 - acc: 0.9660 - val_loss: 0.5286 - val_acc: 0.8681\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.1228 - acc: 0.9630 - val_loss: 0.2433 - val_acc: 0.9080\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.0861 - acc: 0.9715 - val_loss: 0.1554 - val_acc: 0.9340\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.0728 - acc: 0.9774 - val_loss: 0.1013 - val_acc: 0.9705\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.0927 - acc: 0.9716 - val_loss: 0.9139 - val_acc: 0.8490\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.0977 - acc: 0.9686 - val_loss: 0.8743 - val_acc: 0.8594\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.0790 - acc: 0.9728 - val_loss: 0.4347 - val_acc: 0.8906\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 64s 398ms/step - loss: 0.0713 - acc: 0.9761 - val_loss: 0.1090 - val_acc: 0.9688\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.0782 - acc: 0.9742 - val_loss: 0.1607 - val_acc: 0.9462\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.0543 - acc: 0.9836 - val_loss: 0.1408 - val_acc: 0.9566\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0458 - acc: 0.9871 - val_loss: 0.0701 - val_acc: 0.9722\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [0.058167747275064484, 0.9822530864197531]\n",
      "1440/1440 [==============================] - 5s 3ms/step\n",
      "Test [0.07240489174404906, 0.9763888888888889]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 111s 686ms/step - loss: 2.1218 - acc: 0.4925 - val_loss: 3.0826 - val_acc: 0.3646\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.5984 - acc: 0.8218 - val_loss: 0.9802 - val_acc: 0.6875\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 65s 400ms/step - loss: 0.3190 - acc: 0.9039 - val_loss: 1.5033 - val_acc: 0.6736\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.2456 - acc: 0.9244 - val_loss: 0.4023 - val_acc: 0.8854\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.2051 - acc: 0.9377 - val_loss: 0.6412 - val_acc: 0.8333\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1605 - acc: 0.9487 - val_loss: 0.1492 - val_acc: 0.9479\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 398ms/step - loss: 0.1561 - acc: 0.9497 - val_loss: 0.4574 - val_acc: 0.8663\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.1159 - acc: 0.9622 - val_loss: 0.3124 - val_acc: 0.9062\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1131 - acc: 0.9641 - val_loss: 0.1525 - val_acc: 0.9497\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1104 - acc: 0.9628 - val_loss: 0.5213 - val_acc: 0.8872\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 64s 398ms/step - loss: 0.1134 - acc: 0.9639 - val_loss: 0.2756 - val_acc: 0.9115\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0771 - acc: 0.9742 - val_loss: 0.2150 - val_acc: 0.9306\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.0960 - acc: 0.9715 - val_loss: 0.1949 - val_acc: 0.9444\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0773 - acc: 0.9736 - val_loss: 0.4377 - val_acc: 0.8715\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.0766 - acc: 0.9749 - val_loss: 0.2518 - val_acc: 0.9427\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0852 - acc: 0.9749 - val_loss: 0.4156 - val_acc: 0.8559\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.0830 - acc: 0.9742 - val_loss: 0.1473 - val_acc: 0.9531\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0617 - acc: 0.9803 - val_loss: 0.1742 - val_acc: 0.9462\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0565 - acc: 0.9815 - val_loss: 0.2731 - val_acc: 0.9392\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0602 - acc: 0.9824 - val_loss: 0.2592 - val_acc: 0.9219\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [0.21748530647981865, 0.9305555555555556]\n",
      "1440/1440 [==============================] - 5s 3ms/step\n",
      "Test [0.25208789805571236, 0.9277777777777778]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 111s 686ms/step - loss: 2.1726 - acc: 0.4900 - val_loss: 3.8617 - val_acc: 0.3542\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.5422 - acc: 0.8387 - val_loss: 1.4727 - val_acc: 0.6146\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.3614 - acc: 0.8835 - val_loss: 1.3164 - val_acc: 0.6719\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.2398 - acc: 0.9244 - val_loss: 0.5967 - val_acc: 0.8368\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.2094 - acc: 0.9338 - val_loss: 0.4487 - val_acc: 0.8733\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 397ms/step - loss: 0.1836 - acc: 0.9414 - val_loss: 0.9457 - val_acc: 0.7517\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.1537 - acc: 0.9524 - val_loss: 0.4831 - val_acc: 0.8715\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.1423 - acc: 0.9551 - val_loss: 0.4280 - val_acc: 0.8646\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.1287 - acc: 0.9593 - val_loss: 0.5298 - val_acc: 0.8247\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.1026 - acc: 0.9670 - val_loss: 0.3066 - val_acc: 0.9080\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.0848 - acc: 0.9718 - val_loss: 0.1833 - val_acc: 0.9340\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.0874 - acc: 0.9732 - val_loss: 0.1599 - val_acc: 0.9497\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.0790 - acc: 0.9738 - val_loss: 0.7299 - val_acc: 0.8229\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0833 - acc: 0.9743 - val_loss: 0.7091 - val_acc: 0.8472\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0830 - acc: 0.9726 - val_loss: 0.6596 - val_acc: 0.8576\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0740 - acc: 0.9770 - val_loss: 0.1927 - val_acc: 0.9531\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0745 - acc: 0.9755 - val_loss: 0.0942 - val_acc: 0.9670\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0668 - acc: 0.9769 - val_loss: 0.2158 - val_acc: 0.9444\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.0742 - acc: 0.9772 - val_loss: 0.4593 - val_acc: 0.8524\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.0435 - acc: 0.9861 - val_loss: 0.1926 - val_acc: 0.9323\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [0.1293129587005594, 0.9567901234567902]\n",
      "1440/1440 [==============================] - 5s 3ms/step\n",
      "Test [0.14327308663891422, 0.9548611111111112]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 83s 512ms/step - loss: 2.6120 - acc: 0.3304 - val_loss: 1.9069 - val_acc: 0.3906\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 1.1376 - acc: 0.6736 - val_loss: 1.0686 - val_acc: 0.6458\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.6610 - acc: 0.8235 - val_loss: 0.6178 - val_acc: 0.8316\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.4499 - acc: 0.8771 - val_loss: 0.4196 - val_acc: 0.8611\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 36s 220ms/step - loss: 0.3263 - acc: 0.9091 - val_loss: 0.3164 - val_acc: 0.8941\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.2687 - acc: 0.9263 - val_loss: 0.2138 - val_acc: 0.9340\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.2267 - acc: 0.9383 - val_loss: 0.1698 - val_acc: 0.9479\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1901 - acc: 0.9431 - val_loss: 0.2180 - val_acc: 0.9219\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1611 - acc: 0.9564 - val_loss: 0.1123 - val_acc: 0.9670\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1492 - acc: 0.9560 - val_loss: 0.0987 - val_acc: 0.9688\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 0.1352 - acc: 0.9610 - val_loss: 0.1497 - val_acc: 0.9549\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 0.1117 - acc: 0.9666 - val_loss: 0.2001 - val_acc: 0.9340\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 0.0978 - acc: 0.9753 - val_loss: 0.1432 - val_acc: 0.9479\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 0.0985 - acc: 0.9734 - val_loss: 0.1262 - val_acc: 0.9688\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1092 - acc: 0.9684 - val_loss: 0.0827 - val_acc: 0.9705\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 36s 219ms/step - loss: 0.0764 - acc: 0.9784 - val_loss: 0.0986 - val_acc: 0.9635\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 0.0956 - acc: 0.9709 - val_loss: 0.1123 - val_acc: 0.9722\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.0628 - acc: 0.9842 - val_loss: 0.0920 - val_acc: 0.9705\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.0606 - acc: 0.9821 - val_loss: 0.0373 - val_acc: 0.9896\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 0.0497 - acc: 0.9865 - val_loss: 0.1197 - val_acc: 0.9670\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.08563838106361621, 0.9733796296296297]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.09155505452719, 0.9666666666666667]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 82s 509ms/step - loss: 2.6221 - acc: 0.3040 - val_loss: 1.4991 - val_acc: 0.5503\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 1.1015 - acc: 0.6887 - val_loss: 1.0381 - val_acc: 0.6736\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.6561 - acc: 0.8177 - val_loss: 0.5334 - val_acc: 0.8472\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.4558 - acc: 0.8721 - val_loss: 0.4852 - val_acc: 0.8403\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.3512 - acc: 0.8995 - val_loss: 0.2586 - val_acc: 0.9392\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.2863 - acc: 0.9174 - val_loss: 0.2485 - val_acc: 0.9236\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.2187 - acc: 0.9390 - val_loss: 0.2678 - val_acc: 0.9028\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 0.2078 - acc: 0.9402 - val_loss: 0.1958 - val_acc: 0.9375\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 37s 227ms/step - loss: 0.1791 - acc: 0.9491 - val_loss: 0.1432 - val_acc: 0.9583\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 37s 229ms/step - loss: 0.1723 - acc: 0.9483 - val_loss: 0.1316 - val_acc: 0.9722\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 37s 229ms/step - loss: 0.1361 - acc: 0.9568 - val_loss: 0.1644 - val_acc: 0.9497\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1203 - acc: 0.9655 - val_loss: 0.1560 - val_acc: 0.9583\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.0955 - acc: 0.9720 - val_loss: 0.0957 - val_acc: 0.9826\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.1075 - acc: 0.9697 - val_loss: 0.1710 - val_acc: 0.9444\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1001 - acc: 0.9716 - val_loss: 0.1609 - val_acc: 0.9375\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1052 - acc: 0.9680 - val_loss: 0.1454 - val_acc: 0.9462\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.0702 - acc: 0.9809 - val_loss: 0.0661 - val_acc: 0.9757\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.0535 - acc: 0.9873 - val_loss: 0.0394 - val_acc: 0.9861\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1000 - acc: 0.9709 - val_loss: 0.0785 - val_acc: 0.9722\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.0476 - acc: 0.9888 - val_loss: 0.1322 - val_acc: 0.9479\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.10345815666548816, 0.9616126543209876]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.10051100328564644, 0.9659722222222222]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 84s 520ms/step - loss: 2.5612 - acc: 0.3189 - val_loss: 1.5426 - val_acc: 0.5347\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 36s 219ms/step - loss: 1.0940 - acc: 0.6877 - val_loss: 0.9851 - val_acc: 0.6823\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.6676 - acc: 0.8223 - val_loss: 0.6968 - val_acc: 0.8003\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.4506 - acc: 0.8769 - val_loss: 0.3831 - val_acc: 0.8924\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.3564 - acc: 0.8993 - val_loss: 0.3084 - val_acc: 0.9115\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.2800 - acc: 0.9221 - val_loss: 0.3392 - val_acc: 0.9062\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.2499 - acc: 0.9275 - val_loss: 0.1542 - val_acc: 0.9514\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.2089 - acc: 0.9402 - val_loss: 0.1083 - val_acc: 0.9774\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1715 - acc: 0.9510 - val_loss: 0.1161 - val_acc: 0.9688\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.1546 - acc: 0.9560 - val_loss: 0.1366 - val_acc: 0.9514\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1298 - acc: 0.9635 - val_loss: 0.2031 - val_acc: 0.9410\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1294 - acc: 0.9626 - val_loss: 0.0836 - val_acc: 0.9722\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 36s 220ms/step - loss: 0.1032 - acc: 0.9716 - val_loss: 0.1218 - val_acc: 0.9601\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1011 - acc: 0.9728 - val_loss: 0.1070 - val_acc: 0.9757\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.0947 - acc: 0.9743 - val_loss: 0.0595 - val_acc: 0.9861\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.0927 - acc: 0.9782 - val_loss: 0.3023 - val_acc: 0.8802\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.1186 - acc: 0.9622 - val_loss: 0.0778 - val_acc: 0.9826\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.0780 - acc: 0.9788 - val_loss: 0.0469 - val_acc: 0.9913\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 35s 216ms/step - loss: 0.0716 - acc: 0.9815 - val_loss: 0.0611 - val_acc: 0.9844\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 215ms/step - loss: 0.0646 - acc: 0.9801 - val_loss: 0.0580 - val_acc: 0.9705\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.046837644168624175, 0.984375]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.058596673069728746, 0.9777777777777777]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 81s 503ms/step - loss: 4.1804 - acc: 0.0511 - val_loss: 3.3786 - val_acc: 0.1024\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 2.5768 - acc: 0.2652 - val_loss: 2.2228 - val_acc: 0.3281\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 1.7807 - acc: 0.4620 - val_loss: 1.7468 - val_acc: 0.4549\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 1.3768 - acc: 0.5766 - val_loss: 1.5608 - val_acc: 0.4861\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 1.1299 - acc: 0.6443 - val_loss: 1.4316 - val_acc: 0.5729\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.9799 - acc: 0.6971 - val_loss: 1.2530 - val_acc: 0.6319\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 0.8260 - acc: 0.7456 - val_loss: 0.9933 - val_acc: 0.6962\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.7326 - acc: 0.7747 - val_loss: 1.0419 - val_acc: 0.6562\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.6862 - acc: 0.7903 - val_loss: 0.9205 - val_acc: 0.7465\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.6064 - acc: 0.8115 - val_loss: 0.9880 - val_acc: 0.6979\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.5609 - acc: 0.8268 - val_loss: 0.8519 - val_acc: 0.7535\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.5207 - acc: 0.8401 - val_loss: 0.8182 - val_acc: 0.7483\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.4556 - acc: 0.8594 - val_loss: 0.6317 - val_acc: 0.7969\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.4426 - acc: 0.8574 - val_loss: 0.5852 - val_acc: 0.7882\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.4243 - acc: 0.8663 - val_loss: 0.5271 - val_acc: 0.8160\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.3909 - acc: 0.8819 - val_loss: 0.4080 - val_acc: 0.8542\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.3495 - acc: 0.8879 - val_loss: 0.6510 - val_acc: 0.7830\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.3247 - acc: 0.9007 - val_loss: 0.3787 - val_acc: 0.8559\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 32s 199ms/step - loss: 0.3301 - acc: 0.8947 - val_loss: 0.5251 - val_acc: 0.8281\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.3028 - acc: 0.9010 - val_loss: 0.3765 - val_acc: 0.8663\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.403058883032681, 0.8547453703703703]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.39038456976413727, 0.8548611111111111]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 81s 499ms/step - loss: 3.0774 - acc: 0.2141 - val_loss: 1.9859 - val_acc: 0.4201\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 1.3904 - acc: 0.5853 - val_loss: 1.2517 - val_acc: 0.6059\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.9214 - acc: 0.7143 - val_loss: 0.9535 - val_acc: 0.7240\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.7094 - acc: 0.7787 - val_loss: 0.7674 - val_acc: 0.7691\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.5720 - acc: 0.8254 - val_loss: 0.6251 - val_acc: 0.7865\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.4825 - acc: 0.8536 - val_loss: 0.5635 - val_acc: 0.8142\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.4368 - acc: 0.8588 - val_loss: 0.4977 - val_acc: 0.8316\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.3882 - acc: 0.8825 - val_loss: 0.5096 - val_acc: 0.8142\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.3519 - acc: 0.8835 - val_loss: 0.4087 - val_acc: 0.8750\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.3120 - acc: 0.8980 - val_loss: 0.5069 - val_acc: 0.8490\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.3139 - acc: 0.9008 - val_loss: 0.4170 - val_acc: 0.8628\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2682 - acc: 0.9153 - val_loss: 0.3498 - val_acc: 0.8715\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2618 - acc: 0.9144 - val_loss: 0.4042 - val_acc: 0.8802\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2785 - acc: 0.9163 - val_loss: 0.3790 - val_acc: 0.8854\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.2207 - acc: 0.9309 - val_loss: 0.3229 - val_acc: 0.8906\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 32s 199ms/step - loss: 0.2351 - acc: 0.9296 - val_loss: 0.3444 - val_acc: 0.8941\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.2083 - acc: 0.9323 - val_loss: 0.2575 - val_acc: 0.9080\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2244 - acc: 0.9329 - val_loss: 0.2606 - val_acc: 0.9219\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.1847 - acc: 0.9452 - val_loss: 0.3411 - val_acc: 0.9010\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2264 - acc: 0.9248 - val_loss: 0.3510 - val_acc: 0.8889\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.2826614100601018, 0.9066358024691358]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.32933219886488385, 0.8951388888888889]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 86s 533ms/step - loss: 3.1456 - acc: 0.1931 - val_loss: 1.9432 - val_acc: 0.4323\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 1.4046 - acc: 0.5739 - val_loss: 1.1445 - val_acc: 0.6510\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 32s 199ms/step - loss: 0.9070 - acc: 0.7228 - val_loss: 0.8449 - val_acc: 0.7257\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.6772 - acc: 0.7930 - val_loss: 0.6996 - val_acc: 0.7830\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.5715 - acc: 0.8229 - val_loss: 0.6246 - val_acc: 0.8264\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.5046 - acc: 0.8482 - val_loss: 0.4654 - val_acc: 0.8559\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.4423 - acc: 0.8609 - val_loss: 0.5080 - val_acc: 0.8438\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.3811 - acc: 0.8808 - val_loss: 0.3811 - val_acc: 0.8663\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.3685 - acc: 0.8829 - val_loss: 0.4877 - val_acc: 0.8403\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.3338 - acc: 0.8939 - val_loss: 0.3776 - val_acc: 0.8941\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2906 - acc: 0.9097 - val_loss: 0.3646 - val_acc: 0.8819\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2744 - acc: 0.9157 - val_loss: 0.3937 - val_acc: 0.8750\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 0.2550 - acc: 0.9209 - val_loss: 0.3044 - val_acc: 0.8993\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 32s 195ms/step - loss: 0.2417 - acc: 0.9255 - val_loss: 0.3134 - val_acc: 0.9097\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 32s 197ms/step - loss: 0.2372 - acc: 0.9230 - val_loss: 0.3217 - val_acc: 0.9115\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2275 - acc: 0.9327 - val_loss: 0.2336 - val_acc: 0.9184\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.1989 - acc: 0.9360 - val_loss: 0.3597 - val_acc: 0.8906\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.2438 - acc: 0.9230 - val_loss: 0.3404 - val_acc: 0.8785\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 32s 199ms/step - loss: 0.1877 - acc: 0.9377 - val_loss: 0.2494 - val_acc: 0.9323\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 32s 196ms/step - loss: 0.1854 - acc: 0.9402 - val_loss: 0.1869 - val_acc: 0.9427\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.191207101246641, 0.9405864197530864]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.18977223717504077, 0.9368055555555556]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 119s 731ms/step - loss: 2.1496 - acc: 0.5218 - val_loss: 1.1733 - val_acc: 0.6701\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.8261 - acc: 0.8432 - val_loss: 0.5050 - val_acc: 0.9028\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.5013 - acc: 0.9138 - val_loss: 0.3502 - val_acc: 0.9410\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.3895 - acc: 0.9273 - val_loss: 0.2917 - val_acc: 0.9097\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.3043 - acc: 0.9439 - val_loss: 0.1814 - val_acc: 0.9653\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.2527 - acc: 0.9589 - val_loss: 0.1534 - val_acc: 0.9670\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.2098 - acc: 0.9614 - val_loss: 0.1402 - val_acc: 0.9670\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.1833 - acc: 0.9688 - val_loss: 0.1382 - val_acc: 0.9670\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 66s 410ms/step - loss: 0.1620 - acc: 0.9705 - val_loss: 0.0910 - val_acc: 0.9826\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 68s 418ms/step - loss: 0.1346 - acc: 0.9782 - val_loss: 0.0964 - val_acc: 0.9688\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 67s 416ms/step - loss: 0.1318 - acc: 0.9772 - val_loss: 0.0934 - val_acc: 0.9670\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 68s 420ms/step - loss: 0.1256 - acc: 0.9745 - val_loss: 0.0652 - val_acc: 0.9861\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 68s 418ms/step - loss: 0.1036 - acc: 0.9850 - val_loss: 0.0631 - val_acc: 0.9896\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 68s 417ms/step - loss: 0.0941 - acc: 0.9851 - val_loss: 0.0761 - val_acc: 0.9809\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 68s 417ms/step - loss: 0.0875 - acc: 0.9838 - val_loss: 0.0413 - val_acc: 0.9931\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 68s 417ms/step - loss: 0.0842 - acc: 0.9850 - val_loss: 0.1009 - val_acc: 0.9688\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.0773 - acc: 0.9855 - val_loss: 0.0720 - val_acc: 0.9757\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.0663 - acc: 0.9896 - val_loss: 0.0752 - val_acc: 0.9774\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.0684 - acc: 0.9877 - val_loss: 0.0541 - val_acc: 0.9809\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 397ms/step - loss: 0.0658 - acc: 0.9873 - val_loss: 0.0396 - val_acc: 0.9896\n",
      "5184/5184 [==============================] - 17s 3ms/step\n",
      "Train [0.035553011084607816, 0.9909336419753086]\n",
      "1440/1440 [==============================] - 5s 3ms/step\n",
      "Test [0.036458057910203935, 0.9909722222222223]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 119s 732ms/step - loss: 2.1327 - acc: 0.5133 - val_loss: 1.0234 - val_acc: 0.7431\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.8251 - acc: 0.8356 - val_loss: 0.6222 - val_acc: 0.8628\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.5385 - acc: 0.9007 - val_loss: 0.3706 - val_acc: 0.9184\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.3950 - acc: 0.9333 - val_loss: 0.2542 - val_acc: 0.9392\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.3122 - acc: 0.9450 - val_loss: 0.1959 - val_acc: 0.9601\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.2496 - acc: 0.9583 - val_loss: 0.1619 - val_acc: 0.9774\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.2264 - acc: 0.9626 - val_loss: 0.1451 - val_acc: 0.9809\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1865 - acc: 0.9660 - val_loss: 0.1113 - val_acc: 0.9861\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.1646 - acc: 0.9709 - val_loss: 0.1093 - val_acc: 0.9844\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 65s 398ms/step - loss: 0.1495 - acc: 0.9734 - val_loss: 0.0733 - val_acc: 0.9878\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.1337 - acc: 0.9765 - val_loss: 0.0885 - val_acc: 0.9809\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1244 - acc: 0.9749 - val_loss: 0.0744 - val_acc: 0.9844\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.1134 - acc: 0.9769 - val_loss: 0.0842 - val_acc: 0.9809\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0950 - acc: 0.9850 - val_loss: 0.0501 - val_acc: 0.9931\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.1021 - acc: 0.9801 - val_loss: 0.0586 - val_acc: 0.9861\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0859 - acc: 0.9859 - val_loss: 0.0843 - val_acc: 0.9792\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0787 - acc: 0.9857 - val_loss: 0.0660 - val_acc: 0.9861\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0819 - acc: 0.9844 - val_loss: 0.0326 - val_acc: 0.9983\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0685 - acc: 0.9877 - val_loss: 0.0311 - val_acc: 0.9983\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 394ms/step - loss: 0.0693 - acc: 0.9844 - val_loss: 0.0340 - val_acc: 0.9931\n",
      "5184/5184 [==============================] - 17s 3ms/step\n",
      "Train [0.032063601529915574, 0.9936342592592593]\n",
      "1440/1440 [==============================] - 5s 3ms/step\n",
      "Test [0.03170852788930966, 0.9923611111111111]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 118s 729ms/step - loss: 2.1281 - acc: 0.5100 - val_loss: 1.2247 - val_acc: 0.6927\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.8342 - acc: 0.8314 - val_loss: 0.6417 - val_acc: 0.8264\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.5161 - acc: 0.9095 - val_loss: 0.3339 - val_acc: 0.9358\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 65s 399ms/step - loss: 0.3873 - acc: 0.9307 - val_loss: 0.2877 - val_acc: 0.9410\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.2943 - acc: 0.9562 - val_loss: 0.1939 - val_acc: 0.9618\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.2433 - acc: 0.9566 - val_loss: 0.1355 - val_acc: 0.9861\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.2190 - acc: 0.9593 - val_loss: 0.1910 - val_acc: 0.9410\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1827 - acc: 0.9699 - val_loss: 0.1145 - val_acc: 0.9809\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1600 - acc: 0.9751 - val_loss: 0.1072 - val_acc: 0.9670\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.1465 - acc: 0.9734 - val_loss: 0.0757 - val_acc: 0.9844\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 64s 398ms/step - loss: 0.1211 - acc: 0.9801 - val_loss: 0.1144 - val_acc: 0.9670\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.1096 - acc: 0.9824 - val_loss: 0.0765 - val_acc: 0.9896\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1049 - acc: 0.9799 - val_loss: 0.0540 - val_acc: 0.9896\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.1003 - acc: 0.9811 - val_loss: 0.0812 - val_acc: 0.9774\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0787 - acc: 0.9884 - val_loss: 0.0671 - val_acc: 0.9792\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0843 - acc: 0.9830 - val_loss: 0.0804 - val_acc: 0.9809\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 65s 398ms/step - loss: 0.0763 - acc: 0.9867 - val_loss: 0.0470 - val_acc: 0.9878\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 65s 401ms/step - loss: 0.0620 - acc: 0.9882 - val_loss: 0.0404 - val_acc: 0.9878\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 64s 395ms/step - loss: 0.0642 - acc: 0.9882 - val_loss: 0.0482 - val_acc: 0.9913\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 64s 396ms/step - loss: 0.0602 - acc: 0.9898 - val_loss: 0.0384 - val_acc: 0.9913\n",
      "5184/5184 [==============================] - 16s 3ms/step\n",
      "Train [0.0298783322709992, 0.9936342592592593]\n",
      "1440/1440 [==============================] - 5s 3ms/step\n",
      "Test [0.03237972632050514, 0.9930555555555556]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 89s 551ms/step - loss: 3.1459 - acc: 0.2473 - val_loss: 2.3949 - val_acc: 0.4410\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 1.9053 - acc: 0.6316 - val_loss: 1.6485 - val_acc: 0.6476\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 1.3350 - acc: 0.7992 - val_loss: 1.0517 - val_acc: 0.8385\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 217ms/step - loss: 0.9841 - acc: 0.8843 - val_loss: 0.8172 - val_acc: 0.8889\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.7627 - acc: 0.9169 - val_loss: 0.6515 - val_acc: 0.8924\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.6016 - acc: 0.9375 - val_loss: 0.4814 - val_acc: 0.9323\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.4968 - acc: 0.9475 - val_loss: 0.3779 - val_acc: 0.9375\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 36s 221ms/step - loss: 0.4122 - acc: 0.9614 - val_loss: 0.3169 - val_acc: 0.9635\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.3563 - acc: 0.9614 - val_loss: 0.2637 - val_acc: 0.9670\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.3001 - acc: 0.9742 - val_loss: 0.2058 - val_acc: 0.9740\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.2666 - acc: 0.9715 - val_loss: 0.1838 - val_acc: 0.9757\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.2262 - acc: 0.9797 - val_loss: 0.1994 - val_acc: 0.9549\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.2043 - acc: 0.9782 - val_loss: 0.1475 - val_acc: 0.9844\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1784 - acc: 0.9811 - val_loss: 0.1108 - val_acc: 0.9896\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1593 - acc: 0.9867 - val_loss: 0.1020 - val_acc: 0.9965\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1427 - acc: 0.9882 - val_loss: 0.0818 - val_acc: 0.9931\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1312 - acc: 0.9863 - val_loss: 0.0909 - val_acc: 0.9931\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.1147 - acc: 0.9896 - val_loss: 0.0718 - val_acc: 0.9983\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 36s 221ms/step - loss: 0.1036 - acc: 0.9904 - val_loss: 0.0912 - val_acc: 0.9792\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.0956 - acc: 0.9917 - val_loss: 0.0545 - val_acc: 1.0000\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.05062659663136727, 0.9938271604938271]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.05259761433634493, 0.9972222222222222]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 90s 554ms/step - loss: 3.0993 - acc: 0.2685 - val_loss: 2.3107 - val_acc: 0.4618\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 1.8435 - acc: 0.6570 - val_loss: 1.4116 - val_acc: 0.7951\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 1.2619 - acc: 0.8306 - val_loss: 0.9790 - val_acc: 0.8281\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.9267 - acc: 0.8800 - val_loss: 0.6952 - val_acc: 0.9236\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 36s 221ms/step - loss: 0.7091 - acc: 0.9286 - val_loss: 0.5550 - val_acc: 0.9028\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 36s 219ms/step - loss: 0.5587 - acc: 0.9441 - val_loss: 0.4328 - val_acc: 0.9288\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.4589 - acc: 0.9564 - val_loss: 0.3411 - val_acc: 0.9774\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.3781 - acc: 0.9664 - val_loss: 0.2768 - val_acc: 0.9635\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.3204 - acc: 0.9697 - val_loss: 0.2468 - val_acc: 0.9566\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.2766 - acc: 0.9736 - val_loss: 0.1964 - val_acc: 0.9774\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.2367 - acc: 0.9815 - val_loss: 0.2011 - val_acc: 0.9774\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 36s 220ms/step - loss: 0.2095 - acc: 0.9796 - val_loss: 0.1389 - val_acc: 0.9826\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.1811 - acc: 0.9848 - val_loss: 0.1456 - val_acc: 0.9913\n",
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1584 - acc: 0.9861 - val_loss: 0.1313 - val_acc: 0.9774\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.1494 - acc: 0.9832 - val_loss: 0.1265 - val_acc: 0.9792\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1287 - acc: 0.9902 - val_loss: 0.0813 - val_acc: 0.9983\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1190 - acc: 0.9871 - val_loss: 0.0936 - val_acc: 0.9774\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1060 - acc: 0.9900 - val_loss: 0.0833 - val_acc: 0.9792\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.0930 - acc: 0.9907 - val_loss: 0.0643 - val_acc: 1.0000\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.0860 - acc: 0.9925 - val_loss: 0.0502 - val_acc: 0.9948\n",
      "5184/5184 [==============================] - 11s 2ms/step\n",
      "Train [0.04680843500673403, 0.9934413580246914]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.05217651877966192, 0.9909722222222223]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "162/162 [==============================] - 90s 556ms/step - loss: 3.1521 - acc: 0.2537 - val_loss: 2.3535 - val_acc: 0.4601\n",
      "Epoch 2/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 1.8748 - acc: 0.6454 - val_loss: 1.5431 - val_acc: 0.6858\n",
      "Epoch 3/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 1.2746 - acc: 0.8069 - val_loss: 1.0101 - val_acc: 0.8628\n",
      "Epoch 4/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.9303 - acc: 0.8877 - val_loss: 0.7729 - val_acc: 0.9080\n",
      "Epoch 5/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.7109 - acc: 0.9244 - val_loss: 0.5363 - val_acc: 0.9427\n",
      "Epoch 6/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.5667 - acc: 0.9441 - val_loss: 0.4382 - val_acc: 0.9253\n",
      "Epoch 7/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.4647 - acc: 0.9554 - val_loss: 0.3820 - val_acc: 0.9323\n",
      "Epoch 8/20\n",
      "162/162 [==============================] - 36s 219ms/step - loss: 0.3845 - acc: 0.9649 - val_loss: 0.2626 - val_acc: 0.9792\n",
      "Epoch 9/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.3289 - acc: 0.9682 - val_loss: 0.2367 - val_acc: 0.9722\n",
      "Epoch 10/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.2840 - acc: 0.9792 - val_loss: 0.1856 - val_acc: 0.9792\n",
      "Epoch 11/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.2393 - acc: 0.9790 - val_loss: 0.1660 - val_acc: 0.9861\n",
      "Epoch 12/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.2130 - acc: 0.9824 - val_loss: 0.1653 - val_acc: 0.9809\n",
      "Epoch 13/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.1878 - acc: 0.9844 - val_loss: 0.1529 - val_acc: 0.9688\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 14/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1683 - acc: 0.9828 - val_loss: 0.1047 - val_acc: 0.9948\n",
      "Epoch 15/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1462 - acc: 0.9884 - val_loss: 0.1257 - val_acc: 0.9861\n",
      "Epoch 16/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.1345 - acc: 0.9878 - val_loss: 0.0987 - val_acc: 0.9913\n",
      "Epoch 17/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.1218 - acc: 0.9886 - val_loss: 0.0764 - val_acc: 0.9983\n",
      "Epoch 18/20\n",
      "162/162 [==============================] - 36s 220ms/step - loss: 0.1022 - acc: 0.9915 - val_loss: 0.0927 - val_acc: 0.9861\n",
      "Epoch 19/20\n",
      "162/162 [==============================] - 35s 218ms/step - loss: 0.0942 - acc: 0.9931 - val_loss: 0.0731 - val_acc: 0.9913\n",
      "Epoch 20/20\n",
      "162/162 [==============================] - 35s 219ms/step - loss: 0.0925 - acc: 0.9905 - val_loss: 0.0816 - val_acc: 0.9896\n",
      "5184/5184 [==============================] - 12s 2ms/step\n",
      "Train [0.07119307790043546, 0.9884259259259259]\n",
      "1440/1440 [==============================] - 3s 2ms/step\n",
      "Test [0.07364115226599906, 0.9888888888888889]\n"
     ]
    }
   ],
   "source": [
    "# COIL-100\n",
    "import keras.optimizers\n",
    "for lr in [0.1, 0.01, 0.001, 0.0001]:\n",
    "    optimizer = optimizers.Adam(lr=lr)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITHOUT BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetC((128, 128, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH LAYER BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetA((128, 128, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH IDENTITY BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetB((128, 128, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset for CIFAR-10\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import cifar10\n",
    "from keras.utils import to_categorical\n",
    "import numpy as np\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "mean = np.mean(x_train)\n",
    "std  = np.std(x_train)\n",
    "\n",
    "x_train = ((x_train - mean) / std).astype(np.float32)\n",
    "x_test  = ((x_test  - mean) / std).astype(np.float32)\n",
    "\n",
    "y_train = to_categorical(y_train)\n",
    "y_test  = to_categorical(y_test)\n",
    "\n",
    "pivot = int(len(x_train) * 0.9)\n",
    "x_val = x_train[pivot:]\n",
    "y_val = y_train[pivot:]\n",
    "x_train = x_train[:pivot]\n",
    "y_train = y_train[:pivot]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarks for CIFAR-10 for Models 1, 2 and 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4962 - acc: 0.1000 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 26s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "45000/45000 [==============================] - 6s 127us/step\n",
      "Train [14.501987509324815, 0.10026666666666667]\n",
      "10000/10000 [==============================] - 1s 124us/step\n",
      "Test [14.50628564453125, 0.1]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4906 - acc: 0.1004 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.4950 - acc: 0.1007 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4960 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 25s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.4960 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4960 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4971 - acc: 0.1006 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.4982 - acc: 0.1005 - val_loss: 14.5869 - val_acc: 0.0950\n",
      "45000/45000 [==============================] - 6s 127us/step\n",
      "Train [14.49733119591607, 0.10055555555555555]\n",
      "10000/10000 [==============================] - 1s 126us/step\n",
      "Test [14.50628570251465, 0.1]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.4949 - acc: 0.1001 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 16/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 23s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 23s 16ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "45000/45000 [==============================] - 6s 128us/step\n",
      "Train [14.501987517123752, 0.10026666666666667]\n",
      "10000/10000 [==============================] - 1s 128us/step\n",
      "Test [14.506285684204101, 0.1]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 78s 56ms/step - loss: 14.5061 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 73s 52ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 71s 50ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 77s 55ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 69s 49ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 74s 53ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 82s 59ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 78s 56ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 84s 60ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 82s 58ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 79s 56ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 78s 55ms/step - loss: 14.5107 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 77s 54ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 72s 51ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 73s 52ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 81s 57ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 81s 58ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 75s 53ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "45000/45000 [==============================] - 9s 207us/step\n",
      "Train [14.513091099039714, 0.09957777777777778]\n",
      "10000/10000 [==============================] - 2s 206us/step\n",
      "Test [14.50628567199707, 0.1]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.4766 - acc: 0.1015 - val_loss: 14.2548 - val_acc: 0.1156\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 14.4799 - acc: 0.1016 - val_loss: 14.2484 - val_acc: 0.1160\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 75s 53ms/step - loss: 14.4779 - acc: 0.1017 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 74s 53ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 72s 51ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 73s 52ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 71s 51ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 73s 52ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 72s 51ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 72s 51ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 71s 50ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 71s 51ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 69s 49ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 68s 48ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 77s 55ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 70s 50ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 74s 52ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 71s 51ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 73s 52ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "45000/45000 [==============================] - 9s 207us/step\n",
      "Train [14.513091099039714, 0.09957777777777778]\n",
      "10000/10000 [==============================] - 2s 206us/step\n",
      "Test [14.50628567199707, 0.1]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 71s 51ms/step - loss: 14.4931 - acc: 0.1004 - val_loss: 15.2123 - val_acc: 0.0562\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 74s 53ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2123 - val_acc: 0.0562\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2090 - val_acc: 0.0564\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 82s 58ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2058 - val_acc: 0.0566\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 14.4975 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 78s 56ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2026 - val_acc: 0.0568\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 70s 50ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2123 - val_acc: 0.0562\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2123 - val_acc: 0.0562\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 73s 52ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 78s 56ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2252 - val_acc: 0.0554\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 81s 57ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2090 - val_acc: 0.0564\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 81s 58ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2155 - val_acc: 0.0560\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 78s 55ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 15.2058 - val_acc: 0.0566\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 78s 56ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2090 - val_acc: 0.0564\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 81s 58ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2026 - val_acc: 0.0568\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2090 - val_acc: 0.0564\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 78s 56ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2123 - val_acc: 0.0562\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.4986 - acc: 0.1005 - val_loss: 15.2090 - val_acc: 0.0564\n",
      "45000/45000 [==============================] - 9s 207us/step\n",
      "Train [15.14778576880561, 0.0602]\n",
      "10000/10000 [==============================] - 2s 203us/step\n",
      "Test [15.141338537597656, 0.0606]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 14.4122 - acc: 0.1051 - val_loss: 14.4483 - val_acc: 0.1036\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 14.5122 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 27s 20ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 27s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 27s 20ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5129 - acc: 0.0996 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 14.5118 - acc: 0.0997 - val_loss: 14.4450 - val_acc: 0.1038\n",
      "45000/45000 [==============================] - 7s 159us/step\n",
      "Train [14.513091099039714, 0.09957777777777778]\n",
      "10000/10000 [==============================] - 2s 160us/step\n",
      "Test [14.50628567199707, 0.1]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 30s 22ms/step - loss: 14.4897 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.4996 - acc: 0.1004 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "45000/45000 [==============================] - 8s 172us/step\n",
      "Train [14.501987517123752, 0.10026666666666667]\n",
      "10000/10000 [==============================] - 2s 169us/step\n",
      "Test [14.506285684204101, 0.1]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.1 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 31s 22ms/step - loss: 14.4599 - acc: 0.1020 - val_loss: 14.5740 - val_acc: 0.0958\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 14.3229 - acc: 0.1114 - val_loss: 13.9293 - val_acc: 0.1358\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 14.5084 - acc: 0.0999 - val_loss: 14.4031 - val_acc: 0.1064\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4031 - val_acc: 0.1064\n",
      "45000/45000 [==============================] - 8s 171us/step\n",
      "Train [14.51774743923611, 0.09928888888888888]\n",
      "10000/10000 [==============================] - 2s 173us/step\n",
      "Test [14.506285690307617, 0.1]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 14.5042 - acc: 0.0996 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5093 - acc: 0.0998 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5093 - acc: 0.0998 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5104 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5093 - acc: 0.0998 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5082 - acc: 0.0999 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5093 - acc: 0.0998 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5114 - acc: 0.0997 - val_loss: 14.4676 - val_acc: 0.1024\n",
      "45000/45000 [==============================] - 9s 191us/step\n",
      "Train [14.510583856370713, 0.09973333333333333]\n",
      "10000/10000 [==============================] - 2s 186us/step\n",
      "Test [14.506285696411133, 0.1]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 14.5084 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 26s 18ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 25s 17ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 25s 17ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 25s 17ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 19/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 25s 17ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "45000/45000 [==============================] - 9s 206us/step\n",
      "Train [14.51667288224962, 0.09935555555555556]\n",
      "10000/10000 [==============================] - 2s 202us/step\n",
      "Test [14.506285690307617, 0.1]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 14.4949 - acc: 0.1001 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 24s 17ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 25s 17ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5007 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5018 - acc: 0.1003 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 25s 18ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5450 - val_acc: 0.0976\n",
      "45000/45000 [==============================] - 9s 201us/step\n",
      "Train [14.501987509324815, 0.10026666666666667]\n",
      "10000/10000 [==============================] - 2s 213us/step\n",
      "Test [14.50628564453125, 0.1]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 91s 65ms/step - loss: 14.2195 - acc: 0.1170 - val_loss: 14.5127 - val_acc: 0.0996\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 86s 61ms/step - loss: 14.5032 - acc: 0.1002 - val_loss: 14.5063 - val_acc: 0.1000\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 14.5021 - acc: 0.1003 - val_loss: 14.5063 - val_acc: 0.1000\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 85s 61ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5095 - val_acc: 0.0998\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 75s 53ms/step - loss: 14.5036 - acc: 0.1002 - val_loss: 14.5063 - val_acc: 0.1000\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 77s 54ms/step - loss: 14.5046 - acc: 0.1001 - val_loss: 14.5067 - val_acc: 0.0998\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.5021 - acc: 0.1003 - val_loss: 14.5091 - val_acc: 0.0998\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 77s 55ms/step - loss: 14.5046 - acc: 0.1001 - val_loss: 14.5074 - val_acc: 0.0998\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 78s 55ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5072 - val_acc: 0.0998\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 75s 54ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5095 - val_acc: 0.0998\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 82s 58ms/step - loss: 14.5036 - acc: 0.1002 - val_loss: 14.5084 - val_acc: 0.0998\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 81s 58ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5095 - val_acc: 0.0998\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 79s 56ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5079 - val_acc: 0.0998\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 69s 49ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5063 - val_acc: 0.1000\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 78s 55ms/step - loss: 14.5043 - acc: 0.1001 - val_loss: 14.5095 - val_acc: 0.0998\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 14.4402 - acc: 0.1041 - val_loss: 14.3032 - val_acc: 0.1126\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 83s 59ms/step - loss: 13.8874 - acc: 0.1384 - val_loss: 13.7871 - val_acc: 0.1446\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 74s 53ms/step - loss: 13.8839 - acc: 0.1386 - val_loss: 13.7773 - val_acc: 0.1450\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 79s 56ms/step - loss: 13.9020 - acc: 0.1375 - val_loss: 13.8730 - val_acc: 0.1392\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 13.9336 - acc: 0.1355 - val_loss: 13.9937 - val_acc: 0.1318\n",
      "45000/45000 [==============================] - 10s 230us/step\n",
      "Train [13.96292692481147, 0.13371111111111111]\n",
      "10000/10000 [==============================] - 2s 229us/step\n",
      "Test [14.014683865356446, 0.1305]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 82s 58ms/step - loss: 14.4696 - acc: 0.1015 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 77s 55ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 78s 55ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 77s 55ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 78s 56ms/step - loss: 14.5172 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 79s 56ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 89s 63ms/step - loss: 14.5150 - acc: 0.0995 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 14.5157 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 90s 64ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 84s 60ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 82s 58ms/step - loss: 14.5161 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 89s 63ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 90s 64ms/step - loss: 14.5161 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 90s 64ms/step - loss: 14.5168 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 90s 64ms/step - loss: 14.5172 - acc: 0.0993 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 86s 61ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 86s 61ms/step - loss: 14.5157 - acc: 0.0994 - val_loss: 14.4128 - val_acc: 0.1058\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 85s 60ms/step - loss: 14.4696 - acc: 0.1023 - val_loss: 14.3902 - val_acc: 0.1072\n",
      "45000/45000 [==============================] - 11s 240us/step\n",
      "Train [14.503194496832954, 0.10017777777777778]\n",
      "10000/10000 [==============================] - 2s 237us/step\n",
      "Test [14.480496734619141, 0.1016]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 14.5064 - acc: 0.0994 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.5193 - acc: 0.0992 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 82s 58ms/step - loss: 14.5165 - acc: 0.0994 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.5190 - acc: 0.0992 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 78s 55ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 14.5182 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 78s 56ms/step - loss: 14.5182 - acc: 0.0993 - val_loss: 14.4403 - val_acc: 0.1040\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 76s 54ms/step - loss: 14.5154 - acc: 0.0994 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 79s 56ms/step - loss: 14.5182 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 77s 54ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 77s 55ms/step - loss: 14.5179 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 86s 61ms/step - loss: 14.5186 - acc: 0.0992 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 89s 63ms/step - loss: 14.5179 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 84s 60ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 90s 64ms/step - loss: 14.5179 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 91s 64ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 88s 63ms/step - loss: 14.5175 - acc: 0.0993 - val_loss: 14.4386 - val_acc: 0.1042\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 89s 63ms/step - loss: 14.5139 - acc: 0.0995 - val_loss: 14.5514 - val_acc: 0.0972\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 88s 62ms/step - loss: 14.5028 - acc: 0.1002 - val_loss: 14.5514 - val_acc: 0.0972\n",
      "45000/45000 [==============================] - 11s 234us/step\n",
      "Train [14.502345688713921, 0.10024444444444444]\n",
      "10000/10000 [==============================] - 2s 235us/step\n",
      "Test [14.50628564453125, 0.1]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 2.2301 - acc: 0.2064 - val_loss: 2.0798 - val_acc: 0.1956\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.8279 - acc: 0.3182 - val_loss: 1.7696 - val_acc: 0.3694\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.6796 - acc: 0.3771 - val_loss: 1.5213 - val_acc: 0.4352\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.5828 - acc: 0.4162 - val_loss: 1.5288 - val_acc: 0.4368\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.5240 - acc: 0.4393 - val_loss: 1.3997 - val_acc: 0.4820\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.4726 - acc: 0.4634 - val_loss: 1.4693 - val_acc: 0.4718\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.4287 - acc: 0.4787 - val_loss: 1.2915 - val_acc: 0.5258\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3925 - acc: 0.4914 - val_loss: 1.2745 - val_acc: 0.5366\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3725 - acc: 0.5027 - val_loss: 1.2118 - val_acc: 0.5588\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3462 - acc: 0.5118 - val_loss: 1.2723 - val_acc: 0.5400\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3236 - acc: 0.5229 - val_loss: 1.2979 - val_acc: 0.5524\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3015 - acc: 0.5285 - val_loss: 1.3349 - val_acc: 0.5444\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 30s 22ms/step - loss: 1.2823 - acc: 0.5358 - val_loss: 1.1811 - val_acc: 0.5786\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2594 - acc: 0.5445 - val_loss: 1.2492 - val_acc: 0.5598\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2446 - acc: 0.5536 - val_loss: 1.1308 - val_acc: 0.5942\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2267 - acc: 0.5592 - val_loss: 1.1739 - val_acc: 0.5828\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2190 - acc: 0.5636 - val_loss: 1.1209 - val_acc: 0.5988\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2061 - acc: 0.5661 - val_loss: 1.1478 - val_acc: 0.5886\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.1882 - acc: 0.5744 - val_loss: 1.1493 - val_acc: 0.5972\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.1809 - acc: 0.5769 - val_loss: 1.0958 - val_acc: 0.6104\n",
      "45000/45000 [==============================] - 11s 246us/step\n",
      "Train [1.0932263996548124, 0.6123777777777778]\n",
      "10000/10000 [==============================] - 2s 238us/step\n",
      "Test [1.1388080879211426, 0.5942]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 2.1298 - acc: 0.2298 - val_loss: 1.7756 - val_acc: 0.3296\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.7502 - acc: 0.3461 - val_loss: 1.5676 - val_acc: 0.4126\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.6258 - acc: 0.3973 - val_loss: 1.5096 - val_acc: 0.4410\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.5425 - acc: 0.4319 - val_loss: 1.5737 - val_acc: 0.4572\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.4929 - acc: 0.4548 - val_loss: 1.3518 - val_acc: 0.5054\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.4391 - acc: 0.4766 - val_loss: 1.2744 - val_acc: 0.5416\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.4046 - acc: 0.4928 - val_loss: 1.2553 - val_acc: 0.5530\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3737 - acc: 0.5025 - val_loss: 1.2361 - val_acc: 0.5564\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3444 - acc: 0.5165 - val_loss: 1.2113 - val_acc: 0.5618\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3187 - acc: 0.5231 - val_loss: 1.2530 - val_acc: 0.5514\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3025 - acc: 0.5309 - val_loss: 1.2660 - val_acc: 0.5590\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2702 - acc: 0.5445 - val_loss: 1.1959 - val_acc: 0.5844\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2573 - acc: 0.5482 - val_loss: 1.2053 - val_acc: 0.5896\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2398 - acc: 0.5562 - val_loss: 1.2196 - val_acc: 0.5728\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2159 - acc: 0.5621 - val_loss: 1.0643 - val_acc: 0.6230\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2066 - acc: 0.5676 - val_loss: 1.1270 - val_acc: 0.5950\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.1931 - acc: 0.5729 - val_loss: 1.0989 - val_acc: 0.6060\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 31s 22ms/step - loss: 1.1802 - acc: 0.5758 - val_loss: 1.2957 - val_acc: 0.5518\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 30s 22ms/step - loss: 1.1696 - acc: 0.5797 - val_loss: 1.1261 - val_acc: 0.6050\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.1584 - acc: 0.5853 - val_loss: 0.9829 - val_acc: 0.6474\n",
      "45000/45000 [==============================] - 11s 234us/step\n",
      "Train [0.9784303038914999, 0.6512444444444444]\n",
      "10000/10000 [==============================] - 2s 242us/step\n",
      "Test [1.0263099948883057, 0.6333]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.01 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 34s 24ms/step - loss: 2.0587 - acc: 0.2568 - val_loss: 1.8587 - val_acc: 0.2884\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.7236 - acc: 0.3589 - val_loss: 1.5843 - val_acc: 0.4204\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.6201 - acc: 0.4017 - val_loss: 1.4677 - val_acc: 0.4588\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.5498 - acc: 0.4349 - val_loss: 1.4872 - val_acc: 0.4682\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.4995 - acc: 0.4532 - val_loss: 1.3999 - val_acc: 0.5000\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.4529 - acc: 0.4724 - val_loss: 1.4274 - val_acc: 0.5026\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.4186 - acc: 0.4878 - val_loss: 1.2786 - val_acc: 0.5352\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3925 - acc: 0.4939 - val_loss: 1.3044 - val_acc: 0.5398\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3578 - acc: 0.5075 - val_loss: 1.3120 - val_acc: 0.5370\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.3372 - acc: 0.5179 - val_loss: 1.2878 - val_acc: 0.5492\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3206 - acc: 0.5241 - val_loss: 1.2108 - val_acc: 0.5582\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2996 - acc: 0.5330 - val_loss: 1.2837 - val_acc: 0.5538\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2808 - acc: 0.5419 - val_loss: 1.2717 - val_acc: 0.5556\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2609 - acc: 0.5467 - val_loss: 1.1747 - val_acc: 0.5846\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2447 - acc: 0.5507 - val_loss: 1.1460 - val_acc: 0.5960\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2346 - acc: 0.5550 - val_loss: 1.1133 - val_acc: 0.6060\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2250 - acc: 0.5605 - val_loss: 1.2083 - val_acc: 0.5882\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 30s 21ms/step - loss: 1.2109 - acc: 0.5661 - val_loss: 1.0757 - val_acc: 0.6122\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 31s 22ms/step - loss: 1.1955 - acc: 0.5705 - val_loss: 1.3124 - val_acc: 0.5672\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 31s 22ms/step - loss: 1.1904 - acc: 0.5751 - val_loss: 1.1592 - val_acc: 0.5948\n",
      "45000/45000 [==============================] - 12s 267us/step\n",
      "Train [1.145315782154931, 0.5997333333333333]\n",
      "10000/10000 [==============================] - 3s 258us/step\n",
      "Test [1.1954180150985718, 0.586]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 31s 22ms/step - loss: 1.7339 - acc: 0.3562 - val_loss: 1.5009 - val_acc: 0.4660\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.5172 - acc: 0.4441 - val_loss: 1.5240 - val_acc: 0.4548\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.4268 - acc: 0.4828 - val_loss: 1.2725 - val_acc: 0.5470\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3665 - acc: 0.5043 - val_loss: 1.3082 - val_acc: 0.5396\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3231 - acc: 0.5234 - val_loss: 1.2175 - val_acc: 0.5696\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.2861 - acc: 0.5354 - val_loss: 1.2005 - val_acc: 0.5578\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.2502 - acc: 0.5480 - val_loss: 1.1411 - val_acc: 0.5860\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.2344 - acc: 0.5534 - val_loss: 1.1217 - val_acc: 0.6032\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.2140 - acc: 0.5607 - val_loss: 1.2872 - val_acc: 0.5748\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 1.1911 - acc: 0.5698 - val_loss: 1.1920 - val_acc: 0.5918\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.1800 - acc: 0.5755 - val_loss: 1.0855 - val_acc: 0.6094\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.1641 - acc: 0.5815 - val_loss: 1.0865 - val_acc: 0.6202\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.1451 - acc: 0.5892 - val_loss: 1.0925 - val_acc: 0.6142\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.1377 - acc: 0.5925 - val_loss: 1.1253 - val_acc: 0.6126\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 26s 19ms/step - loss: 1.1236 - acc: 0.5979 - val_loss: 1.0800 - val_acc: 0.6338\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.1206 - acc: 0.6005 - val_loss: 1.0207 - val_acc: 0.6380\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.1121 - acc: 0.5994 - val_loss: 1.0114 - val_acc: 0.6404\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.1040 - acc: 0.6032 - val_loss: 1.0821 - val_acc: 0.6230\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.0910 - acc: 0.6096 - val_loss: 1.0734 - val_acc: 0.6260\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.0820 - acc: 0.6152 - val_loss: 1.1841 - val_acc: 0.6184\n",
      "45000/45000 [==============================] - 13s 286us/step\n",
      "Train [1.1297511827362907, 0.6254]\n",
      "10000/10000 [==============================] - 3s 277us/step\n",
      "Test [1.214094896697998, 0.6092]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1407/1406 [==============================] - 31s 22ms/step - loss: 1.9298 - acc: 0.2724 - val_loss: 1.6743 - val_acc: 0.3690\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.7220 - acc: 0.3613 - val_loss: 1.6424 - val_acc: 0.3882\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.6649 - acc: 0.3857 - val_loss: 1.5689 - val_acc: 0.4226\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.6279 - acc: 0.3975 - val_loss: 1.6188 - val_acc: 0.4066\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.5993 - acc: 0.4116 - val_loss: 1.4912 - val_acc: 0.4502\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.5821 - acc: 0.4181 - val_loss: 1.4790 - val_acc: 0.4502\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.5487 - acc: 0.4342 - val_loss: 1.4775 - val_acc: 0.4548\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.5300 - acc: 0.4420 - val_loss: 1.4086 - val_acc: 0.4870\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.5058 - acc: 0.4472 - val_loss: 1.3751 - val_acc: 0.4950\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 26s 19ms/step - loss: 1.4890 - acc: 0.4514 - val_loss: 1.3630 - val_acc: 0.5006\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.4576 - acc: 0.4675 - val_loss: 1.3289 - val_acc: 0.5070\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.4360 - acc: 0.4779 - val_loss: 1.3139 - val_acc: 0.5238\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.4112 - acc: 0.4851 - val_loss: 1.3107 - val_acc: 0.5272\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3937 - acc: 0.4930 - val_loss: 1.3920 - val_acc: 0.5134\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 1.3809 - acc: 0.4961 - val_loss: 1.3288 - val_acc: 0.5248\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3628 - acc: 0.5037 - val_loss: 1.2565 - val_acc: 0.5494\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3565 - acc: 0.5050 - val_loss: 1.3001 - val_acc: 0.5412\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3491 - acc: 0.5088 - val_loss: 1.3178 - val_acc: 0.5254\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3333 - acc: 0.5153 - val_loss: 1.2309 - val_acc: 0.5710\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3340 - acc: 0.5168 - val_loss: 1.2184 - val_acc: 0.5638\n",
      "45000/45000 [==============================] - 13s 290us/step\n",
      "Train [1.2181384427600437, 0.5624444444444444]\n",
      "10000/10000 [==============================] - 3s 280us/step\n",
      "Test [1.240022992515564, 0.5526]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 31s 22ms/step - loss: 1.8503 - acc: 0.3061 - val_loss: 1.6939 - val_acc: 0.3596\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.6555 - acc: 0.3866 - val_loss: 1.5233 - val_acc: 0.4312\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.5667 - acc: 0.4223 - val_loss: 1.5091 - val_acc: 0.4496\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.5189 - acc: 0.4440 - val_loss: 1.3897 - val_acc: 0.4886\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.4848 - acc: 0.4568 - val_loss: 1.3877 - val_acc: 0.4998\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.4523 - acc: 0.4683 - val_loss: 1.5124 - val_acc: 0.4774\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.4337 - acc: 0.4814 - val_loss: 1.3283 - val_acc: 0.5196\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.4144 - acc: 0.4862 - val_loss: 1.3546 - val_acc: 0.5134\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3913 - acc: 0.4934 - val_loss: 1.3371 - val_acc: 0.5196\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 28s 20ms/step - loss: 1.3735 - acc: 0.5004 - val_loss: 1.3916 - val_acc: 0.5222\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3574 - acc: 0.5068 - val_loss: 1.3334 - val_acc: 0.5272\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3407 - acc: 0.5165 - val_loss: 1.3340 - val_acc: 0.5266\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3308 - acc: 0.5191 - val_loss: 1.2716 - val_acc: 0.5542\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3123 - acc: 0.5244 - val_loss: 1.2401 - val_acc: 0.5536\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.3089 - acc: 0.5267 - val_loss: 1.2714 - val_acc: 0.5394\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.2961 - acc: 0.5316 - val_loss: 1.1906 - val_acc: 0.5816\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.2872 - acc: 0.5349 - val_loss: 1.2569 - val_acc: 0.5552\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.2789 - acc: 0.5351 - val_loss: 1.3769 - val_acc: 0.5396\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.2713 - acc: 0.5403 - val_loss: 1.2551 - val_acc: 0.5568\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 27s 19ms/step - loss: 1.2658 - acc: 0.5426 - val_loss: 1.1589 - val_acc: 0.5760\n",
      "45000/45000 [==============================] - 14s 308us/step\n",
      "Train [1.1571586209403144, 0.5835111111111111]\n",
      "10000/10000 [==============================] - 3s 323us/step\n",
      "Test [1.1893019708633423, 0.5781]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.8416 - acc: 0.3460 - val_loss: 1.5356 - val_acc: 0.4504\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 92s 66ms/step - loss: 1.5607 - acc: 0.4342 - val_loss: 1.4275 - val_acc: 0.4870\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 93s 66ms/step - loss: 1.4552 - acc: 0.4719 - val_loss: 1.2980 - val_acc: 0.5412\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 91s 64ms/step - loss: 1.3943 - acc: 0.4973 - val_loss: 1.4211 - val_acc: 0.5182\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 89s 63ms/step - loss: 1.3450 - acc: 0.5154 - val_loss: 1.2299 - val_acc: 0.5588\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 1.3034 - acc: 0.5288 - val_loss: 1.3185 - val_acc: 0.5486\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 77s 55ms/step - loss: 1.2679 - acc: 0.5403 - val_loss: 1.2377 - val_acc: 0.5786\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 85s 61ms/step - loss: 1.2380 - acc: 0.5573 - val_loss: 1.1751 - val_acc: 0.5892\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 90s 64ms/step - loss: 1.2136 - acc: 0.5653 - val_loss: 1.1134 - val_acc: 0.6116\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 85s 61ms/step - loss: 1.1761 - acc: 0.5799 - val_loss: 1.1320 - val_acc: 0.6174\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 90s 64ms/step - loss: 1.1641 - acc: 0.5819 - val_loss: 1.0755 - val_acc: 0.6220\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 84s 60ms/step - loss: 1.1483 - acc: 0.5910 - val_loss: 1.1287 - val_acc: 0.6178\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 77s 55ms/step - loss: 1.1226 - acc: 0.5996 - val_loss: 1.1739 - val_acc: 0.6070\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 89s 63ms/step - loss: 1.1152 - acc: 0.6015 - val_loss: 0.9953 - val_acc: 0.6570\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 91s 65ms/step - loss: 1.0952 - acc: 0.6074 - val_loss: 0.9986 - val_acc: 0.6502\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 85s 60ms/step - loss: 1.0814 - acc: 0.6128 - val_loss: 1.0000 - val_acc: 0.6526\n",
      "Epoch 17/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1407/1406 [==============================] - 83s 59ms/step - loss: 1.0706 - acc: 0.6166 - val_loss: 1.0488 - val_acc: 0.6472\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 85s 60ms/step - loss: 1.0625 - acc: 0.6198 - val_loss: 1.0699 - val_acc: 0.6288\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 84s 60ms/step - loss: 1.0498 - acc: 0.6245 - val_loss: 0.9362 - val_acc: 0.6714\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 84s 60ms/step - loss: 1.0406 - acc: 0.6285 - val_loss: 0.9871 - val_acc: 0.6590\n",
      "45000/45000 [==============================] - 13s 281us/step\n",
      "Train [0.9543800232781304, 0.6711777777777778]\n",
      "10000/10000 [==============================] - 3s 284us/step\n",
      "Test [1.027976955318451, 0.6472]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 88s 62ms/step - loss: 1.8558 - acc: 0.3383 - val_loss: 1.5672 - val_acc: 0.4392\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 81s 57ms/step - loss: 1.5781 - acc: 0.4271 - val_loss: 1.3600 - val_acc: 0.5078\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 81s 58ms/step - loss: 1.4782 - acc: 0.4645 - val_loss: 1.3583 - val_acc: 0.5108\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 90s 64ms/step - loss: 1.4018 - acc: 0.4937 - val_loss: 1.2413 - val_acc: 0.5546\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 85s 60ms/step - loss: 1.3458 - acc: 0.5143 - val_loss: 1.3424 - val_acc: 0.5490\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 71s 51ms/step - loss: 1.3014 - acc: 0.5300 - val_loss: 1.2915 - val_acc: 0.5546\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 95s 67ms/step - loss: 1.2602 - acc: 0.5446 - val_loss: 1.3278 - val_acc: 0.5534\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 92s 65ms/step - loss: 1.2288 - acc: 0.5580 - val_loss: 1.3197 - val_acc: 0.5662\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 90s 64ms/step - loss: 1.2091 - acc: 0.5643 - val_loss: 1.1267 - val_acc: 0.6042\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 94s 67ms/step - loss: 1.1867 - acc: 0.5749 - val_loss: 1.1143 - val_acc: 0.6106\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 94s 67ms/step - loss: 1.1647 - acc: 0.5849 - val_loss: 1.1458 - val_acc: 0.6104\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 92s 66ms/step - loss: 1.1449 - acc: 0.5912 - val_loss: 1.1911 - val_acc: 0.5972\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 93s 66ms/step - loss: 1.1267 - acc: 0.5998 - val_loss: 1.0891 - val_acc: 0.6272\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 93s 66ms/step - loss: 1.1074 - acc: 0.6040 - val_loss: 1.1644 - val_acc: 0.6144\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 94s 67ms/step - loss: 1.1007 - acc: 0.6065 - val_loss: 1.0355 - val_acc: 0.6422\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 93s 66ms/step - loss: 1.0813 - acc: 0.6110 - val_loss: 1.0380 - val_acc: 0.6428\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 96s 68ms/step - loss: 1.0765 - acc: 0.6170 - val_loss: 0.9763 - val_acc: 0.6586\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 96s 68ms/step - loss: 1.0598 - acc: 0.6206 - val_loss: 0.9581 - val_acc: 0.6634\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 95s 68ms/step - loss: 1.0466 - acc: 0.6268 - val_loss: 0.9801 - val_acc: 0.6632\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 96s 68ms/step - loss: 1.0383 - acc: 0.6288 - val_loss: 0.9832 - val_acc: 0.6588\n",
      "45000/45000 [==============================] - 13s 296us/step\n",
      "Train [0.9428587835947673, 0.6737777777777778]\n",
      "10000/10000 [==============================] - 3s 294us/step\n",
      "Test [1.0116162817001342, 0.65]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 102s 73ms/step - loss: 1.8628 - acc: 0.3397 - val_loss: 1.4726 - val_acc: 0.4592\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.5748 - acc: 0.4249 - val_loss: 1.4686 - val_acc: 0.4776\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 97s 69ms/step - loss: 1.4731 - acc: 0.4689 - val_loss: 1.3737 - val_acc: 0.5088\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.3958 - acc: 0.4983 - val_loss: 1.4031 - val_acc: 0.5126\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 93s 66ms/step - loss: 1.3393 - acc: 0.5168 - val_loss: 1.2402 - val_acc: 0.5478\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.3016 - acc: 0.5267 - val_loss: 1.2157 - val_acc: 0.5704\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 97s 69ms/step - loss: 1.2620 - acc: 0.5449 - val_loss: 1.2657 - val_acc: 0.5640\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.2398 - acc: 0.5539 - val_loss: 1.2678 - val_acc: 0.5654\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 99s 71ms/step - loss: 1.2076 - acc: 0.5667 - val_loss: 1.1505 - val_acc: 0.5928\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.1882 - acc: 0.5720 - val_loss: 1.1439 - val_acc: 0.5988\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 99s 70ms/step - loss: 1.1657 - acc: 0.5817 - val_loss: 1.0864 - val_acc: 0.6120\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 99s 70ms/step - loss: 1.1440 - acc: 0.5907 - val_loss: 1.1871 - val_acc: 0.6046\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 98s 69ms/step - loss: 1.1289 - acc: 0.5952 - val_loss: 1.1764 - val_acc: 0.6042\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 93s 66ms/step - loss: 1.1097 - acc: 0.6035 - val_loss: 1.1789 - val_acc: 0.6128\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 97s 69ms/step - loss: 1.0960 - acc: 0.6091 - val_loss: 1.1811 - val_acc: 0.6064\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.0839 - acc: 0.6157 - val_loss: 1.0165 - val_acc: 0.6466\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.0694 - acc: 0.6194 - val_loss: 1.0045 - val_acc: 0.6506\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 97s 69ms/step - loss: 1.0627 - acc: 0.6221 - val_loss: 1.0079 - val_acc: 0.6574\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 97s 69ms/step - loss: 1.0509 - acc: 0.6254 - val_loss: 1.0532 - val_acc: 0.6462\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 96s 68ms/step - loss: 1.0422 - acc: 0.6277 - val_loss: 0.9322 - val_acc: 0.6730\n",
      "45000/45000 [==============================] - 14s 302us/step\n",
      "Train [0.8871748023562961, 0.6881777777777778]\n",
      "10000/10000 [==============================] - 3s 303us/step\n",
      "Test [0.9641754447937012, 0.6688]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.8250 - acc: 0.3281 - val_loss: 1.5517 - val_acc: 0.4368\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.5878 - acc: 0.4195 - val_loss: 1.4005 - val_acc: 0.5034\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.4973 - acc: 0.4551 - val_loss: 1.3965 - val_acc: 0.5034\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.4223 - acc: 0.4857 - val_loss: 1.3220 - val_acc: 0.5338\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.3712 - acc: 0.5058 - val_loss: 1.5035 - val_acc: 0.4990\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.3301 - acc: 0.5192 - val_loss: 1.2186 - val_acc: 0.5760\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.2892 - acc: 0.5349 - val_loss: 1.1949 - val_acc: 0.5770\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.2688 - acc: 0.5427 - val_loss: 1.1332 - val_acc: 0.5994\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.2346 - acc: 0.5554 - val_loss: 1.1453 - val_acc: 0.5960\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.2151 - acc: 0.5629 - val_loss: 1.1567 - val_acc: 0.5940\n",
      "Epoch 11/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1959 - acc: 0.5712 - val_loss: 1.1709 - val_acc: 0.6012\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1800 - acc: 0.5758 - val_loss: 1.1028 - val_acc: 0.6096\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1608 - acc: 0.5847 - val_loss: 1.0914 - val_acc: 0.6262\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1442 - acc: 0.5914 - val_loss: 1.0692 - val_acc: 0.6204\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1317 - acc: 0.5944 - val_loss: 1.0688 - val_acc: 0.6270\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1257 - acc: 0.5965 - val_loss: 1.0454 - val_acc: 0.6380\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1040 - acc: 0.6052 - val_loss: 1.0646 - val_acc: 0.6384\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.0928 - acc: 0.6071 - val_loss: 1.0293 - val_acc: 0.6426\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.0778 - acc: 0.6135 - val_loss: 0.9707 - val_acc: 0.6552\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.0735 - acc: 0.6176 - val_loss: 1.0164 - val_acc: 0.6404\n",
      "45000/45000 [==============================] - 15s 340us/step\n",
      "Train [1.001597176000807, 0.6449111111111111]\n",
      "10000/10000 [==============================] - 3s 332us/step\n",
      "Test [1.0478915488243103, 0.637]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.8139 - acc: 0.3336 - val_loss: 1.5763 - val_acc: 0.4312\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.5699 - acc: 0.4280 - val_loss: 1.6630 - val_acc: 0.4504\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.4808 - acc: 0.4647 - val_loss: 1.4762 - val_acc: 0.4872\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.4177 - acc: 0.4889 - val_loss: 1.3053 - val_acc: 0.5344\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.3742 - acc: 0.5038 - val_loss: 1.2418 - val_acc: 0.5540\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.3321 - acc: 0.5200 - val_loss: 1.2922 - val_acc: 0.5448\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.2982 - acc: 0.5326 - val_loss: 1.3085 - val_acc: 0.5424\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.2762 - acc: 0.5424 - val_loss: 1.4392 - val_acc: 0.5324\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.2460 - acc: 0.5524 - val_loss: 1.3473 - val_acc: 0.5502\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.2258 - acc: 0.5589 - val_loss: 1.1662 - val_acc: 0.5860\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.2062 - acc: 0.5699 - val_loss: 1.1623 - val_acc: 0.5860\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1847 - acc: 0.5746 - val_loss: 1.1875 - val_acc: 0.5954\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1727 - acc: 0.5779 - val_loss: 1.1921 - val_acc: 0.5892\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1514 - acc: 0.5879 - val_loss: 1.1014 - val_acc: 0.6148\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1445 - acc: 0.5899 - val_loss: 1.1793 - val_acc: 0.5924\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1276 - acc: 0.5969 - val_loss: 1.2167 - val_acc: 0.6078\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1166 - acc: 0.5995 - val_loss: 1.1413 - val_acc: 0.6114\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1052 - acc: 0.6018 - val_loss: 0.9695 - val_acc: 0.6554\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.0910 - acc: 0.6102 - val_loss: 1.0139 - val_acc: 0.6390\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.0871 - acc: 0.6137 - val_loss: 1.0920 - val_acc: 0.6162\n",
      "45000/45000 [==============================] - 15s 342us/step\n",
      "Train [1.0734438851038615, 0.6263555555555556]\n",
      "10000/10000 [==============================] - 3s 340us/step\n",
      "Test [1.132124526119232, 0.6108]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.001 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 37s 27ms/step - loss: 1.8244 - acc: 0.3299 - val_loss: 1.6708 - val_acc: 0.4106\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.5821 - acc: 0.4222 - val_loss: 1.6617 - val_acc: 0.4166\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.4909 - acc: 0.4629 - val_loss: 1.4918 - val_acc: 0.4862\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.4312 - acc: 0.4849 - val_loss: 1.3210 - val_acc: 0.5162\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.3775 - acc: 0.5042 - val_loss: 1.3633 - val_acc: 0.5326\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.3359 - acc: 0.5197 - val_loss: 1.1196 - val_acc: 0.5998\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.3039 - acc: 0.5309 - val_loss: 1.1484 - val_acc: 0.5860\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.2658 - acc: 0.5426 - val_loss: 1.1191 - val_acc: 0.5988\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.2420 - acc: 0.5515 - val_loss: 1.3039 - val_acc: 0.5496\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.2207 - acc: 0.5602 - val_loss: 1.1725 - val_acc: 0.5836\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.1915 - acc: 0.5709 - val_loss: 1.1671 - val_acc: 0.5870\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.1747 - acc: 0.5778 - val_loss: 1.2142 - val_acc: 0.5866\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.1572 - acc: 0.5844 - val_loss: 1.1813 - val_acc: 0.5902\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.1430 - acc: 0.5916 - val_loss: 1.1008 - val_acc: 0.6154\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.1245 - acc: 0.5942 - val_loss: 1.2073 - val_acc: 0.5996\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.1125 - acc: 0.6025 - val_loss: 1.1256 - val_acc: 0.6150\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 32s 23ms/step - loss: 1.0993 - acc: 0.6068 - val_loss: 1.1596 - val_acc: 0.6106\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.0887 - acc: 0.6107 - val_loss: 1.1102 - val_acc: 0.6270\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.0754 - acc: 0.6143 - val_loss: 1.1315 - val_acc: 0.6220\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 33s 23ms/step - loss: 1.0713 - acc: 0.6184 - val_loss: 0.9435 - val_acc: 0.6666\n",
      "45000/45000 [==============================] - 15s 343us/step\n",
      "Train [0.9145551048914592, 0.6738444444444445]\n",
      "10000/10000 [==============================] - 3s 342us/step\n",
      "Test [0.9754988464355469, 0.6573]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 34s 24ms/step - loss: 1.9074 - acc: 0.2869 - val_loss: 1.6955 - val_acc: 0.3706\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.6976 - acc: 0.3713 - val_loss: 1.6866 - val_acc: 0.3970\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.6039 - acc: 0.4139 - val_loss: 1.5299 - val_acc: 0.4472\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.5350 - acc: 0.4426 - val_loss: 1.4630 - val_acc: 0.4714\n",
      "Epoch 5/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.4892 - acc: 0.4602 - val_loss: 1.4192 - val_acc: 0.4952\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.4504 - acc: 0.4747 - val_loss: 1.3293 - val_acc: 0.5238\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.4220 - acc: 0.4868 - val_loss: 1.4758 - val_acc: 0.4920\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.3997 - acc: 0.4940 - val_loss: 1.2922 - val_acc: 0.5378\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.3719 - acc: 0.5056 - val_loss: 1.2933 - val_acc: 0.5360\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.3513 - acc: 0.5126 - val_loss: 1.2876 - val_acc: 0.5410\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.3318 - acc: 0.5175 - val_loss: 1.2669 - val_acc: 0.5476\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.3111 - acc: 0.5293 - val_loss: 1.2502 - val_acc: 0.5596\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.3005 - acc: 0.5324 - val_loss: 1.2443 - val_acc: 0.5566\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.2800 - acc: 0.5394 - val_loss: 1.3220 - val_acc: 0.5470\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.2654 - acc: 0.5474 - val_loss: 1.2011 - val_acc: 0.5740\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.2530 - acc: 0.5506 - val_loss: 1.2216 - val_acc: 0.5706\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.2394 - acc: 0.5534 - val_loss: 1.2560 - val_acc: 0.5616\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.2286 - acc: 0.5620 - val_loss: 1.2166 - val_acc: 0.5732\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.2074 - acc: 0.5649 - val_loss: 1.1465 - val_acc: 0.5994\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.1989 - acc: 0.5703 - val_loss: 1.2036 - val_acc: 0.5800\n",
      "45000/45000 [==============================] - 16s 345us/step\n",
      "Train [1.1922927114698623, 0.5857777777777777]\n",
      "10000/10000 [==============================] - 3s 344us/step\n",
      "Test [1.2299964181900025, 0.5753]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 34s 24ms/step - loss: 1.8042 - acc: 0.3244 - val_loss: 1.5929 - val_acc: 0.4200\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.6312 - acc: 0.3982 - val_loss: 1.5616 - val_acc: 0.4292\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.5448 - acc: 0.4392 - val_loss: 1.4516 - val_acc: 0.4820\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.4921 - acc: 0.4579 - val_loss: 1.3583 - val_acc: 0.5168\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.4510 - acc: 0.4767 - val_loss: 1.3681 - val_acc: 0.5158\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.4183 - acc: 0.4856 - val_loss: 1.3831 - val_acc: 0.5142\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3874 - acc: 0.4985 - val_loss: 1.2683 - val_acc: 0.5422\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3615 - acc: 0.5074 - val_loss: 1.3007 - val_acc: 0.5466\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3442 - acc: 0.5115 - val_loss: 1.3185 - val_acc: 0.5442\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.3188 - acc: 0.5249 - val_loss: 1.3724 - val_acc: 0.5256\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2961 - acc: 0.5324 - val_loss: 1.2422 - val_acc: 0.5576\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2848 - acc: 0.5394 - val_loss: 1.1518 - val_acc: 0.5910\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2629 - acc: 0.5430 - val_loss: 1.1433 - val_acc: 0.5946\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2464 - acc: 0.5495 - val_loss: 1.1641 - val_acc: 0.5914\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2346 - acc: 0.5533 - val_loss: 1.1653 - val_acc: 0.5868\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 29s 20ms/step - loss: 1.2226 - acc: 0.5592 - val_loss: 1.1622 - val_acc: 0.5906\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2095 - acc: 0.5648 - val_loss: 1.1332 - val_acc: 0.5990\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.1940 - acc: 0.5677 - val_loss: 1.0619 - val_acc: 0.6226\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.1786 - acc: 0.5766 - val_loss: 1.2462 - val_acc: 0.5748\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.1673 - acc: 0.5803 - val_loss: 1.1034 - val_acc: 0.6086\n",
      "45000/45000 [==============================] - 16s 346us/step\n",
      "Train [1.0981658610873752, 0.6102888888888889]\n",
      "10000/10000 [==============================] - 4s 350us/step\n",
      "Test [1.1359204278945922, 0.6062]\n",
      "RESNET WITHOUT BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 34s 25ms/step - loss: 1.8023 - acc: 0.3296 - val_loss: 1.5845 - val_acc: 0.4118\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.6151 - acc: 0.4094 - val_loss: 1.5479 - val_acc: 0.4332\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.5305 - acc: 0.4449 - val_loss: 1.4363 - val_acc: 0.4850\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.4821 - acc: 0.4628 - val_loss: 1.3564 - val_acc: 0.5120\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.4415 - acc: 0.4757 - val_loss: 1.4126 - val_acc: 0.4940\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.4077 - acc: 0.4906 - val_loss: 1.3322 - val_acc: 0.5234\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3859 - acc: 0.5016 - val_loss: 1.2955 - val_acc: 0.5370\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3616 - acc: 0.5097 - val_loss: 1.3379 - val_acc: 0.5328\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3434 - acc: 0.5151 - val_loss: 1.2672 - val_acc: 0.5496\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3263 - acc: 0.5199 - val_loss: 1.2809 - val_acc: 0.5484\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.3062 - acc: 0.5302 - val_loss: 1.2051 - val_acc: 0.5738\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2919 - acc: 0.5298 - val_loss: 1.2281 - val_acc: 0.5702\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2787 - acc: 0.5383 - val_loss: 1.1925 - val_acc: 0.5760\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2582 - acc: 0.5460 - val_loss: 1.2733 - val_acc: 0.5564\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2460 - acc: 0.5499 - val_loss: 1.1704 - val_acc: 0.5802\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2375 - acc: 0.5539 - val_loss: 1.2646 - val_acc: 0.5664\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2207 - acc: 0.5598 - val_loss: 1.1503 - val_acc: 0.5986\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2082 - acc: 0.5677 - val_loss: 1.2412 - val_acc: 0.5692\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.2004 - acc: 0.5682 - val_loss: 1.1455 - val_acc: 0.5950\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 29s 21ms/step - loss: 1.1843 - acc: 0.5765 - val_loss: 1.1348 - val_acc: 0.6010\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "45000/45000 [==============================] - 16s 350us/step\n",
      "Train [1.1242102628072104, 0.6066888888888889]\n",
      "10000/10000 [==============================] - 4s 351us/step\n",
      "Test [1.1667916519165038, 0.5935]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 102s 72ms/step - loss: 1.8922 - acc: 0.3298 - val_loss: 1.5721 - val_acc: 0.4452\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 100s 71ms/step - loss: 1.6771 - acc: 0.3968 - val_loss: 1.5275 - val_acc: 0.4688\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 91s 65ms/step - loss: 1.6031 - acc: 0.4191 - val_loss: 1.4572 - val_acc: 0.4946\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 86s 61ms/step - loss: 1.5407 - acc: 0.4419 - val_loss: 1.3825 - val_acc: 0.5150\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 85s 61ms/step - loss: 1.4983 - acc: 0.4612 - val_loss: 1.3244 - val_acc: 0.5384\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 88s 63ms/step - loss: 1.4595 - acc: 0.4759 - val_loss: 1.3807 - val_acc: 0.5324\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 99s 71ms/step - loss: 1.4258 - acc: 0.4886 - val_loss: 1.3685 - val_acc: 0.5312\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 91s 65ms/step - loss: 1.3964 - acc: 0.4974 - val_loss: 1.3206 - val_acc: 0.5420\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 89s 63ms/step - loss: 1.3703 - acc: 0.5080 - val_loss: 1.2834 - val_acc: 0.5586\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 1.3452 - acc: 0.5172 - val_loss: 1.3110 - val_acc: 0.5494\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 1.3215 - acc: 0.5263 - val_loss: 1.2295 - val_acc: 0.5740\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 88s 62ms/step - loss: 1.3016 - acc: 0.5340 - val_loss: 1.2369 - val_acc: 0.5704\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 1.2820 - acc: 0.5404 - val_loss: 1.1963 - val_acc: 0.5882\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 85s 60ms/step - loss: 1.2644 - acc: 0.5458 - val_loss: 1.1585 - val_acc: 0.5980\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 86s 61ms/step - loss: 1.2506 - acc: 0.5518 - val_loss: 1.1951 - val_acc: 0.5842\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 82s 58ms/step - loss: 1.2411 - acc: 0.5555 - val_loss: 1.1520 - val_acc: 0.6042\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 82s 58ms/step - loss: 1.2228 - acc: 0.5627 - val_loss: 1.1937 - val_acc: 0.5866\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 91s 65ms/step - loss: 1.2039 - acc: 0.5708 - val_loss: 1.1887 - val_acc: 0.5870\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 92s 66ms/step - loss: 1.1933 - acc: 0.5742 - val_loss: 1.1510 - val_acc: 0.6072\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 1.1762 - acc: 0.5812 - val_loss: 1.1893 - val_acc: 0.5952\n",
      "45000/45000 [==============================] - 16s 358us/step\n",
      "Train [1.1418140314949883, 0.608]\n",
      "10000/10000 [==============================] - 4s 358us/step\n",
      "Test [1.2048329513549805, 0.5909]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 91s 65ms/step - loss: 1.8853 - acc: 0.3307 - val_loss: 1.5473 - val_acc: 0.4450\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 87s 62ms/step - loss: 1.6658 - acc: 0.3951 - val_loss: 1.5082 - val_acc: 0.4696\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 84s 60ms/step - loss: 1.5825 - acc: 0.4272 - val_loss: 1.4386 - val_acc: 0.4948\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 85s 61ms/step - loss: 1.5274 - acc: 0.4489 - val_loss: 1.4285 - val_acc: 0.4982\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 80s 57ms/step - loss: 1.4794 - acc: 0.4656 - val_loss: 1.3691 - val_acc: 0.5310\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 100s 71ms/step - loss: 1.4459 - acc: 0.4789 - val_loss: 1.3442 - val_acc: 0.5424\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 99s 70ms/step - loss: 1.4175 - acc: 0.4902 - val_loss: 1.3688 - val_acc: 0.5316\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 100s 71ms/step - loss: 1.3845 - acc: 0.5015 - val_loss: 1.3031 - val_acc: 0.5508\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 97s 69ms/step - loss: 1.3601 - acc: 0.5092 - val_loss: 1.2714 - val_acc: 0.5604\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 97s 69ms/step - loss: 1.3395 - acc: 0.5159 - val_loss: 1.2833 - val_acc: 0.5658\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 95s 67ms/step - loss: 1.3131 - acc: 0.5274 - val_loss: 1.2590 - val_acc: 0.5672\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.2957 - acc: 0.5336 - val_loss: 1.2037 - val_acc: 0.5788\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.2783 - acc: 0.5414 - val_loss: 1.1843 - val_acc: 0.5920\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 98s 70ms/step - loss: 1.2596 - acc: 0.5458 - val_loss: 1.1411 - val_acc: 0.5980\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 99s 70ms/step - loss: 1.2386 - acc: 0.5549 - val_loss: 1.1728 - val_acc: 0.5912\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 100s 71ms/step - loss: 1.2251 - acc: 0.5615 - val_loss: 1.1548 - val_acc: 0.5966\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 99s 70ms/step - loss: 1.2083 - acc: 0.5659 - val_loss: 1.1118 - val_acc: 0.6102\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 101s 72ms/step - loss: 1.2022 - acc: 0.5669 - val_loss: 1.1508 - val_acc: 0.6062\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 100s 71ms/step - loss: 1.1840 - acc: 0.5770 - val_loss: 1.1532 - val_acc: 0.6050\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 97s 69ms/step - loss: 1.1717 - acc: 0.5795 - val_loss: 1.1352 - val_acc: 0.6086\n",
      "45000/45000 [==============================] - 17s 385us/step\n",
      "Train [1.1101584215799967, 0.6140888888888889]\n",
      "10000/10000 [==============================] - 4s 385us/step\n",
      "Test [1.1599896463394166, 0.6024]\n",
      "RESNET WITH LAYER BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 110s 78ms/step - loss: 1.8854 - acc: 0.3309 - val_loss: 1.5929 - val_acc: 0.4318\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 101s 72ms/step - loss: 1.6698 - acc: 0.3973 - val_loss: 1.5099 - val_acc: 0.4766\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 103s 73ms/step - loss: 1.5958 - acc: 0.4247 - val_loss: 1.4592 - val_acc: 0.4868\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 101s 72ms/step - loss: 1.5322 - acc: 0.4475 - val_loss: 1.4346 - val_acc: 0.5024\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 103s 73ms/step - loss: 1.4923 - acc: 0.4620 - val_loss: 1.3469 - val_acc: 0.5332\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 103s 73ms/step - loss: 1.4571 - acc: 0.4744 - val_loss: 1.3761 - val_acc: 0.5234\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 104s 74ms/step - loss: 1.4127 - acc: 0.4906 - val_loss: 1.3100 - val_acc: 0.5472\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 102s 73ms/step - loss: 1.3868 - acc: 0.5011 - val_loss: 1.3523 - val_acc: 0.5434\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 100s 71ms/step - loss: 1.3616 - acc: 0.5117 - val_loss: 1.2895 - val_acc: 0.5546\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 103s 73ms/step - loss: 1.3375 - acc: 0.5175 - val_loss: 1.2268 - val_acc: 0.5712\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 102s 73ms/step - loss: 1.3114 - acc: 0.5287 - val_loss: 1.2247 - val_acc: 0.5712\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 103s 73ms/step - loss: 1.2965 - acc: 0.5340 - val_loss: 1.2021 - val_acc: 0.5788\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 104s 74ms/step - loss: 1.2765 - acc: 0.5409 - val_loss: 1.2197 - val_acc: 0.5806\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 102s 73ms/step - loss: 1.2578 - acc: 0.5491 - val_loss: 1.2270 - val_acc: 0.5766\n",
      "Epoch 15/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1407/1406 [==============================] - 103s 73ms/step - loss: 1.2422 - acc: 0.5541 - val_loss: 1.1610 - val_acc: 0.6022\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 103s 73ms/step - loss: 1.2228 - acc: 0.5603 - val_loss: 1.1953 - val_acc: 0.5894\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 101s 72ms/step - loss: 1.2163 - acc: 0.5638 - val_loss: 1.1861 - val_acc: 0.5910\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 103s 73ms/step - loss: 1.2051 - acc: 0.5678 - val_loss: 1.1241 - val_acc: 0.6090\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 104s 74ms/step - loss: 1.1863 - acc: 0.5740 - val_loss: 1.1246 - val_acc: 0.6104\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 102s 72ms/step - loss: 1.1808 - acc: 0.5784 - val_loss: 1.0909 - val_acc: 0.6196\n",
      "45000/45000 [==============================] - 18s 398us/step\n",
      "Train [1.059923119714525, 0.6294444444444445]\n",
      "10000/10000 [==============================] - 4s 394us/step\n",
      "Test [1.1193279535293579, 0.6162]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  0\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 43s 30ms/step - loss: 1.7957 - acc: 0.3392 - val_loss: 1.5767 - val_acc: 0.4298\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.6159 - acc: 0.4122 - val_loss: 1.5106 - val_acc: 0.4644\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.5445 - acc: 0.4406 - val_loss: 1.4360 - val_acc: 0.5022\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.4896 - acc: 0.4625 - val_loss: 1.3091 - val_acc: 0.5344\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.4535 - acc: 0.4779 - val_loss: 1.3114 - val_acc: 0.5410\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.4174 - acc: 0.4899 - val_loss: 1.3047 - val_acc: 0.5484\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.3846 - acc: 0.5020 - val_loss: 1.3210 - val_acc: 0.5394\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.3582 - acc: 0.5129 - val_loss: 1.3043 - val_acc: 0.5482\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.3339 - acc: 0.5221 - val_loss: 1.3709 - val_acc: 0.5368\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.3113 - acc: 0.5305 - val_loss: 1.2265 - val_acc: 0.5660\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.2908 - acc: 0.5387 - val_loss: 1.2869 - val_acc: 0.5552\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.2690 - acc: 0.5423 - val_loss: 1.2354 - val_acc: 0.5708\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.2531 - acc: 0.5480 - val_loss: 1.3161 - val_acc: 0.5526\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.2395 - acc: 0.5538 - val_loss: 1.1871 - val_acc: 0.5936\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.2255 - acc: 0.5598 - val_loss: 1.2050 - val_acc: 0.5880\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 35s 25ms/step - loss: 1.2083 - acc: 0.5650 - val_loss: 1.1994 - val_acc: 0.5910\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.1960 - acc: 0.5703 - val_loss: 1.1338 - val_acc: 0.6050\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.1816 - acc: 0.5762 - val_loss: 1.1243 - val_acc: 0.6108\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.1726 - acc: 0.5791 - val_loss: 1.1972 - val_acc: 0.5844\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.1584 - acc: 0.5858 - val_loss: 1.1136 - val_acc: 0.6148\n",
      "45000/45000 [==============================] - 17s 375us/step\n",
      "Train [1.100636676428053, 0.6155111111111111]\n",
      "10000/10000 [==============================] - 4s 371us/step\n",
      "Test [1.1526290788650513, 0.6009]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  1\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 43s 30ms/step - loss: 1.7966 - acc: 0.3343 - val_loss: 1.5174 - val_acc: 0.4488\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.6062 - acc: 0.4153 - val_loss: 1.4775 - val_acc: 0.4810\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.5296 - acc: 0.4461 - val_loss: 1.3960 - val_acc: 0.5056\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.4649 - acc: 0.4716 - val_loss: 1.4120 - val_acc: 0.5078\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.4322 - acc: 0.4841 - val_loss: 1.3870 - val_acc: 0.5164\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.3928 - acc: 0.4991 - val_loss: 1.3917 - val_acc: 0.5214\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.3611 - acc: 0.5096 - val_loss: 1.2151 - val_acc: 0.5678\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.3380 - acc: 0.5184 - val_loss: 1.2176 - val_acc: 0.5682\n",
      "Epoch 9/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.3140 - acc: 0.5279 - val_loss: 1.2986 - val_acc: 0.5554\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.2856 - acc: 0.5390 - val_loss: 1.2830 - val_acc: 0.5604\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.2714 - acc: 0.5438 - val_loss: 1.2278 - val_acc: 0.5714\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.2523 - acc: 0.5510 - val_loss: 1.1903 - val_acc: 0.5788\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.2266 - acc: 0.5593 - val_loss: 1.1456 - val_acc: 0.5900\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.2211 - acc: 0.5638 - val_loss: 1.2505 - val_acc: 0.5768\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.1996 - acc: 0.5704 - val_loss: 1.2893 - val_acc: 0.5736\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.1887 - acc: 0.5754 - val_loss: 1.1256 - val_acc: 0.6046\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.1778 - acc: 0.5800 - val_loss: 1.2460 - val_acc: 0.5810\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.1606 - acc: 0.5846 - val_loss: 1.1804 - val_acc: 0.5900\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.1511 - acc: 0.5891 - val_loss: 1.1964 - val_acc: 0.5992\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 36s 25ms/step - loss: 1.1423 - acc: 0.5918 - val_loss: 1.0737 - val_acc: 0.6210\n",
      "45000/45000 [==============================] - 17s 380us/step\n",
      "Train [1.051509989007314, 0.63]\n",
      "10000/10000 [==============================] - 4s 377us/step\n",
      "Test [1.1077330338478089, 0.615]\n",
      "RESNET WITH IDENTITY BATCHNORM, lr = 0.0001 ITER =  2\n",
      "Epoch 1/20\n",
      "1407/1406 [==============================] - 44s 31ms/step - loss: 1.8015 - acc: 0.3356 - val_loss: 1.6568 - val_acc: 0.4144\n",
      "Epoch 2/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.6119 - acc: 0.4120 - val_loss: 1.5626 - val_acc: 0.4616\n",
      "Epoch 3/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.5353 - acc: 0.4435 - val_loss: 1.4992 - val_acc: 0.4876\n",
      "Epoch 4/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.4766 - acc: 0.4680 - val_loss: 1.4375 - val_acc: 0.5054\n",
      "Epoch 5/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.4353 - acc: 0.4817 - val_loss: 1.3468 - val_acc: 0.5286\n",
      "Epoch 6/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.4021 - acc: 0.4942 - val_loss: 1.3722 - val_acc: 0.5304\n",
      "Epoch 7/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.3713 - acc: 0.5053 - val_loss: 1.4451 - val_acc: 0.5170\n",
      "Epoch 8/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.3514 - acc: 0.5155 - val_loss: 1.3230 - val_acc: 0.5514\n",
      "Epoch 9/20\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.3223 - acc: 0.5248 - val_loss: 1.3066 - val_acc: 0.5540\n",
      "Epoch 10/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.3005 - acc: 0.5322 - val_loss: 1.3125 - val_acc: 0.5486\n",
      "Epoch 11/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.2790 - acc: 0.5418 - val_loss: 1.1986 - val_acc: 0.5834\n",
      "Epoch 12/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.2612 - acc: 0.5466 - val_loss: 1.1505 - val_acc: 0.6002\n",
      "Epoch 13/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.2375 - acc: 0.5558 - val_loss: 1.1631 - val_acc: 0.5892\n",
      "Epoch 14/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.2181 - acc: 0.5646 - val_loss: 1.2079 - val_acc: 0.5888\n",
      "Epoch 15/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.2068 - acc: 0.5659 - val_loss: 1.2320 - val_acc: 0.5856\n",
      "Epoch 16/20\n",
      "1407/1406 [==============================] - 36s 26ms/step - loss: 1.1944 - acc: 0.5677 - val_loss: 1.1866 - val_acc: 0.5922\n",
      "Epoch 17/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.1792 - acc: 0.5769 - val_loss: 1.1351 - val_acc: 0.6110\n",
      "Epoch 18/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.1641 - acc: 0.5825 - val_loss: 1.1131 - val_acc: 0.6154\n",
      "Epoch 19/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.1540 - acc: 0.5882 - val_loss: 1.1701 - val_acc: 0.6032\n",
      "Epoch 20/20\n",
      "1407/1406 [==============================] - 37s 26ms/step - loss: 1.1419 - acc: 0.5921 - val_loss: 1.1844 - val_acc: 0.6064\n",
      "45000/45000 [==============================] - 17s 382us/step\n",
      "Train [1.1783335713704428, 0.5991333333333333]\n",
      "10000/10000 [==============================] - 4s 382us/step\n",
      "Test [1.2289922315597535, 0.5864]\n"
     ]
    }
   ],
   "source": [
    "# CIFAR-10\n",
    "import keras.optimizers as optimizers\n",
    "for lr in [0.1, 0.01, 0.001, 0.0001]:\n",
    "    optimizer = optimizers.Adam(lr=lr)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITHOUT BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetC((32, 32, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH LAYER BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetA((32, 32, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)\n",
    "    \n",
    "    for i in range(3):\n",
    "        print(\"RESNET WITH IDENTITY BATCHNORM, lr =\", lr, \"ITER = \", i)\n",
    "        model = ConvNetB((32, 32, 3), y_train.shape[1])\n",
    "        model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\n",
    "        datagen = Feeder()\n",
    "        Train(model, datagen, x_train, y_train, x_val, y_val, epochs=20, batch_size=32)\n",
    "\n",
    "        scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "        print(\"Test\", scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}