Visión Artificial/Cuadernos/clasificacion_moderna.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<img src=\"../Imagenes/logo-final-ap.png\"  width=\"80\" height=\"80\" align=\"left\"/> \n",
    "</figure>\n",
    "\n",
    "# <span style=\"color:blue\"><left>Aprendizaje Profundo</left></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:red\"><center>Visión por Computadora</center></span>\n",
    "## <span style=\"color:red\"><center>Inception y DenseNet: Clasificación Moderna</center></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##   <span style=\"color:blue\">Profesores</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coordinador\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Campo Elías Pardo Turriago, cepardot@unal.edu.co "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conferencistas\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Alvaro  Montenegro, PhD, ammontenegrod@unal.edu.co\n",
    "- Daniel  Montenegro, Msc, dextronomo@gmail.com \n",
    "- Oleg Jarma, Estadístico, ojarmam@unal.edu.co "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Asesora Medios y Marketing digital</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Maria del Pilar Montenegro, pmontenegro88@gmail.com \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Asistentes</span>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Nayibe Yesenia Arias, naariasc@unal.edu.co\n",
    "- Venus Celeste Puertas, vpuertasg@unal.edu.co "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Introducción</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En el año 1989 Se propone la estructura LeNet5, considerada la primera red neuronal convolucional. \n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/LeNet.png\" width=\"700\" height=\"200\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>\n",
    "\n",
    "Este tipo de arquitecturas fueron \"muy avanzados\" para la época, siendo incapaces de encontrar buenos métodos de aprendizaje con la tecnología del momento."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En 2010 se inicia el \"ImageNet Large Scale visual Recognition Challenge\"(ILSVRC), un concurso para promover la mejora en las tareas de clasificación\n",
    "\n",
    "En 2012, en la tercera edición del reto, Apareció AlexNet, que trajo el diseño de LeNet al siglo XXI. Ganando por una diferencia considerable\n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/alexnet-vs-lenet.png\" width=\"600\" height=\"450\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En 2014, la arquitectura VGG construyó sobre alexnet, haciéndola más grande y más poderosa. Aquí nació el termino de \"bloques\", grupos de capas con una estructura determinada, logrando agregar más capas convolucionales. Aunque se puede decir que este tipo de red fue la que más inspiración ha generado, no fue la verdadera ganadora."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/vgg-vs-alexnet.png\" width=\"450\" height=\"600\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En 2015, buscando superar el número de capas de VGG, nacen los bloques residuales y ResNet, la cual [hemos hablado más a fondo](../Cuadernos/redes_residuales.ipynb)\n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/vgg-vs-resnet.png\" width=\"1100\" height=\"500\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Antes de cambiar de tema, es importante revisar ciertos modelos que no siguieron necesariamente la filosofía de AlexNet. Y al mismo tiempo, vamos a ver las estructuras que se construyeron a partir de ResNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">GoogLeNet y los bloques Inception</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Supongamos, por un momento, que estamos en 2014. Aún no tenemos los avances de VGG y ResNet. ¿Cuáles son los obstáculos a superar en las redes convolucionales profundas?\n",
    "\n",
    "Primero, para hacer un clasificador robusto, es necesario pensar en la variabilidad de tamaños de los objetos a encontrar. \n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/dogs_different_areas.png\" width=\"800\" height=\"300\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>\n",
    "\n",
    "Todos estos perros tienen que ser identificados, a pesar de las diferencias en posiciones y tamaños relativos a la imagen. Ya hemos visto técnicas como [MTCNN](../Cuadernos/rostros_cnn.ipynb) y la \"pirámide de imágenes\", pero ¿Qué tal si, en lugar de variar el tamaño de la imagen, variaramos los tamaños de los kernel?\n",
    "\n",
    "Segundo, Según se ha visto en la teoría, agregar más capas a la red puede causar sobreajuste. Así que, qué tal si aplicaramos capas en el mismo nivel, de manera simultanea?\n",
    "\n",
    "Por último, trabajar con múltiples canales (como son imágenes a color) y grandes tamaños de Kernel es computacionalmente costo. ¿Habrá alguna forma de resumir esta información?\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:blue\">Inception ingenuo</span>\n",
    "\n",
    "Enfoquémonos en los dos primeros obstáculos.\n",
    "\n",
    "La proposición inicial de los bloques Inception era (recordando el nombre) \"Redes dentro de Redes\".\n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/naive_inception.png\" width=\"800\" height=\"400\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Como dijimos anteriormente, en lugar de obtener diferentes tamaños de la imagen, para que un mismo kernel encuentre diferentes tamaños en esta, vamos a pasar la misma imagen sobre kernel de distintos tamaños, de tal forma que se encuentren los objetivos sin importar el tamaño. Luego se concatenan los resultados, uniendo todo lo aprendido.\n",
    "\n",
    "Todo esto sucede al mismo tiempo, Necesitando relativamente menos capas de manera secuencial, evitando el sobreajuste"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:blue\">Inception con reducción de dimensiones</span>\n",
    "\n",
    "A pesar de tener \"menos\" capas, aún estamos aplicando varios procesos computacionales. De hecho, podemos pensar que el costo es mayor debido a la cantidad de procesos simultáneos. Así que es necesario reducir esto.\n",
    "\n",
    "La forma que encontraron de lograr esto es a partir de la reducción de dimensiones con kernel 1x1.\n",
    "\n",
    "En los caminos de kernel 3x3 y 5x5, se aplica con anterioridad este pequeño Kernel. Se pensaría que esto no reduciría nada, ya que se está haciendo un análisis pixel por pixel, pero lo que hacemos es enfocarnos en una dimensión diferente. Usualmente, la red busca la información general sobre regiones de la imagen sobre cada canal por separado. En este caso, buscamos la información de la imagen sobre todos los canales a la vez. \n",
    " Así los datos a procesar por los kernel es, por lo menos, 3 veces menor. \n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/full_inception.png\" width=\"850\" height=\"400\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GoogLeNet no es más que varios de estos bloques conectados, con average pooling al final.\n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/googlenet.png\" width=\"1200\" height=\"400\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vamos a hacer una implementación desde 0. (PELIGRO, CORRER ÚNICAMENTE EN COLAB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import random\n",
    "from types import SimpleNamespace\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.utils.data as data\n",
    "import torch.optim as optim\n",
    "\n",
    "import torchvision\n",
    "from torchvision import models\n",
    "from torchvision.datasets import CIFAR10\n",
    "from torchvision import transforms\n",
    "\n",
    "device = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device(\"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib.request\n",
    "from urllib.error import HTTPError\n",
    "# URL de github donde están los modelos pre entrenados\n",
    "base_url = \"https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/\"\n",
    "# Archivos a descargar\n",
    "pretrained_files = [\"GoogleNet.ckpt\", \"DenseNet.ckpt\"]\n",
    "# Crear directorio de checkopoints si no existe\n",
    "os.makedirs('../modelos', exist_ok=True)\n",
    "\n",
    "# Revisar si cada archivo ya existe, y descargarlo si no\n",
    "for file_name in pretrained_files:\n",
    "    file_path = os.path.join('../modelos', file_name)\n",
    "    if \"/\" in file_name:\n",
    "        os.makedirs(file_path.rsplit(\"/\",1)[0], exist_ok=True)\n",
    "    if not os.path.isfile(file_path):\n",
    "        file_url = base_url + file_name\n",
    "        print(f\"Descargando {file_url}...\")\n",
    "        try:\n",
    "            urllib.request.urlretrieve(file_url, file_path)\n",
    "        except HTTPError as e:\n",
    "            print(\"Errores encontrados durante la descarga:\\n\", e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "transform = transforms.Compose([\n",
    "    transforms.Resize(256),\n",
    "    transforms.CenterCrop(224),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n",
    "])\n",
    "\n",
    "train_dataset = CIFAR10(root='/mnt/storage/Datasets', transform=transform, train=True, download=True)\n",
    "test_dataset = CIFAR10(root='/mnt/storage/Datasets', transform=transform, train=False, download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set, val_set = torch.utils.data.random_split(train_dataset, [45000, 5000])\n",
    "\n",
    "train_loader = data.DataLoader(train_set, batch_size=128, shuffle=True, pin_memory=True, num_workers=4)\n",
    "val_loader = data.DataLoader(val_set, batch_size=128, shuffle=False, num_workers=4)\n",
    "test_loader = data.DataLoader(test_dataset, batch_size=128, shuffle=False, num_workers=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 13\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pytorch_lightning as pl\n",
    "\n",
    "pl.seed_everything(13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CIFARModule(pl.LightningModule):\n",
    "    \n",
    "    def __init__(self, model_name, model_hparams, optimizer_name, optimizer_hparams):\n",
    "        \n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        self.model = create_model(model_name, model_hparams)\n",
    "        self.loss_module = nn.CrossEntropyLoss()\n",
    "        \n",
    "    def forward(self, imgs):\n",
    "        return self.model(imgs)\n",
    "    \n",
    "    def configure_optimizers(self):\n",
    "        if self.hparams.optimizer_name == 'Adam':\n",
    "            optimizer = optim.AdamW(\n",
    "                self.parameters(), **self.hparams.optimizer_hparams)\n",
    "        elif self.hparams.optimizer_name == \"SGD\":\n",
    "            optimizer = optim.SGD(self.parameters(), **self.hparams.optimizer_hparams)\n",
    "        else:\n",
    "            assert False, f\"Optimizador desconocido: \\\"{self.hparams.optimizer_name}\\\"\" \n",
    "            \n",
    "        scheduler = optim.lr_scheduler.MultiStepLR(\n",
    "            optimizer, milestones=[100, 150], gamma=0.1)\n",
    "        return [optimizer], [scheduler]\n",
    "    \n",
    "    def training_step(self, batch, batch_idx):\n",
    "        # \"batch\" es la salida del dataloader de entrenamiento.\n",
    "        imgs, labels = batch\n",
    "        preds = self.model(imgs)\n",
    "        loss = self.loss_module(preds, labels)\n",
    "        acc = (preds.argmax(dim=-1) == labels).float().mean()\n",
    "\n",
    "        self.log('train_acc', acc, on_step=False, on_epoch=True)\n",
    "        self.log('train_loss', loss)\n",
    "        return loss\n",
    "    \n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        imgs, labels = batch\n",
    "        preds = self.model(imgs).argmax(dim=-1)\n",
    "        acc = (labels == preds).float().mean()\n",
    "        self.log('val_acc', acc)\n",
    "        \n",
    "    def test_step(self, batch, batch_idx):\n",
    "        imgs, labels = batch\n",
    "        preds = self.model(imgs).argmax(dim=-1)\n",
    "        acc = (labels == preds).float().mean()\n",
    "        self.log('test_acc', acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint\n",
    "\n",
    "model_dict = {}\n",
    "\n",
    "def create_model(model_name, model_hparams):\n",
    "    if model_name in model_dict:\n",
    "        return model_dict[model_name](**model_hparams)\n",
    "    else:\n",
    "        assert False, f\"Modelo desconocido \\\"{model_name}\\\". Los modelos disponibles son: {str(model_dict.keys())}\"\n",
    "        \n",
    "act_fn_by_name = {\n",
    "    \"tanh\": nn.Tanh,\n",
    "    \"relu\": nn.ReLU,\n",
    "    \"leakyrelu\": nn.LeakyReLU,\n",
    "    \"gelu\": nn.GELU\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model_name, save_name=None, **kwargs):\n",
    "\n",
    "    if save_name is None:\n",
    "        save_name = model_name\n",
    "\n",
    "    # Se crea el entrenador de lightning\n",
    "    trainer = pl.Trainer(default_root_dir=os.path.join('../modelos/', save_name),                            # Donde se guardan los modelos\n",
    "                         gpus=1 if str(device)==\"cuda:0\" else 0,                                             # Corremos en una sola GPU(si es posible)\n",
    "                         max_epochs=180,                                                                     # número de epocas a entrenar si no hay un límite de paciencia\n",
    "                         callbacks=[ModelCheckpoint(save_weights_only=True, mode=\"max\", monitor=\"val_acc\"),  # Se guarda el mejor checkpoint basado en la precisión de validación. Se guardan los pesos\n",
    "                                    LearningRateMonitor(\"epoch\")])                                           # Se crean logs de la rata de aprendizaje\n",
    "\n",
    "    # Se revisa si existe un modelo pre entrenado existe. En caso de si, se salta el entrenamiento\n",
    "    pretrained_filename = os.path.join('../modelos/', save_name + \".ckpt\")\n",
    "    if os.path.isfile(pretrained_filename):\n",
    "        print(f\"Modelo encontrado en {pretrained_filename}, cargando...\")\n",
    "        model = CIFARModule.load_from_checkpoint(pretrained_filename) # Se carga el modelo con los hiperparámetros\n",
    "    else:\n",
    "        pl.seed_everything(42) # Para mantener reproductibilidad\n",
    "        model = CIFARModule(model_name=model_name, **kwargs)\n",
    "        trainer.fit(model, train_loader, val_loader)\n",
    "        model = CIFARModule.load_from_checkpoint(trainer.checkpoint_callback.best_model_path) # Se carga el mejor checkpoint después del entrenamiento\n",
    "\n",
    "    # Se revisa el mejor modelo en los datos de validación y prueba\n",
    "    val_result = trainer.test(model, test_dataloaders=val_loader, verbose=False)\n",
    "    test_result = trainer.test(model, test_dataloaders=test_loader, verbose=False)\n",
    "    result = {\"test\": test_result[0][\"test_acc\"], \"val\": val_result[0][\"test_acc\"]}\n",
    "\n",
    "    return model, result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class InceptionBlock(nn.Module):\n",
    "\n",
    "    def __init__(self, c_in, c_red : dict, c_out : dict, act_fn):\n",
    "        \"\"\"\n",
    "        Inputs:\n",
    "            c_in - número de mapas de características de las capas anteriores\n",
    "            c_red - diccionario con llaves \"3x3\" and \"5x5\" que especifican la salida de los kernel 1x1\n",
    "            c_out - diccionario con llaves \"1x1\", \"3x3\", \"5x5\", y \"max\"\n",
    "            act_fn - función de activación a usar (e.g. nn.ReLU)\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "\n",
    "        # rama de convolución 1x1 \n",
    "        self.conv_1x1 = nn.Sequential(\n",
    "            nn.Conv2d(c_in, c_out[\"1x1\"], kernel_size=1),\n",
    "            nn.BatchNorm2d(c_out[\"1x1\"]),\n",
    "            act_fn()\n",
    "        )\n",
    "\n",
    "        # rama de convolución 3x3\n",
    "        self.conv_3x3 = nn.Sequential(\n",
    "            nn.Conv2d(c_in, c_red[\"3x3\"], kernel_size=1),\n",
    "            nn.BatchNorm2d(c_red[\"3x3\"]),\n",
    "            act_fn(),\n",
    "            nn.Conv2d(c_red[\"3x3\"], c_out[\"3x3\"], kernel_size=3, padding=1),\n",
    "            nn.BatchNorm2d(c_out[\"3x3\"]),\n",
    "            act_fn()\n",
    "        )\n",
    "\n",
    "        # rama de convolución 5x5\n",
    "        self.conv_5x5 = nn.Sequential(\n",
    "            nn.Conv2d(c_in, c_red[\"5x5\"], kernel_size=1),\n",
    "            nn.BatchNorm2d(c_red[\"5x5\"]),\n",
    "            act_fn(),\n",
    "            nn.Conv2d(c_red[\"5x5\"], c_out[\"5x5\"], kernel_size=5, padding=2),\n",
    "            nn.BatchNorm2d(c_out[\"5x5\"]),\n",
    "            act_fn()\n",
    "        )\n",
    "\n",
    "        # rama Max-pool\n",
    "        self.max_pool = nn.Sequential(\n",
    "            nn.MaxPool2d(kernel_size=3, padding=1, stride=1),\n",
    "            nn.Conv2d(c_in, c_out[\"max\"], kernel_size=1),\n",
    "            nn.BatchNorm2d(c_out[\"max\"]),\n",
    "            act_fn()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x_1x1 = self.conv_1x1(x)\n",
    "        x_3x3 = self.conv_3x3(x)\n",
    "        x_5x5 = self.conv_5x5(x)\n",
    "        x_max = self.max_pool(x)\n",
    "        x_out = torch.cat([x_1x1, x_3x3, x_5x5, x_max], dim=1)\n",
    "        return x_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GoogleNet(nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes=10, act_fn_name=\"relu\", **kwargs):\n",
    "        super(GoogleNet, self).__init__()\n",
    "        self.hparams = SimpleNamespace(num_classes=num_classes,\n",
    "                                       act_fn_name=act_fn_name,\n",
    "                                       act_fn=act_fn_by_name[act_fn_name])\n",
    "        self._create_network()\n",
    "        self._init_params()\n",
    "\n",
    "    def _create_network(self):\n",
    "        #Primera convolución de la imagen original para escalar al tamaño del canal\n",
    "        self.input_net = nn.Sequential(\n",
    "            nn.Conv2d(3, 64, kernel_size=3, padding=1),\n",
    "            nn.BatchNorm2d(64),\n",
    "            self.hparams.act_fn()\n",
    "        )\n",
    "        # Se apilan los bloques inception\n",
    "        self.inception_blocks = nn.Sequential(\n",
    "            InceptionBlock(64, c_red={\"3x3\": 32, \"5x5\": 16}, c_out={\"1x1\": 16, \"3x3\": 32, \"5x5\": 8, \"max\": 8}, act_fn=self.hparams.act_fn),\n",
    "            InceptionBlock(64, c_red={\"3x3\": 32, \"5x5\": 16}, c_out={\"1x1\": 24, \"3x3\": 48, \"5x5\": 12, \"max\": 12}, act_fn=self.hparams.act_fn),\n",
    "            nn.MaxPool2d(3, stride=2, padding=1),  # 32x32 => 16x16\n",
    "            InceptionBlock(96, c_red={\"3x3\": 32, \"5x5\": 16}, c_out={\"1x1\": 24, \"3x3\": 48, \"5x5\": 12, \"max\": 12}, act_fn=self.hparams.act_fn),\n",
    "            InceptionBlock(96, c_red={\"3x3\": 32, \"5x5\": 16}, c_out={\"1x1\": 16, \"3x3\": 48, \"5x5\": 16, \"max\": 16}, act_fn=self.hparams.act_fn),\n",
    "            InceptionBlock(96, c_red={\"3x3\": 32, \"5x5\": 16}, c_out={\"1x1\": 16, \"3x3\": 48, \"5x5\": 16, \"max\": 16}, act_fn=self.hparams.act_fn),\n",
    "            InceptionBlock(96, c_red={\"3x3\": 32, \"5x5\": 16}, c_out={\"1x1\": 32, \"3x3\": 48, \"5x5\": 24, \"max\": 24}, act_fn=self.hparams.act_fn),\n",
    "            nn.MaxPool2d(3, stride=2, padding=1),  # 16x16 => 8x8\n",
    "            InceptionBlock(128, c_red={\"3x3\": 48, \"5x5\": 16}, c_out={\"1x1\": 32, \"3x3\": 64, \"5x5\": 16, \"max\": 16}, act_fn=self.hparams.act_fn),\n",
    "            InceptionBlock(128, c_red={\"3x3\": 48, \"5x5\": 16}, c_out={\"1x1\": 32, \"3x3\": 64, \"5x5\": 16, \"max\": 16}, act_fn=self.hparams.act_fn)\n",
    "        )\n",
    "        # sección completamente conectada\n",
    "        self.output_net = nn.Sequential(\n",
    "            nn.AdaptiveAvgPool2d((1, 1)),\n",
    "            nn.Flatten(),\n",
    "            nn.Linear(128, self.hparams.num_classes)\n",
    "        )\n",
    "\n",
    "    def _init_params(self):\n",
    "        for m in self.modules():\n",
    "            if isinstance(m, nn.Conv2d):\n",
    "                nn.init.kaiming_normal_(\n",
    "                    m.weight, nonlinearity=self.hparams.act_fn_name)\n",
    "            elif isinstance(m, nn.BatchNorm2d):\n",
    "                nn.init.constant_(m.weight, 1)\n",
    "                nn.init.constant_(m.bias, 0)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.input_net(x)\n",
    "        x = self.inception_blocks(x)\n",
    "        x = self.output_net(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_dict[\"GoogleNet\"] = GoogleNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "googlenet_model, googlenet_results = train_model(model_name=\"GoogleNet\",\n",
    "                                                 model_hparams={\"num_classes\": 10,\n",
    "                                                                \"act_fn_name\": \"relu\"},\n",
    "                                                 optimizer_name=\"Adam\",\n",
    "                                                 optimizer_hparams={\"lr\": 1e-3,\n",
    "                                                                    \"weight_decay\": 1e-4})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Resultados GoogleNet\", googlenet_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Densenet</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recordando las redes residuales, un bloque de esta red da una especie de pista para no tener que encontrar la mejor función y parámetros de la nada. Dando en cierta forma la capacidad de \"comparar\" con los resultados inmediatamente anteriores.\n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/residual-block.svg\" width=\"600\" height=\"400\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>\n",
    "\n",
    "Estas redes dieron una enseñanza importante: Podemos obtener redes más profundas, con mayor precisión y menor tiempo de entrenamiento si logramos hacer pequeñas conexiones entre las capas cercanas a la entrada y la salida. La evolución lógica de esta enseñanza es: Hagan más conexiones, entre más mejor, CONECTENLO TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:blue\">Bloque Denso</span>\n",
    "\n",
    "En el bloque residual, se utiliza una suma entre los valores originales de entrada y los datos procesados por las capas convolucionales. Esto se hace para mejorar la convergencia de un bloque al otro.\n",
    "\n",
    "En este nuevo caso, no queremos \"sumar\" la información, simplemente capturarlos. Aquí, todos los valores de entrada del bloque serán \"concatenados\", aumentando el número de características. Todo este proceso es \"recursivo\", así que las concatenaciones de las capas anteriores serán agregadas a la siguiente. Tendremos cada vez más y más mapas de características a medida que se procesan los bloques.\n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/densenet.gif\" width=\"600\" height=\"200\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>\n",
    "\n",
    "Gracias a esto necesitamos menos canales en cada capa. Definimos un parametro \"k\", que será el número de canales por capa. Vamos a llamarlo la \"Rata de crecimiento\".\n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/growth_rate.png\" width=\"600\" height=\"300\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:blue\">La arquitectura DenseNet</span>\n",
    "\n",
    "- Capa de transición: \n",
    "    - Normalización por lotes\n",
    "    - Activación ReLU\n",
    "    - convolución con kernel 1x1\n",
    "- Capa de composición:\n",
    "    - Normalización por lotes\n",
    "    - Activación ReLU\n",
    "    - convolución con kernel 3x3\n",
    "\n",
    "\n",
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/full_densenet.png\" width=\"800\" height=\"300\" align=\"center\"/> \n",
    "</center>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseLayer(nn.Module):\n",
    "\n",
    "    def __init__(self, c_in, bn_size, growth_rate, act_fn):\n",
    "        \"\"\"\n",
    "        Inputs:\n",
    "            c_in - Número de canales de entrada\n",
    "            bn_size - Tamaño de la boca de botella (factor de Rata de crecimiento) para el kernel 1x1. Usualmente entre 2 y 4\n",
    "            growth_rate - número de canales para los kernel 3x3\n",
    "            act_fn - función de activación (e.g. nn.ReLU)\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(\n",
    "            nn.BatchNorm2d(c_in),\n",
    "            act_fn(),\n",
    "            nn.Conv2d(c_in, bn_size * growth_rate, kernel_size=1, bias=False),\n",
    "            nn.BatchNorm2d(bn_size * growth_rate),\n",
    "            act_fn(),\n",
    "            nn.Conv2d(bn_size * growth_rate, growth_rate, kernel_size=3, padding=1, bias=False)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.net(x)\n",
    "        out = torch.cat([out, x], dim=1)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseBlock(nn.Module):\n",
    "\n",
    "    def __init__(self, c_in, num_layers, bn_size, growth_rate, act_fn):\n",
    "        \"\"\"\n",
    "        Inputs:\n",
    "            c_in - Número de canales de entrada\n",
    "            num_layers - Número de capas Densas dentro del bloque\n",
    "            bn_size - Tamaño de la boca de botella para usar en las capas densas\n",
    "            growth_rate - Rata de crecimiento dentro de las capas densas\n",
    "            act_fn - Función de activación\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        layers = []\n",
    "        for layer_idx in range(num_layers):\n",
    "            layers.append(\n",
    "                DenseLayer(c_in=c_in + layer_idx * growth_rate, # Los canales de entrada son los originales y los mapas de características anteriores \n",
    "                           bn_size=bn_size,\n",
    "                           growth_rate=growth_rate,\n",
    "                           act_fn=act_fn)\n",
    "            )\n",
    "        self.block = nn.Sequential(*layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.block(x)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransitionLayer(nn.Module):\n",
    "\n",
    "    def __init__(self, c_in, c_out, act_fn):\n",
    "        super().__init__()\n",
    "        self.transition = nn.Sequential(\n",
    "            nn.BatchNorm2d(c_in),\n",
    "            act_fn(),\n",
    "            nn.Conv2d(c_in, c_out, kernel_size=1, bias=False),\n",
    "            nn.AvgPool2d(kernel_size=2, stride=2) \n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.transition(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseNet(nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes=10, num_layers=[6,6,6,6], bn_size=2, growth_rate=16, act_fn_name=\"relu\", **kwargs):\n",
    "        super().__init__()\n",
    "        self.hparams = SimpleNamespace(num_classes=num_classes,\n",
    "                                       num_layers=num_layers,\n",
    "                                       bn_size=bn_size,\n",
    "                                       growth_rate=growth_rate,\n",
    "                                       act_fn_name=act_fn_name,\n",
    "                                       act_fn=act_fn_by_name[act_fn_name])\n",
    "        self._create_network()\n",
    "        self._init_params()\n",
    "\n",
    "    def _create_network(self):\n",
    "        c_hidden = self.hparams.growth_rate * self.hparams.bn_size # The start number of hidden channels\n",
    "\n",
    "        # primera convolución para cambiar el tamaño\n",
    "        self.input_net = nn.Sequential(\n",
    "            nn.Conv2d(3, c_hidden, kernel_size=3, padding=1)  # no se hace normalización de lotes ni función de activación\n",
    "           \n",
    "        )\n",
    "\n",
    "        # Se crean los bloques densos, eventualmente agregando las capas de transición\n",
    "        blocks = []\n",
    "        for block_idx, num_layers in enumerate(self.hparams.num_layers):\n",
    "            blocks.append(\n",
    "                DenseBlock(c_in=c_hidden,\n",
    "                           num_layers=num_layers,\n",
    "                           bn_size=self.hparams.bn_size,\n",
    "                           growth_rate=self.hparams.growth_rate,\n",
    "                           act_fn=self.hparams.act_fn)\n",
    "            )\n",
    "            c_hidden = c_hidden + num_layers * self.hparams.growth_rate #salida general de los bloques densos\n",
    "            if block_idx < len(self.hparams.num_layers)-1: #  No se aplica transición en el último bloque\n",
    "                blocks.append(\n",
    "                    TransitionLayer(c_in=c_hidden,\n",
    "                                    c_out=c_hidden // 2,\n",
    "                                    act_fn=self.hparams.act_fn))\n",
    "                c_hidden = c_hidden // 2\n",
    "\n",
    "        self.blocks = nn.Sequential(*blocks)\n",
    "\n",
    "        # Sección completamente conectada\n",
    "        self.output_net = nn.Sequential(\n",
    "            nn.BatchNorm2d(c_hidden), \n",
    "            self.hparams.act_fn(),\n",
    "            nn.AdaptiveAvgPool2d((1,1)),\n",
    "            nn.Flatten(),\n",
    "            nn.Linear(c_hidden, self.hparams.num_classes)\n",
    "        )\n",
    "\n",
    "    def _init_params(self):\n",
    "        for m in self.modules():\n",
    "            if isinstance(m, nn.Conv2d):\n",
    "                nn.init.kaiming_normal_(m.weight, nonlinearity=self.hparams.act_fn_name)\n",
    "            elif isinstance(m, nn.BatchNorm2d):\n",
    "                nn.init.constant_(m.weight, 1)\n",
    "                nn.init.constant_(m.bias, 0)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.input_net(x)\n",
    "        x = self.blocks(x)\n",
    "        x = self.output_net(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_dict[\"DenseNet\"] = DenseNet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "densenet_model, densenet_results = train_model(model_name=\"DenseNet\",\n",
    "                                               model_hparams={\"num_classes\": 10,\n",
    "                                                              \"num_layers\": [6,6,6,6],\n",
    "                                                              \"bn_size\": 2,\n",
    "                                                              \"growth_rate\": 16,\n",
    "                                                              \"act_fn_name\": \"relu\"},\n",
    "                                               optimizer_name=\"Adam\",\n",
    "                                               optimizer_hparams={\"lr\": 1e-3,\n",
    "                                                                  \"weight_decay\": 1e-4})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "3a35811627f2d849a94a681c597b35b034065b47c984a0e4ef87a04e993e3cd2"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}