4-HPC-Simulations/03-hpc-simulations-pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2gMnY4ESvfEA"
   },
   "source": [
    "# HPC as a solution for AI: PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7wHXE1ij1bIA"
   },
   "source": [
    "<p style='text-align: justify;'>\n",
    "In this section, it will be shown how to optimize PyTorch models, accelerating training and execution using GPUs.\n",
    "</p>    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2FfzlD24wAC0"
   },
   "source": [
    "## Objectives\n",
    "* **Use** the PyTorch library on GPU environments for the first time to accelerate the training of image classification models,\n",
    "* **Familiarize** yourself with the CIFAR-10 and CIFAR-100 dataset by classifying their various classes,\n",
    "* **Evaluate** and **Compare** the performance of your models on GPU and CPU environments to understand the benefits of GPU acceleration in AI tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fmwBTXpn1bIA"
   },
   "source": [
    "## The problem: Resource-intensive training and model scalability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vP0wr4R51bIA"
   },
   "source": [
    "<p style='text-align: justify;'>\n",
    "As AI research progresses, deep neural networks have become critical for tasks like image generation and language translation. However, resource-intensive training challenges arise as networks become more complex and demanding in performance.\n",
    "</p>\n",
    "\n",
    "<p style='text-align: justify;'>\n",
    "Research and development in artificial intelligence have made remarkable strides in recent decades, mainly driven by deep neural networks. These networks are computational structures loosely inspired by the functioning of the human brain. They are particularly well-suited for tasks that involve large volumes of data, such as pattern recognition in images, natural language processing, and more.\n",
    "</p>\n",
    "\n",
    "<p style='text-align: justify;'>\n",
    "However, as the problems being addressed become more complex and performance demands increase, the need for computational resources also grows exponentially. Additionally, the scalability of these models becomes a concern as they grow in size and complexity. Maintaining and optimizing constantly expanding AI models becomes challenging for the research and development community.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OJ4Teb2m1bIA"
   },
   "source": [
    "## The solution: GPUs and PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ST3ronAy1bIA"
   },
   "source": [
    "<p style='text-align: justify;'>\n",
    "Using libraries like PyTorch, a popular machine learning and AI framework, offers a flexible interface for designing, training, and evaluating neural networks using GPUs, especially when harnessed with computational prowess.\n",
    "</p>\n",
    "<p style='text-align: justify;'>\n",
    "Furthermore, Intel® PyTorch is well equipped to fully utilize the optimizations and hardware support of Intel® processors and GPUs. This synergy results in an even more efficient and performance-oriented machine-learning experience. It enables practitioners to extract maximum computational throughput from their hardware infrastructure.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BMvo8TG2WWZE"
   },
   "source": [
    "##  ☆ Challenge: Zoo breakout!☆"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hn3hps3AwAC1"
   },
   "source": [
    "<p style='text-align: justify;'>\n",
    "    Recently, an unexpected incident occurred at the local zoo, <b>Orange Grove Zoo</b>: all the animals escaped from their enclosures and are now roaming freely. To deal with this situation, we need your help locating and classifying the escaped animals, distinguishing each animal class, and identifying possible vehicles in the same environment.\n",
    "</p>\n",
    "<p style='text-align: justify;'>\n",
    "You have been assigned as the person responsible for developing a computer vision system capable of identifying and classifying the escaped animals and identifying the presence of vehicles in the images. We will use the CIFAR-10 dataset and the TensorFlow library to train a deep-learning model for this challenge.\n",
    "</p>\n",
    "CIFAR-10 and CIFAR-100 datasets comprehensively collect $32$x$32$ pixel images grouped into $10$ and $100$ distinct classes, respectively.\n",
    "\n",
    "- [CIFAR-10 Dataset](https://www.cs.toronto.edu/~kriz/cifar.html): CIFAR-10 consists of $60,000$ images, each belonging to one of the ten classes: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. This dataset offers a diverse set of images representing everyday objects.\n",
    "\n",
    "- [CIFAR-100 Dataset](https://www.cs.toronto.edu/~kriz/cifar.html): CIFAR-100 expands upon the CIFAR-10 concept. However, it introduces a more challenging task by categorizing images into $100$ classes. These classes include various subcategories such as fruits, animals, vehicles, and more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X9pJ2tDOCaN6"
   },
   "source": [
    "a) **Create** deep neural network model utilizing the PyTorch library for the classification of animals and vehicles on a GPU environment using the CIFAR-10 dataset.\n",
    "\n",
    "b) **Conduct** a comparative analysis between models trained on a CPU and GPU to highlight disparities in results.\n",
    "\n",
    "c) Now, use the CIFAR-100 dataset for the classification of animals and vehicles on a GPU. Would it be a good decision to use a GPU or CPU environment for the training process?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n0vnSMPdIx2c"
   },
   "source": [
    "### ☆ Solution for `CIFAR-10` using PyTorch on CPU☆"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t9Ud6RMBwAC2"
   },
   "source": [
    "#### ⊗ Importing packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "yZjcPbwQwAC2"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ⊗ Define processing device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "zfgQLNaRI7Hu"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cpu:0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tTr-T8mYJG_9"
   },
   "source": [
    "#### ⊗ Transformations to the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2915d780-1d61-4f6a-9ec4-154e64ac4896"
   },
   "source": [
    "<p style='text-align: justify;'>\n",
    "    As part of the data preparation process, we create a <b>transforms</b> object to apply specific transformations to the data. These transformations are commonly used in training datasets to enhance data diversity and ready images for utilization in a deep learning model, such as a convolutional neural network (CNN).\n",
    "    </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "210c3993-fedc-4018-88b8-48dc30df64da"
   },
   "outputs": [],
   "source": [
    "transform = transforms.Compose([\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomCrop(32, padding=4),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "131d6960-4abe-48b3-9167-3bd9454e037d"
   },
   "source": [
    "#### ⊗ Downloading the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dc889d5f-3ace-4c72-bf7c-5ec768c8bcc1"
   },
   "source": [
    "<p style='text-align: justify;'>\n",
    "Following that, download the CIFAR-10 dataset and load it into the code. Define the neural network as we have done in previous notebooks, and remember to move this network instance to the previously defined device.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d89db0f4-fd04-4b3e-a59e-79220d531953",
    "outputId": "b9ec4363-afc1-4f35-dc94-ddb29b11f6dd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, num_workers=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9be0375d-d0f8-41f0-9530-5459d2c125f4"
   },
   "source": [
    "#### ⊗ Creating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ee0c9319-c3a3-4895-9c60-58ae53e2f6ff"
   },
   "source": [
    "Now it is necessary to create the model for our neural network using PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9295417d-ad2f-421c-b053-16f5b4db5b5d",
    "outputId": "7b713c9c-0138-489f-d6da-5ae2c5fb7c02"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Net(\n",
       "  (conv1): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (conv2): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (fc1): Linear(in_features=8192, out_features=512, bias=True)\n",
       "  (fc2): Linear(in_features=512, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)\n",
    "        self.conv2 = nn.Conv2d(64, 128, kernel_size=3, padding=1)\n",
    "        self.fc1 = nn.Linear(128 * 8 * 8, 512)\n",
    "        self.fc2 = nn.Linear(512, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = torch.relu(self.conv1(x))\n",
    "        x = torch.max_pool2d(x, 2)\n",
    "        x = torch.relu(self.conv2(x))\n",
    "        x = torch.max_pool2d(x, 2)\n",
    "        x = x.view(-1, 128 * 8 * 8)\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "net = Net()\n",
    "net.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cb622a0f-9d02-4d7c-aaf5-572e4529ebed"
   },
   "source": [
    "#### ⊗ Training the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "99d4fa56-591a-418b-b52e-ff58d543565c"
   },
   "source": [
    "Now we will train our neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "be67ba50-0e8d-435f-ada7-b0a98b91aa8a",
    "outputId": "0da42ee2-faf8-4fbc-a7d0-7820bdc8626c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1, Loss: 1.7964948424902718\n",
      "Epoch 2, Loss: 1.417594108740082\n",
      "Epoch 3, Loss: 1.2418243392654087\n",
      "Epoch 4, Loss: 1.1093364748198662\n",
      "Epoch 5, Loss: 1.0149216093980442\n",
      "Epoch 6, Loss: 0.9433749182449888\n",
      "Epoch 7, Loss: 0.8874704087786662\n",
      "Epoch 8, Loss: 0.8377322353365476\n",
      "Epoch 9, Loss: 0.8021479407539758\n",
      "Epoch 10, Loss: 0.7553412200849684\n",
      "\n",
      "CPU Training time: 462.89 seconds)\n"
     ]
    }
   ],
   "source": [
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)\n",
    "\n",
    "device = torch.device(\"cpu\")\n",
    "net.to(device)\n",
    "\n",
    "cpu_start_time = time.time()\n",
    "\n",
    "for epoch in range(10):\n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        inputs, labels = data\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        optimizer.zero_grad()\n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        running_loss += loss.item()\n",
    "\n",
    "    print(f'Epoch {epoch+1}, Loss: {running_loss / len(trainloader)}')\n",
    "\n",
    "cpu_end_time = time.time()\n",
    "\n",
    "total_cpu_time_cifar_10 = cpu_end_time - cpu_start_time\n",
    "\n",
    "print(f\"\\nCPU Training time: {total_cpu_time_cifar_10:.2f} seconds)\")\n",
    "\n",
    "torch.save(net.state_dict(), 'cifar10_cpu_model.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8_Ch1ZPO1bIC"
   },
   "source": [
    "### ☆  Solution for `CIFAR-10` using PyTorch on GPU  ☆"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t67WI7bSwAC3"
   },
   "source": [
    "#### ⊗ Transformations to the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "VqrPy46pwAC3"
   },
   "outputs": [],
   "source": [
    "transform = transforms.Compose([\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomCrop(32, padding=4),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DidyB70kwAC4",
    "outputId": "cdfe72e8-a2fd-4bf7-8f09-b24bb0df0563"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Net(\n",
       "  (conv1): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (conv2): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
       "  (fc1): Linear(in_features=8192, out_features=512, bias=True)\n",
       "  (fc2): Linear(in_features=512, out_features=10, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, num_workers=2)\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)\n",
    "        self.conv2 = nn.Conv2d(64, 128, kernel_size=3, padding=1)\n",
    "        self.fc1 = nn.Linear(128 * 8 * 8, 512)\n",
    "        self.fc2 = nn.Linear(512, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = torch.relu(self.conv1(x))\n",
    "        x = torch.max_pool2d(x, 2)\n",
    "        x = torch.relu(self.conv2(x))\n",
    "        x = torch.max_pool2d(x, 2)\n",
    "        x = x.view(-1, 128 * 8 * 8)\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "net = Net()\n",
    "net.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tN6p-GqvwAC4"
   },
   "source": [
    "#### ⊗ Training the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NPhkUlBuwAC4",
    "outputId": "04acf14d-7a1f-443f-d3dd-59818527e7c9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1, Loss: 1.7745253969641293\n",
      "Epoch 2, Loss: 1.4122941140323648\n",
      "Epoch 3, Loss: 1.239821930827997\n",
      "Epoch 4, Loss: 1.10916560552919\n",
      "Epoch 5, Loss: 1.0204177968337407\n",
      "Epoch 6, Loss: 0.9522925040606037\n",
      "Epoch 7, Loss: 0.8932674217711934\n",
      "Epoch 8, Loss: 0.8396686213400663\n",
      "Epoch 9, Loss: 0.7989676483451863\n",
      "Epoch 10, Loss: 0.7566478979557066\n",
      "GPU Training time: 54.03 seconds\n"
     ]
    }
   ],
   "source": [
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)\n",
    "\n",
    "gpu_start_time = time.time()\n",
    "\n",
    "for epoch in range(10):\n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        inputs, labels = data\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        optimizer.zero_grad()\n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        running_loss += loss.item()\n",
    "    print(f'Epoch {epoch+1}, Loss: {running_loss / len(trainloader)}')\n",
    "\n",
    "gpu_end_time = time.time()\n",
    "total_gpu_cifar_10 = gpu_end_time - gpu_start_time\n",
    "\n",
    "print(f'GPU Training time: {total_gpu_cifar_10:.2f} seconds')\n",
    "\n",
    "torch.save(net.state_dict(), 'cifar10_gpu_model.pth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Nf08ji33Msa0",
    "outputId": "f0b73eb0-8f0d-463a-c359-012f222d8187"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Speedup: 8.57X\n"
     ]
    }
   ],
   "source": [
    "print(f\"\\nSpeedup:{total_cpu_time_cifar_10 / total_gpu_cifar_10 : .2f}X\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ☆ Solution for `CIFAR-100` using PyTorch on CPU☆"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g8zX7lXBwAC5"
   },
   "source": [
    "<p style='text-align: justify;'>\n",
    "Utilizing the CIFAR-100 dataset for the classification of animals and vehicles is a significantly more computationally demanding task compared to CIFAR-10 training. This heightened computational demand arises from the larger number of images and classes present in CIFAR-100. Additionally, training a deep neural network for this purpose typically necessitates a higher number of epochs to ensure effective model training.\n",
    "</p>\n",
    "\n",
    "<p style='text-align: justify;'>\n",
    "Let's repeat the process on the journey of training a new model for the CIFAR-100 dataset using the CPU. It is important to keep in mind that this process can be time-consuming, especially if your CPU doesn't have high computational capabilities. Patience may be required as we proceed with this task.\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ⊗ Define processing device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ⊗ Training the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5cRqMR271DNc",
    "outputId": "059e1614-3be2-42c4-a378-d9344612e123"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz to ./data/cifar-100-python.tar.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100.0%\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./data/cifar-100-python.tar.gz to ./data\n",
      "Epoch 1, Loss: 4.134092291297815\n",
      "Epoch 2, Loss: 3.595558230529356\n",
      "Epoch 3, Loss: 3.3053552775126893\n",
      "Epoch 4, Loss: 3.0862986937813135\n",
      "Epoch 5, Loss: 2.9105644085827995\n",
      "Epoch 6, Loss: 2.7574649564445477\n",
      "Epoch 7, Loss: 2.622874382511734\n",
      "Epoch 8, Loss: 2.4997932191395087\n",
      "Epoch 9, Loss: 2.3941552340222136\n",
      "Epoch 10, Loss: 2.3102799943645898\n",
      "\n",
      "CPU Training time for CIFAR-100: 476.2612404823303 seconds\n"
     ]
    }
   ],
   "source": [
    "transform = transforms.Compose([\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomCrop(32, padding=4),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "])\n",
    "\n",
    "trainset = torchvision.datasets.CIFAR100(root='./data', train=True, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, num_workers=4)\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)\n",
    "        self.conv2 = nn.Conv2d(64, 128, kernel_size=3, padding=1)\n",
    "        self.fc1 = nn.Linear(128 * 8 * 8, 512)\n",
    "        self.fc2 = nn.Linear(512, 100) # Redefine for 100 outputs.\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = torch.relu(self.conv1(x))\n",
    "        x = torch.max_pool2d(x, 2)\n",
    "        x = torch.relu(self.conv2(x))\n",
    "        x = torch.max_pool2d(x, 2)\n",
    "        x = x.view(-1, 128 * 8 * 8)\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "net = Net()\n",
    "net.to(device)\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)\n",
    "\n",
    "cpu_start_time = time.time()\n",
    "\n",
    "for epoch in range(10):\n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        inputs, labels = data\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        optimizer.zero_grad()\n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        running_loss += loss.item()\n",
    "\n",
    "    print(f'Epoch {epoch+1}, Loss: {running_loss / len(trainloader)}')\n",
    "\n",
    "cpu_end_time = time.time()\n",
    "\n",
    "total_cpu_time_cifar_100 = cpu_end_time - cpu_start_time\n",
    "\n",
    "print(f\"\\nCPU Training time for CIFAR-100: {total_cpu_time_cifar_100} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ☆ Solution for `CIFAR-100` using PyTorch on GPU☆"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you have seen previously, training simple neural networks with CPUs is not a feasible practice. Therefore, utilizing a GPU is a wise choice if you have access to one, as it will significantly accelerate the training process and empower you to experiment with more intricate models and larger datasets in the future. So, let's proceed to repeat the process, this time loading the CIFAR-100 dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ⊗ Define processing device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lFKsxPAEwAC5"
   },
   "source": [
    "#### ⊗ Training the network "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xWTIejQKwAC5",
    "outputId": "6b34f250-91d0-4b8a-87d5-3e92d4fe36b1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Epoch 1, Loss: 4.17650096495743\n",
      "Epoch 2, Loss: 3.5885800492123265\n",
      "Epoch 3, Loss: 3.290251217839663\n",
      "Epoch 4, Loss: 3.0836600952441127\n",
      "Epoch 5, Loss: 2.921369514197035\n",
      "Epoch 6, Loss: 2.773668039180434\n",
      "Epoch 7, Loss: 2.6351819642059637\n",
      "Epoch 8, Loss: 2.5177373129998326\n",
      "Epoch 9, Loss: 2.4155612860799143\n",
      "Epoch 10, Loss: 2.315835479580228\n",
      "GPU Training time: 53.77\n"
     ]
    }
   ],
   "source": [
    "transform = transforms.Compose([\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomCrop(32, padding=4),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "])\n",
    "\n",
    "trainset = torchvision.datasets.CIFAR100(root='./data', train=True, download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, num_workers=2)\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)\n",
    "        self.conv2 = nn.Conv2d(64, 128, kernel_size=3, padding=1)\n",
    "        self.fc1 = nn.Linear(128 * 8 * 8, 512)\n",
    "        # Change output layers to 100 units\n",
    "        self.fc2 = nn.Linear(512, 100)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = torch.relu(self.conv1(x))\n",
    "        x = torch.max_pool2d(x, 2)\n",
    "        x = torch.relu(self.conv2(x))\n",
    "        x = torch.max_pool2d(x, 2)\n",
    "        x = x.view(-1, 128 * 8 * 8)\n",
    "        x = torch.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x\n",
    "\n",
    "net = Net()\n",
    "net.to(device)\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)\n",
    "\n",
    "gpu_start_time = time.time()\n",
    "\n",
    "for epoch in range(10):\n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        inputs, labels = data\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        optimizer.zero_grad()\n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        running_loss += loss.item()\n",
    "    print(f'Epoch {epoch+1}, Loss: {running_loss / len(trainloader)}')\n",
    "\n",
    "gpu_end_time = time.time()\n",
    "\n",
    "total_gpu_cifar_100 = gpu_end_time - gpu_start_time\n",
    "\n",
    "print(f'GPU Training time: {total_gpu_cifar_100:.2f}')\n",
    "\n",
    "# Save the model\n",
    "torch.save(net.state_dict(), 'cifar100_gpu_model.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "at1Slw5dK0Ps"
   },
   "source": [
    "Now we will evaluate the speedup by comparing the GPU and CPU execution times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1E6ehQp4K1Xp",
    "outputId": "9780b4e4-2ff0-42f3-8b0e-48c278de2930"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Speedup: 8.86X\n"
     ]
    }
   ],
   "source": [
    "print(f\"\\nSpeedup:{total_cpu_time_cifar_100 / total_gpu_cifar_100 : .2f}X\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xnZ2iKrSwAC5"
   },
   "source": [
    "### Comments about the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MvVsjO15wAC5"
   },
   "source": [
    "<p style='text-align: justify;'>\n",
    "We explored training neural networks with PyTorch, comparing CPU and GPU performance on the CIFAR-10 and CIFAR-100 dataset using 10 epochs. When training with CIFAR-10 and utilizing the CPU, and GPU environments, the process can be executed in approximately (in seconds):\n",
    "</p>\n",
    "\n",
    "|  Pytorch |      CIFAR-10    |  CIFAR-100 |\n",
    "|----------|:-------------:   |-----------:|\n",
    "| CPU      |  462.89          |   472.26   |\n",
    "| GPU      |  54.03           |   53.77    |\n",
    "| Speedup  |  8.56X           |    8.78X   |\n",
    "\n",
    "<p style='text-align: justify;'>\n",
    "This outcome illustrates that the GPU has achieved nearly a <b>Speedup of 9X</b> compared to the CPU when running with 10 epochs in the algorithm with the highest computational cost (CIFAR-100). Using the Pytorch the GPU has substantially enhanced the training speed, which is particularly advantageous.\n",
    "</p>    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oCCcZ7Q31bIC"
   },
   "source": [
    "## Summary\n",
    "In this notebook we have shown:\n",
    "\n",
    "- Install and use PyTorch using GPU environments,\n",
    "- Comparative performance tests between CPU and GPU on model training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4FolLqE_1bID"
   },
   "source": [
    "## Clear the memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tXiePpX31bID"
   },
   "source": [
    "Before moving on, please execute the following cell to clear up the CPU memory. This is required to move on to the next notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kMB90WpX1bID"
   },
   "outputs": [],
   "source": [
    "import IPython\n",
    "app = IPython.Application.instance()\n",
    "app.kernel.do_shutdown(True)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "machine_shape": "hm",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}