tutorials/building_image_classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": "true"
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Building-an-Image-Classifier-with-Differential-Privacy\" data-toc-modified-id=\"Building-an-Image-Classifier-with-Differential-Privacy-1\">Building an Image Classifier with Differential Privacy</a></span><ul class=\"toc-item\"><li><span><a href=\"#Overview\" data-toc-modified-id=\"Overview-1.1\">Overview</a></span></li><li><span><a href=\"#Hyper-parameters\" data-toc-modified-id=\"Hyper-parameters-1.2\">Hyper-parameters</a></span></li><li><span><a href=\"#Data\" data-toc-modified-id=\"Data-1.3\">Data</a></span></li><li><span><a href=\"#Model\" data-toc-modified-id=\"Model-1.4\">Model</a></span></li><li><span><a href=\"#Prepare-for-Training\" data-toc-modified-id=\"Prepare-for-Training-1.5\">Prepare for Training</a></span></li><li><span><a href=\"#Train-the-network\" data-toc-modified-id=\"Train-the-network-1.6\">Train the network</a></span></li><li><span><a href=\"#Test-the-network-on-test-data\" data-toc-modified-id=\"Test-the-network-on-test-data-1.7\">Test the network on test data</a></span></li><li><span><a href=\"#Tips-and-Tricks\" data-toc-modified-id=\"Tips-and-Tricks-1.8\">Tips and Tricks</a></span></li><li><span><a href=\"#Private-Model-vs-Non-Private-Model-Performance\" data-toc-modified-id=\"Private-Model-vs-Non-Private-Model-Performance-1.9\">Private Model vs Non-Private Model Performance</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building an Image Classifier with Differential Privacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In this tutorial we will learn to do the following:\n",
    "  1. Learn about privacy-specific hyper-parameters related to DP-SGD \n",
    "  2. Learn about ModelInspector, incompatible layers, and use model rewriting utility. \n",
    "  3. Train a differentially private ResNet18 for image classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyper-parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a model with Opacus there are three privacy-specific hyper-parameters that must be tuned for better performance:\n",
    "\n",
    "* Max Grad Norm: The maximum L2 norm of per-sample gradients before they are aggregated by the averaging step.\n",
    "* Noise Multiplier: The amount of noise sampled and added to the average of the gradients in a batch.\n",
    "* Delta: The target δ of the (ϵ,δ)-differential privacy guarantee. Generally, it should be set to be less than the inverse of the size of the training dataset. In this tutorial, it is set to $10^{−5}$ as the CIFAR10 dataset has 50,000 training points.\n",
    "\n",
    "We use the hyper-parameter values below to obtain results in the last section:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.simplefilter(\"ignore\")\n",
    "\n",
    "MAX_GRAD_NORM = 1.2\n",
    "EPSILON = 50.0\n",
    "DELTA = 1e-5\n",
    "EPOCHS = 20\n",
    "\n",
    "LR = 1e-3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's another constraint we should be mindful of&mdash;memory. To balance peak memory requirement, which is proportional to `batch_size^2`, and training performance, we will be using BatchMemoryManager. It separates the logical batch size (which defines how often the model is updated and how much DP noise is added), and a physical batch size (which defines how many samples we process at a time).\n",
    "\n",
    "With BatchMemoryManager you will create your DataLoader with a logical batch size, and then provide the maximum physical batch size to the memory manager."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 512\n",
    "MAX_PHYSICAL_BATCH_SIZE = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's load the CIFAR10 dataset. We don't use data augmentation here because, in our experiments, we found that data augmentation lowers utility when training with DP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "# These values, specific to the CIFAR10 dataset, are assumed to be known.\n",
    "# If necessary, they can be computed with modest privacy budgets.\n",
    "CIFAR10_MEAN = (0.4914, 0.4822, 0.4465)\n",
    "CIFAR10_STD_DEV = (0.2023, 0.1994, 0.2010)\n",
    "\n",
    "transform = transforms.Compose([\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(CIFAR10_MEAN, CIFAR10_STD_DEV),\n",
    "])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using torchvision datasets, we can load CIFAR10 and transform the PILImage images to Tensors of normalized range [-1, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "from torchvision.datasets import CIFAR10\n",
    "\n",
    "DATA_ROOT = '../cifar10'\n",
    "\n",
    "train_dataset = CIFAR10(\n",
    "    root=DATA_ROOT, train=True, download=True, transform=transform)\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(\n",
    "    train_dataset,\n",
    "    batch_size=BATCH_SIZE,\n",
    ")\n",
    "\n",
    "test_dataset = CIFAR10(\n",
    "    root=DATA_ROOT, train=False, download=True, transform=transform)\n",
    "\n",
    "test_loader = torch.utils.data.DataLoader(\n",
    "    test_dataset,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    shuffle=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import models\n",
    "\n",
    "model = models.resnet18(num_classes=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let’s check if the model is compatible with Opacus. Opacus does not support all types of Pytorch layers. To check if your model is compatible with the privacy engine, we have provided a util class to validate your model.\n",
    "\n",
    "When you run the code below, you're presented with a list of errors, indicating which modules are incompatible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[opacus.validators.errors.ShouldReplaceModuleError(\"BatchNorm cannot support training with differential privacy. The reason for it is that BatchNorm makes each sample's normalized value depend on its peers in a batch, ie the same sample x will get normalized to a different value depending on who else is on its batch. Privacy-wise, this means that we would have to put a privacy mechanism there too. While it can in principle be done, there are now multiple normalization layers that do not have this issue: LayerNorm, InstanceNorm and their generalization GroupNorm are all privacy-safe since they don't have this property.We offer utilities to automatically replace BatchNorms to GroupNorms and we will release pretrained models to help transition, such as GN-ResNet ie a ResNet using GroupNorm, pretrained on ImageNet\"),\n",
       " opacus.validators.errors.ShouldReplaceModuleError(\"BatchNorm cannot support training with differential privacy. The reason for it is that BatchNorm makes each sample's normalized value depend on its peers in a batch, ie the same sample x will get normalized to a different value depending on who else is on its batch. Privacy-wise, this means that we would have to put a privacy mechanism there too. While it can in principle be done, there are now multiple normalization layers that do not have this issue: LayerNorm, InstanceNorm and their generalization GroupNorm are all privacy-safe since they don't have this property.We offer utilities to automatically replace BatchNorms to GroupNorms and we will release pretrained models to help transition, such as GN-ResNet ie a ResNet using GroupNorm, pretrained on ImageNet\"),\n",
       " opacus.validators.errors.ShouldReplaceModuleError(\"BatchNorm cannot support training with differential privacy. The reason for it is that BatchNorm makes each sample's normalized value depend on its peers in a batch, ie the same sample x will get normalized to a different value depending on who else is on its batch. Privacy-wise, this means that we would have to put a privacy mechanism there too. While it can in principle be done, there are now multiple normalization layers that do not have this issue: LayerNorm, InstanceNorm and their generalization GroupNorm are all privacy-safe since they don't have this property.We offer utilities to automatically replace BatchNorms to GroupNorms and we will release pretrained models to help transition, such as GN-ResNet ie a ResNet using GroupNorm, pretrained on ImageNet\"),\n",
       " opacus.validators.errors.ShouldReplaceModuleError(\"BatchNorm cannot support training with differential privacy. The reason for it is that BatchNorm makes each sample's normalized value depend on its peers in a batch, ie the same sample x will get normalized to a different value depending on who else is on its batch. Privacy-wise, this means that we would have to put a privacy mechanism there too. While it can in principle be done, there are now multiple normalization layers that do not have this issue: LayerNorm, InstanceNorm and their generalization GroupNorm are all privacy-safe since they don't have this property.We offer utilities to automatically replace BatchNorms to GroupNorms and we will release pretrained models to help transition, such as GN-ResNet ie a ResNet using GroupNorm, pretrained on ImageNet\"),\n",
       " opacus.validators.errors.ShouldReplaceModuleError(\"BatchNorm cannot support training with differential privacy. The reason for it is that BatchNorm makes each sample's normalized value depend on its peers in a batch, ie the same sample x will get normalized to a different value depending on who else is on its batch. Privacy-wise, this means that we would have to put a privacy mechanism there too. While it can in principle be done, there are now multiple normalization layers that do not have this issue: LayerNorm, InstanceNorm and their generalization GroupNorm are all privacy-safe since they don't have this property.We offer utilities to automatically replace BatchNorms to GroupNorms and we will release pretrained models to help transition, such as GN-ResNet ie a ResNet using GroupNorm, pretrained on ImageNet\")]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from opacus.validators import ModuleValidator\n",
    "\n",
    "errors = ModuleValidator.validate(model, strict=False)\n",
    "errors[-5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us modify the model to work with Opacus. From the output above, you can see that the BatchNorm layers are not supported because they compute the mean and variance across the batch, creating a dependency between samples in a batch, a privacy violation.\n",
    "\n",
    "Recommended approach to deal with it is calling `ModuleValidator.fix(model)` - it tries to find the best replacement for incompatible modules. For example, for BatchNorm modules, it replaces them with GroupNorm.\n",
    "You can see, that after this, no exception is raised"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = ModuleValidator.fix(model)\n",
    "ModuleValidator.validate(model, strict=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For maximal speed, we can check if CUDA is available and supported by the PyTorch installation. If GPU is available, set the `device` variable to your CUDA-compatible device. We can then transfer the neural network onto that device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then define our optimizer and loss function. Opacus’ privacy engine can attach to any (first-order) optimizer.  You can use your favorite&mdash;Adam, Adagrad, RMSprop&mdash;as long as it has an implementation derived from [torch.optim.Optimizer](https://pytorch.org/docs/stable/optim.html). In this tutorial, we're going to use [RMSprop](https://pytorch.org/docs/stable/optim.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.RMSprop(model.parameters(), lr=LR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare for Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define a util function to calculate accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy(preds, labels):\n",
    "    return (preds == labels).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now attach the privacy engine initialized with the privacy hyperparameters defined earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using sigma=0.39066894531249996 and C=1.2\n"
     ]
    }
   ],
   "source": [
    "from opacus import PrivacyEngine\n",
    "\n",
    "privacy_engine = PrivacyEngine()\n",
    "\n",
    "model, optimizer, train_loader = privacy_engine.make_private_with_epsilon(\n",
    "    module=model,\n",
    "    optimizer=optimizer,\n",
    "    data_loader=train_loader,\n",
    "    epochs=EPOCHS,\n",
    "    target_epsilon=EPSILON,\n",
    "    target_delta=DELTA,\n",
    "    max_grad_norm=MAX_GRAD_NORM,\n",
    ")\n",
    "\n",
    "print(f\"Using sigma={optimizer.noise_multiplier} and C={MAX_GRAD_NORM}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will then define our train function. This function will train the model for one epoch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from opacus.utils.batch_memory_manager import BatchMemoryManager\n",
    "\n",
    "\n",
    "def train(model, train_loader, optimizer, epoch, device):\n",
    "    model.train()\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "    losses = []\n",
    "    top1_acc = []\n",
    "    \n",
    "    with BatchMemoryManager(\n",
    "        data_loader=train_loader, \n",
    "        max_physical_batch_size=MAX_PHYSICAL_BATCH_SIZE, \n",
    "        optimizer=optimizer\n",
    "    ) as memory_safe_data_loader:\n",
    "\n",
    "        for i, (images, target) in enumerate(memory_safe_data_loader):   \n",
    "            optimizer.zero_grad()\n",
    "            images = images.to(device)\n",
    "            target = target.to(device)\n",
    "\n",
    "            # compute output\n",
    "            output = model(images)\n",
    "            loss = criterion(output, target)\n",
    "\n",
    "            preds = np.argmax(output.detach().cpu().numpy(), axis=1)\n",
    "            labels = target.detach().cpu().numpy()\n",
    "\n",
    "            # measure accuracy and record loss\n",
    "            acc = accuracy(preds, labels)\n",
    "\n",
    "            losses.append(loss.item())\n",
    "            top1_acc.append(acc)\n",
    "\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            if (i+1) % 200 == 0:\n",
    "                epsilon = privacy_engine.get_epsilon(DELTA)\n",
    "                print(\n",
    "                    f\"\\tTrain Epoch: {epoch} \\t\"\n",
    "                    f\"Loss: {np.mean(losses):.6f} \"\n",
    "                    f\"Acc@1: {np.mean(top1_acc) * 100:.6f} \"\n",
    "                    f\"(ε = {epsilon:.2f}, δ = {DELTA})\"\n",
    "                )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will define our test function to validate our model on our test dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(model, test_loader, device):\n",
    "    model.eval()\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    losses = []\n",
    "    top1_acc = []\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for images, target in test_loader:\n",
    "            images = images.to(device)\n",
    "            target = target.to(device)\n",
    "\n",
    "            output = model(images)\n",
    "            loss = criterion(output, target)\n",
    "            preds = np.argmax(output.detach().cpu().numpy(), axis=1)\n",
    "            labels = target.detach().cpu().numpy()\n",
    "            acc = accuracy(preds, labels)\n",
    "\n",
    "            losses.append(loss.item())\n",
    "            top1_acc.append(acc)\n",
    "\n",
    "    top1_avg = np.mean(top1_acc)\n",
    "\n",
    "    print(\n",
    "        f\"\\tTest set:\"\n",
    "        f\"Loss: {np.mean(losses):.6f} \"\n",
    "        f\"Acc: {top1_avg * 100:.6f} \"\n",
    "    )\n",
    "    return np.mean(top1_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bfa9bcd7c0854e71b2b30d9f963b75c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Epoch:   0%|          | 0/20 [00:00<?, ?epoch/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Epoch: 1 \tLoss: 2.771490 Acc@1: 15.429688 (ε = 13.64, δ = 1e-05)\n",
      "\tTrain Epoch: 2 \tLoss: 1.755194 Acc@1: 38.804688 (ε = 17.68, δ = 1e-05)\n",
      "\tTrain Epoch: 3 \tLoss: 1.724797 Acc@1: 45.769531 (ε = 20.62, δ = 1e-05)\n",
      "\tTrain Epoch: 4 \tLoss: 1.706076 Acc@1: 48.921875 (ε = 22.94, δ = 1e-05)\n",
      "\tTrain Epoch: 5 \tLoss: 1.682414 Acc@1: 51.664062 (ε = 25.25, δ = 1e-05)\n",
      "\tTrain Epoch: 6 \tLoss: 1.671187 Acc@1: 53.234375 (ε = 26.99, δ = 1e-05)\n",
      "\tTrain Epoch: 7 \tLoss: 1.657112 Acc@1: 55.324219 (ε = 28.73, δ = 1e-05)\n",
      "\tTrain Epoch: 8 \tLoss: 1.633768 Acc@1: 56.277344 (ε = 30.46, δ = 1e-05)\n",
      "\tTrain Epoch: 9 \tLoss: 1.647288 Acc@1: 57.203125 (ε = 32.20, δ = 1e-05)\n",
      "\tTrain Epoch: 10 \tLoss: 1.639933 Acc@1: 58.191406 (ε = 33.83, δ = 1e-05)\n",
      "\tTrain Epoch: 11 \tLoss: 1.639214 Acc@1: 59.140625 (ε = 35.17, δ = 1e-05)\n",
      "\tTrain Epoch: 12 \tLoss: 1.629374 Acc@1: 59.613281 (ε = 36.51, δ = 1e-05)\n",
      "\tTrain Epoch: 13 \tLoss: 1.636143 Acc@1: 60.246094 (ε = 37.85, δ = 1e-05)\n",
      "\tTrain Epoch: 14 \tLoss: 1.634575 Acc@1: 60.550781 (ε = 39.18, δ = 1e-05)\n",
      "\tTrain Epoch: 15 \tLoss: 1.611133 Acc@1: 61.378906 (ε = 40.52, δ = 1e-05)\n",
      "\tTrain Epoch: 16 \tLoss: 1.604075 Acc@1: 62.015625 (ε = 41.86, δ = 1e-05)\n",
      "\tTrain Epoch: 17 \tLoss: 1.601270 Acc@1: 62.140625 (ε = 43.20, δ = 1e-05)\n",
      "\tTrain Epoch: 18 \tLoss: 1.599596 Acc@1: 62.437500 (ε = 44.53, δ = 1e-05)\n",
      "\tTrain Epoch: 19 \tLoss: 1.587946 Acc@1: 63.097656 (ε = 45.87, δ = 1e-05)\n",
      "\tTrain Epoch: 20 \tLoss: 1.583897 Acc@1: 63.250000 (ε = 47.21, δ = 1e-05)\n"
     ]
    }
   ],
   "source": [
    "from tqdm.notebook import tqdm\n",
    "\n",
    "for epoch in tqdm(range(EPOCHS), desc=\"Epoch\", unit=\"epoch\"):\n",
    "    train(model, train_loader, optimizer, epoch + 1, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the network on test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTest set:Loss: 1.711833 Acc: 60.753676 \n"
     ]
    }
   ],
   "source": [
    "top1_acc = test(model, test_loader, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tips and Tricks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Generally speaking, differentially private training is enough of a regularizer by itself. Adding any more regularization (such as dropouts or data augmentation) is unnecessary and typically hurts performance.\n",
    "2. Tuning MAX_GRAD_NORM is very important. Start with a low noise multiplier like .1, this should give comparable performance to a non-private model. Then do a grid search for the optimal MAX_GRAD_NORM value. The grid can be in the range [.1, 10].\n",
    "3. You can play around with the level of privacy, EPSILON.  Smaller EPSILON means more privacy, more noise -- and hence lower accuracy.  Reducing EPSILON to 5.0 reduces the Top 1 Accuracy to around 53%.  One useful technique is to pre-train a model on public (non-private) data, before completing the training on the private training data.  See the workbook at [bit.ly/opacus-dev-day](https://bit.ly/opacus-dev-day) for an example.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Private Model vs Non-Private Model Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us compare how our private model compares with the non-private ResNet18.\n",
    "\n",
    "We trained a non-private ResNet18 model for 20 epochs using the same hyper-parameters as above and with BatchNorm replaced with GroupNorm. The results of that training and the training that is discussed in this tutorial are summarized in the table below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Model          | Top 1 Accuracy (%) |  ϵ |\n",
    "|----------------|--------------------|---|\n",
    "| ResNet         | 76                 | ∞ |\n",
    "| Private ResNet |         61         |  47.21  |"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "740px",
    "left": "0px",
    "right": "1628px",
    "top": "161px",
    "width": "253px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}