tutorials/building_image_classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": "true"
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Building-an-Image-Classifier-with-Differential-Privacy\" data-toc-modified-id=\"Building-an-Image-Classifier-with-Differential-Privacy-1\">Building an Image Classifier with Differential Privacy</a></span><ul class=\"toc-item\"><li><span><a href=\"#Overview\" data-toc-modified-id=\"Overview-1.1\">Overview</a></span></li><li><span><a href=\"#Hyper-parameters\" data-toc-modified-id=\"Hyper-parameters-1.2\">Hyper-parameters</a></span></li><li><span><a href=\"#Data\" data-toc-modified-id=\"Data-1.3\">Data</a></span></li><li><span><a href=\"#Model\" data-toc-modified-id=\"Model-1.4\">Model</a></span></li><li><span><a href=\"#Prepare-for-Training\" data-toc-modified-id=\"Prepare-for-Training-1.5\">Prepare for Training</a></span></li><li><span><a href=\"#Train-the-network\" data-toc-modified-id=\"Train-the-network-1.6\">Train the network</a></span></li><li><span><a href=\"#Test-the-network-on-test-data\" data-toc-modified-id=\"Test-the-network-on-test-data-1.7\">Test the network on test data</a></span></li><li><span><a href=\"#Tips-and-Tricks\" data-toc-modified-id=\"Tips-and-Tricks-1.8\">Tips and Tricks</a></span></li><li><span><a href=\"#Private-Model-vs-Non-Private-Model-Performance\" data-toc-modified-id=\"Private-Model-vs-Non-Private-Model-Performance-1.9\">Private Model vs Non-Private Model Performance</a></span></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building an Image Classifier with Differential Privacy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In this tutorial we will learn to do the following:\n",
    "  1. Learn about privacy specific hyper-parameters related to DP-SGD \n",
    "  2. Learn about ModelInspector, incompatible layers, and use model rewriting utility. \n",
    "  3. Train a differentially private ResNet18 for image classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyper-parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a model with Opacus there are three privacy-specific hyper-parameters that must be tuned for better performance:\n",
    "\n",
    "* Max Grad Norm: The maximum L2 norm of per-sample gradients before they are aggregated by the averaging step.\n",
    "* Noise Multiplier: The amount of noise sampled and added to the average of the gradients in a batch.\n",
    "* Delta: The target δ of the (ϵ,δ)-differential privacy guarantee. Generally, it should be set to be less than the inverse of the size of the training dataset. In this tutorial, it is set to $10^{−5}$ as the CIFAR10 dataset has 50,000 training points.\n",
    "\n",
    "We use the hyper-parameter values below to obtain results in the last section:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_GRAD_NORM = 1.2\n",
    "NOISE_MULTIPLIER = .38\n",
    "DELTA = 1e-5\n",
    "\n",
    "LR = 1e-3\n",
    "NUM_WORKERS = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's another constraint we should be mindful of&mdash;memory. To balance peak memory requirement, which is proportional to `batch_size^2`, and training performance, we use virtual batches. With virtual batches we can separate physical steps (gradient computation) and logical steps (noise addition and parameter updates): use larger batches for training, while keeping memory footprint low. Below we will specify two constants:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 128\n",
    "VIRTUAL_BATCH_SIZE = 512"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's load the CIFAR10 dataset. We don't use data augmentation here because, in our experiments, we found that data augmentation lowers utility when training with DP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "# These values, specific to the CIFAR10 dataset, are assumed to be known.\n",
    "# If necessary, they can be computed with modest privacy budget.\n",
    "CIFAR10_MEAN = (0.4914, 0.4822, 0.4465)\n",
    "CIFAR10_STD_DEV = (0.2023, 0.1994, 0.2010)\n",
    "\n",
    "transform = transforms.Compose([\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(CIFAR10_MEAN, CIFAR10_STD_DEV),\n",
    "])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using torchvision datasets, we can load CIFAR10 and transform the PILImage images to Tensors of normalized range [-1, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.datasets import CIFAR10\n",
    "\n",
    "DATA_ROOT = '../cifar10'\n",
    "\n",
    "train_dataset = CIFAR10(\n",
    "    root=DATA_ROOT, train=True, download=True, transform=transform)\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(\n",
    "    train_dataset,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    shuffle=True,\n",
    "    num_workers=NUM_WORKERS,\n",
    ")\n",
    "\n",
    "test_dataset = CIFAR10(\n",
    "    root=DATA_ROOT, train=False, download=True, transform=transform)\n",
    "\n",
    "test_loader = torch.utils.data.DataLoader(\n",
    "    test_dataset,\n",
    "    batch_size=BATCH_SIZE,\n",
    "    shuffle=False,\n",
    "    num_workers=NUM_WORKERS,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import models\n",
    "\n",
    "model = models.resnet18(num_classes=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let’s check if the model is compatible with Opacus. Opacus does not support all type of Pytorch layers. To check if your model is compatible with the privacy engine, we have provided a util class to validate your model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you run these commands, you will get the following error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "IncompatibleModuleException",
     "evalue": "Model contains incompatible modules.\nSome modules are not valid.: ['Main.bn1', 'Main.layer1.0.bn1', 'Main.layer1.0.bn2', 'Main.layer1.1.bn1', 'Main.layer1.1.bn2', 'Main.layer2.0.bn1', 'Main.layer2.0.bn2', 'Main.layer2.0.downsample.1', 'Main.layer2.1.bn1', 'Main.layer2.1.bn2', 'Main.layer3.0.bn1', 'Main.layer3.0.bn2', 'Main.layer3.0.downsample.1', 'Main.layer3.1.bn1', 'Main.layer3.1.bn2', 'Main.layer4.0.bn1', 'Main.layer4.0.bn2', 'Main.layer4.0.downsample.1', 'Main.layer4.1.bn1', 'Main.layer4.1.bn2']",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIncompatibleModuleException\u001b[0m               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-11-c3648acc319a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0minspector\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mDPModelInspector\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0minspector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalidate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/mnt/xarfuse/uid-179429/a7a74ae2-seed-nspid4026531836-ns-4026531840/opacus/dp_model_inspector.py\u001b[0m in \u001b[0;36mvalidate\u001b[0;34m(self, model)\u001b[0m\n\u001b[1;32m    115\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0minspector\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mviolators\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    116\u001b[0m                     \u001b[0mmessage\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;34mf\"\\n{inspector.message}: {inspector.violators}\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 117\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mIncompatibleModuleException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    118\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mvalid\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    119\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIncompatibleModuleException\u001b[0m: Model contains incompatible modules.\nSome modules are not valid.: ['Main.bn1', 'Main.layer1.0.bn1', 'Main.layer1.0.bn2', 'Main.layer1.1.bn1', 'Main.layer1.1.bn2', 'Main.layer2.0.bn1', 'Main.layer2.0.bn2', 'Main.layer2.0.downsample.1', 'Main.layer2.1.bn1', 'Main.layer2.1.bn2', 'Main.layer3.0.bn1', 'Main.layer3.0.bn2', 'Main.layer3.0.downsample.1', 'Main.layer3.1.bn1', 'Main.layer3.1.bn2', 'Main.layer4.0.bn1', 'Main.layer4.0.bn2', 'Main.layer4.0.downsample.1', 'Main.layer4.1.bn1', 'Main.layer4.1.bn2']"
     ]
    }
   ],
   "source": [
    "from opacus.dp_model_inspector import DPModelInspector\n",
    "\n",
    "inspector = DPModelInspector()\n",
    "inspector.validate(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us modify the model to work with Opacus. From the output above, you can see that the BatchNorm layers are not supported because they compute the mean and variance across the batch, creating a dependency between samples in a batch, a privacy violation. One way to modify our model is to replace all the BatchNorm layers with [GroupNorm](https://arxiv.org/pdf/1803.08494.pdf) using the `convert_batchnorm_modules` util function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is the model valid? True\n"
     ]
    }
   ],
   "source": [
    "from opacus.utils import module_modification\n",
    "\n",
    "model = module_modification.convert_batchnorm_modules(model)\n",
    "inspector = DPModelInspector()\n",
    "print(f\"Is the model valid? {inspector.validate(model)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For maximal speed, we can check if CUDA is available and supported by the PyTorch installation. If GPU is available, set the `device` variable to your CUDA-compatible device. We can then transfer the neural network onto that device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then define our optimizer and loss function. Opacus’ privacy engine can attach to any (first-order) optimizer.  You can use your favorite&mdash;Adam, Adagrad, RMSprop&mdash;as long as it has an implementation derived from [torch.optim.Optimizer](https://pytorch.org/docs/stable/optim.html). In this tutorial, we're going to use [RMSprop](https://pytorch.org/docs/stable/optim.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.RMSprop(model.parameters(), lr=LR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare for Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define a util function to calculate accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy(preds, labels):\n",
    "    return (preds == labels).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now attach the privacy engine initialized with the privacy hyperparameters defined earlier. There’s also the enigmatic-looking parameter `alphas`, which we won’t touch for the time being."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using sigma=0.38 and C=1.2\n"
     ]
    }
   ],
   "source": [
    "from opacus import PrivacyEngine\n",
    "\n",
    "print(f\"Using sigma={NOISE_MULTIPLIER} and C={MAX_GRAD_NORM}\")\n",
    "\n",
    "privacy_engine = PrivacyEngine(\n",
    "    model,\n",
    "    batch_size=VIRTUAL_BATCH_SIZE,\n",
    "    sample_size=len(train_dataset),\n",
    "    alphas=[1 + x / 10.0 for x in range(1, 100)] + list(range(12, 64)),\n",
    "    noise_multiplier=NOISE_MULTIPLIER,\n",
    "    max_grad_norm=MAX_GRAD_NORM,\n",
    ")\n",
    "privacy_engine.attach(optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will then define our train function. This function will train the model for one epoch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert VIRTUAL_BATCH_SIZE % BATCH_SIZE == 0 # VIRTUAL_BATCH_SIZE should be divisible by BATCH_SIZE\n",
    "virtual_batch_rate = int(VIRTUAL_BATCH_SIZE / BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def train(model, train_loader, optimizer, epoch, device):\n",
    "    model.train()\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "    losses = []\n",
    "    top1_acc = []\n",
    "\n",
    "    for i, (images, target) in enumerate(train_loader):\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        images = images.to(device)\n",
    "        target = target.to(device)\n",
    "\n",
    "        # compute output\n",
    "        output = model(images)\n",
    "        loss = criterion(output, target)\n",
    "        \n",
    "        preds = np.argmax(output.detach().cpu().numpy(), axis=1)\n",
    "        labels = target.detach().cpu().numpy()\n",
    "        \n",
    "        # measure accuracy and record loss\n",
    "        acc = accuracy(preds, labels)\n",
    "\n",
    "        losses.append(loss.item())\n",
    "        top1_acc.append(acc)\n",
    "        \n",
    "        loss.backward()\n",
    "        \t\n",
    "        # take a real optimizer step after N_VIRTUAL_STEP steps t\n",
    "        if ((i + 1) % virtual_batch_rate == 0) or ((i + 1) == len(train_loader)):\n",
    "            optimizer.step()\n",
    "        else:\n",
    "            optimizer.virtual_step() # take a virtual step\n",
    "\n",
    "        if i % 200 == 0:\n",
    "            epsilon, best_alpha = optimizer.privacy_engine.get_privacy_spent(DELTA)\n",
    "            print(\n",
    "                f\"\\tTrain Epoch: {epoch} \\t\"\n",
    "                f\"Loss: {np.mean(losses):.6f} \"\n",
    "                f\"Acc@1: {np.mean(top1_acc) * 100:.6f} \"\n",
    "                f\"(ε = {epsilon:.2f}, δ = {DELTA})\"\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will define our test function to validate our model on our test dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test(model, test_loader, device):\n",
    "    model.eval()\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    losses = []\n",
    "    top1_acc = []\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for images, target in test_loader:\n",
    "            images = images.to(device)\n",
    "            target = target.to(device)\n",
    "\n",
    "            output = model(images)\n",
    "            loss = criterion(output, target)\n",
    "            preds = np.argmax(output.detach().cpu().numpy(), axis=1)\n",
    "            labels = target.detach().cpu().numpy()\n",
    "            acc = accuracy(preds, labels)\n",
    "\n",
    "            losses.append(loss.item())\n",
    "            top1_acc.append(acc)\n",
    "\n",
    "    top1_avg = np.mean(top1_acc)\n",
    "\n",
    "    print(\n",
    "        f\"\\tTest set:\"\n",
    "        f\"Loss: {np.mean(losses):.6f} \"\n",
    "        f\"Acc: {top1_avg * 100:.6f} \"\n",
    "    )\n",
    "    return np.mean(top1_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a0521999fa9d45219724acf70b3b4469",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch', max=20.0, style=ProgressStyle(description_width='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTrain Epoch: 1 \tLoss: 2.732535 Acc@1: 7.812500 (ε = 0.19, δ = 1e-05)\n",
      "\tTrain Epoch: 1 \tLoss: 2.782132 Acc@1: 14.128576 (ε = 16.04, δ = 1e-05)\n",
      "\tTrain Epoch: 2 \tLoss: 1.730190 Acc@1: 36.718750 (ε = 18.66, δ = 1e-05)\n",
      "\tTrain Epoch: 2 \tLoss: 1.761741 Acc@1: 38.355100 (ε = 20.83, δ = 1e-05)\n",
      "\tTrain Epoch: 3 \tLoss: 1.542238 Acc@1: 50.781250 (ε = 22.43, δ = 1e-05)\n",
      "\tTrain Epoch: 3 \tLoss: 1.723531 Acc@1: 45.565143 (ε = 23.96, δ = 1e-05)\n",
      "\tTrain Epoch: 4 \tLoss: 1.949715 Acc@1: 42.187500 (ε = 25.43, δ = 1e-05)\n",
      "\tTrain Epoch: 4 \tLoss: 1.720563 Acc@1: 49.109919 (ε = 26.85, δ = 1e-05)\n",
      "\tTrain Epoch: 5 \tLoss: 1.774229 Acc@1: 51.562500 (ε = 27.92, δ = 1e-05)\n",
      "\tTrain Epoch: 5 \tLoss: 1.714577 Acc@1: 51.698539 (ε = 29.03, δ = 1e-05)\n",
      "\tTrain Epoch: 6 \tLoss: 1.405191 Acc@1: 60.156250 (ε = 30.10, δ = 1e-05)\n",
      "\tTrain Epoch: 6 \tLoss: 1.675316 Acc@1: 53.976213 (ε = 31.21, δ = 1e-05)\n",
      "\tTrain Epoch: 7 \tLoss: 1.554456 Acc@1: 52.343750 (ε = 32.28, δ = 1e-05)\n",
      "\tTrain Epoch: 7 \tLoss: 1.685776 Acc@1: 54.909049 (ε = 33.39, δ = 1e-05)\n",
      "\tTrain Epoch: 8 \tLoss: 1.778377 Acc@1: 53.125000 (ε = 34.46, δ = 1e-05)\n",
      "\tTrain Epoch: 8 \tLoss: 1.696506 Acc@1: 56.012904 (ε = 35.36, δ = 1e-05)\n",
      "\tTrain Epoch: 9 \tLoss: 1.904890 Acc@1: 54.687500 (ε = 36.17, δ = 1e-05)\n",
      "\tTrain Epoch: 9 \tLoss: 1.639168 Acc@1: 57.668688 (ε = 37.00, δ = 1e-05)\n",
      "\tTrain Epoch: 10 \tLoss: 1.614231 Acc@1: 53.906250 (ε = 37.81, δ = 1e-05)\n",
      "\tTrain Epoch: 10 \tLoss: 1.641604 Acc@1: 58.652052 (ε = 38.65, δ = 1e-05)\n",
      "\tTrain Epoch: 11 \tLoss: 1.678383 Acc@1: 55.468750 (ε = 39.45, δ = 1e-05)\n",
      "\tTrain Epoch: 11 \tLoss: 1.627174 Acc@1: 59.343905 (ε = 40.29, δ = 1e-05)\n",
      "\tTrain Epoch: 12 \tLoss: 1.534639 Acc@1: 62.500000 (ε = 41.09, δ = 1e-05)\n",
      "\tTrain Epoch: 12 \tLoss: 1.627389 Acc@1: 59.732587 (ε = 41.93, δ = 1e-05)\n",
      "\tTrain Epoch: 13 \tLoss: 1.565778 Acc@1: 63.281250 (ε = 42.74, δ = 1e-05)\n",
      "\tTrain Epoch: 13 \tLoss: 1.611232 Acc@1: 60.743159 (ε = 43.58, δ = 1e-05)\n",
      "\tTrain Epoch: 14 \tLoss: 1.529223 Acc@1: 62.500000 (ε = 44.38, δ = 1e-05)\n",
      "\tTrain Epoch: 14 \tLoss: 1.614406 Acc@1: 60.886971 (ε = 45.22, δ = 1e-05)\n",
      "\tTrain Epoch: 15 \tLoss: 1.445196 Acc@1: 66.406250 (ε = 46.02, δ = 1e-05)\n",
      "\tTrain Epoch: 15 \tLoss: 1.589295 Acc@1: 61.427239 (ε = 46.86, δ = 1e-05)\n",
      "\tTrain Epoch: 16 \tLoss: 1.733013 Acc@1: 59.375000 (ε = 47.67, δ = 1e-05)\n",
      "\tTrain Epoch: 16 \tLoss: 1.577568 Acc@1: 61.866449 (ε = 48.46, δ = 1e-05)\n",
      "\tTrain Epoch: 17 \tLoss: 1.845230 Acc@1: 55.468750 (ε = 49.09, δ = 1e-05)\n",
      "\tTrain Epoch: 17 \tLoss: 1.578402 Acc@1: 62.010261 (ε = 49.73, δ = 1e-05)\n",
      "\tTrain Epoch: 18 \tLoss: 1.565229 Acc@1: 64.843750 (ε = 50.35, δ = 1e-05)\n",
      "\tTrain Epoch: 18 \tLoss: 1.542843 Acc@1: 63.432836 (ε = 51.00, δ = 1e-05)\n",
      "\tTrain Epoch: 19 \tLoss: 1.447579 Acc@1: 65.625000 (ε = 51.62, δ = 1e-05)\n",
      "\tTrain Epoch: 19 \tLoss: 1.541139 Acc@1: 63.219061 (ε = 52.27, δ = 1e-05)\n",
      "\tTrain Epoch: 20 \tLoss: 1.561364 Acc@1: 65.625000 (ε = 52.89, δ = 1e-05)\n",
      "\tTrain Epoch: 20 \tLoss: 1.576318 Acc@1: 62.624378 (ε = 53.54, δ = 1e-05)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm_notebook\n",
    "\n",
    "for epoch in tqdm_notebook(range(20), desc=\"Epoch\", unit=\"epoch\"):\n",
    "    train(model, train_loader, optimizer, epoch + 1, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the network on test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tTest set:Loss: 2.015386 Acc: 56.615902 \n"
     ]
    }
   ],
   "source": [
    "top1_acc = test(model, test_loader, device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tips and Tricks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Generally speaking, differentially private training is enough of a regularizer by itself. Adding any more regularization (such as dropouts or data augmentation) is unnecessary and typically hurts performance.\n",
    "2. Tuning MAX_GRAD_NORM is very important. Start with a low noise multiplier like .1, this should give comparable performance to a non-private model. Then do a grid search for the optimal MAX_GRAD_NORM value. The grid can be in the range [.1, 10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Private Model vs Non-Private Model Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us compare how our private model compares with the non-private ResNet18.\n",
    "\n",
    "We trained a non-private ResNet18 model for 20 epochs using the same hyper-parameters as above and with BatchNorm replaced with GroupNorm. The results of that training and the training that is discussed in this tutorial are summarized in the table below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Model          | Top 1 Accuracy (%) |  ϵ |\n",
    "|----------------|--------------------|---|\n",
    "| ResNet         | 76                 | ∞ |\n",
    "| Private ResNet |         56.61         |  53.54  |"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5+"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "740px",
    "left": "0px",
    "right": "1628px",
    "top": "161px",
    "width": "253px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}