Hold_Me_Tight.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "# Hold me tight! Influence of discriminative features on deep network boundaries\n\n**Authors**: Guillermo Ortiz-Jimenez, Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli and Pascal Frossard"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Requirements"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "For executing the code, please make sure that you meet the following requirements:\n\n* python (Successfully tested on v3.8.3)\n* [PyTorch](https://pytorch.org/get-started/previous-versions/) (Successfully tested on v1.5.0 with CUDA v10.0.130)\n* [Torchvision](https://pytorch.org/get-started/previous-versions/) (Successfully tested on v0.6.0 with CUDA v10.0.130)\n* [NumPy](https://numpy.org/) (Successfully tested on v1.18.1)\n* [torch-dct](https://github.com/zh217/torch-dct) (Successfully tested on v0.1.5)\n* [Matplotlib](https://matplotlib.org/) (Successfully tested on v3.1.3)\n\nIn our experiments, every package was installed through a Conda environment. Assuming CUDA v10.0.130 and Conda v4.8.1 (installed through [Miniconda3](https://docs.conda.io/en/latest/miniconda.html) on CentOS Linux 7), these are the corresponding commands:\n\n\u0026nbsp;\u0026nbsp;\u0026nbsp; ```conda create -n hold_me_tight python\u003d\u003d3.8.3```  \n\u0026nbsp;\u0026nbsp;\u0026nbsp; ```conda activate hold_me_tight```  \n\u0026nbsp;\u0026nbsp;\u0026nbsp; ```conda install numpy\u003d\u003d1.18.1```  \n\u0026nbsp;\u0026nbsp;\u0026nbsp; ```conda install pytorch\u003d1.5.0 torchvision\u003d0.6.0 cudatoolkit\u003d10.1 -c pytorch```  \n\u0026nbsp;\u0026nbsp;\u0026nbsp; ```pip install torch-dct\u003d\u003d0.1.5```  \n\u0026nbsp;\u0026nbsp;\u0026nbsp; ```conda install matplotlib\u003d\u003d3.1.3```"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## Table of contents\n",
        "\n",
        "- [Margin distribution on diagonal of DCT](#margin)\n",
        "    - [Training LeNet on MNIST](#mnist)\n",
        "    - [Evaluating LeNet on MNIST](#margin)\n",
        "    - [LeNet on flipped MNIST](#flipped)\n",
        "- [Frequency manipulated image examples](#images)\n",
        "    - [Flipped data](#flipped_im)\n",
        "    - [Filtered data](#filtered_im)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## \u003ca name\u003dmargin\u003eMargin distribution on diagonal of DCT\u003ca/\u003e\n",
        "\n",
        "We first give an example of our training procedure and how we obtain the margin distribution of a network. We show this for the LeNet architecture trained on MNIST."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Let\u0027s load the data. Please fix the root path of the project `TREE_ROOT` where all results will be saved."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "TREE_ROOT \u003d \u0027./\u0027"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Set the default PyTorch device"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "\n",
        "DEVICE \u003d \u0027cuda\u0027 if torch.cuda.is_available() else \u0027cpu\u0027"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Create the loaders"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "import torchvision\n",
        "import os\n",
        "\n",
        "DATASET_DIR \u003d os.path.join(TREE_ROOT,\u0027Datasets/\u0027)\n",
        "os.makedirs(DATASET_DIR, exist_ok\u003dTrue)\n",
        "\n",
        "BATCH_SIZE \u003d 128\n",
        "\n",
        "trainset \u003d torchvision.datasets.MNIST(root\u003dDATASET_DIR, download\u003dTrue, train\u003dTrue, transform\u003dtorchvision.transforms.ToTensor())\n",
        "testset \u003d torchvision.datasets.MNIST(root\u003dDATASET_DIR, download\u003dTrue, train\u003dFalse, transform\u003dtorchvision.transforms.ToTensor())\n",
        "\n",
        "pin_memory \u003d True if DEVICE \u003d\u003d \u0027cuda\u0027 else False\n",
        "trainloader \u003d torch.utils.data.DataLoader(trainset, batch_size\u003dBATCH_SIZE, shuffle\u003dTrue,\n",
        "                                              num_workers\u003d2, pin_memory\u003dpin_memory)\n",
        "testloader \u003d torch.utils.data.DataLoader(testset, batch_size\u003dBATCH_SIZE, shuffle\u003dFalse,\n",
        "                                             num_workers\u003d2, pin_memory\u003dpin_memory)\n",
        "\n",
        "mean \u003d torch.tensor([0.1307], device\u003dDEVICE)[None, :, None, None]\n",
        "std \u003d torch.tensor([0.3081], device\u003dDEVICE)[None, :, None, None]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Let\u0027s create the model and the normalization layer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "from model_classes.mnist import LeNet\nfrom model_classes import TransformLayer\n\nmodel \u003d LeNet()\nmodel \u003d model.to(DEVICE)\n\ntrans \u003d TransformLayer(mean\u003dmean, std\u003dstd)"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "The network will be saved here"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "SAVE_TRAIN_DIR \u003d os.path.join(TREE_ROOT, \u0027Models/Generated/%s/%s/\u0027 % (\u0027MNIST\u0027, model.__class__.__name__))\nos.makedirs(SAVE_TRAIN_DIR, exist_ok\u003dTrue)"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "### \u003ca name\u003dmnist\u003eTraining loop\u003c/a\u003e\n\nWe use a standard SGD optimizer with a cyclic learning rate schedule to optimize this network, and a Cross-Entropy loss function. Let\u0027s setup the hyperparameters"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "import torch.nn as nn\nimport numpy as np\nfrom utils import train\n\nEPOCHS \u003d 30\nMAX_LR \u003d 0.21\nMOMENTUM \u003d 0.9\nWEIGHT_DECAY \u003d 5e-4\n\nopt \u003d torch.optim.SGD(model.parameters(), lr\u003dMAX_LR, momentum\u003dMOMENTUM, weight_decay\u003dWEIGHT_DECAY)\nloss_fun \u003d nn.CrossEntropyLoss()\nlr_schedule \u003d lambda t: np.interp([t], [0, EPOCHS * 2 // 5, EPOCHS], [0, MAX_LR, 0])[0]\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": false,
        "pycharm": {}
      },
      "outputs": [],
      "source": "import time\nt0 \u003d time.time()\n\nprint(\u0027---\u003e Training a LeNet architecture on MNIST\u0027)\nmodel \u003d train(model, trans, trainloader, testloader, EPOCHS, opt, loss_fun, lr_schedule, SAVE_TRAIN_DIR)\nprint(\u0027---\u003e Training is done! Elapsed time: %.5f minutes\\n\u0027 % ((time.time() - t0) / 60.))\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "### \u003ca name\u003dmargin\u003eCompute margin distribution\u003c/a\u003e\nOnce we have our trained model we can proceed to measure its margin distribution. In order to approximate the distance to the boundary, we will use the subspace-constrained version of DeepFool. \nThen, we can use a procedure to perform this measurement on a sequence of subspaces.\n\nBut first, we need to generate a list of the desired subspaces.\n"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "from utils import generate_subspace_list\nfrom utils import compute_margin_distribution\n\nSUBSPACE_DIM \u003d 8\nDIM \u003d 28\nSUBSPACE_STEP \u003d 1\n\nsubspace_list \u003d generate_subspace_list(SUBSPACE_DIM, DIM, SUBSPACE_STEP, channels\u003d1)"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Let\u0027s get the result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "RESULTS_DIR \u003d os.path.join(TREE_ROOT, \u0027Results/margin_%s/%s/\u0027 % (\u0027MNIST\u0027, model.__class__.__name__))\nos.makedirs(RESULTS_DIR, exist_ok\u003dTrue)\n\nNUM_SAMPLES_EVAL \u003d 100\nindices \u003d np.random.choice(len(testset), NUM_SAMPLES_EVAL, replace\u003dFalse)\n\neval_dataset \u003d torch.utils.data.Subset(testset, indices[:NUM_SAMPLES_EVAL])\neval_loader \u003d torch.utils.data.DataLoader(eval_dataset, batch_size\u003dBATCH_SIZE,\n                                          shuffle\u003dFalse, num_workers\u003d2, pin_memory\u003dpin_memory)"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "margins \u003d compute_margin_distribution(model, trans, eval_loader, subspace_list, RESULTS_DIR + \u0027margins.npy\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Finally let\u0027s plot the result"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "from graphics import swarmplot\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "swarmplot(margins, color\u003d\u0027tab:blue\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "## \u003ca name\u003dflipped\u003eFlipped MNIST\u003c/a\u003e\n\nWe now repeat the same procedure for the frequency flipped version of MNIST. We start first by flipping the data."
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "from utils_dct import dct_flip\n\nflip_train_data \u003d dct_flip(trainset.data.type(torch.float32).view([-1, 1, 28, 28]) / 255.)\nflip_test_data \u003d dct_flip(testset.data.type(torch.float32).view([-1, 1, 28, 28]) / 255.)\n\nflip_trainset \u003d torch.utils.data.TensorDataset(flip_train_data, trainset.targets)\nflip_testset \u003d torch.utils.data.TensorDataset(flip_test_data, testset.targets)\n\nflip_trainloader \u003d torch.utils.data.DataLoader(flip_trainset, batch_size\u003dBATCH_SIZE, shuffle\u003dTrue,\n                                               num_workers\u003d2, pin_memory\u003dpin_memory)\nflip_testloader \u003d torch.utils.data.DataLoader(flip_testset, batch_size\u003dBATCH_SIZE, shuffle\u003dFalse,\n                                              num_workers\u003d2, pin_memory\u003dpin_memory)"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "Because we want to simply rotate (flip) the data we feed the network, we need to modify the normalization layer to account for the new representation"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "from model_classes import TransformFlippedLayer\nflip_trans \u003d TransformFlippedLayer(mean, std, [1, 28, 28], DEVICE)"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": "Let\u0027s train again a LeNet but on the data of the new (flipped) representation"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "flip_model \u003d LeNet()\nflip_model \u003d flip_model.to(DEVICE)\n\nSAVE_TRAIN_DIR \u003d os.path.join(TREE_ROOT, \u0027Models/Generated/%s_flipped/%s/\u0027 % (\u0027MNIST\u0027, model.__class__.__name__))\nos.makedirs(SAVE_TRAIN_DIR, exist_ok\u003dTrue)\n\nopt \u003d torch.optim.SGD(flip_model.parameters(), lr\u003dMAX_LR, momentum\u003dMOMENTUM, weight_decay\u003dWEIGHT_DECAY)\nloss_fun \u003d nn.CrossEntropyLoss()\nlr_schedule \u003d lambda t: np.interp([t], [0, EPOCHS * 2 // 5, EPOCHS], [0, MAX_LR, 0])[0]\n\nt0 \u003d time.time()\n\nprint(\u0027\\n---\u003e Training a LeNet architecture on Flipped MNIST\u0027)\nflip_model \u003d train(flip_model, flip_trans, flip_trainloader, flip_testloader, EPOCHS, opt, loss_fun, lr_schedule, SAVE_TRAIN_DIR)\nprint(\u0027---\u003e Training is done! Elapsed time: %.5f minutes\\n\u0027 % ((time.time() - t0) / 60.))"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "And compute the margin distribution of this model for the same data as before, but with a flipped representation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "flip_eval_dataset \u003d torch.utils.data.Subset(flip_testset, indices[:NUM_SAMPLES_EVAL])\nflip_eval_loader \u003d torch.utils.data.DataLoader(flip_eval_dataset, batch_size\u003dBATCH_SIZE,\n                                          shuffle\u003dFalse, num_workers\u003d2, pin_memory\u003dpin_memory)\n\nRESULTS_DIR \u003d os.path.join(TREE_ROOT, \u0027Results/margin_%s_flipped/%s/\u0027 % (\u0027MNIST\u0027, model.__class__.__name__))\nos.makedirs(RESULTS_DIR, exist_ok\u003dTrue)\nflip_margins \u003d compute_margin_distribution(flip_model, flip_trans, flip_eval_loader, subspace_list, RESULTS_DIR + \u0027margins.npy\u0027)"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "swarmplot(flip_margins, color\u003d\u0027tab:red\u0027)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "## \u003ca name\u003dimages\u003eFrequency manipulated image examples\u003c/a\u003e\n",
        "\n",
        "We now show a few image examples from the modified versions of the standard datasets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "### \u003ca name\u003dflipped_im\u003eFlipped images\u003c/a\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Reproduce the flipped ImageNet examples of the Supplementary material"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "from PIL import Image\n",
        "import os\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "IMAGE_DIR \u003d os.path.join(TREE_ROOT, \u0027Images/\u0027)\n",
        "DATASET \u003d \u0027ImageNet\u0027\n",
        "\n",
        "for file in os.listdir(IMAGE_DIR + DATASET):\n",
        "    im \u003d Image.open(os.path.join(IMAGE_DIR, DATASET, file))\n",
        "\n",
        "    im_flipped \u003d dct_flip(torch.from_numpy(np.asarray(im).transpose((2, 0, 1))).float())\n",
        "    im_flipped \u003d im_flipped - im_flipped.min()\n",
        "    im_flipped \u003d im_flipped / im_flipped.max()\n",
        "    im_flipped \u003d Image.fromarray((255 * im_flipped.permute(1, 2, 0).numpy()).astype(np.uint8))\n",
        "\n",
        "    cat_im \u003d Image.new(im.mode, (2 * im.size[0], im.size[1]))\n",
        "    cat_im.paste(im, (0, 0))\n",
        "    cat_im.paste(im_flipped, (im.size[0], 0))\n",
        "\n",
        "    display(cat_im)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Reproduce the flipped CIFAR-10 examples of the Supplementary material"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "DATASET \u003d \u0027CIFAR10\u0027\n",
        "\n",
        "for file in os.listdir(IMAGE_DIR + DATASET):\n",
        "    im \u003d Image.open(os.path.join(IMAGE_DIR, DATASET, file))\n",
        "\n",
        "    im_flipped \u003d dct_flip(torch.from_numpy(np.asarray(im).transpose((2, 0, 1))).float())\n",
        "    im_flipped \u003d im_flipped - im_flipped.min()\n",
        "    im_flipped \u003d im_flipped / im_flipped.max()\n",
        "    im_flipped \u003d Image.fromarray((255 * im_flipped.permute(1, 2, 0).numpy()).astype(np.uint8))\n",
        "\n",
        "    cat_im \u003d Image.new(im.mode, (2 * im.size[0], im.size[1]))\n",
        "    cat_im.paste(im, (0, 0))\n",
        "    cat_im.paste(im_flipped, (im.size[0], 0))\n",
        "\n",
        "    display(cat_im)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Reproduce the flipped MNIST examples of the Supplementary material"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "DATASET \u003d \u0027MNIST\u0027\n",
        "\n",
        "for file in os.listdir(IMAGE_DIR + DATASET):\n",
        "    im \u003d Image.open(os.path.join(IMAGE_DIR, DATASET, file))\n",
        "\n",
        "    im_flipped \u003d dct_flip(torch.from_numpy(np.asarray(im)).float())\n",
        "    im_flipped \u003d im_flipped - im_flipped.min()\n",
        "    im_flipped \u003d im_flipped / im_flipped.max()\n",
        "    im_flipped \u003d Image.fromarray((255 * im_flipped.numpy()).astype(np.uint8))\n",
        "\n",
        "    cat_im \u003d Image.new(im.mode, (2 * im.size[0], im.size[1]))\n",
        "    cat_im.paste(im, (0, 0))\n",
        "    cat_im.paste(im_flipped, (im.size[0], 0))\n",
        "\n",
        "    display(cat_im)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "### \u003ca name\u003dfiltered_im\u003eFiltered images\u003c/a\u003e"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": "from utils_dct import dct_low_pass\nfrom utils_dct import dct_high_pass\nfrom utils_dct import dct_cutoff_low\n"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Reproduce the low-pass and high-pass CIFAR-10 examples of the Supplementary material"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "DATASET \u003d \u0027CIFAR10\u0027\n",
        "BANDWIDTH \u003d 16\n",
        "\n",
        "for file in os.listdir(IMAGE_DIR + DATASET):\n",
        "    im \u003d Image.open(os.path.join(IMAGE_DIR, DATASET, file))\n",
        "\n",
        "    im_lp \u003d dct_low_pass(torch.from_numpy(np.asarray(im).transpose((2, 0, 1))).float(), bandwidth\u003dBANDWIDTH)\n",
        "    im_lp \u003d im_lp - im_lp.min()\n",
        "    im_lp \u003d im_lp / im_lp.max()\n",
        "    im_lp \u003d Image.fromarray((255 * im_lp.permute(1, 2, 0).numpy()).astype(np.uint8))\n",
        "    \n",
        "    im_hp \u003d dct_high_pass(torch.from_numpy(np.asarray(im).transpose((2, 0, 1))).float(), bandwidth\u003dBANDWIDTH)\n",
        "    im_hp \u003d im_hp - im_hp.min()\n",
        "    im_hp \u003d im_hp / im_hp.max()\n",
        "    im_hp \u003d Image.fromarray((255 * im_hp.permute(1, 2, 0).numpy()).astype(np.uint8))\n",
        "\n",
        "    cat_im \u003d Image.new(im.mode, (3 * im.size[0], im.size[1]))\n",
        "    cat_im.paste(im, (0, 0))\n",
        "    cat_im.paste(im_lp, (im.size[0], 0))\n",
        "    cat_im.paste(im_hp, (2 * im.size[0], 0))\n",
        "\n",
        "    display(cat_im)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "pycharm": {}
      },
      "source": [
        "Reproduce the high-pass MNIST examples of the Supplementary material"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [],
      "source": [
        "DATASET \u003d \u0027MNIST\u0027\n",
        "BANDWIDTH \u003d 14\n",
        "\n",
        "for file in os.listdir(IMAGE_DIR + DATASET):\n",
        "    im \u003d Image.open(os.path.join(IMAGE_DIR, DATASET, file))\n",
        "    \n",
        "    im_hp \u003d dct_cutoff_low(torch.from_numpy(np.asarray(im)).float(), bandwidth\u003dBANDWIDTH)\n",
        "    im_hp \u003d im_hp - im_hp.min()\n",
        "    im_hp \u003d im_hp / im_hp.max()\n",
        "    im_hp \u003d Image.fromarray((255 * im_hp.numpy()).astype(np.uint8))\n",
        "\n",
        "    cat_im \u003d Image.new(im.mode, (2 * im.size[0], im.size[1]))\n",
        "    cat_im.paste(im, (0, 0))\n",
        "    cat_im.paste(im_hp, (im.size[0], 0))\n",
        "\n",
        "    display(cat_im)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}