advanced_functionality/pytorch_bring_your_own_gan/build_gan_with_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build GAN (Generative Adversarial Networks) with PyTorch and SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### About GAN\n",
    "\n",
    "Generative Adversarial Network (GAN) is a generative machine learning model, which is widely used in advertising, games, entertainment, media, pharmaceuticals and other industries. It can be used to create fictional characters and scenes, simulate facial aging, and change image styles, and produce chemical formulas and so on.\n",
    "\n",
    "GAN was proposed by Ian Goodfellow in 2014, it is a deep neural network architecture consisting of a generative network and a discriminant network. The generation network generates \"fake\" data and tries to deceive the discrimination network; the discrimination network authenticates the generated data and tries to correctly identify all \"fake\" data. In the process of training iterations, the two networks continue to evolve and confront until they reach an equilibrium state (reference: Nash equilibrium), the discriminant network can no longer recognize \"fake\" data, and the training ends.\n",
    "\n",
    "This example will lead you to build a GAN model leveraging the PyTorch framework, introducing GAN from the perspective of engineering practice, and opening a new and interesting AI/ML experience in generative models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Environment setup\n",
    "Upgrade packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade pip sagemaker awscli boto3 numpy ipywidgets\n",
    "!pip install Pillow==7.1.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create folders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p data src tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download data\n",
    "There are many public datasets on the Internet, which are very helpful for machine learning engineering and scientific research, such as algorithm study and evaluation. We will use MNIST dataset, which is a handwritten digits dataset, we will use it to train a GAN model, and eventually generate some fake \"handwritten\" digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from sagemaker.s3 import S3Downloader as s3down\n",
    "\n",
    "s3down.download(\n",
    "    f\"s3://sagemaker-example-files-prod-{boto3.session.Session().region_name}/datasets/image/MNIST/pytorch/\",\n",
    "    \"./data\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "isConfigCell": true
   },
   "source": [
    "### Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "isConfigCell": true
   },
   "source": [
    "PyTorch framework has a torchvision.datasets package, which provides access to a number of datasets, you may use the following commands to read MNIST pre-downloaded dataset from local storage, for later use.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from torchvision import datasets\n",
    "\n",
    "dataroot = \"./data\"\n",
    "trainset = datasets.MNIST(root=dataroot, train=True, download=False)\n",
    "testset = datasets.MNIST(root=dataroot, train=False, download=False)\n",
    "print(trainset)\n",
    "print(testset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "isConfigCell": true
   },
   "source": [
    "SageMaker SDK will create a default Amazon S3 bucket for you to access various files and data, that you may need in the machine learning engineering lifecycle. We can get the name of this bucket through the default_bucket method of the sagemaker.session.Session class in the SageMaker SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.session import Session\n",
    "\n",
    "sess = Session()\n",
    "\n",
    "# S3 bucket for saving code and model artifacts.\n",
    "# Feel free to specify a different bucket here if you wish.\n",
    "bucket = sess.default_bucket()\n",
    "prefix = \"byos-pytorch-gan\"\n",
    "\n",
    "# Location to save your custom code in tar.gz format.\n",
    "s3_custom_code_upload_location = f\"s3://{bucket}/{prefix}/customcode\"\n",
    "\n",
    "# Location where results of model training are saved.\n",
    "s3_model_artifacts_location = f\"s3://{bucket}/{prefix}/artifacts/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SageMaker SDK provides tools for operating AWS services. For example, the S3Downloader class is used to download objects in S3, and the S3Uploader is used to upload local files to S3. You will upload the dataset files to Amazon S3 for model training. During model training, we do not download data from the Internet to avoid network latency caused by fetching data from the Internet, and at the same time avoiding possible security risks due to direct access to the Internet.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from sagemaker.s3 import S3Uploader as s3up\n",
    "\n",
    "s3_data_location = s3up.upload(\n",
    "    os.path.join(dataroot, \"MNIST\"), f\"s3://{bucket}/{prefix}/data/mnist\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DCGAN (Deep Convolutional Generative Adversarial Networks) is a variant of the GAN families. This architecture essentially leverages Deep Convolutional Neural Networks to generate images belonging to a given distribution from noisy data using the Generator-Discriminator framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile src/train.py\n",
    "\n",
    "from __future__ import print_function\n",
    "import argparse\n",
    "import json\n",
    "import logging\n",
    "import os\n",
    "import sys\n",
    "import random\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.parallel\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import torch.optim as optim\n",
    "import torch.backends.cudnn as cudnn\n",
    "import torch.utils.data\n",
    "import torchvision.datasets as dset\n",
    "import torchvision.transforms as transforms\n",
    "import torchvision.utils as vutils\n",
    "\n",
    "\n",
    "cudnn.benchmark = True\n",
    "\n",
    "logger = logging.getLogger(__name__)\n",
    "logger.setLevel(logging.DEBUG)\n",
    "logger.addHandler(logging.StreamHandler(sys.stdout))\n",
    "\n",
    "\n",
    "class Generator(nn.Module):\n",
    "    def __init__(self, *, nz, nc, ngf, ngpu=1):\n",
    "        super(Generator, self).__init__()\n",
    "        self.ngpu = ngpu\n",
    "        self.main = nn.Sequential(\n",
    "            # input is Z, going into a convolution\n",
    "            nn.ConvTranspose2d(nz, ngf * 8, 4, 1, 0, bias=False),\n",
    "            nn.BatchNorm2d(ngf * 8),\n",
    "            nn.ReLU(True),\n",
    "            # state size. (ngf*8) x 4 x 4\n",
    "            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),\n",
    "            nn.BatchNorm2d(ngf * 4),\n",
    "            nn.ReLU(True),\n",
    "            # state size. (ngf*4) x 8 x 8\n",
    "            nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),\n",
    "            nn.BatchNorm2d(ngf * 2),\n",
    "            nn.ReLU(True),\n",
    "            # state size. (ngf*2) x 16 x 16\n",
    "            nn.ConvTranspose2d(ngf * 2, ngf, 4, 2, 1, bias=False),\n",
    "            nn.BatchNorm2d(ngf),\n",
    "            nn.ReLU(True),\n",
    "            # state size. (ngf) x 32 x 32\n",
    "            nn.ConvTranspose2d(ngf, nc, 4, 2, 1, bias=False),\n",
    "            nn.Tanh()\n",
    "            # state size. (nc) x 64 x 64\n",
    "        )\n",
    "\n",
    "    def forward(self, input):\n",
    "        if input.is_cuda and self.ngpu > 1:\n",
    "            output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))\n",
    "        else:\n",
    "            output = self.main(input)\n",
    "        return output\n",
    "\n",
    "    def save(self, path, *, filename=None, device=\"cpu\"):\n",
    "        # recommended way from http://pytorch.org/docs/master/notes/serialization.html\n",
    "        self.to(device)\n",
    "        if not filename is None:\n",
    "            path = os.path.join(path, filename)\n",
    "        torch.save(self.state_dict(), path)\n",
    "\n",
    "    def load(self, path, *, filename=None):\n",
    "        if not filename is None:\n",
    "            path = os.path.join(path, filename)\n",
    "        with open(path, \"rb\") as f:\n",
    "            self.load_state_dict(torch.load(f))\n",
    "\n",
    "\n",
    "class Discriminator(nn.Module):\n",
    "    def __init__(self, *, nc, ndf, ngpu=1):\n",
    "        super(Discriminator, self).__init__()\n",
    "        self.ngpu = ngpu\n",
    "        self.main = nn.Sequential(\n",
    "            # input is (nc) x 64 x 64\n",
    "            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),\n",
    "            nn.LeakyReLU(0.2, inplace=True),\n",
    "            # state size. (ndf) x 32 x 32\n",
    "            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),\n",
    "            nn.BatchNorm2d(ndf * 2),\n",
    "            nn.LeakyReLU(0.2, inplace=True),\n",
    "            # state size. (ndf*2) x 16 x 16\n",
    "            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),\n",
    "            nn.BatchNorm2d(ndf * 4),\n",
    "            nn.LeakyReLU(0.2, inplace=True),\n",
    "            # state size. (ndf*4) x 8 x 8\n",
    "            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),\n",
    "            nn.BatchNorm2d(ndf * 8),\n",
    "            nn.LeakyReLU(0.2, inplace=True),\n",
    "            # state size. (ndf*8) x 4 x 4\n",
    "            nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),\n",
    "            nn.Sigmoid(),\n",
    "        )\n",
    "\n",
    "    def forward(self, input):\n",
    "        if input.is_cuda and self.ngpu > 1:\n",
    "            output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))\n",
    "        else:\n",
    "            output = self.main(input)\n",
    "\n",
    "        return output.view(-1, 1).squeeze(1)\n",
    "\n",
    "    def save(self, path, *, filename=None, device=\"cpu\"):\n",
    "        # recommended way from http://pytorch.org/docs/master/notes/serialization.html\n",
    "        self.to(device)\n",
    "        if not filename is None:\n",
    "            path = os.path.join(path, filename)\n",
    "        torch.save(self.state_dict(), path)\n",
    "\n",
    "    def load(self, path, *, filename=None):\n",
    "        if not filename is None:\n",
    "            path = os.path.join(path, filename)\n",
    "        with open(path, \"rb\") as f:\n",
    "            self.load_state_dict(torch.load(f))\n",
    "\n",
    "\n",
    "class DCGAN(object):\n",
    "    \"\"\"\n",
    "    A wrapper class for Generator and Discriminator,\n",
    "    'train_step' method is for single batch training.\n",
    "    \"\"\"\n",
    "\n",
    "    fixed_noise = None\n",
    "    criterion = None\n",
    "    device = None\n",
    "    netG = None\n",
    "    netD = None\n",
    "    optimizerG = None\n",
    "    optimizerD = None\n",
    "    nz = None\n",
    "    nc = None\n",
    "    ngf = None\n",
    "    ndf = None\n",
    "    real_cpu = None\n",
    "\n",
    "    def __init__(\n",
    "        self,\n",
    "        *,\n",
    "        batch_size,\n",
    "        nz,\n",
    "        nc,\n",
    "        ngf,\n",
    "        ndf,\n",
    "        device,\n",
    "        weights_init,\n",
    "        learning_rate,\n",
    "        betas,\n",
    "        real_label,\n",
    "        fake_label,\n",
    "    ):\n",
    "        super(DCGAN, self).__init__()\n",
    "\n",
    "        import torch\n",
    "\n",
    "        self.nz = nz\n",
    "        self.nc = nc\n",
    "        self.ngf = ngf\n",
    "        self.ndf = ndf\n",
    "\n",
    "        self.real_label = real_label\n",
    "        self.fake_label = fake_label\n",
    "\n",
    "        self.fixed_noise = torch.randn(batch_size, nz, 1, 1, device=device)\n",
    "        self.criterion = nn.BCELoss()\n",
    "        self.device = device\n",
    "\n",
    "        self.netG = Generator(nz=nz, nc=nc, ngf=ngf).to(device)\n",
    "        # print(netG)\n",
    "        self.netD = Discriminator(nc=nc, ndf=ndf).to(device)\n",
    "        # print(netD)\n",
    "\n",
    "        self.netG.apply(weights_init)\n",
    "        self.netD.apply(weights_init)\n",
    "\n",
    "        # setup optimizer\n",
    "        self.optimizerG = optim.Adam(self.netG.parameters(), lr=learning_rate, betas=betas)\n",
    "        self.optimizerD = optim.Adam(self.netD.parameters(), lr=learning_rate, betas=betas)\n",
    "\n",
    "    def train_step(self, data, *, epoch, epochs):\n",
    "        import torch\n",
    "\n",
    "        ############################\n",
    "        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))\n",
    "        ###########################\n",
    "        # train with real\n",
    "        self.netD.zero_grad()\n",
    "        self.real_cpu = data[0]\n",
    "        real = data[0].to(self.device)\n",
    "        batch_size = real.size(0)\n",
    "        label = torch.full((batch_size,), self.real_label, device=self.device)\n",
    "\n",
    "        output = self.netD(real).view(-1)\n",
    "        errD_real = self.criterion(output, label)\n",
    "        errD_real.backward()\n",
    "        D_x = output.mean().item()\n",
    "\n",
    "        # train with fake\n",
    "        noise = torch.randn(batch_size, self.nz, 1, 1, device=self.device)\n",
    "        fake = self.netG(noise)\n",
    "        label.fill_(self.fake_label)\n",
    "        output = self.netD(fake.detach()).view(-1)\n",
    "        errD_fake = self.criterion(output, label)\n",
    "        errD_fake.backward()\n",
    "        D_G_z1 = output.mean().item()\n",
    "        errD = errD_real + errD_fake\n",
    "        self.optimizerD.step()\n",
    "\n",
    "        ############################\n",
    "        # (2) Update G network: maximize log(D(G(z)))\n",
    "        ###########################\n",
    "        self.netG.zero_grad()\n",
    "        label.fill_(self.real_label)  # fake labels are real for generator cost\n",
    "        output = self.netD(fake).view(-1)\n",
    "        errG = self.criterion(output, label)\n",
    "        errG.backward()\n",
    "        D_G_z2 = output.mean().item()\n",
    "        self.optimizerG.step()\n",
    "\n",
    "        return errG.item(), errD.item(), D_x, D_G_z1, D_G_z2\n",
    "\n",
    "\n",
    "# custom weights initialization called on netG and netD\n",
    "def weights_init(m):\n",
    "    classname = m.__class__.__name__\n",
    "    if classname.find(\"Conv\") != -1:\n",
    "        torch.nn.init.normal_(m.weight, 0.0, 0.02)\n",
    "    elif classname.find(\"BatchNorm\") != -1:\n",
    "        torch.nn.init.normal_(m.weight, 1.0, 0.02)\n",
    "        torch.nn.init.zeros_(m.bias)\n",
    "\n",
    "\n",
    "def log_batch(\n",
    "    epoch, epochs, batch, batches, errD, errG, D_x, D_G_z1, D_G_z2, *, log_interval=10, output_dir\n",
    "):\n",
    "    if batch % log_interval == 0:\n",
    "        logger.info(\n",
    "            f\"Epoch[{epoch}/{epochs}], Batch[{batch}/{batches}], \"\n",
    "            + f\"Loss_D: {errD:.4}, Loss_G: {errG:.4}, D(x): {D_x:.4}, D(G(z)): {D_G_z1:.4}/{D_G_z2:.4}\"\n",
    "        )\n",
    "\n",
    "\n",
    "def get_device(use_cuda):\n",
    "    import torch\n",
    "\n",
    "    device = \"cpu\"\n",
    "    num_gpus = 0\n",
    "\n",
    "    if torch.cuda.is_available():\n",
    "        if use_cuda:\n",
    "            device = \"cuda\"\n",
    "            torch.cuda.set_device(0)\n",
    "            num_gpus = torch.cuda.device_count()\n",
    "        else:\n",
    "            logger.debug(\n",
    "                \"WARNING: You have a CUDA device, so you should probably run with --cuda 1\"\n",
    "            )\n",
    "\n",
    "    logger.debug(f\"Number of gpus available: {num_gpus}\")\n",
    "\n",
    "    return device, num_gpus\n",
    "\n",
    "\n",
    "def train(dataloader, hps, test_batch_size, device, model_dir, output_dir, seed, log_interval):\n",
    "    epochs = hps[\"epochs\"]\n",
    "    batch_size = hps[\"batch-size\"]\n",
    "    nz = hps[\"nz\"]\n",
    "    ngf = hps[\"ngf\"]\n",
    "    ndf = hps[\"ndf\"]\n",
    "    learning_rate = hps[\"learning-rate\"]\n",
    "    beta1 = hps[\"beta1\"]\n",
    "\n",
    "    dcgan = DCGAN(\n",
    "        batch_size=batch_size,\n",
    "        nz=nz,\n",
    "        nc=1,\n",
    "        ngf=ngf,\n",
    "        ndf=ndf,\n",
    "        device=device,\n",
    "        weights_init=weights_init,\n",
    "        learning_rate=learning_rate,\n",
    "        betas=(beta1, 0.999),\n",
    "        real_label=1,\n",
    "        fake_label=0,\n",
    "    )\n",
    "\n",
    "    for epoch in range(epochs):\n",
    "        batches = len(dataloader)\n",
    "        for batch, data in enumerate(dataloader, 0):\n",
    "            errG, errD, D_x, D_G_z1, D_G_z2 = dcgan.train_step(data, epoch=epoch, epochs=epochs)\n",
    "\n",
    "            log_batch(\n",
    "                epoch,\n",
    "                epochs,\n",
    "                batch,\n",
    "                batches,\n",
    "                errD,\n",
    "                errG,\n",
    "                D_x,\n",
    "                D_G_z1,\n",
    "                D_G_z2,\n",
    "                log_interval=log_interval,\n",
    "                output_dir=output_dir,\n",
    "            )\n",
    "\n",
    "    save_model(model_dir, dcgan.netG)\n",
    "\n",
    "    return\n",
    "\n",
    "\n",
    "def save_model(model_dir, model):\n",
    "    logger.info(\"Saving the model.\")\n",
    "    model.save(model_dir, filename=\"model.pth\")\n",
    "\n",
    "\n",
    "def load_model(model_dir, device=None):\n",
    "    logger.info(\"Loading the model.\")\n",
    "    if device is None:\n",
    "        device = get_training_device_name(1)\n",
    "\n",
    "    netG.load(model_dir, filename=\"model.pth\", device=device)\n",
    "\n",
    "    return netG\n",
    "\n",
    "\n",
    "def parse_args():\n",
    "    # Training settings\n",
    "    parser = argparse.ArgumentParser(description=\"PyTorch Example\")\n",
    "\n",
    "    parser.add_argument(\n",
    "        \"--batch-size\", type=int, default=1000, metavar=\"N\", help=\"input batch size (default: 1000)\"\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        \"--test-batch-size\",\n",
    "        type=int,\n",
    "        default=1000,\n",
    "        metavar=\"N\",\n",
    "        help=\"input batch size for testing (default: 1000)\",\n",
    "    )\n",
    "    parser.add_argument(\"--seed\", type=int, default=None, metavar=\"S\", help=\"random seed\")\n",
    "    parser.add_argument(\n",
    "        \"--log-interval\",\n",
    "        type=int,\n",
    "        default=10,\n",
    "        metavar=\"N\",\n",
    "        help=\"how many batches to wait before logging training status\",\n",
    "    )\n",
    "\n",
    "    parser.add_argument(\n",
    "        \"--save-model\", action=\"store_true\", default=False, help=\"For Saving the current Model\"\n",
    "    )\n",
    "\n",
    "    parser.add_argument(\"--model-dir\", type=str, default=os.environ.get(\"SM_MODEL_DIR\", None))\n",
    "\n",
    "    parser.add_argument(\"--cuda\", type=int, default=1)\n",
    "    parser.add_argument(\"--num-gpus\", type=int, default=os.environ.get(\"SM_NUM_GPUS\", None))\n",
    "\n",
    "    parser.add_argument(\"--pin-memory\", type=bool, default=os.environ.get(\"SM_PIN_MEMORY\", False))\n",
    "\n",
    "    parser.add_argument(\"--data-dir\", required=False, default=None, help=\"path to data dir\")\n",
    "    parser.add_argument(\"--workers\", type=int, help=\"number of data loading workers\", default=2)\n",
    "    parser.add_argument(\n",
    "        \"--output-dir\",\n",
    "        default=os.environ.get(\"SM_OUTPUT_DATA_DIR\", None),\n",
    "        help=\"folder to output images and model checkpoints\",\n",
    "    )\n",
    "    parser.add_argument(\"--hps\", default=os.environ.get(\"SM_HPS\", None), help=\"Hyperparameters\")\n",
    "\n",
    "    return parser.parse_known_args()\n",
    "\n",
    "\n",
    "def get_datasets(*, dataroot=\"/opt/ml/input/data\", classes=None):\n",
    "    dataset = dset.MNIST(\n",
    "        root=dataroot,\n",
    "        transform=transforms.Compose(\n",
    "            [\n",
    "                transforms.Resize(64),\n",
    "                transforms.ToTensor(),\n",
    "                transforms.Normalize((0.5,), (0.5,)),\n",
    "            ]\n",
    "        ),\n",
    "    )\n",
    "\n",
    "    return dataset\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    args, unknown = parse_args()\n",
    "\n",
    "    # get training options\n",
    "    hps = json.loads(args.hps)\n",
    "\n",
    "    try:\n",
    "        os.makedirs(args.output_dir)\n",
    "    except OSError:\n",
    "        pass\n",
    "\n",
    "    if args.seed is None:\n",
    "        random_seed = random.randint(1, 10000)\n",
    "        logger.debug(f\"Generated Random Seed: {random_seed}\")\n",
    "        cudnn.benchmark = True\n",
    "    else:\n",
    "        logger.debug(f\"Provided Random Seed: {args.seed}\")\n",
    "        random_seed = args.seed\n",
    "        cudnn.deterministic = True\n",
    "        cudnn.benchmark = False\n",
    "\n",
    "    random.seed(random_seed)\n",
    "    torch.manual_seed(random_seed)\n",
    "\n",
    "    pin_memory = args.pin_memory\n",
    "    num_workers = int(args.workers)\n",
    "\n",
    "    device, num_gpus = get_device(args.cuda)\n",
    "\n",
    "    if device == \"cuda\":\n",
    "        num_workers = 1\n",
    "        pin_memory = True\n",
    "\n",
    "    if args.data_dir is None:\n",
    "        input_dir = os.environ.get(\"SM_INPUT_DIR\", None)\n",
    "        if input_dir is None and str(args.dataset).lower() != \"fake\":\n",
    "            raise ValueError(f'`--data-dir` parameter is required for dataset \"{args.dataset}\"')\n",
    "\n",
    "        dataroot = input_dir + \"/data\"\n",
    "    else:\n",
    "        dataroot = args.data_dir\n",
    "\n",
    "    dataset = get_datasets(dataroot=dataroot)\n",
    "\n",
    "    assert dataset\n",
    "    dataloader = torch.utils.data.DataLoader(\n",
    "        dataset,\n",
    "        batch_size=args.batch_size,\n",
    "        shuffle=True,\n",
    "        num_workers=num_workers,\n",
    "        pin_memory=pin_memory,\n",
    "    )\n",
    "\n",
    "    train(\n",
    "        dataloader,\n",
    "        hps,\n",
    "        args.test_batch_size,\n",
    "        device,\n",
    "        args.model_dir,\n",
    "        args.output_dir,\n",
    "        args.seed,\n",
    "        args.log_interval,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Per sagemaker.get_execution_role() method, the notebook can get the role pre-assigned to the notebook instance. This role will be used to obtain training resources, such as downloading training framework images, allocating Amazon EC2 instances, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sagemaker import get_execution_role\n",
    "\n",
    "# IAM execution role that gives SageMaker access to resources in your AWS account.\n",
    "# We can use the SageMaker Python SDK to get the role from our notebook environment.\n",
    "role = get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hyperparameters, that used in the model training tasks, can be defined in the notebook so that it is separated from the algorithm and training code. The hyperparameters are passed in when the training task is created and dynamically combined with the training task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "hps = {\n",
    "    \"seed\": 0,\n",
    "    \"learning-rate\": 0.0002,\n",
    "    \"epochs\": 18,\n",
    "    \"pin-memory\": 1,\n",
    "    \"beta1\": 0.5,\n",
    "    \"nz\": 100,\n",
    "    \"ngf\": 28,\n",
    "    \"ndf\": 28,\n",
    "    \"batch-size\": 128,\n",
    "    \"log-interval\": 20,\n",
    "}\n",
    "\n",
    "\n",
    "str_hps = json.dumps(hps, indent=4)\n",
    "print(str_hps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```PyTorch``` class from sagemaker.pytorch package, is an estimator for PyTorch framework, it can be used to create and execute training tasks, as well as to deploy trained models. In the parameter list, ``instance_type`` is used to specify the instance type, such as CPU or GPU instances. The directory containing training script and the model code are specified by ``source_dir``, and the training script file name must be clearly defined by ``entry_point``. These parameters will be passed to the training task along with other parameters, and they determine the environment settings of the training task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.pytorch import PyTorch\n",
    "\n",
    "estimator = PyTorch(\n",
    "    role=role,\n",
    "    entry_point=\"train.py\",\n",
    "    source_dir=\"./src\",\n",
    "    output_path=s3_model_artifacts_location,\n",
    "    code_location=s3_custom_code_upload_location,\n",
    "    instance_count=1,\n",
    "    instance_type=\"ml.g4dn.2xlarge\",\n",
    "    framework_version=\"1.5.0\",\n",
    "    py_version=\"py3\",\n",
    "    hyperparameters=hps,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please pay special attention to the ``train_use_spot_instances`` parameter. The value of ``True`` means that you want to use SPOT instances first. Since machine learning training usually requires a large amount of computing resources to run for a long time, leveraging SPOT instances can help you control your cost. The SPOT instances may save cost up to 90% of the on-demand instances, depending on the instance type, region, and time, the actual price might be different."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have created a PyTorch object, and you can use it to fit pre-uploaded data on Amazon S3. The following command will initiate the training task, and the training data will be imported into the training environment in the form of an input channel named **MNIST**. When the training task starts, the training data was already downloaded from S3 to the local file system of the training instance, and the training script ```train.py``` will load the data from the local disk afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Start training\n",
    "estimator.fit({\"MNIST\": s3_data_location}, wait=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on the training instance you choose, the training process may last from tens of minutes to several hours. It is recommended to set the ``wait`` parameter to ``False``, this option will detach the notebook from the training task. In scenarios with long training time and many training logs, it can prevent the notebook context from being lost due to network interruption or session timeout. After the notebook detached from the training task, the output will be temporarily invisible. You can execute the following code, and the notebook will obtain and resume the previous training session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "from sagemaker.estimator import Estimator\n",
    "\n",
    "# Attaching previous training session\n",
    "training_job_name = estimator.latest_training_job.name\n",
    "attached_estimator = Estimator.attach(training_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the model was designed to leverage the GPU power to accelerate training, it will be much faster on GPU instances than on CPU instances. For example, the g4dn.2xlarge instance will take about 12 minutes, while the c5.xlarge instance may take more than 6 hours. The current model does not support multi-instance training, so instance_count parameter, with value more than 1, will not bring extra benefits in training time optimisation.\n",
    "\n",
    "When the training completes, the trained model will be uploaded to S3. The upload location is specified by the `output_path` parameter provided when creating the `PyTorch` object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "isConfigCell": true
   },
   "source": [
    "### Model verification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "isConfigCell": true
   },
   "source": [
    "You will download the trained model from Amazon S3 to the local file system of the instance where the notebook is located. The following code will load the model, and then generate a picture with a random number as input, then display picture.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "from sagemaker.s3 import S3Downloader as s3down\n",
    "\n",
    "model_url = attached_estimator.model_data\n",
    "s3down.download(model_url, \"./tmp\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "!tar -zxf tmp/model.tar.gz -C ./tmp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "isConfigCell": true
   },
   "source": [
    "Execute the following instructions to load the trained model, and generate a set of \"handwritten\" digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "def generate_fake_handwriting(model, *, num_images, nz, device=None):\n",
    "    import torch\n",
    "    import torchvision.utils as vutils\n",
    "    from io import BytesIO\n",
    "    from PIL import Image\n",
    "\n",
    "    z = torch.randn(num_images, nz, 1, 1, device=device)\n",
    "    fake = model(z)\n",
    "\n",
    "    imgio = BytesIO()\n",
    "    vutils.save_image(fake.detach(), imgio, normalize=True, format=\"PNG\")\n",
    "    img = Image.open(imgio)\n",
    "\n",
    "    return img\n",
    "\n",
    "\n",
    "def load_model(path, *, model_cls=None, params=None, filename=None, device=None, strict=True):\n",
    "    import os\n",
    "    import torch\n",
    "\n",
    "    model_pt_path = path\n",
    "    if not filename is None:\n",
    "        model_pt_path = os.path.join(path, filename)\n",
    "\n",
    "    if device is None:\n",
    "        device = \"cpu\"\n",
    "\n",
    "    if not model_cls is None:\n",
    "        model = model_cls(**params)\n",
    "        model.load_state_dict(\n",
    "            torch.load(model_pt_path, map_location=torch.device(device)), strict=strict\n",
    "        )\n",
    "    else:\n",
    "        model = torch.jit.load(model_pt_path, map_location=torch.device(device))\n",
    "\n",
    "    model.to(device)\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import torch\n",
    "from src.train import Generator\n",
    "\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "params = {\"nz\": hps[\"nz\"], \"nc\": 1, \"ngf\": hps[\"ngf\"]}\n",
    "model = load_model(\n",
    "    \"./tmp/model.pth\", model_cls=Generator, params=params, device=device, strict=False\n",
    ")\n",
    "img = generate_fake_handwriting(model, num_images=64, nz=hps[\"nz\"], device=device)\n",
    "\n",
    "plt.imshow(np.asarray(img))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "isConfigCell": true
   },
   "source": [
    "### Clean up\n",
    "\n",
    "Run the following commandline in a terminal, to remove files generated by this notebook from S3 and local storage\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "print(f\"aws s3 rm --recursive s3://{bucket}/{prefix}\")\n",
    "print(f\"rm -rf {os.path.abspath(dataroot)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The PyTorch framework, as one of the most popular deep learning framework, is being widely recognised and applied, has become one of the de facto mainstream frameworks.\n",
    "\n",
    "Amazon SageMaker is tightly integrated with a variety of AWS services, such as Amazon EC2 instances of various types and sizes, Amazon S3, Amazon ECR, etc., providing an end-to-end, consistent machine learning experience for all framework practitioners. Amazon SageMaker continues to support mainstream machine learning frameworks, including PyTorch. Machine learning algorithms and models developed with PyTorch can be easily transplanted to Amazon SageMaker environment, by using Amazon SageMaker's fully managed Jupyter Notebook, SPOT training instances, Amazon Elastic Container Registry, SageMaker SDK, and so on, the complexity of machine learning engineering and infrastracture operation are simplified, productivity and efficiency are improved, operation and maintenance costs reduced.\n",
    "\n",
    "DCGAN is a landmark in the field of generative adversarial networks, and it is the cornerstone of many complex GANs today. We will explore some of the most recent and interesting variants of GAN in later examples.\n",
    "\n",
    "I believe that through the introduction and engineering practice of this example, it will be helpful for you to understand the principles and engineering methods for GAN in general."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/advanced_functionality|pytorch_bring_your_own_gan|build_gan_with_pytorch.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "kernelspec": {
   "display_name": "Python 3 (PyTorch 1.13 Python 3.9 CPU Optimized)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/pytorch-1.13-cpu-py39"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.",
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}