CNN_compression_demo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/K0EKJE/CNN_compression_with_Tensor_Decomposition/blob/main/CNN_compression_demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Network Compression With Tensor Decomposition Demo"
      ],
      "metadata": {
        "id": "zsX6wsQWXB_C"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "First install `tensorly` and clone the repo to access pretrained model weights and scripts."
      ],
      "metadata": {
        "id": "mTVwSUCLXIAm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install tensorly"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IdWTtnGUY1X4",
        "outputId": "c0649c8e-a2a6-4b06-a622-24653e5fbee6"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting tensorly\n",
            "  Downloading tensorly-0.8.1-py3-none-any.whl (229 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/229.7 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m \u001b[32m225.3/229.7 kB\u001b[0m \u001b[31m7.8 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m229.7/229.7 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from tensorly) (1.23.5)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from tensorly) (1.11.3)\n",
            "Installing collected packages: tensorly\n",
            "Successfully installed tensorly-0.8.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rvRAw_m0Hmwr",
        "outputId": "30bd179a-ef7d-49a1-815a-c945624d26f0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'CNN_compression_with_Tensor_Decomposition'...\n",
            "remote: Enumerating objects: 303, done.\u001b[K\n",
            "remote: Counting objects: 100% (153/153), done.\u001b[K\n",
            "remote: Compressing objects: 100% (84/84), done.\u001b[K\n",
            "remote: Total 303 (delta 77), reused 141 (delta 69), pack-reused 150\u001b[K\n",
            "Receiving objects: 100% (303/303), 1016.19 MiB | 35.62 MiB/s, done.\n",
            "Resolving deltas: 100% (129/129), done.\n"
          ]
        }
      ],
      "source": [
        "!git clone https://github.com/K0EKJE/CNN_compression_with_Tensor_Decomposition.git"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## CP decomposition"
      ],
      "metadata": {
        "id": "3gpwDnIoZG8K"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run the follwoing to start decomposition on the second convolutional layer with CP decomposition. It will download the dataset first and print the decomposed model  structure. Check the index for convolutional layers. A `decomposed_model` file will be automatically generated in the directory.\n",
        "\n",
        "**Note:** change `layer_to_decomp_cp` in config file to choose layer for decomposition. As for CP decomposition, it requires large RAM so try to test with only the frist first two convolutiaon layers."
      ],
      "metadata": {
        "id": "1MbcX_BCZKxY"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4U9Pfz4PsrzU",
        "outputId": "b240cd8f-6041-48cf-ded5-36edc3ae1e23"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==> Preparing data..\n",
            "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data/cifar-10-python.tar.gz\n",
            "100% 170498071/170498071 [00:05<00:00, 30635929.26it/s]\n",
            "Extracting ./data/cifar-10-python.tar.gz to ./data\n",
            "Files already downloaded and verified\n",
            "Decomposing layer 3: Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "CP rank = 21\n",
            "Decomposition of layer 3 Completed. Ratio = 0.7589842\n",
            "\n",
            "====================================================================================================\n",
            "\n",
            "==> Building decomposed model..\n",
            "VGG(\n",
            "  (features): Sequential(\n",
            "    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (2): ReLU(inplace=True)\n",
            "    (3): Sequential(\n",
            "      (0): Conv2d(64, 21, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(21, 21, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), groups=21, bias=False)\n",
            "      (2): Conv2d(21, 21, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), groups=21, bias=False)\n",
            "      (3): Conv2d(21, 64, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (5): ReLU(inplace=True)\n",
            "    (6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (7): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (8): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (9): ReLU(inplace=True)\n",
            "    (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (11): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (12): ReLU(inplace=True)\n",
            "    (13): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (14): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (15): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (16): ReLU(inplace=True)\n",
            "    (17): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (18): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (19): ReLU(inplace=True)\n",
            "    (20): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (21): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (22): ReLU(inplace=True)\n",
            "    (23): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (24): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (25): ReLU(inplace=True)\n",
            "    (26): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (27): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (28): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (29): ReLU(inplace=True)\n",
            "    (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (31): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (32): ReLU(inplace=True)\n",
            "    (33): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (34): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (35): ReLU(inplace=True)\n",
            "    (36): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (37): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (38): ReLU(inplace=True)\n",
            "    (39): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (40): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (41): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (42): ReLU(inplace=True)\n",
            "    (43): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (44): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (45): ReLU(inplace=True)\n",
            "    (46): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (47): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (48): ReLU(inplace=True)\n",
            "    (49): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (50): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (51): ReLU(inplace=True)\n",
            "    (52): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (53): AvgPool2d(kernel_size=1, stride=1, padding=0)\n",
            "  )\n",
            "  (classifier): Linear(in_features=512, out_features=10, bias=True)\n",
            ")\n",
            "====================================================================================================\n",
            "\n",
            "Accuracy of new model: \n",
            " [================================================================>]  Step: 18ms  | Tot: 2s 627ms  | Loss: 0.837 | Acc: 84.240% (8424/10000) 100/100 \n"
          ]
        }
      ],
      "source": [
        "!python /content/CNN_compression_with_Tensor_Decomposition/main.py --decompose"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have successfully used a rank-21 CP decomposition to decomposed the second convolutional layer of the model, and we can see that the accuracy right after decomposition is 84.240%(it may vary for each run). For reference, the base accuracy is 93.11%. After three epochs, the accuracy recovers to 91.950%.\n",
        "\n",
        "Now we can finetune for 3 epochs, and we will see a quick recovery of accuracy after the first epoch. The accruacy might not keep reducing as we are using a relatively large learning rate. Usually we will keep a small learning rate to avoid over-training."
      ],
      "metadata": {
        "id": "Pzf8TyBcaHjl"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "7pP0G0WuUA_m",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5994bab1-22f1-440e-d764-bfb3403dbbbb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==> Preparing data..\n",
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n",
            "VGG(\n",
            "  (features): Sequential(\n",
            "    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (2): ReLU(inplace=True)\n",
            "    (3): Sequential(\n",
            "      (0): Conv2d(64, 21, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(21, 21, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), groups=21, bias=False)\n",
            "      (2): Conv2d(21, 21, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), groups=21, bias=False)\n",
            "      (3): Conv2d(21, 64, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (5): ReLU(inplace=True)\n",
            "    (6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (7): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (8): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (9): ReLU(inplace=True)\n",
            "    (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (11): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (12): ReLU(inplace=True)\n",
            "    (13): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (14): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (15): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (16): ReLU(inplace=True)\n",
            "    (17): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (18): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (19): ReLU(inplace=True)\n",
            "    (20): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (21): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (22): ReLU(inplace=True)\n",
            "    (23): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (24): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (25): ReLU(inplace=True)\n",
            "    (26): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (27): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (28): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (29): ReLU(inplace=True)\n",
            "    (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (31): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (32): ReLU(inplace=True)\n",
            "    (33): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (34): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (35): ReLU(inplace=True)\n",
            "    (36): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (37): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (38): ReLU(inplace=True)\n",
            "    (39): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (40): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (41): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (42): ReLU(inplace=True)\n",
            "    (43): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (44): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (45): ReLU(inplace=True)\n",
            "    (46): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (47): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (48): ReLU(inplace=True)\n",
            "    (49): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (50): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (51): ReLU(inplace=True)\n",
            "    (52): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (53): AvgPool2d(kernel_size=1, stride=1, padding=0)\n",
            "  )\n",
            "  (classifier): Linear(in_features=512, out_features=10, bias=True)\n",
            ")\n",
            "Starting learning rate is: 0.001\n",
            "\n",
            "Epoch: 1\n",
            " [================================================================>]  Step: 1s 172ms  | Tot: 28s 417ms  | Loss: 0.083 | Acc: 97.780% (48890/50000) 391/391 \n",
            " [================================================================>]  Step: 18ms  | Tot: 2s 613ms  | Loss: 0.365 | Acc: 91.420% (9142/10000) 100/100 \n",
            "Saving..\n",
            "\n",
            "Epoch: 2\n",
            " [================================================================>]  Step: 45ms  | Tot: 27s 156ms  | Loss: 0.015 | Acc: 99.568% (49784/50000) 391/391 \n",
            " [================================================================>]  Step: 20ms  | Tot: 3s 471ms  | Loss: 0.397 | Acc: 91.230% (9123/10000) 100/100 \n",
            "\n",
            "Epoch: 3\n",
            " [================================================================>]  Step: 45ms  | Tot: 27s 610ms  | Loss: 0.011 | Acc: 99.662% (49831/50000) 391/391 \n",
            " [================================================================>]  Step: 18ms  | Tot: 2s 787ms  | Loss: 0.367 | Acc: 91.950% (9195/10000) 100/100 \n",
            "Saving..\n"
          ]
        }
      ],
      "source": [
        "!python /content/CNN_compression_with_Tensor_Decomposition/main.py --run_model --fine_tune --lr=0.001"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Partial Tucker Decomposition"
      ],
      "metadata": {
        "id": "UEjO3CGjZ8OA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Add a `--tucker` parser to use the Tucker decomposition.The rank selection method can be chosen from VBMF and SVD by changing the `tucker_rank_selection_method` setting in the config file, and similarly `layer_to_decomp_tucker` will control layers to decompose.\n",
        "\n",
        "By default, it will apply SVT to all the layers."
      ],
      "metadata": {
        "id": "ysbt3y0ldjMO"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gJy1JFLVC44i",
        "outputId": "4fa84cae-f973-43d9-e256-4878a42c1422"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==> Preparing data..\n",
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n",
            "Decomposing layer 3: Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [15, 14]\n",
            "Number of params before: 36928 || after: 3839\n",
            "Decomposition of layer 3 Completed. Ratio = 0.78914803\n",
            "\n",
            "Decomposing layer 7: Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [27, 22]\n",
            "Number of params before: 73856 || after: 10387\n",
            "Decomposition of layer 7 Completed. Ratio = 0.7905847\n",
            "\n",
            "Decomposing layer 10: Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [47, 41]\n",
            "Number of params before: 147584 || after: 28823\n",
            "Decomposition of layer 10 Completed. Ratio = 0.7721972\n",
            "\n",
            "Decomposing layer 14: Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [79, 66]\n",
            "Number of params before: 295168 || after: 75999\n",
            "Decomposition of layer 14 Completed. Ratio = 0.7958527\n",
            "\n",
            "Decomposing layer 17: Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [111, 106]\n",
            "Number of params before: 590080 || after: 161919\n",
            "Decomposition of layer 17 Completed. Ratio = 0.77876246\n",
            "\n",
            "Decomposing layer 20: Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [111, 114]\n",
            "Number of params before: 590080 || after: 171967\n",
            "Decomposition of layer 20 Completed. Ratio = 0.8014925\n",
            "\n",
            "Decomposing layer 23: Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [95, 95]\n",
            "Number of params before: 590080 || after: 130311\n",
            "Decomposition of layer 23 Completed. Ratio = 0.7764052\n",
            "\n",
            "Decomposing layer 27: Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [79, 73]\n",
            "Number of params before: 1180160 || after: 111703\n",
            "Decomposition of layer 27 Completed. Ratio = 0.7959165\n",
            "\n",
            "Decomposing layer 30: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [63, 63]\n",
            "Number of params before: 2359808 || after: 100871\n",
            "Decomposition of layer 30 Completed. Ratio = 0.79998255\n",
            "\n",
            "Decomposing layer 33: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [39, 47]\n",
            "Number of params before: 2359808 || after: 61127\n",
            "Decomposition of layer 33 Completed. Ratio = 0.78537756\n",
            "\n",
            "Decomposing layer 36: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [63, 50]\n",
            "Number of params before: 2359808 || after: 86831\n",
            "Decomposition of layer 36 Completed. Ratio = 0.8031716\n",
            "\n",
            "Decomposing layer 40: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [27, 27]\n",
            "Number of params before: 2359808 || after: 34775\n",
            "Decomposition of layer 40 Completed. Ratio = 0.7715779\n",
            "\n",
            "Decomposing layer 43: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [35, 33]\n",
            "Number of params before: 2359808 || after: 45791\n",
            "Decomposition of layer 43 Completed. Ratio = 0.7916469\n",
            "\n",
            "Decomposing layer 46: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [35, 32]\n",
            "Number of params before: 2359808 || after: 44963\n",
            "Decomposition of layer 46 Completed. Ratio = 0.78685236\n",
            "\n",
            "Decomposing layer 49: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "SVD Estimated ranks:  [39, 38]\n",
            "Number of params before: 2359808 || after: 53351\n",
            "Decomposition of layer 49 Completed. Ratio = 0.7989307\n",
            "\n",
            "Total param reduction: 20022592 ==> 1122657(X17.84)\n",
            "====================================================================================================\n",
            "\n",
            "==> Building decomposed model..\n",
            "VGG(\n",
            "  (features): Sequential(\n",
            "    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (2): ReLU(inplace=True)\n",
            "    (3): Sequential(\n",
            "      (0): Conv2d(64, 14, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(14, 15, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(15, 64, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (5): ReLU(inplace=True)\n",
            "    (6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (7): Sequential(\n",
            "      (0): Conv2d(64, 22, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(22, 27, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(27, 128, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (8): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (9): ReLU(inplace=True)\n",
            "    (10): Sequential(\n",
            "      (0): Conv2d(128, 41, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(41, 47, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(47, 128, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (11): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (12): ReLU(inplace=True)\n",
            "    (13): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (14): Sequential(\n",
            "      (0): Conv2d(128, 66, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(66, 79, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(79, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (15): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (16): ReLU(inplace=True)\n",
            "    (17): Sequential(\n",
            "      (0): Conv2d(256, 106, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(106, 111, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(111, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (18): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (19): ReLU(inplace=True)\n",
            "    (20): Sequential(\n",
            "      (0): Conv2d(256, 114, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(114, 111, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(111, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (21): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (22): ReLU(inplace=True)\n",
            "    (23): Sequential(\n",
            "      (0): Conv2d(256, 95, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(95, 95, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(95, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (24): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (25): ReLU(inplace=True)\n",
            "    (26): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (27): Sequential(\n",
            "      (0): Conv2d(256, 73, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(73, 79, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(79, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (28): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (29): ReLU(inplace=True)\n",
            "    (30): Sequential(\n",
            "      (0): Conv2d(512, 63, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(63, 63, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(63, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (31): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (32): ReLU(inplace=True)\n",
            "    (33): Sequential(\n",
            "      (0): Conv2d(512, 47, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(47, 39, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(39, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (34): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (35): ReLU(inplace=True)\n",
            "    (36): Sequential(\n",
            "      (0): Conv2d(512, 50, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(50, 63, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(63, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (37): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (38): ReLU(inplace=True)\n",
            "    (39): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (40): Sequential(\n",
            "      (0): Conv2d(512, 27, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(27, 27, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(27, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (41): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (42): ReLU(inplace=True)\n",
            "    (43): Sequential(\n",
            "      (0): Conv2d(512, 33, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(33, 35, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(35, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (44): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (45): ReLU(inplace=True)\n",
            "    (46): Sequential(\n",
            "      (0): Conv2d(512, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(32, 35, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(35, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (47): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (48): ReLU(inplace=True)\n",
            "    (49): Sequential(\n",
            "      (0): Conv2d(512, 38, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(38, 39, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(39, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (50): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (51): ReLU(inplace=True)\n",
            "    (52): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (53): AvgPool2d(kernel_size=1, stride=1, padding=0)\n",
            "  )\n",
            "  (classifier): Linear(in_features=512, out_features=10, bias=True)\n",
            ")\n",
            "====================================================================================================\n",
            "\n",
            "Accuracy of new model: \n",
            " [================================================================>]  Step: 32ms  | Tot: 3s 541ms  | Loss: 3.611 | Acc: 10.000% (1000/10000) 100/100 \n"
          ]
        }
      ],
      "source": [
        "!python /content/CNN_compression_with_Tensor_Decomposition/main.py --decompose --tucker"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can see that the decomposed model achieves 20022592 ==> 1122657(X17.84) compression, but it currenly loses predictive ability. We will see the accuracy recovering quickly with merely one epoch of training."
      ],
      "metadata": {
        "id": "PRPQ71vgiaFV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's also fine-tune for a few epochs. Change `fine_tune_epochs` to vary the number of epochs. The fine-tuned model will be automatically saved as `fine_tuned_model` after training complete.."
      ],
      "metadata": {
        "id": "w0A-toS4eA5F"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nypcciVptpXu",
        "outputId": "187369ac-f60c-4c56-dea8-c3402e3b2d96"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==> Preparing data..\n",
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n",
            "VGG(\n",
            "  (features): Sequential(\n",
            "    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (2): ReLU(inplace=True)\n",
            "    (3): Sequential(\n",
            "      (0): Conv2d(64, 14, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(14, 15, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(15, 64, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (5): ReLU(inplace=True)\n",
            "    (6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (7): Sequential(\n",
            "      (0): Conv2d(64, 22, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(22, 27, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(27, 128, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (8): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (9): ReLU(inplace=True)\n",
            "    (10): Sequential(\n",
            "      (0): Conv2d(128, 41, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(41, 47, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(47, 128, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (11): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (12): ReLU(inplace=True)\n",
            "    (13): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (14): Sequential(\n",
            "      (0): Conv2d(128, 66, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(66, 79, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(79, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (15): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (16): ReLU(inplace=True)\n",
            "    (17): Sequential(\n",
            "      (0): Conv2d(256, 106, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(106, 111, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(111, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (18): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (19): ReLU(inplace=True)\n",
            "    (20): Sequential(\n",
            "      (0): Conv2d(256, 114, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(114, 111, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(111, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (21): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (22): ReLU(inplace=True)\n",
            "    (23): Sequential(\n",
            "      (0): Conv2d(256, 95, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(95, 95, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(95, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (24): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (25): ReLU(inplace=True)\n",
            "    (26): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (27): Sequential(\n",
            "      (0): Conv2d(256, 73, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(73, 79, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(79, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (28): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (29): ReLU(inplace=True)\n",
            "    (30): Sequential(\n",
            "      (0): Conv2d(512, 63, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(63, 63, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(63, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (31): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (32): ReLU(inplace=True)\n",
            "    (33): Sequential(\n",
            "      (0): Conv2d(512, 47, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(47, 39, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(39, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (34): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (35): ReLU(inplace=True)\n",
            "    (36): Sequential(\n",
            "      (0): Conv2d(512, 50, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(50, 63, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(63, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (37): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (38): ReLU(inplace=True)\n",
            "    (39): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (40): Sequential(\n",
            "      (0): Conv2d(512, 27, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(27, 27, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(27, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (41): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (42): ReLU(inplace=True)\n",
            "    (43): Sequential(\n",
            "      (0): Conv2d(512, 33, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(33, 35, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(35, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (44): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (45): ReLU(inplace=True)\n",
            "    (46): Sequential(\n",
            "      (0): Conv2d(512, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(32, 35, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(35, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (47): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (48): ReLU(inplace=True)\n",
            "    (49): Sequential(\n",
            "      (0): Conv2d(512, 38, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(38, 39, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(39, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (50): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (51): ReLU(inplace=True)\n",
            "    (52): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (53): AvgPool2d(kernel_size=1, stride=1, padding=0)\n",
            "  )\n",
            "  (classifier): Linear(in_features=512, out_features=10, bias=True)\n",
            ")\n",
            "Starting learning rate is: 0.001\n",
            "\n",
            "Epoch: 1\n",
            " [================================================================>]  Step: 839ms  | Tot: 25s 390ms  | Loss: 0.146 | Acc: 95.640% (47820/50000) 391/391 \n",
            " [================================================================>]  Step: 22ms  | Tot: 3s 602ms  | Loss: 0.348 | Acc: 90.420% (9042/10000) 100/100 \n",
            "Saving..\n",
            "\n",
            "Epoch: 2\n",
            " [================================================================>]  Step: 28ms  | Tot: 25s 290ms  | Loss: 0.082 | Acc: 97.362% (48681/50000) 391/391 \n",
            " [================================================================>]  Step: 23ms  | Tot: 2s 712ms  | Loss: 0.361 | Acc: 90.270% (9027/10000) 100/100 \n",
            "\n",
            "Epoch: 3\n",
            " [================================================================>]  Step: 29ms  | Tot: 26s 204ms  | Loss: 0.070 | Acc: 97.698% (48849/50000) 391/391 \n",
            " [================================================================>]  Step: 19ms  | Tot: 2s 876ms  | Loss: 0.354 | Acc: 90.890% (9089/10000) 100/100 \n",
            "Saving..\n"
          ]
        }
      ],
      "source": [
        "!python /content/CNN_compression_with_Tensor_Decomposition/main.py --run_model --fine_tune --lr=0.001"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "An example using VBMF. We can see that the ratio is fluctuating greatly over the training process, which means for some of the layers, the algorithm preserves more information than the other. It may have an advantage of ignoring less important weights, but it may also introduce the bias from VBMF method that omits information from certain layers. Also, though VBMF would excute faster than the SVT method, which involves a binary search to ensure approximation ratio, it is an one-shot scheme so the trade-off between accuracy and compression rate."
      ],
      "metadata": {
        "id": "AjOKnVAkjSUI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python /content/CNN_compression_with_Tensor_Decomposition/main.py --decompose --tucker"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BW0wfzf0jM7G",
        "outputId": "94b830fc-1686-420a-ca98-c5c6396fedb2"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==> Preparing data..\n",
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n",
            "Decomposing layer 3: Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [29, 34]\n",
            "Number of params before: 36928 || after: 13033\n",
            "Decomposition of layer 3 Completed. Ratio = 0.92890704\n",
            "\n",
            "Decomposing layer 7: Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [38, 32]\n",
            "Number of params before: 73856 || after: 18054\n",
            "Decomposition of layer 7 Completed. Ratio = 0.8615357\n",
            "\n",
            "Decomposing layer 10: Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [35, 40]\n",
            "Number of params before: 147584 || after: 22403\n",
            "Decomposition of layer 10 Completed. Ratio = 0.7280691\n",
            "\n",
            "Decomposing layer 14: Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [56, 53]\n",
            "Number of params before: 295168 || after: 48197\n",
            "Decomposition of layer 14 Completed. Ratio = 0.7179657\n",
            "\n",
            "Decomposing layer 17: Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [57, 65]\n",
            "Number of params before: 590080 || after: 64955\n",
            "Decomposition of layer 17 Completed. Ratio = 0.6248904\n",
            "\n",
            "Decomposing layer 20: Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [64, 66]\n",
            "Number of params before: 590080 || after: 71682\n",
            "Decomposition of layer 20 Completed. Ratio = 0.66403157\n",
            "\n",
            "Decomposing layer 23: Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [62, 71]\n",
            "Number of params before: 590080 || after: 74055\n",
            "Decomposition of layer 23 Completed. Ratio = 0.6962589\n",
            "\n",
            "Decomposing layer 27: Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [94, 67]\n",
            "Number of params before: 1180160 || after: 122635\n",
            "Decomposition of layer 27 Completed. Ratio = 0.8063507\n",
            "\n",
            "Decomposing layer 30: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [91, 88]\n",
            "Number of params before: 2359808 || after: 164411\n",
            "Decomposition of layer 30 Completed. Ratio = 0.84497374\n",
            "\n",
            "Decomposing layer 33: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [69, 75]\n",
            "Number of params before: 2359808 || after: 120959\n",
            "Decomposition of layer 33 Completed. Ratio = 0.8532271\n",
            "\n",
            "Decomposing layer 36: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [81, 65]\n",
            "Number of params before: 2359808 || after: 122795\n",
            "Decomposition of layer 36 Completed. Ratio = 0.83434904\n",
            "\n",
            "Decomposing layer 40: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [44, 41]\n",
            "Number of params before: 2359808 || after: 60353\n",
            "Decomposition of layer 40 Completed. Ratio = 0.84946936\n",
            "\n",
            "Decomposing layer 43: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [44, 44]\n",
            "Number of params before: 2359808 || after: 63080\n",
            "Decomposition of layer 43 Completed. Ratio = 0.8242535\n",
            "\n",
            "Decomposing layer 46: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [44, 45]\n",
            "Number of params before: 2359808 || after: 63989\n",
            "Decomposition of layer 46 Completed. Ratio = 0.83373624\n",
            "\n",
            "Decomposing layer 49: Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "VBMF Estimated ranks:  [63, 63]\n",
            "Number of params before: 2359808 || after: 100871\n",
            "Decomposition of layer 49 Completed. Ratio = 0.84039944\n",
            "\n",
            "Total param reduction: 20022592 ==> 1131472(X17.7)\n",
            "====================================================================================================\n",
            "\n",
            "==> Building decomposed model..\n",
            "VGG(\n",
            "  (features): Sequential(\n",
            "    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (2): ReLU(inplace=True)\n",
            "    (3): Sequential(\n",
            "      (0): Conv2d(64, 34, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(34, 29, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(29, 64, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (5): ReLU(inplace=True)\n",
            "    (6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (7): Sequential(\n",
            "      (0): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(32, 38, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(38, 128, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (8): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (9): ReLU(inplace=True)\n",
            "    (10): Sequential(\n",
            "      (0): Conv2d(128, 40, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(40, 35, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(35, 128, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (11): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (12): ReLU(inplace=True)\n",
            "    (13): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (14): Sequential(\n",
            "      (0): Conv2d(128, 53, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(53, 56, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(56, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (15): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (16): ReLU(inplace=True)\n",
            "    (17): Sequential(\n",
            "      (0): Conv2d(256, 65, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(65, 57, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(57, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (18): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (19): ReLU(inplace=True)\n",
            "    (20): Sequential(\n",
            "      (0): Conv2d(256, 66, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(66, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (21): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (22): ReLU(inplace=True)\n",
            "    (23): Sequential(\n",
            "      (0): Conv2d(256, 71, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(71, 62, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(62, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (24): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (25): ReLU(inplace=True)\n",
            "    (26): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (27): Sequential(\n",
            "      (0): Conv2d(256, 67, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(67, 94, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(94, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (28): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (29): ReLU(inplace=True)\n",
            "    (30): Sequential(\n",
            "      (0): Conv2d(512, 88, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(88, 91, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(91, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (31): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (32): ReLU(inplace=True)\n",
            "    (33): Sequential(\n",
            "      (0): Conv2d(512, 75, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(75, 69, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(69, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (34): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (35): ReLU(inplace=True)\n",
            "    (36): Sequential(\n",
            "      (0): Conv2d(512, 65, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(65, 81, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(81, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (37): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (38): ReLU(inplace=True)\n",
            "    (39): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (40): Sequential(\n",
            "      (0): Conv2d(512, 41, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(41, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(44, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (41): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (42): ReLU(inplace=True)\n",
            "    (43): Sequential(\n",
            "      (0): Conv2d(512, 44, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(44, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (44): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (45): ReLU(inplace=True)\n",
            "    (46): Sequential(\n",
            "      (0): Conv2d(512, 45, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(45, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(44, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (47): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (48): ReLU(inplace=True)\n",
            "    (49): Sequential(\n",
            "      (0): Conv2d(512, 63, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(63, 63, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(63, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (50): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (51): ReLU(inplace=True)\n",
            "    (52): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (53): AvgPool2d(kernel_size=1, stride=1, padding=0)\n",
            "  )\n",
            "  (classifier): Linear(in_features=512, out_features=10, bias=True)\n",
            ")\n",
            "====================================================================================================\n",
            "\n",
            "Accuracy of new model: \n",
            " [================================================================>]  Step: 8ms  | Tot: 2s 577ms  | Loss: 3.964 | Acc: 10.000% (1000/10000) 100/100 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This has a similar compression rate of 20022592 ==> 1131472(X17.7) compared with the SVD method with a approximation ratio of 0.79. Similarly, the model will temporarily lose its predictive ability."
      ],
      "metadata": {
        "id": "XQf6pOGkkChr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python /content/CNN_compression_with_Tensor_Decomposition/main.py --run_model --fine_tune --lr=0.001"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zy7zELBwjNnH",
        "outputId": "d8549d43-78d3-4fa1-ca44-6d310f1ec13c"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "==> Preparing data..\n",
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n",
            "VGG(\n",
            "  (features): Sequential(\n",
            "    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
            "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (2): ReLU(inplace=True)\n",
            "    (3): Sequential(\n",
            "      (0): Conv2d(64, 34, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(34, 29, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(29, 64, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (5): ReLU(inplace=True)\n",
            "    (6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (7): Sequential(\n",
            "      (0): Conv2d(64, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(32, 38, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(38, 128, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (8): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (9): ReLU(inplace=True)\n",
            "    (10): Sequential(\n",
            "      (0): Conv2d(128, 40, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(40, 35, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(35, 128, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (11): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (12): ReLU(inplace=True)\n",
            "    (13): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (14): Sequential(\n",
            "      (0): Conv2d(128, 53, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(53, 56, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(56, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (15): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (16): ReLU(inplace=True)\n",
            "    (17): Sequential(\n",
            "      (0): Conv2d(256, 65, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(65, 57, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(57, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (18): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (19): ReLU(inplace=True)\n",
            "    (20): Sequential(\n",
            "      (0): Conv2d(256, 66, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(66, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (21): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (22): ReLU(inplace=True)\n",
            "    (23): Sequential(\n",
            "      (0): Conv2d(256, 71, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(71, 62, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(62, 256, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (24): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (25): ReLU(inplace=True)\n",
            "    (26): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (27): Sequential(\n",
            "      (0): Conv2d(256, 67, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(67, 94, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(94, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (28): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (29): ReLU(inplace=True)\n",
            "    (30): Sequential(\n",
            "      (0): Conv2d(512, 88, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(88, 91, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(91, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (31): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (32): ReLU(inplace=True)\n",
            "    (33): Sequential(\n",
            "      (0): Conv2d(512, 75, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(75, 69, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(69, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (34): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (35): ReLU(inplace=True)\n",
            "    (36): Sequential(\n",
            "      (0): Conv2d(512, 65, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(65, 81, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(81, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (37): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (38): ReLU(inplace=True)\n",
            "    (39): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (40): Sequential(\n",
            "      (0): Conv2d(512, 41, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(41, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(44, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (41): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (42): ReLU(inplace=True)\n",
            "    (43): Sequential(\n",
            "      (0): Conv2d(512, 44, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(44, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(44, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (44): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (45): ReLU(inplace=True)\n",
            "    (46): Sequential(\n",
            "      (0): Conv2d(512, 45, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(45, 44, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(44, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (47): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (48): ReLU(inplace=True)\n",
            "    (49): Sequential(\n",
            "      (0): Conv2d(512, 63, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (1): Conv2d(63, 63, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (2): Conv2d(63, 512, kernel_size=(1, 1), stride=(1, 1))\n",
            "    )\n",
            "    (50): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    (51): ReLU(inplace=True)\n",
            "    (52): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "    (53): AvgPool2d(kernel_size=1, stride=1, padding=0)\n",
            "  )\n",
            "  (classifier): Linear(in_features=512, out_features=10, bias=True)\n",
            ")\n",
            "Starting learning rate is: 0.001\n",
            "\n",
            "Epoch: 1\n",
            " [================================================================>]  Step: 883ms  | Tot: 26s 735ms  | Loss: 0.258 | Acc: 92.316% (46158/50000) 391/391 \n",
            " [================================================================>]  Step: 9ms  | Tot: 2s 715ms  | Loss: 0.359 | Acc: 89.140% (8914/10000) 100/100 \n",
            "Saving..\n",
            "\n",
            "Epoch: 2\n",
            " [================================================================>]  Step: 23ms  | Tot: 25s 860ms  | Loss: 0.165 | Acc: 94.488% (47244/50000) 391/391 \n",
            " [================================================================>]  Step: 10ms  | Tot: 2s 766ms  | Loss: 0.342 | Acc: 89.580% (8958/10000) 100/100 \n",
            "Saving..\n",
            "\n",
            "Epoch: 3\n",
            " [================================================================>]  Step: 24ms  | Tot: 25s 867ms  | Loss: 0.135 | Acc: 95.508% (47754/50000) 391/391 \n",
            " [================================================================>]  Step: 19ms  | Tot: 3s 821ms  | Loss: 0.356 | Acc: 89.910% (8991/10000) 100/100 \n",
            "Saving..\n"
          ]
        }
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "3gpwDnIoZG8K"
      ],
      "authorship_tag": "ABX9TyPNNwBwmkOZMKJG/6N4ExeY",
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}