project-notebook-codes.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"jupyter":{"source_hidden":true}},"source":["# Readme for the Code\n","The provided code is a deep learning pipeline for training a model using the SeResNet architecture on a dataset of face images. The pipeline includes data loading, model creation, training, validation, and testing. Here's a breakdown of the major components and processes in the code:\n","\n","#### Libraries\n","The code begins by installing and importing necessary libraries such as PyTorch, torchvision, wandb, and other relevant packages.\n","\n","#### Data Loading and Preprocessing\n","The code then sets up the data directories, defines transformations for the training and validation datasets, and creates data loaders for the training, validation, and test sets.\n","\n","#### Visualization\n","A visualization section is included to display a few images from the dataset as a sanity check for data augmentation.\n","\n","#### Network Architecture\n","The SeResNetNetwork class is defined, which creates an instance of the SeResNet model using the timm library. The model is then moved to the GPU if available, and the total number of parameters is calculated.\n","\n","#### Training\n","The training process is defined, including the train function for training the model, and the validate function for evaluating the model on the validation set. The code also includes the use of mixed precision training and a learning rate scheduler.\n","\n","#### Wandb Integration\n","The code integrates with Weights & Biases (wandb) for experiment tracking and visualization of training and validation metrics.\n","\n","#### Testing\n","The testing process is defined, including the test function for evaluating the model on the test set and calculating the test accuracy.\n","\n","#### Results\n","The code concludes by printing the test accuracy achieved by the trained model.\n","\n","Overall, the code provides a comprehensive deep learning pipeline for training and evaluating a SeResNet model on a face image dataset.\n","\n","The code is well-structured and includes detailed comments to explain each section and its purpose. It also integrates with wandb for experiment tracking and visualization of training and validation metrics.\n","\n","The provided code demonstrates best practices for deep learning model training, including data loading, model creation, training, validation, testing, and experiment tracking using wandb."]},{"cell_type":"markdown","metadata":{"_cell_guid":"e0d54fa7-da4e-442b-a929-afb55a4c8773","_uuid":"9a24d0a0-9f0b-49cf-b26f-2c615da48968","trusted":true},"source":["# Libraries"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"6ff93977-4dcd-48cb-a513-2b31ba9d64df","_uuid":"fd4ca40f-adcd-4804-88c5-2288b7fb07b8","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:25:59.435602Z","iopub.status.busy":"2023-12-08T16:25:59.435316Z","iopub.status.idle":"2023-12-08T16:26:12.297568Z","shell.execute_reply":"2023-12-08T16:26:12.296394Z","shell.execute_reply.started":"2023-12-08T16:25:59.435579Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["!pip install wandb --quiet"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"d76f5cbb-9588-4cf9-8a36-bb9313653aaa","_uuid":"5f57162e-6d99-4ea5-96c8-674592811803","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:12.299883Z","iopub.status.busy":"2023-12-08T16:26:12.299558Z","iopub.status.idle":"2023-12-08T16:26:17.251065Z","shell.execute_reply":"2023-12-08T16:26:17.250133Z","shell.execute_reply.started":"2023-12-08T16:26:12.299855Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["import torch\n","from torch import nn\n","import torchvision #This library is used for image-based operations (Augmentations)\n","import os\n","import gc\n","from tqdm import tqdm\n","from PIL import Image\n","import numpy as np\n","import pandas as pd\n","from sklearn.metrics import accuracy_score\n","import wandb\n","import glob\n","import matplotlib.pyplot as plt\n","from sklearn.metrics import accuracy_score\n","DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\n","print(\"Device: \", DEVICE)"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b0333ad6-5a27-46c8-8785-2e49e4b9e112","_uuid":"17712327-b74b-4150-96b6-8a75069ae0f3","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:17.252577Z","iopub.status.busy":"2023-12-08T16:26:17.252253Z","iopub.status.idle":"2023-12-08T16:26:17.256879Z","shell.execute_reply":"2023-12-08T16:26:17.256006Z","shell.execute_reply.started":"2023-12-08T16:26:17.252551Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["config = {\n","    'batch_size': 64, # Increase this if your GPU can handle it\n","    'lr': 1e-3,\n","    'epochs': 100, \n","}"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b181245b-b252-49ee-a658-00c10a5f538d","_uuid":"d69f891b-9d47-4f8e-ae9d-44abadfd72ba","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:17.259161Z","iopub.status.busy":"2023-12-08T16:26:17.258894Z","iopub.status.idle":"2023-12-08T16:26:34.039554Z","shell.execute_reply":"2023-12-08T16:26:34.038580Z","shell.execute_reply.started":"2023-12-08T16:26:17.259137Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["DATA_DIR    = '/kaggle/input/data-files/kaggle/working/dataset/Faceswap_images'\n","TRAIN_DIR   = os.path.join(DATA_DIR, \"train\")\n","VAL_DIR     = os.path.join(DATA_DIR, \"val\")\n","TEST_DIR    = os.path.join(DATA_DIR, \"test\")\n","\n","\n","train_transforms = torchvision.transforms.Compose([\n","    torchvision.transforms.Resize((224, 224)),\n","    torchvision.transforms.RandomPerspective(0.3, 0.3),\n","    torchvision.transforms.RandomRotation(degrees=20),\n","    torchvision.transforms.RandomHorizontalFlip(p=0.3),\n","    torchvision.transforms.ToTensor()\n","])\n","\n","valid_transforms = torchvision.transforms.Compose([\n","    torchvision.transforms.Resize((224, 224)),\n","    torchvision.transforms.ToTensor(),\n","])\n","\n","\n","train_dataset   = torchvision.datasets.ImageFolder(TRAIN_DIR, transform= train_transforms)\n","valid_dataset   = torchvision.datasets.ImageFolder(VAL_DIR, transform= valid_transforms)\n","test_dataset   = torchvision.datasets.ImageFolder(TEST_DIR, transform= valid_transforms)\n","\n","\n","# Create data loaders\n","train_loader = torch.utils.data.DataLoader(\n","    dataset     = train_dataset,\n","    batch_size  = config['batch_size'],\n","    shuffle     = True,\n","    num_workers = 2,\n","    pin_memory  = True\n",")\n","\n","valid_loader = torch.utils.data.DataLoader(\n","    dataset     = valid_dataset,\n","    batch_size  = config['batch_size'],\n","    shuffle     = False,\n","    num_workers = 2\n",")\n","\n","test_loader = torch.utils.data.DataLoader(\n","    dataset     = test_dataset,\n","    batch_size  = config['batch_size'],\n","    shuffle     = False,\n","    num_workers = 2\n",")"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"837800e6-d49b-4e91-a6dc-90a4506a28e6","_uuid":"c9c2b8cd-7ab2-41c9-b21e-ed397b613bda","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:34.041238Z","iopub.status.busy":"2023-12-08T16:26:34.040870Z","iopub.status.idle":"2023-12-08T16:26:34.055623Z","shell.execute_reply":"2023-12-08T16:26:34.054647Z","shell.execute_reply.started":"2023-12-08T16:26:34.041206Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["print(train_dataset.classes)\n","print(valid_dataset.classes)\n","print(test_dataset.classes)"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"9e6ec7a0-c53c-45fa-a31c-a1ca9e2121d7","_uuid":"ea1a9c70-9686-493b-9979-5b53e85f17db","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:34.057319Z","iopub.status.busy":"2023-12-08T16:26:34.056956Z","iopub.status.idle":"2023-12-08T16:26:34.138345Z","shell.execute_reply":"2023-12-08T16:26:34.136940Z","shell.execute_reply.started":"2023-12-08T16:26:34.057284Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["print(\"Number of classes    : \", len(train_dataset.classes))\n","print(\"No. of train images  : \", train_dataset.__len__())\n","print(\"Shape of image       : \", train_dataset[0][0].shape)\n","print(\"Batch size           : \", config['batch_size'])\n","print(\"Train batches        : \", train_loader.__len__())\n","print(\"Val batches          : \", valid_loader.__len__())"]},{"cell_type":"markdown","metadata":{"_cell_guid":"48161737-a6ea-4be8-a33d-53a7f76cd115","_uuid":"3d71ac85-3746-4e8b-ace2-14108da88de0","trusted":true},"source":["# Visualization"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"82f6c0ca-9206-40c0-8c01-27a977667135","_uuid":"90548137-f186-4f43-82a7-5e9679b5ae90","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:34.140070Z","iopub.status.busy":"2023-12-08T16:26:34.139779Z","iopub.status.idle":"2023-12-08T16:26:40.700732Z","shell.execute_reply":"2023-12-08T16:26:40.699467Z","shell.execute_reply.started":"2023-12-08T16:26:34.140045Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["# Visualize a few images in the dataset\n","\n","r, c = [5, 5]\n","fig, ax = plt.subplots(r, c, figsize=(15, 15))\n","\n","k = 0\n","dtl = torch.utils.data.DataLoader(\n","    dataset=torchvision.datasets.ImageFolder(TRAIN_DIR, transform=train_transforms),\n","    batch_size=config['batch_size'],\n","    shuffle=True,\n",")\n","\n","for data in dtl:\n","    x, y = data\n","    break\n","\n","for i in range(r):\n","    for j in range(c):\n","        img = x[k].numpy().transpose(1, 2, 0)\n","        ax[i, j].imshow(img)\n","        ax[i, j].axis('off')\n","        \n","        # Get the class label for the current image\n","        class_label = dtl.dataset.classes[y[k].item()]  # Assuming y[k] contains class indices\n","        ax[i, j].set_title(class_label)  # Set the title to the class label\n","        \n","        k += 1\n","        if k >= r * c:  # Break if all subplots are filled\n","            break\n","    if k >= r * c:\n","        break\n","\n","del dtl"]},{"cell_type":"markdown","metadata":{"_cell_guid":"cd902ad8-9c9a-4fd0-9159-98cb2017335a","_uuid":"4fbcec10-1e01-4db8-9611-73531081e005","trusted":true},"source":["# Network architecture"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"ffff7b19-7b91-4a8a-9dc1-86c1c5b87579","_uuid":"c3d92219-1a12-489a-ad4e-f0fe2190d68a","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:40.702233Z","iopub.status.busy":"2023-12-08T16:26:40.701954Z","iopub.status.idle":"2023-12-08T16:26:40.706473Z","shell.execute_reply":"2023-12-08T16:26:40.705607Z","shell.execute_reply.started":"2023-12-08T16:26:40.702208Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["# !pip install timm"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"e2781b1f-f7a9-4820-b7d3-cd0dbc2845fe","_uuid":"1f43b4d3-df82-498d-b1dd-58f6293bbbd0","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:40.708432Z","iopub.status.busy":"2023-12-08T16:26:40.707703Z","iopub.status.idle":"2023-12-08T16:26:41.234908Z","shell.execute_reply":"2023-12-08T16:26:41.233930Z","shell.execute_reply.started":"2023-12-08T16:26:40.708396Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["import timm"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"27a48629-e13f-4baa-a7ca-68ef755acecc","_uuid":"295cd63e-5a57-43bb-beba-7d2bddffde5d","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:41.239086Z","iopub.status.busy":"2023-12-08T16:26:41.238792Z","iopub.status.idle":"2023-12-08T16:26:44.581344Z","shell.execute_reply":"2023-12-08T16:26:44.580332Z","shell.execute_reply.started":"2023-12-08T16:26:41.239061Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["class SeResNetNetwork(nn.Module):\n","    def __init__(self, num_classes=2):\n","        super(SeResNetNetwork, self).__init__()\n","        self.backbone = timm.create_model(\"seresnet34\", pretrained=False)\n","        num_features = self.backbone.fc.in_features  # Access the default module\n","        self.backbone.fc = nn.Linear(num_features, num_classes)\n","\n","    def forward(self, x, return_feats=False):\n","        if return_feats:\n","            feats = self.backbone(x)\n","            return feats\n","        else:\n","            out = self.backbone(x)\n","            return out\n","\n","# Create an instance of the model and move it to the GPU if available\n","model = SeResNetNetwork().to(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n","\n","# Calculate the total number of parameters\n","total_params = sum(p.numel() for p in model.parameters())\n","print(f\"Total Parameters: {total_params}\")"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"480a5b94-1c45-461f-bd8b-e80adb62b5fb","_uuid":"4238d510-1c9d-4013-99d5-c89ac3ba6c6b","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:44.637026Z","iopub.status.busy":"2023-12-08T16:26:44.636674Z","iopub.status.idle":"2023-12-08T16:26:44.648879Z","shell.execute_reply":"2023-12-08T16:26:44.648076Z","shell.execute_reply.started":"2023-12-08T16:26:44.637001Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["criterion = torch.nn.CrossEntropyLoss(label_smoothing=0.15)\n","optimizer = torch.optim.Adam(model.parameters(), lr=config['lr'],betas=(0.5, 0.999),weight_decay=1e-4)\n","scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor=0.5, patience=2)\n","scaler = torch.cuda.amp.GradScaler() "]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"2c7008ab-1ea6-4fb8-9c1e-bed1c9c3ebf2","_uuid":"5efe73e0-e0f6-45af-bb1c-5d2cafc676d6","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:44.650202Z","iopub.status.busy":"2023-12-08T16:26:44.649939Z","iopub.status.idle":"2023-12-08T16:26:44.658534Z","shell.execute_reply":"2023-12-08T16:26:44.657760Z","shell.execute_reply.started":"2023-12-08T16:26:44.650156Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["# Track the initial learning rate\n","initial_lr = optimizer.param_groups[0]['lr']\n","print(f\"Initial learning rate: {initial_lr}\")\n","\n","# Define a function to check if the learning rate changes\n","def check_lr_change():\n","    current_lr = optimizer.param_groups[0]['lr']\n","    if current_lr != initial_lr:\n","        print(f\"Learning rate changed to: {current_lr}\")# Track the initial learning rate"]},{"cell_type":"markdown","metadata":{"_cell_guid":"f52a40c8-2ce7-4922-a5fd-95747b0d869d","_uuid":"3e8722e3-1cfd-48f1-b773-1bbcc2652738","trusted":true},"source":["## Train"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"a686afa5-17ca-4601-b1a5-fbc03e81ba92","_uuid":"a03381e1-f4e8-4e9c-a6dd-775d7157aac6","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:44.659937Z","iopub.status.busy":"2023-12-08T16:26:44.659655Z","iopub.status.idle":"2023-12-08T16:26:44.672370Z","shell.execute_reply":"2023-12-08T16:26:44.671600Z","shell.execute_reply.started":"2023-12-08T16:26:44.659914Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["def train(model, dataloader, optimizer, criterion):\n","\n","    model.train()\n","\n","    # Progress Bar\n","    batch_bar   = tqdm(total=len(dataloader), dynamic_ncols=True, leave=False, position=0, desc='Train', ncols=5)\n","\n","    num_correct = 0\n","    total_loss  = 0\n","\n","    for i, (images, labels) in enumerate(dataloader):\n","\n","        optimizer.zero_grad() # Zero gradients\n","\n","        images, labels = images.to(DEVICE), labels.to(DEVICE)\n","\n","        with torch.cuda.amp.autocast(): # This implements mixed precision. Thats it!\n","            outputs = model(images)\n","            loss    = criterion(outputs, labels)\n","\n","        # Update no. of correct predictions & loss as we iterate\n","        num_correct     += int((torch.argmax(outputs, axis=1) == labels).sum())\n","        total_loss      += float(loss.item())\n","\n","        # tqdm lets you add some details so you can monitor training as you train.\n","        batch_bar.set_postfix(\n","            acc         = \"{:.04f}%\".format(100 * num_correct / (config['batch_size']*(i + 1))),\n","            loss        = \"{:.04f}\".format(float(total_loss / (i + 1))),\n","            num_correct = num_correct,\n","            lr          = \"{:.04f}\".format(float(optimizer.param_groups[0]['lr']))\n","        )\n","\n","        scaler.scale(loss).backward() # a replacement for loss.backward()\n","        scaler.step(optimizer) # a replacement for optimizer.step()\n","        scaler.update()\n","\n","        batch_bar.update() # Update tqdm bar\n","\n","    batch_bar.close() # close the tqdm bar\n","\n","    acc         = 100 * num_correct / (config['batch_size']* len(dataloader))\n","    total_loss  = float(total_loss / len(dataloader))\n","\n","    return acc, total_loss"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"867559da-e3af-40c8-96aa-65ca114538d4","_uuid":"31cd060c-160d-4cb0-abe1-d721eac4a554","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:44.673781Z","iopub.status.busy":"2023-12-08T16:26:44.673458Z","iopub.status.idle":"2023-12-08T16:26:44.686926Z","shell.execute_reply":"2023-12-08T16:26:44.686084Z","shell.execute_reply.started":"2023-12-08T16:26:44.673749Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["def validate(model, dataloader, criterion):\n","\n","    model.eval()\n","    batch_bar = tqdm(total=len(dataloader), dynamic_ncols=True, position=0, leave=False, desc='Val', ncols=5)\n","\n","    num_correct = 0.0\n","    total_loss = 0.0\n","\n","    for i, (images, labels) in enumerate(dataloader):\n","\n","        # Move images to device\n","        images, labels = images.to(DEVICE), labels.to(DEVICE)\n","\n","        # Get model outputs\n","        with torch.inference_mode():\n","            outputs = model(images)\n","            loss = criterion(outputs, labels)\n","\n","        num_correct += int((torch.argmax(outputs, axis=1) == labels).sum())\n","        total_loss += float(loss.item())\n","\n","        batch_bar.set_postfix(\n","            acc=\"{:.04f}%\".format(100 * num_correct / (config['batch_size']*(i + 1))),\n","            loss=\"{:.04f}\".format(float(total_loss / (i + 1))),\n","            num_correct=num_correct)\n","\n","        batch_bar.update()\n","\n","    batch_bar.close()\n","    acc = 100 * num_correct / (config['batch_size']* len(dataloader))\n","    total_loss = float(total_loss / len(dataloader))\n","    return acc, total_loss"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"b97ec8d7-56bf-489a-92b3-a6400b09d8d9","_uuid":"871a9dd6-fb27-44f2-b3c5-222e536ee4c1","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:44.688338Z","iopub.status.busy":"2023-12-08T16:26:44.688014Z","iopub.status.idle":"2023-12-08T16:26:44.876110Z","shell.execute_reply":"2023-12-08T16:26:44.874933Z","shell.execute_reply.started":"2023-12-08T16:26:44.688265Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["gc.collect() # These commands help you when you face CUDA OOM error\n","torch.cuda.empty_cache()"]},{"cell_type":"markdown","metadata":{"_cell_guid":"538f6d6f-a76a-4e59-94c7-206a8cfa5c96","_uuid":"068821b1-a8bb-4466-8ee5-d28c1e47b3ca","trusted":true},"source":["## Wandb"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"f891c71b-e252-4946-84db-d6d66e237003","_uuid":"7254d612-e497-4299-8ec3-525f4d1ecbb6","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:44.877764Z","iopub.status.busy":"2023-12-08T16:26:44.877466Z","iopub.status.idle":"2023-12-08T16:26:47.534859Z","shell.execute_reply":"2023-12-08T16:26:47.533910Z","shell.execute_reply.started":"2023-12-08T16:26:44.877740Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["wandb.login(key=\"Insert wandb API key\") "]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"95e2aa70-d564-4732-9fe6-96b8718e7c64","_uuid":"94651b71-c673-427b-b6f4-5bf55c3836d9","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:26:47.536782Z","iopub.status.busy":"2023-12-08T16:26:47.536159Z","iopub.status.idle":"2023-12-08T16:27:18.586891Z","shell.execute_reply":"2023-12-08T16:27:18.585764Z","shell.execute_reply.started":"2023-12-08T16:26:47.536745Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["# Create your wandb run\n","run = wandb.init(\n","    name = \"Resnet50 Pretrained\", #\n","    reinit = True, ### Allows reinitalizing runs when you re-run this cell\n","    project = \"Final_project\", \n","    config = config ### Wandb Config for your run\n",")"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"4108ac85-e035-496c-a93c-a2e5db524f1e","_uuid":"de91280a-eaaf-4bad-abc2-dc8e7f2b30ce","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T16:27:18.596707Z","iopub.status.busy":"2023-12-08T16:27:18.596394Z","iopub.status.idle":"2023-12-08T18:00:42.421690Z","shell.execute_reply":"2023-12-08T18:00:42.420814Z","shell.execute_reply.started":"2023-12-08T16:27:18.596679Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["best_valacc = 0\n","\n","for epoch in range(config['epochs']):\n","\n","    curr_lr = float(optimizer.param_groups[0]['lr'])\n","\n","    train_acc, train_loss = train(model, train_loader, optimizer, criterion)\n","\n","    print(\"\\nEpoch {}/{}: \\nTrain Acc {:.04f}%\\t Train Loss {:.04f}\\t Learning Rate {:.04f}\".format(\n","        epoch + 1,\n","        config['epochs'],\n","        train_acc,\n","        train_loss,\n","        curr_lr))\n","\n","    val_acc, val_loss = validate(model, valid_loader, criterion)\n","\n","    print(\"Val Acc {:.04f}%\\t Val Loss {:.04f}\".format(val_acc, val_loss))\n","\n","    wandb.log({\"train_loss\":train_loss, 'train_Acc': train_acc, 'validation_Acc':val_acc,\n","               'validation_loss': val_loss, \"learning_Rate\": curr_lr})\n","\n","    scheduler.step(val_loss)  # Adjust the learning rate based on validation loss\n","    check_lr_change()\n","    \n","    \n","\n","    # Save model in a drive location if val_acc is better than the best recorded val_acc\n","    if val_acc >= best_valacc:\n","        print(\"Saving model\")\n","        torch.save({'model_state_dict':model.state_dict(),\n","                    'optimizer_state_dict':optimizer.state_dict(),\n","                    'scheduler_state_dict':scheduler.state_dict(),\n","                    'val_acc': val_acc,\n","                    'epoch': epoch}, './checkpoint.pth')\n","        best_valacc = val_acc\n","        wandb.save('checkpoint.pth')\n","\n","run.finish()"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"3190dc7d-97db-4263-b089-41025624ff26","_uuid":"4a7d8072-bb57-4d2c-960c-6d92348abfa6","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T18:00:42.423567Z","iopub.status.busy":"2023-12-08T18:00:42.423169Z","iopub.status.idle":"2023-12-08T18:00:42.969324Z","shell.execute_reply":"2023-12-08T18:00:42.968342Z","shell.execute_reply.started":"2023-12-08T18:00:42.423539Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["checkpoint = torch.load('./checkpoint.pth')\n","\n","# Load model state dict\n","model.load_state_dict(checkpoint['model_state_dict'])\n","val_acc = checkpoint['val_acc']"]},{"cell_type":"markdown","metadata":{"_cell_guid":"fe5c52b2-4b52-469a-a0c4-55ba01e04431","_uuid":"f5795cf0-02e5-4627-8ff0-c85090f23a56","trusted":true},"source":["## Testing"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"44e98d96-54c8-4cf1-a304-1a21974d52e2","_uuid":"001b0a9b-067b-4ddb-9c32-faaf09dc72dd","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T18:00:42.971234Z","iopub.status.busy":"2023-12-08T18:00:42.970860Z","iopub.status.idle":"2023-12-08T18:00:42.979321Z","shell.execute_reply":"2023-12-08T18:00:42.978325Z","shell.execute_reply.started":"2023-12-08T18:00:42.971200Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["def test(model,dataloader):\n","\n","  model.eval()\n","  batch_bar = tqdm(total=len(dataloader), dynamic_ncols=True, position=0, leave=False, desc='Test')\n","  test_results = []\n","\n","  for i, (images) in enumerate(dataloader):\n","      # predicting on the test set.\n","      images = images.to(DEVICE)\n","\n","      with torch.inference_mode():\n","        outputs = model(images)\n","\n","      outputs = torch.argmax(outputs, axis=1).detach().cpu().numpy().tolist()\n","      test_results.extend(outputs)\n","\n","      batch_bar.update()\n","\n","  batch_bar.close()\n","  return test_results"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c73b05f7-0ee6-496f-9047-94fe60ca62e6","_uuid":"e4b4844f-75ce-4e8c-8f0d-c4e8f36eb6cd","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T18:00:42.980949Z","iopub.status.busy":"2023-12-08T18:00:42.980606Z","iopub.status.idle":"2023-12-08T18:00:42.990809Z","shell.execute_reply":"2023-12-08T18:00:42.989973Z","shell.execute_reply.started":"2023-12-08T18:00:42.980915Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["\n","\n","def test(model, dataloader):\n","    model.eval()\n","    batch_bar = tqdm(total=len(dataloader), dynamic_ncols=True, position=0, leave=False, desc='Test')\n","    test_results = []\n","    true_labels = []\n","\n","    for i, (images, labels) in enumerate(dataloader):  # assuming labels are present in the dataloader\n","        images = images.to(DEVICE)\n","        labels = labels.to(DEVICE)\n","\n","        with torch.inference_mode():\n","            outputs = model(images)\n","\n","        predicted = torch.argmax(outputs, axis=1).detach().cpu().numpy().tolist()\n","        test_results.extend(predicted)\n","        true_labels.extend(labels.cpu().numpy().tolist())\n","\n","        batch_bar.update()\n","\n","    batch_bar.close()\n","\n","    accuracy = accuracy_score(true_labels, test_results)\n","    return accuracy"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"06a6de52-5dd6-47bc-8831-f1c220912e4c","_uuid":"0b5de956-f2f0-43a8-8715-15d26728b9d5","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T18:02:35.679101Z","iopub.status.busy":"2023-12-08T18:02:35.678386Z","iopub.status.idle":"2023-12-08T18:02:42.725088Z","shell.execute_reply":"2023-12-08T18:02:42.723993Z","shell.execute_reply.started":"2023-12-08T18:02:35.679068Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["test_results = test(model, test_loader)"]},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"2818e7dc-d743-43c8-b387-abb542c9187e","_uuid":"24582257-456c-45fe-8c41-7d0b05bd9999","collapsed":false,"execution":{"iopub.execute_input":"2023-12-08T18:02:48.497748Z","iopub.status.busy":"2023-12-08T18:02:48.496856Z","iopub.status.idle":"2023-12-08T18:02:48.502713Z","shell.execute_reply":"2023-12-08T18:02:48.501718Z","shell.execute_reply.started":"2023-12-08T18:02:48.497712Z"},"jupyter":{"outputs_hidden":false},"trusted":true},"outputs":[],"source":["print (f\"The test accuracy is {test_results*100} %\")"]}],"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"datasetId":4125866,"sourceId":7146970,"sourceType":"datasetVersion"},{"datasetId":4130674,"sourceId":7153488,"sourceType":"datasetVersion"}],"dockerImageVersionId":30588,"isGpuEnabled":true,"isInternetEnabled":true,"language":"python","sourceType":"notebook"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"}},"nbformat":4,"nbformat_minor":4}