Colab_Version_of_Main.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Colab_Version_of_Main.ipynb","provenance":[],"collapsed_sections":[],"authorship_tag":"ABX9TyOkfT59kse3XGsGWWHLYV5p"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"pmMmfCYWfmjx"},"source":["## Honest-but-Curious Nets (ACM CCS'21)\n","\n","https://github.com/mmalekzadeh/honest-but-curious-nets\n","\n","This notebook is prepared for those who want to run an expermints on Google Colab GPUs.\n","> The result is the same as running `main.py` on your machine."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"9EkjjP5DUfIx","executionInfo":{"status":"ok","timestamp":1637181844683,"user_tz":0,"elapsed":1250,"user":{"displayName":"Mohammad Malekzadeh","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhyyTTQ0So37s4ik4Tk9Pfpi6S1BXAqGjSafRcO=s64","userId":"11054559934326118576"}},"outputId":"760ae924-6d22-498c-994b-71af6710777d"},"source":["from google.colab import drive\n","drive.mount('/content/gdrive/', force_remount= True)"],"execution_count":1,"outputs":[{"output_type":"stream","name":"stdout","text":["Mounted at /content/gdrive/\n"]}]},{"cell_type":"code","metadata":{"id":"LH0aMA3CiB4b","executionInfo":{"status":"ok","timestamp":1637181844684,"user_tz":0,"elapsed":6,"user":{"displayName":"Mohammad Malekzadeh","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhyyTTQ0So37s4ik4Tk9Pfpi6S1BXAqGjSafRcO=s64","userId":"11054559934326118576"}}},"source":["# !pip install torch==1.9.1"],"execution_count":2,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"QMxA_muqgdjp"},"source":["I assume that you have copied the [GitHub Folder](https://github.com/mmalekzadeh/honest-but-curious-nets) in your Google drive. So, we just open the directory:"]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"pneovht-WyG8","executionInfo":{"status":"ok","timestamp":1637181844684,"user_tz":0,"elapsed":5,"user":{"displayName":"Mohammad Malekzadeh","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhyyTTQ0So37s4ik4Tk9Pfpi6S1BXAqGjSafRcO=s64","userId":"11054559934326118576"}},"outputId":"486ee3cd-5d0b-43c0-b18e-d0b798e14062"},"source":["cd gdrive/My\\ Drive/honest-but-curious-nets"],"execution_count":3,"outputs":[{"output_type":"stream","name":"stdout","text":["/content/gdrive/My Drive/honest-but-curious-nets\n"]}]},{"cell_type":"code","metadata":{"id":"bepPP8kyXD7B","executionInfo":{"status":"ok","timestamp":1637181844684,"user_tz":0,"elapsed":4,"user":{"displayName":"Mohammad Malekzadeh","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhyyTTQ0So37s4ik4Tk9Pfpi6S1BXAqGjSafRcO=s64","userId":"11054559934326118576"}}},"source":["import sys\n","sys.path.append(\".\")"],"execution_count":4,"outputs":[]},{"cell_type":"code","metadata":{"id":"zyOSZAk2XGr3","executionInfo":{"status":"ok","timestamp":1637181845775,"user_tz":0,"elapsed":1095,"user":{"displayName":"Mohammad Malekzadeh","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhyyTTQ0So37s4ik4Tk9Pfpi6S1BXAqGjSafRcO=s64","userId":"11054559934326118576"}}},"source":["import os\n","import numpy as np\n","import torch\n","from torch.utils.data import TensorDataset, DataLoader, TensorDataset\n","from torchsummary import summary\n","import copy\n","import torchvision \n","##\n","from hbcnets import datasets, models, utils, trainers, constants\n","## Optional: For making your results reproducible\n","# torch.manual_seed(constants.RANDOM_SEED)\n","# torch.cuda.manual_seed(constants.RANDOM_SEED)\n","# torch.backends.cudnn.deterministic = True\n","# torch.backends.cudnn.benchmark = False\n","# np.random.seed(constants.RANDOM_SEED)"],"execution_count":5,"outputs":[]},{"cell_type":"code","metadata":{"id":"epu-rBoRbXpK","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1637181845776,"user_tz":0,"elapsed":7,"user":{"displayName":"Mohammad Malekzadeh","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhyyTTQ0So37s4ik4Tk9Pfpi6S1BXAqGjSafRcO=s64","userId":"11054559934326118576"}},"outputId":"6419c56a-16c0-4167-e4ee-af68cca35a24"},"source":["print(torch.__version__)"],"execution_count":6,"outputs":[{"output_type":"stream","name":"stdout","text":["1.9.1+cu102\n"]}]},{"cell_type":"markdown","metadata":{"id":"wuNQWcyOhJBH"},"source":["Here I copied the file `setting.py` in a way that I can pass the desired `args` to the `main(args)` method.\n","> This will do the same thing as we do with `from setting import args_parser`"]},{"cell_type":"code","metadata":{"id":"HuuxVsDHXfWv","executionInfo":{"status":"ok","timestamp":1637181845776,"user_tz":0,"elapsed":5,"user":{"displayName":"Mohammad Malekzadeh","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhyyTTQ0So37s4ik4Tk9Pfpi6S1BXAqGjSafRcO=s64","userId":"11054559934326118576"}}},"source":["import argparse\n","parser = argparse.ArgumentParser()\n","\n","## Training parameters\n","parser.add_argument('--server_epochs', type=int, default=50,\n","                    help=\"Number of Epochs\")    \n","parser.add_argument('--server_batch', type=int, default=100,\n","                    help=\"Batch Size\")\n","parser.add_argument('--server_lr', type=int, default=0.001,\n","                    help=\"Learning Rate\")                                                    \n","\n","## Attack Type\n","parser.add_argument('--attack', type=str, default=\"parameterized\",\n","                    help=\"either 'parameterized' or 'regularized'\")  \n","\n","## Etc.\n","parser.add_argument('--root_dir', type=str, default=\"hbcnets\",\n","                    help=\"The root directory for saving data and results.\")    \n","parser.add_argument('--gpu', default=0, \n","                    help=\"When using GPU, set --gpu=0\")\n","parser.add_argument('--device', default='cuda', \n","                    help=\"When using GPU, set --device='cuda'\")\n","\n","args = parser.parse_args([])"],"execution_count":7,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"lhA6hZK_h6uW"},"source":["The below is exactly the `main.py`, but as a method."]},{"cell_type":"code","metadata":{"id":"aT8fkQCzYd3k","executionInfo":{"status":"ok","timestamp":1637181845777,"user_tz":0,"elapsed":5,"user":{"displayName":"Mohammad Malekzadeh","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhyyTTQ0So37s4ik4Tk9Pfpi6S1BXAqGjSafRcO=s64","userId":"11054559934326118576"}}},"source":["# if __name__ == '__main__':\n","def main(args):\n","        \n","    # args = args_parser() ## Reading the input arguments (see setting.py)\n","    if args.gpu:\n","        torch.cuda.set_device(args.gpu)\n","    ## Fetch the datasets\n","    if constants.DATASET == \"utk_face\":\n","        (train_images, train_labels), (test_images, test_labels) = datasets.get_dataset(args)     \n","        train_labels, test_labels = datasets.prepare_labels(train_labels, test_labels)  \n","        train_labels = train_labels[:,[constants.HONEST,constants.CURIOUS]].astype(int)\n","        test_labels  =  test_labels[:,[constants.HONEST,constants.CURIOUS]].astype(int)\n","        train_dataset = (train_images, train_labels)    \n","    elif constants.DATASET == \"celeba\":\n","        (train_images, train_labels), (valid_images, valid_labels), (test_images, test_labels) = datasets.get_dataset(args)\n","        train_labels, valid_labels, test_labels = datasets.prepare_labels(train_labels, test_labels, valid_labels)\n","        train_dataset = ((train_images, train_labels), (valid_images, valid_labels))    \n","\n","    print(\"\\n*** Dataset's Info\") \n","    print(\"Training\")\n","    utils.print_dataset_info((train_images, train_labels))\n","    if constants.DATASET == \"celeba\":\n","        print(\"Validation\")\n","        utils.print_dataset_info((valid_images, valid_labels))\n","    print(\"Testing\")\n","    utils.print_dataset_info((test_images, test_labels))                    \n","      \n","\n","    ## For logging\n","    exp_name = str(constants.HONEST)+\"_\"+str(constants.CURIOUS)+\"_\"+str(constants.K_Y)+\\\n","                \"_\"+str(constants.K_S)+\"_\"+str(int(constants.BETA_X))+\"_\"+str(int(constants.BETA_Y))+\\\n","                \"_\"+str(constants.SOFTMAX)+\"/\"+str(constants.IMAGE_SIZE)+\"_\"+str(constants.RANDOM_SEED) \n","\n","    ## Model\n","    model = models.Classifier(num_classes=constants.K_Y, with_softmax=constants.SOFTMAX)        \n","    model.to(args.device)\n","    summary(model, input_size=(3, constants.IMAGE_SIZE, constants.IMAGE_SIZE),device=args.device)            \n","                 \n","    if args.attack == \"parameterized\":\n","        param_G = models.Parameterized(num_inputs=constants.K_Y, num_classes=constants.K_S)        \n","        param_G.to(args.device)\n","        summary(param_G, input_size=(constants.K_Y,), device=args.device)       \n","\n","        save_dir = args.root_dir+\"/results_par/\"+constants.DATASET+\"/\"+exp_name+\"/\"\n","        if not os.path.exists(save_dir):\n","            os.makedirs(save_dir)            \n","        model, param_G = trainers.train_model_par(args, model, param_G, train_dataset, save_dir)  \n","\n","        ## Test    \n","        model.load_state_dict(torch.load(save_dir+\"best_model.pt\", map_location=torch.device(args.device)))         \n","        param_G.load_state_dict(torch.load(save_dir+\"best_param_G.pt\", map_location=torch.device(args.device)))         \n","        test_data_loader = DataLoader(TensorDataset(torch.Tensor(test_images), torch.Tensor(test_labels).long()),\n","                                batch_size=len(test_images)//50, shuffle=False, drop_last=False)  \n","        eval_acc_1, eval_acc_2, cf_mat_1, cf_mat_2  = utils.evaluate_acc_par(args, model, param_G, test_data_loader, cf_mat=True, roc=False)\n","        print(\"\\n$$$ Test Accuracy of the BEST model Y {:.2f}\".format(eval_acc_1))\n","        print(\"     Confusion Matrix Y:\\n\", (cf_mat_1*100).round(2))    \n","        print(\"\\n$$$ Test Accuracy of the BEST model S {:.2f}\".format(eval_acc_2))\n","        print(\"     Confusion Matrix S:\\n\", (cf_mat_2*100).round(2))\n","\n","        ### Optional: to report the avg entropy\n","        yh = utils.evaluate_acc_par(args, model, param_G, test_data_loader, preds=True)\n","        yh = torch.tensor(yh)\n","        yh_entropies = (-torch.sum(yh * torch.log2(yh), dim=1))\n","        norm_ent = torch.linalg.norm(yh_entropies, ord=1)/len(yh_entropies)\n","        print(\"\\n$$$ The average of the entropy of classifiers output {:.4f}\".format(norm_ent))    \n","    \n","    elif args.attack == \"regularized\":\n","        save_dir = args.root_dir+\"/results_reg/\"+constants.DATASET+\"/\"+exp_name+\"/\"\n","        if not os.path.exists(save_dir):\n","            os.makedirs(save_dir)            \n","        model = trainers.train_model_reg(args, model, \n","                        train_dataset, save_dir)  \n","\n","        ## Valid   \n","        best_REGTAU = 0.\n","        best_acc_valid = 0.\n","        model.load_state_dict(torch.load(save_dir+\"best_model.pt\", map_location=torch.device(args.device)))             \n","        for REGTAU in np.arange(.04,.91,.02):        \n","            valid_data_loader =  trainers.get_data_loader(args, train_dataset, train=False)\n","            eval_acc_1, eval_acc_2  = utils.evaluate_acc_reg(args, model, valid_data_loader,\n","                                                                                    beTau=REGTAU)    \n","            if eval_acc_2 > best_acc_valid:\n","                best_REGTAU = REGTAU\n","                best_acc_valid = eval_acc_2            \n","        print(\"\\n$$$ Tau {} Valid Acc G , {:.2f}\".format(best_REGTAU, best_acc_valid))  \n","        ## Test\n","        test_data_loader = DataLoader(TensorDataset(torch.Tensor(test_images), torch.Tensor(test_labels).long()),\n","                                batch_size=len(test_images)//50, shuffle=False, drop_last=False)  \n","        eval_acc_1, eval_acc_2, cf_mat_1, cf_mat_2  = utils.evaluate_acc_reg(args, model, test_data_loader, cf_mat=True, roc=False, \n","                                                                                    beTau=best_REGTAU)\n","        print(\"\\n$$$ Test Accuracy of the BEST model Y {:.2f}\".format(eval_acc_1))\n","        print(\"     Confusion Matrix Y:\\n\", (cf_mat_1*100).round(2))    \n","        print(\"\\n$$$ Test Accuracy of the BEST model S {:.2f}\".format(eval_acc_2))\n","        print(\"     Confusion Matrix S:\\n\", (cf_mat_2*100).round(2))"],"execution_count":8,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"PphADhsjiCVd"},"source":["Before running the final command, make sure that you have set up your desired experiment in the `hbcnets/constants.py`. e.g., This note book is for Age vs. Race (both 3-class) on UTKFace dataset.\n","> That's it :-) Hope you enjoy it."]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"cZcjP6W_YeM3","executionInfo":{"status":"ok","timestamp":1637182310176,"user_tz":0,"elapsed":464404,"user":{"displayName":"Mohammad Malekzadeh","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GhyyTTQ0So37s4ik4Tk9Pfpi6S1BXAqGjSafRcO=s64","userId":"11054559934326118576"}},"outputId":"be5d7136-61d1-4a80-95f7-dfeff106cb5b"},"source":["main(args)"],"execution_count":9,"outputs":[{"output_type":"stream","name":"stdout","text":["\n","*** Dataset's Info\n","Training\n","Data Dimensions:  (18964, 3, 64, 64)\n","Honest:\n"," [[   0 8213]\n"," [   1 6883]\n"," [   2 3868]]\n","Curious:\n"," [[   0 8002]\n"," [   1 8196]\n"," [   2 2766]]\n","Testing\n","Data Dimensions:  (4741, 3, 64, 64)\n","Honest:\n"," [[   0 2017]\n"," [   1 1715]\n"," [   2 1009]]\n","Curious:\n"," [[   0 2076]\n"," [   1 1997]\n"," [   2  668]]\n","----------------------------------------------------------------\n","        Layer (type)               Output Shape         Param #\n","================================================================\n","            Conv2d-1          [-1, 128, 32, 32]           1,664\n","       BatchNorm2d-2          [-1, 128, 32, 32]             256\n","         LeakyReLU-3          [-1, 128, 32, 32]               0\n","           Dropout-4          [-1, 128, 32, 32]               0\n","            Conv2d-5          [-1, 128, 16, 16]          65,664\n","       BatchNorm2d-6          [-1, 128, 16, 16]             256\n","         LeakyReLU-7          [-1, 128, 16, 16]               0\n","           Dropout-8          [-1, 128, 16, 16]               0\n","            Conv2d-9             [-1, 64, 8, 8]          32,832\n","      BatchNorm2d-10             [-1, 64, 8, 8]             128\n","        LeakyReLU-11             [-1, 64, 8, 8]               0\n","          Dropout-12             [-1, 64, 8, 8]               0\n","           Conv2d-13             [-1, 64, 4, 4]          16,448\n","      BatchNorm2d-14             [-1, 64, 4, 4]             128\n","        LeakyReLU-15             [-1, 64, 4, 4]               0\n","          Dropout-16             [-1, 64, 4, 4]               0\n","           Linear-17                  [-1, 128]         131,200\n","        LeakyReLU-18                  [-1, 128]               0\n","          Dropout-19                  [-1, 128]               0\n","           Linear-20                    [-1, 3]             387\n","================================================================\n","Total params: 248,963\n","Trainable params: 248,963\n","Non-trainable params: 0\n","----------------------------------------------------------------\n","Input size (MB): 0.05\n","Forward/backward pass size (MB): 5.16\n","Params size (MB): 0.95\n","Estimated Total Size (MB): 6.16\n","----------------------------------------------------------------\n","----------------------------------------------------------------\n","        Layer (type)               Output Shape         Param #\n","================================================================\n","            Linear-1                   [-1, 60]             240\n","         LeakyReLU-2                   [-1, 60]               0\n","           Dropout-3                   [-1, 60]               0\n","            Linear-4                   [-1, 30]           1,830\n","         LeakyReLU-5                   [-1, 30]               0\n","           Dropout-6                   [-1, 30]               0\n","            Linear-7                    [-1, 3]              93\n","================================================================\n","Total params: 2,163\n","Trainable params: 2,163\n","Non-trainable params: 0\n","----------------------------------------------------------------\n","Input size (MB): 0.00\n","Forward/backward pass size (MB): 0.00\n","Params size (MB): 0.01\n","Estimated Total Size (MB): 0.01\n","----------------------------------------------------------------\n","_________ Epoch:  1\n","**** Best Weighted Valid Acc Y & S on Epoch 1 is 71.09\n","- Train Loss Y: 0.83677, \n","- Train Acc Y: 61.82\n","- Valid Acc Y: 72.89\n","- Train Loss S: 0.91485, \n","- Train Acc S: 55.51\n","- Valid Acc S: 66.88\n","- Train Entropy Loss: 0.74382\n","_________ Epoch:  2\n","**** Best Weighted Valid Acc Y & S on Epoch 2 is 75.50\n","- Train Loss Y: 0.66185, \n","- Train Acc Y: 71.98\n","- Valid Acc Y: 76.85\n","- Train Loss S: 0.78196, \n","- Train Acc S: 65.94\n","- Valid Acc S: 72.36\n","- Train Entropy Loss: 0.56796\n","_________ Epoch:  3\n","**** Best Weighted Valid Acc Y & S on Epoch 3 is 76.84\n","- Train Loss Y: 0.61955, \n","- Train Acc Y: 74.26\n","- Valid Acc Y: 77.85\n","- Train Loss S: 0.73054, \n","- Train Acc S: 69.35\n","- Valid Acc S: 74.47\n","- Train Entropy Loss: 0.51297\n","_________ Epoch:  4\n","**** Best Weighted Valid Acc Y & S on Epoch 4 is 77.70\n","- Train Loss Y: 0.57827, \n","- Train Acc Y: 76.28\n","- Valid Acc Y: 78.27\n","- Train Loss S: 0.68972, \n","- Train Acc S: 71.60\n","- Valid Acc S: 76.37\n","- Train Entropy Loss: 0.48418\n","_________ Epoch:  5\n","**** Best Weighted Valid Acc Y & S on Epoch 5 is 79.34\n","- Train Loss Y: 0.55929, \n","- Train Acc Y: 77.08\n","- Valid Acc Y: 79.75\n","- Train Loss S: 0.65914, \n","- Train Acc S: 72.89\n","- Valid Acc S: 78.38\n","- Train Entropy Loss: 0.45733\n","_________ Epoch:  6\n","- Train Loss Y: 0.54231, \n","- Train Acc Y: 77.99\n","- Valid Acc Y: 78.85\n","- Train Loss S: 0.63348, \n","- Train Acc S: 74.32\n","- Valid Acc S: 79.17\n","- Train Entropy Loss: 0.45146\n","_________ Epoch:  7\n","**** Best Weighted Valid Acc Y & S on Epoch 7 is 79.61\n","- Train Loss Y: 0.52066, \n","- Train Acc Y: 78.43\n","- Valid Acc Y: 79.64\n","- Train Loss S: 0.62234, \n","- Train Acc S: 75.34\n","- Valid Acc S: 79.54\n","- Train Entropy Loss: 0.42865\n","_________ Epoch:  8\n","**** Best Weighted Valid Acc Y & S on Epoch 8 is 80.44\n","- Train Loss Y: 0.50786, \n","- Train Acc Y: 79.35\n","- Valid Acc Y: 80.54\n","- Train Loss S: 0.61058, \n","- Train Acc S: 75.78\n","- Valid Acc S: 80.22\n","- Train Entropy Loss: 0.41895\n","_________ Epoch:  9\n","- Train Loss Y: 0.49783, \n","- Train Acc Y: 79.75\n","- Valid Acc Y: 79.80\n","- Train Loss S: 0.59398, \n","- Train Acc S: 76.53\n","- Valid Acc S: 80.01\n","- Train Entropy Loss: 0.41164\n","_________ Epoch:  10\n","- Train Loss Y: 0.48519, \n","- Train Acc Y: 80.67\n","- Valid Acc Y: 79.80\n","- Train Loss S: 0.58973, \n","- Train Acc S: 76.53\n","- Valid Acc S: 81.65\n","- Train Entropy Loss: 0.39834\n","_________ Epoch:  11\n","**** Best Weighted Valid Acc Y & S on Epoch 11 is 80.84\n","- Train Loss Y: 0.48082, \n","- Train Acc Y: 80.57\n","- Valid Acc Y: 81.17\n","- Train Loss S: 0.57449, \n","- Train Acc S: 77.37\n","- Valid Acc S: 80.06\n","- Train Entropy Loss: 0.39767\n","_________ Epoch:  12\n","**** Best Weighted Valid Acc Y & S on Epoch 12 is 80.96\n","- Train Loss Y: 0.46815, \n","- Train Acc Y: 81.23\n","- Valid Acc Y: 80.96\n","- Train Loss S: 0.55710, \n","- Train Acc S: 77.70\n","- Valid Acc S: 80.96\n","- Train Entropy Loss: 0.38645\n","_________ Epoch:  13\n","**** Best Weighted Valid Acc Y & S on Epoch 13 is 81.44\n","- Train Loss Y: 0.46347, \n","- Train Acc Y: 81.22\n","- Valid Acc Y: 81.01\n","- Train Loss S: 0.56272, \n","- Train Acc S: 77.73\n","- Valid Acc S: 82.44\n","- Train Entropy Loss: 0.37799\n","_________ Epoch:  14\n","- Train Loss Y: 0.44966, \n","- Train Acc Y: 81.72\n","- Valid Acc Y: 80.91\n","- Train Loss S: 0.54800, \n","- Train Acc S: 78.15\n","- Valid Acc S: 80.96\n","- Train Entropy Loss: 0.37243\n","_________ Epoch:  15\n","- Train Loss Y: 0.44108, \n","- Train Acc Y: 82.38\n","- Valid Acc Y: 81.12\n","- Train Loss S: 0.55389, \n","- Train Acc S: 78.24\n","- Valid Acc S: 80.17\n","- Train Entropy Loss: 0.36536\n","_________ Epoch:  16\n","- Train Loss Y: 0.43644, \n","- Train Acc Y: 82.40\n","- Valid Acc Y: 81.33\n","- Train Loss S: 0.54417, \n","- Train Acc S: 78.22\n","- Valid Acc S: 81.65\n","- Train Entropy Loss: 0.35716\n","_________ Epoch:  17\n","**** Best Weighted Valid Acc Y & S on Epoch 17 is 81.72\n","- Train Loss Y: 0.43563, \n","- Train Acc Y: 82.35\n","- Valid Acc Y: 81.33\n","- Train Loss S: 0.55173, \n","- Train Acc S: 77.61\n","- Valid Acc S: 82.65\n","- Train Entropy Loss: 0.36454\n","_________ Epoch:  18\n","- Train Loss Y: 0.42280, \n","- Train Acc Y: 82.82\n","- Valid Acc Y: 80.85\n","- Train Loss S: 0.52652, \n","- Train Acc S: 79.38\n","- Valid Acc S: 82.33\n","- Train Entropy Loss: 0.35097\n","_________ Epoch:  19\n","**** Best Weighted Valid Acc Y & S on Epoch 19 is 81.86\n","- Train Loss Y: 0.41869, \n","- Train Acc Y: 82.96\n","- Valid Acc Y: 81.43\n","- Train Loss S: 0.53256, \n","- Train Acc S: 79.01\n","- Valid Acc S: 82.86\n","- Train Entropy Loss: 0.34688\n","_________ Epoch:  20\n","- Train Loss Y: 0.41135, \n","- Train Acc Y: 83.55\n","- Valid Acc Y: 80.54\n","- Train Loss S: 0.52603, \n","- Train Acc S: 79.32\n","- Valid Acc S: 82.59\n","- Train Entropy Loss: 0.34343\n","_________ Epoch:  21\n","**** Best Weighted Valid Acc Y & S on Epoch 21 is 82.22\n","- Train Loss Y: 0.41319, \n","- Train Acc Y: 83.95\n","- Valid Acc Y: 81.70\n","- Train Loss S: 0.51945, \n","- Train Acc S: 79.48\n","- Valid Acc S: 83.44\n","- Train Entropy Loss: 0.33931\n","_________ Epoch:  22\n","- Train Loss Y: 0.40711, \n","- Train Acc Y: 83.48\n","- Valid Acc Y: 81.54\n","- Train Loss S: 0.51950, \n","- Train Acc S: 79.35\n","- Valid Acc S: 83.18\n","- Train Entropy Loss: 0.33711\n","_________ Epoch:  23\n","- Train Loss Y: 0.39861, \n","- Train Acc Y: 84.04\n","- Valid Acc Y: 80.70\n","- Train Loss S: 0.51679, \n","- Train Acc S: 79.76\n","- Valid Acc S: 82.96\n","- Train Entropy Loss: 0.33176\n","_________ Epoch:  24\n","- Train Loss Y: 0.39904, \n","- Train Acc Y: 83.85\n","- Valid Acc Y: 80.43\n","- Train Loss S: 0.49905, \n","- Train Acc S: 80.56\n","- Valid Acc S: 82.49\n","- Train Entropy Loss: 0.32647\n","_________ Epoch:  25\n","- Train Loss Y: 0.39160, \n","- Train Acc Y: 84.34\n","- Valid Acc Y: 81.07\n","- Train Loss S: 0.51401, \n","- Train Acc S: 80.32\n","- Valid Acc S: 83.65\n","- Train Entropy Loss: 0.32610\n","_________ Epoch:  26\n","- Train Loss Y: 0.38081, \n","- Train Acc Y: 84.63\n","- Valid Acc Y: 81.59\n","- Train Loss S: 0.50311, \n","- Train Acc S: 80.24\n","- Valid Acc S: 83.23\n","- Train Entropy Loss: 0.31756\n","_________ Epoch:  27\n","- Train Loss Y: 0.38901, \n","- Train Acc Y: 84.21\n","- Valid Acc Y: 81.49\n","- Train Loss S: 0.51126, \n","- Train Acc S: 79.90\n","- Valid Acc S: 83.23\n","- Train Entropy Loss: 0.32385\n","_________ Epoch:  28\n","- Train Loss Y: 0.37832, \n","- Train Acc Y: 84.83\n","- Valid Acc Y: 81.33\n","- Train Loss S: 0.50110, \n","- Train Acc S: 80.35\n","- Valid Acc S: 83.18\n","- Train Entropy Loss: 0.32050\n","_________ Epoch:  29\n","**** Best Weighted Valid Acc Y & S on Epoch 29 is 82.53\n","- Train Loss Y: 0.37079, \n","- Train Acc Y: 85.28\n","- Valid Acc Y: 82.23\n","- Train Loss S: 0.49056, \n","- Train Acc S: 80.35\n","- Valid Acc S: 83.23\n","- Train Entropy Loss: 0.30947\n","_________ Epoch:  30\n","- Train Loss Y: 0.37698, \n","- Train Acc Y: 85.06\n","- Valid Acc Y: 81.80\n","- Train Loss S: 0.49117, \n","- Train Acc S: 80.85\n","- Valid Acc S: 83.97\n","- Train Entropy Loss: 0.31135\n","_________ Epoch:  31\n","- Train Loss Y: 0.36805, \n","- Train Acc Y: 85.21\n","- Valid Acc Y: 81.33\n","- Train Loss S: 0.49142, \n","- Train Acc S: 80.71\n","- Valid Acc S: 84.23\n","- Train Entropy Loss: 0.30934\n","_________ Epoch:  32\n","- Train Loss Y: 0.36413, \n","- Train Acc Y: 85.54\n","- Valid Acc Y: 81.80\n","- Train Loss S: 0.49334, \n","- Train Acc S: 80.46\n","- Valid Acc S: 84.18\n","- Train Entropy Loss: 0.30510\n","_________ Epoch:  33\n","- Train Loss Y: 0.36827, \n","- Train Acc Y: 85.29\n","- Valid Acc Y: 81.22\n","- Train Loss S: 0.48458, \n","- Train Acc S: 81.19\n","- Valid Acc S: 84.34\n","- Train Entropy Loss: 0.30681\n","_________ Epoch:  34\n","**** Best Weighted Valid Acc Y & S on Epoch 34 is 82.66\n","- Train Loss Y: 0.35921, \n","- Train Acc Y: 85.62\n","- Valid Acc Y: 82.44\n","- Train Loss S: 0.49331, \n","- Train Acc S: 80.64\n","- Valid Acc S: 83.18\n","- Train Entropy Loss: 0.30195\n","_________ Epoch:  35\n","- Train Loss Y: 0.34855, \n","- Train Acc Y: 85.93\n","- Valid Acc Y: 81.54\n","- Train Loss S: 0.49401, \n","- Train Acc S: 80.75\n","- Valid Acc S: 83.65\n","- Train Entropy Loss: 0.29873\n","_________ Epoch:  36\n","- Train Loss Y: 0.34812, \n","- Train Acc Y: 86.35\n","- Valid Acc Y: 80.49\n","- Train Loss S: 0.48465, \n","- Train Acc S: 81.18\n","- Valid Acc S: 83.70\n","- Train Entropy Loss: 0.29847\n","_________ Epoch:  37\n","**** Best Weighted Valid Acc Y & S on Epoch 37 is 82.80\n","- Train Loss Y: 0.34437, \n","- Train Acc Y: 86.32\n","- Valid Acc Y: 82.44\n","- Train Loss S: 0.47933, \n","- Train Acc S: 81.18\n","- Valid Acc S: 83.65\n","- Train Entropy Loss: 0.29740\n","_________ Epoch:  38\n","- Train Loss Y: 0.34388, \n","- Train Acc Y: 86.44\n","- Valid Acc Y: 80.75\n","- Train Loss S: 0.48271, \n","- Train Acc S: 81.22\n","- Valid Acc S: 83.97\n","- Train Entropy Loss: 0.29452\n","_________ Epoch:  39\n","- Train Loss Y: 0.33376, \n","- Train Acc Y: 86.79\n","- Valid Acc Y: 81.28\n","- Train Loss S: 0.46446, \n","- Train Acc S: 82.09\n","- Valid Acc S: 83.07\n","- Train Entropy Loss: 0.29035\n","_________ Epoch:  40\n","- Train Loss Y: 0.34334, \n","- Train Acc Y: 86.59\n","- Valid Acc Y: 81.17\n","- Train Loss S: 0.47418, \n","- Train Acc S: 81.49\n","- Valid Acc S: 84.02\n","- Train Entropy Loss: 0.29009\n","_________ Epoch:  41\n","- Train Loss Y: 0.34234, \n","- Train Acc Y: 86.19\n","- Valid Acc Y: 82.49\n","- Train Loss S: 0.46674, \n","- Train Acc S: 81.55\n","- Valid Acc S: 82.96\n","- Train Entropy Loss: 0.29272\n","_________ Epoch:  42\n","- Train Loss Y: 0.33125, \n","- Train Acc Y: 87.05\n","- Valid Acc Y: 80.75\n","- Train Loss S: 0.47369, \n","- Train Acc S: 81.69\n","- Valid Acc S: 83.86\n","- Train Entropy Loss: 0.29332\n","_________ Epoch:  43\n","- Train Loss Y: 0.33041, \n","- Train Acc Y: 87.05\n","- Valid Acc Y: 82.17\n","- Train Loss S: 0.47319, \n","- Train Acc S: 81.46\n","- Valid Acc S: 82.12\n","- Train Entropy Loss: 0.29016\n","_________ Epoch:  44\n","- Train Loss Y: 0.33543, \n","- Train Acc Y: 86.78\n","- Valid Acc Y: 82.28\n","- Train Loss S: 0.46861, \n","- Train Acc S: 81.94\n","- Valid Acc S: 82.38\n","- Train Entropy Loss: 0.29123\n","_________ Epoch:  45\n","- Train Loss Y: 0.32870, \n","- Train Acc Y: 86.85\n","- Valid Acc Y: 81.28\n","- Train Loss S: 0.47075, \n","- Train Acc S: 81.58\n","- Valid Acc S: 83.97\n","- Train Entropy Loss: 0.29211\n","_________ Epoch:  46\n","- Train Loss Y: 0.32995, \n","- Train Acc Y: 86.84\n","- Valid Acc Y: 81.75\n","- Train Loss S: 0.47036, \n","- Train Acc S: 81.72\n","- Valid Acc S: 85.18\n","- Train Entropy Loss: 0.29993\n","_________ Epoch:  47\n","- Train Loss Y: 0.32890, \n","- Train Acc Y: 87.10\n","- Valid Acc Y: 81.91\n","- Train Loss S: 0.45915, \n","- Train Acc S: 82.39\n","- Valid Acc S: 83.81\n","- Train Entropy Loss: 0.30238\n","_________ Epoch:  48\n","**** Best Weighted Valid Acc Y & S on Epoch 48 is 83.12\n","- Train Loss Y: 0.32467, \n","- Train Acc Y: 87.43\n","- Valid Acc Y: 82.59\n","- Train Loss S: 0.48174, \n","- Train Acc S: 81.69\n","- Valid Acc S: 84.34\n","- Train Entropy Loss: 0.30632\n","_________ Epoch:  49\n","- Train Loss Y: 0.32013, \n","- Train Acc Y: 87.38\n","- Valid Acc Y: 82.23\n","- Train Loss S: 0.46919, \n","- Train Acc S: 81.63\n","- Valid Acc S: 83.60\n","- Train Entropy Loss: 0.29871\n","_________ Epoch:  50\n","- Train Loss Y: 0.32309, \n","- Train Acc Y: 87.51\n","- Valid Acc Y: 81.54\n","- Train Loss S: 0.47554, \n","- Train Acc S: 81.34\n","- Valid Acc S: 84.76\n","- Train Entropy Loss: 0.30147\n","\n","$$$ Test Accuracy of the BEST model Y 81.21\n","     Confusion Matrix Y:\n"," [[85.67  9.92  4.41]\n"," [22.39 76.27  1.34]\n"," [17.84  1.49 80.67]]\n","\n","$$$ Test Accuracy of the BEST model S 82.49\n","     Confusion Matrix S:\n"," [[82.76 14.07  3.18]\n"," [10.87 86.43  2.7 ]\n"," [11.38 18.71 69.91]]\n","\n","$$$ The average of the entropy of classifiers output 0.4846\n"]}]}]}