group4_final.ipynb

{
  "cells": [
    {
      "metadata": {
        "_cell_guid": "c3b96391-2148-43c8-9eb1-6f53c9c8a704",
        "_uuid": "2c2c966557b24c189b2e0688ea2c27bacae5be5b"
      },
      "cell_type": "markdown",
      "source": "# Using Ensemble Pretrained Models to classify Human Protein Atlas Image\n\n## Motivation and Background\n\n#### [CNN](http://cs231n.github.io/convolutional-networks/)\n\nWe can identify two main blocks inside of a typical CNN: \n - Feature extraction\n - Classification\n \nThe feature extraction is made of a series of convolutional and pooling layers which extract features from the image, increasing in complexity in each layer (i.e. from simpler features in the first layers as points, to more complex ones in the last layers like edges and shapes. These features are then fed to a fully connected network (classifier), which learns to classify them.\n\nConvolutional Neural Networks is a sequence of layers, and every layer of a ConvNet transforms one volume of activations to another through a differentialble function. \nThere are three main types of layers to build the ConvNet(architecture[INPUT-CONV-RELU-POOL-FC]):\n1. INPUT [$ 32  \\times  32 \\times 3 $]: An image of width 32, height 32, and with three color channels, R,G,B\n2. Convolutional Layer: Compute the output of neurons that are connected to local regions in the input. This may redult in volume such as [$ 32  \\times  32 \\times 12 $] if we decided to use 12 filters.\n3. RELU Layer: Activation function $ max(0,x) $ \n4. Pooling Layer: Perform a downsampling operation along the spatial dimensions, resulting in volume such as [$ 16  \\times  16 \\times 12 $] \n5. Fully-Connected Layer (Regular Neural Networks): Compute the class scores, resulting in colume of size [$ 1  \\times  1 \\times 28 $], where each of the 10 numbers correspond to a class score.\n<img src=\"https://www.mdpi.com/entropy/entropy-19-00242/article_deploy/html/images/entropy-19-00242-g001.png\" width=800>\n\n\n#### [Transfer learning](https://en.wikipedia.org/wiki/Transfer_learning) \n\nMost deep learning networks learn to detect edges in the earlier layers, shapes in the middle layer and some spefific features in the later layer. Therefore, with certain data types it is possible to use the weights learned in one task to be transfered to another task. \nIt turns out to be useful when dealing with relatively small datasets; for examples medical images, which are harder to obtain in large numbers than other datasets. \nInstead of training a deep neural network from scratch, which would require a significant amount of data, power and time, it's often convenient to use a pretrained model and just finetune its performance to simplify and speed up the process. It is heavily used in Image recognition and Natural Language Processing related tasks.\n\n\nEnsemble Pretrained models\n1. Using a model (VGG-16)  which is already capable of extracting features from an image and train its fully connected network in order to classify different types of retinal damage instead of objects.\n\nThe model we'll use is a [VGG-16](https://www.quora.com/What-is-the-VGG-neural-network)- convolutional network, built by Visual Geometry Group and trained on [ImageNet](http://www.image-net.org/) dataset. \n\n\n<img src=\"https://cdn-images-1.medium.com/max/1600/1*0Tk4JclhGOCR_uLe6RKvUQ.png\" width=500>\n\n\n\n2. ResNet\nResidual Neural Network introduced a concept called 'skip connections'. In residual network, it directly copy the input matrix to the second transformation output and sum the output in final RELU function.\n<img src=\"https://cdn-images-1.medium.com/max/1600/1*beczGvPaBBnyauKItYGTYQ.png\" width=800>\n\n3. [Inception](https://towardsdatascience.com/a-simple-guide-to-the-versions-of-the-inception-network-7fc52b863202)\n\n<img src=\"https://cdn-images-1.medium.com/max/1600/1*uW81y16b-ptBDV8SIT1beQ.png\" width=800>\n\n"
    },
    {
      "metadata": {
        "_cell_guid": "a2276b70-8eae-4709-9af1-e08df6bab2c1",
        "_uuid": "0fca07bf66c325a944b4866601538e7c4e976e50"
      },
      "cell_type": "markdown",
      "source": "## About Human Protein Atlas Image Classification Challenge\n\n#### Our goal and challenge\n1. Unlike most image labeling tasks, where binary or multiclass labeling is considered, in this competition each image have multiple labels. Multiclass multilabel task has its own specific affecting the design of the model and the loass function.Our models should be able to classify mixed patterns of proteins in microscope images. It means that is a mutilabel classifiication problem so each image could have more than one possible labels. \n2. Also, we need to develop a model that is fast during prediction while maintaining high accuracy and can run on minimum hardware resources. Since the final model submission would be executed in the Docker container and the Docker container will be run on hardware meeting the following specificationsand must generate a submission result within 1 hour.\n\nSubmission hardware limits:\n* CPU Cores: 2\n* RAM: 4 GB\n* GPUs: Integrated Intel Graphics\n\n#### [About Human Protein Atlas Project](https://www.youtube.com/watch?v=GUvHrs5lKtU) \nHuman protein, human being's building blocks, they build, repair and give signals to our bodies, executing many functions that tohether enable life.  Different cells take on completely different functions which is due to different genes in the DNA being activated in accordance with teh part of the body in which the cell is located. It is the genes that give the instructions, but not for the cells,  for the proteins. The active gene is a code for a certein type of protein to be created which will determine the functins of the cells. The porject want to understand where are the 20,000 proteins are and to understand their functions. There only few proteins are only expressed uniquely in one of the tissues or organs in the body, only 10% of the proteins are tissue secific, and over the half of all the proteins are found everywhere. Almost all of today's medications are directed against proteins but so far they are only targeting 3% of 20,000 different proteins. If there is a treatment for liver disease, the risk is that the protein which the medicine is aimed at is also found in the brain. This will cause series consequences and also explains why many drugs have side effects. The goal of the project is to fully understand the complexity of the human cell, models must classify mixed patterns across a range of different human cells.\n\n[The cell atlas](https://www.proteinatlas.org/humanproteome/cell)\n\n<img src=\"http://science.sciencemag.org/content/sci/early/2017/05/10/science.aal3321/F1.large.jpg?width=800&height=600&carousel=1\" width=500>\n\n- Each image is of size $ 512 \\times 512 $. All image samples are represented by four filters.\n##### Green \nIndicates the protein of interest. It shows thw stained target proteins and consequently it's the most informative one.\n##### Blue\nIndicates the nucleus\n##### Red\nIndicates the microtubules\n##### Yellow\nIndicates the endoplasmic reticulum\n\nNotice, the green filter should be used to predict the label, other filters are used as references. \n\nThis work will use [PyTorch](https://pytorch.org/) as deep learning framework and [CUDA](https://developer.nvidia.com/cuda-zone) for GPU acceleration. "
    },
    {
      "metadata": {
        "_uuid": "0e41c777f4b3304901a20d9a4d34dea1a14866e8"
      },
      "cell_type": "markdown",
      "source": "###### This protein_pretrained_models_Ensemble_in_pytorch notebook will includes:\n- Our own pytorch dataset/loader class with the protein dataset\n- Several pretrained networks with Pytorch weights from this [dataset](https://www.kaggle.com/pvlima/pretrained-pytorch-models)\n- Our own ensemble method \n- Our own loss-function (Micro-F1)\n- Setup the training loop\n\n##### Also, many studies have showed that exploiting label correlations helps  to classify multi-label image. However, most of the work focus on analysis pairwise labels while ignore the complex structure of labels. This kernel would add a basic CNN model which take advantage of the high-order co-occurrence dependencies between labels to enhance clasification performance.\nRef: [A Dual-CNN Model for Multi-label\nClassification by Leveraging Co-occurrence\nDependencies Between Labels](https://pdfs.semanticscholar.org/c2a2/093b4163616b83398e503ae9ed948f4f6a2b.pdf)\n\n<img src='https://media.springernature.com/lw785/springer-static/image/chp%3A10.1007%2F978-3-319-77380-3_30/MediaObjects/465508_1_En_30_Fig2_HTML.gif'>\n\nThe image pretrained models accepts  images as input to extract the image representation , and the matrix CNN takes label co-occurrence matrix as input to get matrix embedding. Then the two representations are imported into fushion layer to get the joint representation which would be the final input to prediction layer."
    },
    {
      "metadata": {
        "_cell_guid": "b4eb5ad5-03ba-4fc4-b417-451e30a01fd1",
        "_uuid": "be3002e45fc687fc33e7c7ae9d869ac7ee926b1a"
      },
      "cell_type": "markdown",
      "source": "Import Python Modules"
    },
    {
      "metadata": {
        "_cell_guid": "0e05b5f8-fc51-404d-a9d2-5197aa283b73",
        "_uuid": "76fd0ec2a5eb7fbe49b51147eabbd109c61279c0",
        "trusted": true
      },
      "cell_type": "code",
      "source": "from __future__ import print_function, division\n\nimport os\nfrom os.path import isfile, join, abspath, exists, isdir, expanduser\nfrom os import listdir, makedirs, getcwd, remove\nimport sys\nimport time\nimport copy\nimport random\nimport math\nimport logging\nimport numpy as np\nimport pandas as pd\nimport PIL\nfrom PIL import Image\nfrom PIL import ImageChops\nfrom textwrap import wrap\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom collections import OrderedDict, defaultdict\nfrom scipy.misc import imread\nfrom sklearn.model_selection import RepeatedKFold\nfrom skimage.transform import resize\nfrom sklearn.preprocessing import MultiLabelBinarizer\nfrom sklearn.metrics import f1_score\nfrom sklearn.model_selection import train_test_split\n\nimport torch\nimport torch.optim as optim\nimport torch.nn as nn\nimport torch.optim as optim\nimport torch.backends.cudnn as cudnn\nimport torch.nn.functional as F\nfrom torch.autograd import Variable\nfrom torchvision import datasets, transforms, models\nfrom torch.utils.data import Dataset, DataLoader\nfrom torch.optim import lr_scheduler\n\nfrom IPython.display import HTML\nimport base64\n\n\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "4ab249acf2a99eb8bf8cfa08ab47f5e729695eab"
      },
      "cell_type": "code",
      "source": "!ls ../input/pretrained-pytorch-models",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "34949e6f18810466229443a9e304767656cafd2b"
      },
      "cell_type": "markdown",
      "source": "We have to copy the pretrained models to the cache directory (~/.torch/models) where Pytorch is looking for them."
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "ce135cb578f269906bb75baf518fa3a3551734d6"
      },
      "cell_type": "code",
      "source": "cache_dir = expanduser(join('~', '.torch'))\nif not exists(cache_dir):\n    makedirs(cache_dir)\nmodels_dir = join(cache_dir, 'models')\nif not exists(models_dir):\n    makedirs(models_dir)\n\n!cp ../input/pretrained-pytorch-models/* ~/.torch/models/\n!ls ~/.torch/models",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "ddf8d0ee12c408e85aecf7e11b7b3255acb52470"
      },
      "cell_type": "markdown",
      "source": "#### EDA"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b5d06ae920375fc79fb626b3e68ae48c589b265f"
      },
      "cell_type": "code",
      "source": "path=(\"../input/human-protein-atlas-image-classification\")\nprint (\"1. Loading data & Converting data\")\nprint (\"----------------------------------------\")\nprint('Data files: ')\nfor file in os.listdir(path):\n    print(file) \nprint (\"----------------------------------------\")\ntrain_class = pd.read_csv(path+\"/train.csv\")\nprint(\"There is %s samples in traininig set.\"%train_class.shape[0])\ntest_label = pd.read_csv(path+\"/sample_submission.csv\")\nprint(\"There is %s samples in testing set.\"%test_label.shape[0])\nprint (\"----------------------------------------\")\nprint (\"There are in total 28 different labels present in the dataset, each image has various possible labels.\")\n# train\ntrain_class.head()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c1815e4750272978862d5b8da96dc3124f13379a"
      },
      "cell_type": "code",
      "source": "label_names = {\n    0:  \"Nucleoplasm\",  \n    1:  \"Nuclear membrane\",   \n    2:  \"Nucleoli\",   \n    3:  \"Nucleoli fibrillar center\",   \n    4:  \"Nuclear speckles\",\n    5:  \"Nuclear bodies\",   \n    6:  \"Endoplasmic reticulum\",   \n    7:  \"Golgi apparatus\",   \n    8:  \"Peroxisomes\",   \n    9:  \"Endosomes\",   \n    10:  \"Lysosomes\",   \n    11:  \"Intermediate filaments\",   \n    12:  \"Actin filaments\",   \n    13:  \"Focal adhesion sites\",   \n    14:  \"Microtubules\",   \n    15:  \"Microtubule ends\",   \n    16:  \"Cytokinetic bridge\",   \n    17:  \"Mitotic spindle\",   \n    18:  \"Microtubule organizing center\",   \n    19:  \"Centrosome\",   \n    20:  \"Lipid droplets\",   \n    21:  \"Plasma membrane\",   \n    22:  \"Cell junctions\",   \n    23:  \"Mitochondria\",   \n    24:  \"Aggresome\",   \n    25:  \"Cytosol\",   \n    26:  \"Cytoplasmic bodies\",   \n    27:  \"Rods & rings\"\n}\nLABELS = []\n\nfor label in label_names.values():\n    LABELS.append(label)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0656547ee6632286c5202d4453af3d4778de3341"
      },
      "cell_type": "markdown",
      "source": "###### Show the highest frequent single label in the dataset"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "367cbadaec0e4922eb858cc4880d63b5a5a2e3ec"
      },
      "cell_type": "code",
      "source": "def fill_targets(row):\n    row.Target = np.array(row.Target.split(\" \")).astype(np.int)\n    for num in row.Target:\n        name = label_names[int(num)]\n        row.loc[name] = 1\n    return row\nfor key in label_names.keys():\n    train_class[label_names[key]] = 0\ntrain_label = train_class.apply(fill_targets, axis=1)\n#train_label.head()\nn=28\ntarget_counts = train_label.drop([\"Id\", \"Target\"],axis=1).sum(axis=0).sort_values(ascending=False)\nplt.figure(figsize=(10,6))\npal = sns.cubehelix_palette(n, start=2, rot=-0.1, dark=0, light=.65, reverse=True)\nsns.barplot(y=target_counts.index.values, x=target_counts.values, order=target_counts.index, palette=pal).set_title('The distribution Labels')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "381bc88c2fa59fa2c7a62f0ef2e5c1cf81ebae8b"
      },
      "cell_type": "markdown",
      "source": "####  Show the most common label combinations"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "51b8e04cd88130be9fdc90ddff6587c9680dcc22"
      },
      "cell_type": "code",
      "source": "print(\"There are \"+  str(len(train_class.Target.unique())) + \" different combinations of labels in our dataset\")\nsns.set(style=\"dark\")\nn=40 # top 40 common label combinations\nvalues = train_class['Target'].value_counts()[:n].keys().tolist()\ncounts = train_class['Target'].value_counts()[:n].tolist()\n\nplt.figure(figsize=(12,5))\npal = sns.cubehelix_palette(n, start=2, rot=-0.1, dark=0, light=.65, reverse=True)\nsns.barplot(y=counts, x=values, palette=pal).set_title(str(n)+\" MOST COMMON LABEL COMBINATIONS\")\nplt.xticks(rotation=45)\nplt.show()\n\n## the least populated class combinations counts\nprint(\"There are %s label combinations only has one sample.\" % str(sum(train_class['Target'].value_counts(ascending=True)<=1)))\n\n## ratio of single label to multiple label\nn=len(train_class.Target.unique())\nvalues = train_class['Target'].value_counts()[:n].keys().tolist()\ncounts = train_class['Target'].value_counts()[:n].tolist()\nsingle_count=0\ncomb_count=0\nfor labels in values:\n    if len(labels.split())==1:\n        single_count+=train_class['Target'].value_counts()[labels]\n    else:\n        comb_count+=train_class['Target'].value_counts()[labels]\nprint(\"There are \" + str(single_count)+\" samples have single label\")\nprint(\"There are \" + str(comb_count)+\" samples have multiple labels\")\nprint(\"The percentage of images which has single label is: \" + str(single_count/train_class.shape[0]))\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "0533b8583d5b5e04af6ec96324bf4991991036d1"
      },
      "cell_type": "markdown",
      "source": "###### According to the different targets (582 kinds of combination of labels) for each sample, we can compute a label co-occurence matrix "
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "b32a85ad5ed26dd2ae9a3987462885b050d5494f"
      },
      "cell_type": "code",
      "source": "train_corr=train_label.drop(['Id','Target'],axis=1)\ndf_asint=train_corr.astype(int)\ncoocc=df_asint.T.dot(df_asint)\nnp.fill_diagonal(coocc.values,0)\ncoocc",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "07763a82-f724-46ec-a323-e1f26101dcf3",
        "_uuid": "c060d3a7eae2502855e84ae5eabd1a8c72fc4ea2",
        "colab_type": "text",
        "id": "69iuBpidwheJ"
      },
      "cell_type": "markdown",
      "source": "## Dataset loader in Pytorch\n\nThe dataset is divided in three categories: training, validation and test. \n\n#### Dataset Class\nTo create a dataset class for the human protein atlas image dataset, we will read the csv in __init__ but leave the reading of image to __getitem__. This is memory efficient because all the images are not stored in the memory at once but read as required.\n\n#### Transforms\nMost neural networks expect the images of a fixed size. Therefore, we need to provide a preprocessing function which includes:\n* Rescale: to scale the image\n* RandomCrop: to crop from image randomly which is data augmentation.\n* ToTensor: to convert the numpy images to torch images\n* Normalize: normalize a tensor image with mean and standard deviation. \n"
    },
    {
      "metadata": {
        "_cell_guid": "e9d7ef88-fdbd-4b73-b64f-70294976d238",
        "colab_type": "code",
        "id": "xvsy0IR4wheJ",
        "executionInfo": {
          "elapsed": 1074,
          "user": {
            "displayName": "Carlo Alberto",
            "userId": "107843268563316278814",
            "photoUrl": "//lh5.googleusercontent.com/-_sBZsyc315U/AAAAAAAAAAI/AAAAAAAAAgg/b4D9SE9jgD0/s50-c-k-no/photo.jpg"
          },
          "timestamp": 1525007461873,
          "user_tz": -120,
          "status": "ok"
        },
        "_uuid": "83371f19c7f1e261bb0f9cc71f7a265c37a4c16a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68,
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "outputId": "bb02efaa-518c-4342-d6e5-7275a7d7fdd5",
        "trusted": true
      },
      "cell_type": "code",
      "source": "def load_image(basepath, image_id):\n    images = np.zeros(shape=(512,512,3))\n    r = np.array(Image.open(basepath+image_id+\"_red.png\"))\n    g = np.array(Image.open(basepath+image_id+\"_green.png\"))\n    b = np.array(Image.open(basepath+image_id+\"_blue.png\"))\n    y =np.array( Image.open(basepath+image_id+\"_yellow.png\"))\n\n    ## add yellow channel to red channel\n    r+=y\n    \n    images[:,:,0] = r.astype(np.uint8) \n    images[:,:,1] = g.astype(np.uint8)\n    images[:,:,2] = b.astype(np.uint8)\n    #images[:,:,3] = y.astype(np.uint8)\n    images = images.astype(np.uint8)\n    return images",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "c3f7f06c-97fe-4f8e-8b47-f512d4989ecb",
        "colab_type": "code",
        "id": "hGLvyjZ2whe0",
        "executionInfo": {
          "elapsed": 4437,
          "user": {
            "displayName": "Carlo Alberto",
            "userId": "107843268563316278814",
            "photoUrl": "//lh5.googleusercontent.com/-_sBZsyc315U/AAAAAAAAAAI/AAAAAAAAAgg/b4D9SE9jgD0/s50-c-k-no/photo.jpg"
          },
          "timestamp": 1525025142751,
          "user_tz": -120,
          "status": "ok"
        },
        "_uuid": "4f2ed1c58098075949f93eadc7ba3b5786c7b307",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1122,
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        },
        "outputId": "497f6621-ee2c-43a5-9b54-917bbbe9b22c",
        "trusted": true
      },
      "cell_type": "code",
      "source": "targets = train_class['Target'].value_counts().keys()\ncounts =train_class['Target'].value_counts().values\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "3773b14be3aeac9bb7600ec8768d39e7ba6cad37"
      },
      "cell_type": "markdown",
      "source": "#### Define a dataset class for Human Protein Dataset\n*  read the csv in ' __init__ ' but leave the reading of images to  ' __getitem__ '\n*  This is memory efficient because all the images are not stored in the memory at once but read as required\n*  any required processing can be applied by '__transform__' argument "
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "9b79885898b99dc0611bb55a35d8fdf0f051084d"
      },
      "cell_type": "code",
      "source": "## parameters\nnum_classes = 28\n# image size\nBATCH_SIZE =16\nW = H = 224\nclass HumanProteinDataset(Dataset):\n\n    def __init__(self, df,set_path,transform=False,test=False):\n        self.set_path=set_path\n        self.test = test\n        self.df = df.copy()\n        #self.mlb= MultiLabelBinarizer(num_classes)\n        #self.mlb.fit(num_classes)\n        self.transform = transform\n           \n    def __getitem__(self, idx):\n        \n        image = load_image(self.set_path, self.df['Id'].iloc[idx])\n        \n        sample = {'image': image}\n\n        if not self.test:\n            target=np.array(list(map(int, self.df.iloc[idx].Target.split(' '))))\n            sample['target'] = np.eye(num_classes,dtype=np.float)[target].sum(axis=0) \n        else:\n            sample['Id'] = self.df['Id'].iloc[idx]\n        \n        sample['image']=transforms.ToPILImage()(sample['image'])\n        \n        if self.transform:\n            sample['image'] = self.image_transform(sample['image'])\n        \n        totensor = transforms.Compose([transforms.Resize(size=(W,H)),transforms.ToTensor(), transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])\n        sample['image']=totensor(sample['image'])\n        return sample\n        \n       # ret = {'image': totensor(image)}\n       #\n       # if \"target\" in sample.keys():\n       #     target = sample['target'][0]\n       #     ret['target'] = target\n       # else:\n       #     ret['Id'] = sample['Id']\n       #          \n       # return ret\n    \n    def __len__(self):\n        return self.df.shape[0]\n    \n    def shape(self):\n        return self.df.shape\n## Apply a list of transformations in a random order\n    def image_transform(self,image):\n        transform=transforms.RandomOrder([\n        transforms.RandomRotation(30),\n        transforms.RandomHorizontalFlip(),\n        transforms.RandomVerticalFlip()\n    ])\n        t_image=transform(image)\n        return t_image\n    \n## split train set into trainning set and validation set\ntrain_path= path+'/train/'\ntrain=pd.read_csv(path+'/train.csv')\n\n#####--------------------------------------\n##### delete those class combinations with less than 5 samples\n#####--------------------------------------\ncount=train['Target'].value_counts(ascending=True)\ncondition=count<=5\none_sample_label=count[condition].index\none_sample_label.tolist()\nprint(\"the sample numbers before dropping four samples target is : \"+ str(len(train)))\ntrain=train[~train['Target'].isin(one_sample_label)]\nprint(\"the sample numbers after dropping four samples target is : \"+ str(len(train)))\n\n\n#### ----------------------------------------\n#### Downsample majority class: ['0', '0,25', '23', '25']\n#### ----------------------------------------\n\nfrom sklearn.utils import resample\n## Downsample class '0'\ndf_majority=train[train.Target=='0']\ndf_minority=train[train.Target!='0']\ndf_majority_downsampled = resample(df_majority, \n                                 replace=False,    # sample without replacement\n                                 n_samples=1500,     # to match minority class\n                                 random_state=123) # reproducible results\n\ntrain = pd.concat([df_majority_downsampled, df_minority])\nprint(len(train))\n## Downsample class '0,25'\ndf_majority=train[train.Target=='25 0']\ndf_minority=train[train.Target!='25 0']\ndf_majority_downsampled = resample(df_majority, \n                                 replace=False,    # sample without replacement\n                                 n_samples=1300,     # to match minority class\n                                 random_state=12) # reproducible results\ntrain = pd.concat([df_majority_downsampled, df_minority])\nprint(len(train))\n## Downsample class '23'\ndf_majority=train[train.Target=='23']\ndf_minority=train[train.Target!='23']\ndf_majority_downsampled = resample(df_majority, \n                                 replace=False,    # sample without replacement\n                                 n_samples=1200,     # to match minority class\n                                 random_state=23) # reproducible results\ntrain = pd.concat([df_majority_downsampled, df_minority])\nprint(len(train))\n## Downsample class '25'\ndf_majority=train[train.Target=='25']\ndf_minority=train[train.Target!='25']\ndf_majority_downsampled = resample(df_majority, \n                                 replace=False,    # sample without replacement\n                                 n_samples=1300,     # to match minority class\n                                 random_state=13) # reproducible results\ntrain = pd.concat([df_majority_downsampled, df_minority])\nprint(len(train))\nid=train['Id']\ny=train['Target']\n## randomly pick 80% of the data for training\n#train_id,val_id,train_y,val_y = train_test_split(id,y,stratify=y,test_size = 0.2)\n## randomly split the picked data into training set and validation set\ntrain_id,val_id,train_y,val_y = train_test_split(id,y,stratify=y,test_size = 0.2)\n\ntrain_df=pd.DataFrame({'Id':train_id, 'Target':train_y})\nval_df=pd.DataFrame({'Id':val_id, 'Target':val_y})\n\n## load dataset\ntrain_set= HumanProteinDataset(train_df,train_path,transform=True,test=False)\nval_set= HumanProteinDataset(val_df,train_path,transform=False,test=False)\n\ndataset_sizes = {}\n\ndataset_sizes['train'] = len(train_df)\ndataset_sizes['val'] = len(val_df)\n    \ntrain_loader = DataLoader(train_set, batch_size=BATCH_SIZE,num_workers=0,shuffle=True)\nvalidation_loader = DataLoader(val_set, batch_size=BATCH_SIZE, num_workers=0,shuffle=False)\n\ndataloaders = {}\n\ndataloaders['train'] = train_loader\ndataloaders['val'] = validation_loader\n\nprint(len(train_df))\nprint(len(val_df))\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "d7108ced4631ed533e2a36b0f162f16248c73004"
      },
      "cell_type": "code",
      "source": "print(train_set.df.head())\nprint(val_set.df.head())\n###### --------------------------------------------------------\n###### plot the target distribution in train set and val set\n##### -------------------------------------------------------\nprint(\"There are \"+  str(len(train_df.Target.unique())) + \" different combinations of labels in our dataset in train set and validation set\")\nsns.set(style=\"dark\")\nn=40 # top 40 common label combinations\ntrain_values = train_df['Target'].value_counts()[:n].keys().tolist()\ntrain_counts = train_df['Target'].value_counts()[:n].tolist()\n\nval_values = val_df['Target'].value_counts()[:n].keys().tolist()\nval_counts = val_df['Target'].value_counts()[:n].tolist()\n\nplt.figure(figsize=(16,6))\nsns.barplot(y=train_counts, x=train_values, palette=pal).set_title(str(n)+\" MOST COMMON LABEL COMBINATIONS DISTRIBUTIONS BETWEEN TRAIN SET AND VALIDATION SET \")\npal_val = sns.cubehelix_palette(n, start=0.5, rot=-.4, dark=0, light=.75, reverse=False)\nsns.barplot(y=val_counts, x=val_values, palette=pal_val).set_title(str(n)+\" MOST COMMON LABEL COMBINATIONS DISTRIBUTIONS BETWEEN TRAIN SET AND VALIDATION SET\")\nplt.xticks(rotation=45)\nplt.legend(['Train','Validation'], loc = \"upper right\", framealpha=1, shadow=True)\nplt.show()\n",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "56d5cb7dca553db848c56c26633d6e146b0cb9e0"
      },
      "cell_type": "code",
      "source": "## Customize the color maps for imshow\n\ncdict1 = {'red':((0.0,  0.0, 0.0),(0.75, 1.0, 1.0),(1.0,  1.0, 1.0)),\n\n         'green': ((0.0,  0.0, 0.0),(1.0,  0.0, 0.0)),\n\n         'blue':  ((0.0,  0.0, 0.0),(1.0,  0.0, 0.0))}\n\ncdict2 = {'red':   ((0.0,  0.0, 0.0),(1.0,  0.0, 0.0)),\n\n         'green': ((0.0,  0.0, 0.0),(0.75, 1.0, 1.0),(1.0,  1.0, 1.0)),\n\n         'blue':  ((0.0,  0.0, 0.0),(1.0,  0.0, 0.0))}\n\ncdict3 = {'red':   ((0.0,  0.0, 0.0),(1.0,  0.0, 0.0)),\n\n         'green': ((0.0,  0.0, 0.0),(1.0,  0.0, 0.0)),\n\n         'blue':  ((0.0,  0.0, 0.0),(0.75, 1.0, 1.0),(1.0,  1.0, 1.0))}\n\n\nplt.register_cmap(name='reds', data=cdict1)\nplt.register_cmap(name='greens', data=cdict2)\nplt.register_cmap(name='blues', data=cdict3)\n\n\ndef Show(sample):\n    f, (ax1,ax2,ax3,ax4) = plt.subplots(1, 4, figsize=(16,16), sharey=True)\n    #f, (ax1,ax2,ax3) = plt.subplots(1, 3, figsize=(16,16), sharey=True)\n    title = ''\n    \n    labels =sample['target']\n                \n    for i, label in enumerate(LABELS):\n        if labels[i] == 1:\n            if title == '':\n                title += label\n            else:\n                title += \" & \" + label\n            \n    ax1.imshow(sample['image'][0,:,:],cmap=\"reds\")\n    ax1.set_title(\"\\n\".join(wrap('(Red&Yellow)- Mocrotublues & Endoplasmic Reticulum', 30)),fontsize=12)\n    ax2.imshow(sample['image'][1,:,:],cmap=\"greens\")\n    ax2.set_title('(Green)The Protein of interest',fontsize=12)\n    ax3.imshow(sample['image'][2,:,:],cmap=\"blues\")\n    ax3.set_title('(Blue)nucleus',fontsize=12)\n### plot stack images\n    x1=sample['image'][0,:,:]\n    x2=sample['image'][1,:,:]\n    x3=sample['image'][2,:,:]\n    ax4.imshow(x1, alpha=0.33, cmap=\"reds\")\n    ax4.imshow(x2, alpha=0.33,cmap=\"greens\")\n    ax4.imshow(x3, alpha=0.33, cmap=\"blues\")\n    ax4.set_title('Combined RGBY images',fontsize=12)\n    f.suptitle(title, fontsize=16, y=0.65)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "c26f83ec417c9432363e6002995a18b44215e882"
      },
      "cell_type": "code",
      "source": "idxs = random.sample(range(1, train_set.df.shape[0]), 3)\n\nfor idx in idxs:\n    Show( train_set[idx])",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "4a18a7c2cb7df278ef6542b2e2b5aa7f490bc4fd"
      },
      "cell_type": "markdown",
      "source": "### ResNet50"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "f26aabf1d15f290f5e988f28da09abf831cea4b7"
      },
      "cell_type": "code",
      "source": "!nvidia-smi",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "4f785ca8dd180d1c6676d83cee51ef8d75aa3f85"
      },
      "cell_type": "markdown",
      "source": "##### As the challenge has indicated that tthe final submission would be evaluated by macro F1 Score\n[macro F1 Score](https://sebastianraschka.com/faq/docs/multiclass-metric.html)\n\n<img src='https://sebastianraschka.com/images/faq/multiclass-metric/conf_mat.png' width=300>\n<img src='https://sebastianraschka.com/images/faq/multiclass-metric/pre-rec.png' width=300>\n#### Note that  PRE=precision, REC=recall, F1=F1-score\n<img src='https://sebastianraschka.com/images/faq/multiclass-metric/micro.png' width=300>\n<img src='https://sebastianraschka.com/images/faq/multiclass-metric/macro.png' width=200>"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "d68d4dc7c804b3efe0e9415329f333337453d403"
      },
      "cell_type": "code",
      "source": "def train_model(dataloders, model, criterion, optimizer,num_epochs=0):\n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    best_model_wts = model.state_dict()\n    best_f1= 0\n    losses=AverageMeter()\n    epoch_f1=AverageMeter()\n    dataset_sizes = {'train': len(dataloders['train'].dataset), \n                     'val': len(dataloders['val'].dataset)}\n\n    for epoch in range(num_epochs):\n        for phase in ['train', 'val']:\n            if phase == 'train':\n                model.train(True)\n            else:\n                model.train(False)\n            for i, data in enumerate(dataloaders[phase], 0):            \n                # get the inputs\n                inputs, labels = data['image'], data['target']\n                if use_gpu:\n                     inputs, labels = inputs.to(device,dtype=torch.float), labels.to(device,dtype=torch.float)\n                else:\n                    inputs, labels = Variable(inputs), Variable(labels)\n\n                optimizer.zero_grad()\n                outputs = model(inputs)\n                outputs=outputs.to(device,dtype=torch.float)\n               \n                loss = criterion(outputs,labels)\n                losses.update(loss.item(),inputs.size(0))\n                f1=f1_score(labels.cpu().data.numpy(),(outputs.sigmoid()>0.15).cpu().data.numpy(),average='macro')\n                epoch_f1.update(f1,inputs.size(0))\n\n                if phase == 'train':\n                    loss.backward()\n                    optimizer.step()\n\n                labels = labels.data.byte()\n            \n            if phase == 'train':\n                train_epoch_loss = losses.avg\n                train_epoch_f1 = epoch_f1.avg\n            else:\n                valid_epoch_loss =  losses.avg\n                valid_epoch_f1 = epoch_f1.avg\n                \n            if phase == 'val' and valid_epoch_f1 > best_f1:\n                best_f1= valid_epoch_f1\n                best_model_wts = model.state_dict()\n        \n        print('Epoch [{}/{}] train loss: {:.4f} train f1: {:.4f} ,' \n              'valid loss: {:.4f} valid f1: {:.4f}'.format(\n                epoch, num_epochs - 1,\n                train_epoch_loss, train_epoch_f1,\n                valid_epoch_loss, valid_epoch_f1))\n            \n    print('Best val Acc: {:4f}'.format(best_f1))\n\n    model.load_state_dict(best_model_wts)\n    return model",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "76a7f9b53f0dfe7ca8d1cb4654b07bfd129d8c82"
      },
      "cell_type": "code",
      "source": "# resnet = models.resnet50(pretrained=True)\n# freeze all the layers except the final one\nresnet = models.resnet50(pretrained=True)\nfor param in resnet.parameters():\n    param.requires_grad = False\nresnet.dropout=nn.Dropout(0.3)\n# add new final layer with 28 classes\nnum_ftrs = resnet.fc.in_features\nresnet.fc = nn.Linear(num_ftrs,28)\nuse_gpu = torch.cuda.is_available()\n\nprint(resnet)\nif use_gpu:\n    print('CUDA')\n    resnet = resnet.cuda()\n### loss function based on micro f1-score\n#def f1_loss(target, output, epsilon=1e-7):\n#    y_pred = nn.Sigmoid()(output)\n#    y_true = target.double()\n\n#    TP = (y_pred * y_true).sum(1)\n#    prec = TP / (y_pred.sum(1) + epsilon)\n#    rec = TP / (y_true.sum(1) + epsilon)\n#    res = 2 * prec * rec / (prec + rec + epsilon)\n\n#    f1 = res\n#    f1 = f1.clamp(min=0)\n#    return 1 - f1.mean()\n\nclass AverageMeter:\n    \"\"\"Computes and stores the average and current value\"\"\"\n\n    def __init__(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def init(self):\n        self.val = 0\n        self.avg = 0\n        self.sum = 0\n        self.count = 0\n\n    def update(self, val, n=1):\n        self.val = val\n        self.sum += val * n\n        self.count += n\n        self.avg = self.sum / self.count\n\n    def __repr__(self):\n        return str(self.avg)\ncriterion = torch.nn.BCEWithLogitsLoss().cuda()\nlr=0.09\noptimizer = torch.optim.SGD(resnet.fc.parameters(), lr=lr,momentum=0.9,weight_decay=1e-4)\ndloaders = {'train':dataloaders['train'], 'val':dataloaders['val']}\nstart_time = time.time()\nmodel = train_model(dloaders, resnet, criterion, optimizer, num_epochs=25)\nprint('Training time: {:10f} minutes'.format((time.time()-start_time)/60))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_uuid": "9f19f7e15b2e6df2ee8ef917c08f6b8496ee06d2"
      },
      "cell_type": "markdown",
      "source": "### Predict test set"
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "3bc33a047206712a819bbf7d4643f99d40e26b67"
      },
      "cell_type": "code",
      "source": "test_df=pd.read_csv(path+'/sample_submission.csv')\ntest_path=path+'/test/'\ndataset_test = HumanProteinDataset(test_df,test_path,transform=False,test=True)\n\ndataloader_test = DataLoader(dataset_test, 1,shuffle=False, num_workers=0)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "12cd983a2f88bc82adf852341aa4219913cb2fff"
      },
      "cell_type": "code",
      "source": "def run_model(model,batch):\n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    model.to(device)\n    inputs = batch\n    inputs = inputs.to(device,dtype=torch.float)\n    out = model(inputs)\n    out = out.cpu()\n    return out\nids = []\npredictions = []\n\nmodel = model.cuda()\n\nfor sample_batched in dataloader_test:\n        out = run_model(model,sample_batched['image'])\n        preds = []\n        out = out.detach().numpy()\n        for sample in out:\n            p = \"\"\n            for i, label in enumerate(sample):\n                if label > 0.15:\n                    p += \" \" + str(i)\n            if p == \"\":\n                p = \"0\"\n            else:\n                p = p[1:]\n            preds.append(p)\n\n        ids += list(sample_batched['Id'])\n        predictions += preds",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true,
        "_uuid": "a41c7344796d0dcbdc386b0488404f61e0f9709e"
      },
      "cell_type": "code",
      "source": "## Since kaggle kernel is read only file system, we need to create a function that\n## takes in a dataframe and creates a link to download it\ndf = pd.DataFrame({'Id':ids,'Predicted':predictions})\ndf.to_csv('protein_classification.csv', header=True, index=False)\n\n# function that takes in a dataframe and creates a text link to  \n# download it (will only work for files < 2MB or so)\ndef create_download_link(df, title = \"Download Submission CSV file\", filename = \"protein_classification.csv\"):  \n    csv = df.to_csv( sep=',', encoding='utf-8', index=False)\n    b64 = base64.b64encode(csv.encode())\n    payload = b64.decode()\n    html = '<a download=\"{filename}\" href=\"data:text/csv;base64,{payload}\" target=\"_blank\">{title}</a>'\n    html = html.format(payload=payload,title=title,filename=filename)\n    return HTML(html)\n\n\n# create a link to download the dataframe\ncreate_download_link(df)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "_cell_guid": "c36cfd33-f857-419d-8508-2a4441c68614",
        "_uuid": "39d5b7019cd132b26b95d250d5983f6ada032c92"
      },
      "cell_type": "markdown",
      "source": "## TO DO LIST\n - Stack the features extracted frommltiple Pretrained models.\n - Ref: [Multi-label-Inception-net](https://github.com/BartyzalRadek/Multi-label-Inception-net)\n - dd Model Diversities Embedding: High model complexity \n - Fix class imbalances issues: downsampling/upsampling (why focalLoss doesn't help?)\n - implement Powerset Methods\n - CNN + RNN doesn't work well?\n\n\n"
    }
  ],
  "metadata": {
    "colab": {
      "name": "VGG16_v2-OCT2017_Retina.ipynb",
      "version": "0.3.2",
      "toc_visible": true,
      "views": {},
      "default_view": {},
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "accelerator": "GPU",
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 1
}