kaggle_histopathology_a-complete-ml-pipeline-fast-ai.ipynb

{"cells":[{"metadata":{"_uuid":"4c08ef44cc9bdc0d491b6c5d53387105186c2ab8"},"cell_type":"markdown","source":"<a href=\"https://ibb.co/5Mpm3ZF\"><img src=\"https://i.ibb.co/gjqN8YV/metastases.png\" alt=\"metastases\" border=\"0\" align=”right”></a>\n**Sections of this kernel**\n- Project understanding\n- Data understanding\n- Data visualization\n- Baseline model\n- Validation and analysis\n    - Metrics\n    - Prediction visualizations\n    - Confusion matrix\n    - ROC & AUC\n- Submit\n\n---------------------------------------------------\n# Project understanding\n###  What exactly is the problem?\n\n**Binary image classification problem.** Identify the presence of metastases from 96 x 96px digital histopathology images. One key challenge is that the metastases can be as small as single cells in a large area of tissue.\n\n### How would a solution look like?\n\n**Our evaluation metric is [area under the ROC curve](http://en.wikipedia.org/wiki/Receiver_operating_characteristic).** The ROC curve is a plot of *True positive rate* against *False positive rate* at various thresholds and the area under the curve (AUC) is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one. The best possible solution would yield an AUC of 1 which means we would classify all positive samples correctly without getting any false positives. \n\n![ROC curve example](https://i.ibb.co/mBKh6ZB/roc.png)\n<p style=\"text-align: center;\"> ROC curve from a previous run of this kernel </p>\n\n### What is known about the domain?\n\n**The histopathological images are glass slide microscope images of lymph nodes that are stained with hematoxylin and eosin (H&E).** This staining method is one of the most widely used in medical diagnosis and it produces blue, violet and red colors. Dark blue hematoxylin binds to negatively charged substances such as nucleic acids and pink eosin to positively charged substances like amino-acid side chains (most proteins). Typically nuclei are stained blue, whereas cytoplasm and extracellular parts in various shades of pink.\n\n**Low-resolution**             | **Mid-resolution**            | **High-resolution** \n:-------------------------:|:-------------------------:|:-------------------------:\n![](https://camelyon17.grand-challenge.org/site/CAMELYON17/serve/public_html/example_low_resolution.png) | ![Example of a metastatic region](https://camelyon17.grand-challenge.org/site/CAMELYON17/serve/public_html/example_mid_resolution.png) | ![Example of a metastatic region](https://camelyon17.grand-challenge.org/site/CAMELYON17/serve/public_html/example_high_resolution.png)\n**[<p style=\"text-align: center;\"> Example of a metastatic region in lymph nodes, CHAMELYON17 </p>](https://camelyon17.grand-challenge.org/Background/)**\n\nLymph nodes are small glands that filter the fluid in the lymphatic system and they are the first place a breast cancer is likely to spread. Histological assessment of lymph node metastases is part of determining the stage of breast cancer in TNM classification which is a globally recognized standard for classifying the extent of spread of cancer. The diagnostic procedure for pathologists is tedious and time-consuming as a large area of tissue has to be examined and small metastases can be easily missed.\n\n**Useful links for background knowledge**\n- [Patch Camelyon (PCam)](https://github.com/basveeling/pcam)\n- [Hematoxylin and eosin staining of tissue and cell sections](https://www.ncbi.nlm.nih.gov/pubmed/21356829)\n- [H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset](https://academic.oup.com/gigascience/article/7/6/giy065/5026175)\n- [CAMELYON16 - background](https://camelyon16.grand-challenge.org/Background/)\n- [CAMELYON17 - background](https://camelyon17.grand-challenge.org/Background/)\n- [TNM classification](https://www.uicc.org/resources/tnm)"},{"metadata":{"_uuid":"19fdd1381f06e397810c3fef5a31be0c9c9024bc"},"cell_type":"markdown","source":"----------------------------------------------\n# Data understanding\n### What data do we have available?\n\n**220k training images and 57k evaluation images.** The dataset is a subset of the [PCam dataset](https://github.com/basveeling/pcam) and the only difference between these two is that all duplicate images have been removed. The PCam dataset is derived from the [Camelyon16 Challenge dataset](https://camelyon16.grand-challenge.org/Data/) which contains 400 H&E stained whole slide images of sentinel lymph node sections that were acquired and digitized at 2 different centers using a 40x objective. The PCam's dataset including this one uses 10x undersampling to increase the field of view, which gives the resultant pixel resolution of 2.43 microns.\n\nAccording to the data description, there is a 50/50 balance between positive and negative examples in the training and test splits. However, **the training distribution seems to be 60/40 (negatives/positives)**. A positive label means that there is at least one pixel of tumor tissue in the center region (32 x 32px) of the image. **Tumor tissue in the outer region of the patch does not influence the label.** This means that a negatively labeled image could contain metastases in the outer region. Thus, it would be a good idea to crop the images to the center region.\n\n**Image file descriptors**\n\nDescription | \n:--------:|:-------:\nFormat | TIF\nSize | 96 x 96\nChannels | 3\nBits per channel | 8\nData type | Unsigned char\nCompression | Jpeg\n\n### Is the data relevant to the problem?\n\nThis dataset is a combination of two independent datasets collected in Radboud University Medical Center (Nijmegen, the Netherlands), and the University Medical Center Utrecht (Utrecht, the Netherlands). The slides are produced by routine clinical practices and a trained pathologist would examine similar images for identifying metastases. However, some relevant information about the surroundings might be left out with these small-sized image samples.\n\n### Is it valid? Does it reflect our expectations?\n\nAccording to the data description, the dataset has been stripped of duplicates. However, this has not been confirmed by testing.\n\n> For the entire dataset, when the slide-level label was unclear during the inspection of the H&E-stained slide, an additional WSI with a consecutive tissue section, immunohistochemically stained for cytokeratin, was used to confirm the classification.\n- [1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset](https://academic.oup.com/gigascience/article/7/6/giy065/5026175)\n\n### Is the data quality, quantity, recency sufficient?\n\n> All glass slides included in the CAMELYON dataset were part of routine clinical care and are thus of diagnostic quality. However, during the acquisition process, scanning can fail or result in out-of-focus images. As a quality-control measure, all slides were inspected manually after scanning. The inspection was performed by an experienced technician (Q.M. and N.S. for UMCU, M.H. or R.vd.L. for the other centers) to assess the quality of the scan; when in doubt, a pathologist was consulted on whether scanning issues might affect diagnosis.\n- [1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset](https://academic.oup.com/gigascience/article/7/6/giy065/5026175)"},{"metadata":{"_uuid":"dfc41ba2b70e88b92ff41a94f0b6504336f2360d"},"cell_type":"markdown","source":"-----------------------------------------\n# Data visualization"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport random\nfrom sklearn.utils import shuffle\n\ndata = pd.read_csv('/kaggle/input/train_labels.csv')\ntrain_path = '/kaggle/input/train/'\ntest_path = '/kaggle/input/test/'\n# quick look at the label stats\ndata.describe()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"We can see that the negative/positive ratio is not entirely 50/50 as the label mean is well below 0.5. The ratio is closer to 60/40 meaning that there are 1.5 times more negative images than positives.\n\n### Plot some images with and without cancer tissue for comparison"},{"metadata":{"trusted":true,"_uuid":"90811cbf83e65ba2679ed1044798a71b2dd89818","_kg_hide-input":true},"cell_type":"code","source":"def readImage(path):\n    # OpenCV reads the image in bgr format by default\n    bgr_img = cv2.imread(path)\n    # We flip it to rgb for visualization purposes\n    b,g,r = cv2.split(bgr_img)\n    rgb_img = cv2.merge([r,g,b])\n    return rgb_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"20e119981333d59b0ba4a52b46a9aa5b4bb2436f","_kg_hide-input":true},"cell_type":"code","source":"# random sampling\nshuffled_data = shuffle(data)\n\nfig, ax = plt.subplots(2,5, figsize=(20,8))\nfig.suptitle('Histopathologic scans of lymph node sections',fontsize=20)\n# Negatives\nfor i, idx in enumerate(shuffled_data[shuffled_data['label'] == 0]['id'][:5]):\n    path = os.path.join(train_path, idx)\n    ax[0,i].imshow(readImage(path + '.tif'))\n    # Create a Rectangle patch\n    box = patches.Rectangle((32,32),32,32,linewidth=4,edgecolor='b',facecolor='none', linestyle=':', capstyle='round')\n    ax[0,i].add_patch(box)\nax[0,0].set_ylabel('Negative samples', size='large')\n# Positives\nfor i, idx in enumerate(shuffled_data[shuffled_data['label'] == 1]['id'][:5]):\n    path = os.path.join(train_path, idx)\n    ax[1,i].imshow(readImage(path + '.tif'))\n    # Create a Rectangle patch\n    box = patches.Rectangle((32,32),32,32,linewidth=4,edgecolor='r',facecolor='none', linestyle=':', capstyle='round')\n    ax[1,i].add_patch(box)\nax[1,0].set_ylabel('Tumor tissue samples', size='large')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c63783a335ea6f078c07fcd1024baf86e0255234"},"cell_type":"markdown","source":"Classifying metastases is probably not an easy task for a trained pathologist and extremely difficult for an untrained eye. According to [Libre Pathology](https://librepathology.org/wiki/Lymph_node_metastasis), lymph node metastases can have these features:\n\n> - Foreign cell population - key feature (Classic location: subcapsular sinuses)\n- Cells with cytologic features of malignancy\n    - Nuclear pleomorphism (variation in size, shape and staining).\n    - Nuclear atypia:\n        - **Nuclear enlargement**.\n        - **Irregular nuclear membrane**.\n        - **Irregular chromatin pattern, esp. asymmetry**.\n        - **Large or irregular nucleolus**.\n     - Abundant mitotic figures.\n- Cells in architectural arrangements seen in malignancy; highly variable - dependent on tumour type and differentiation.\n    - Gland formation.\n    - Single cells.\n    - Small clusters of cells.\n  \n**The takeaway from this is probably that irregular nuclear shapes, sizes or staining shades can indicate metastases.**\n\n### How is the data best transformed for modeling?\n\nWe know that the label of the image is influenced only by the center region (32 x 32px) so it would make sense to crop our data to that region only. However, some useful information about the surroundings could be lost if we crop too close.  This hypothesis could be confirmed by training models with varying crop sizes. My initial results with 32 x 32px size showed worse performance than with 48 x 48px but I haven't done a search for optimal size.\n\n### How may we increase the data quality?\n\nWe could inspect if the data contains bad data (too unfocused or corrupted) and remove those to increase the overall quality. *TODO*"},{"metadata":{"_uuid":"74f542bfe934472aa9ff06a4031b9a79738f5db5"},"cell_type":"markdown","source":"### Preprocessing and augmentation\nThere are couple of ways we can use to avoid overfitting; more data, augmentation, regularization and less complex model architectures. Here we will define what image augmentations to use and add them directly to our image loader function. Note that if we apply augmentation here, augmentations will also be applied when we are predicting (inference). This is called test time augmentation (TTA) and it can improve our results if we run inference multiple times for each image and average out the predictions. \n\n**The augmentations we can use for this type of data:**\n- random rotation\n- random crop\n- random flip (horizontal and vertical both)\n- random lighting\n- random zoom (not implemented here)\n- Gaussian blur (not implemented here)\n\nWe will use OpenCV with image operations because in my experience, OpenCV is a lot faster than *PIL* or *scikit-image*."},{"metadata":{"trusted":true,"_uuid":"d08cfe0546459f2d9b5c8fd4c5d4994527677173"},"cell_type":"code","source":"import random\nORIGINAL_SIZE = 96      # original size of the images - do not change\n\n# AUGMENTATION VARIABLES\nCROP_SIZE = 68          # final size after crop\nRANDOM_ROTATION = 180   # range (0-180), 180 allows all rotation variations, 0=no change\nRANDOM_SHIFT = 4        # center crop shift in x and y axes, 0=no change\nRANDOM_BRIGHTNESS = 5   # range (0-100), 0=no change\nRANDOM_CONTRAST = 5     # range (0-100), 0=no change\n\ndef readCroppedImage(path):\n    # OpenCV reads the image in bgr format by default\n    bgr_img = cv2.imread(path)\n    # We flip it to rgb for visualization purposes\n    b,g,r = cv2.split(bgr_img)\n    rgb_img = cv2.merge([r,g,b])\n    \n    #random rotation\n    rotation = random.randint(-RANDOM_ROTATION,RANDOM_ROTATION)  \n    M = cv2.getRotationMatrix2D((48,48),rotation,1)\n    rgb_img = cv2.warpAffine(rgb_img,M,(96,96))\n    \n    #random x,y-shift\n    x = random.randint(-RANDOM_SHIFT, RANDOM_SHIFT)\n    y = random.randint(-RANDOM_SHIFT, RANDOM_SHIFT)\n    \n    # crop to center and normalize to 0-1 range\n    start_crop = (ORIGINAL_SIZE - CROP_SIZE) // 2\n    end_crop = start_crop + CROP_SIZE\n    rgb_img = rgb_img[(start_crop + x):(end_crop + x), (start_crop + y):(end_crop + y)] / 255\n    \n    # Random flip\n    flip_hor = bool(random.getrandbits(1))\n    flip_ver = bool(random.getrandbits(1))\n    if(flip_hor):\n        rgb_img = rgb_img[:, ::-1]\n    if(flip_ver):\n        rgb_img = rgb_img[::-1, :]\n        \n    # Random brightness\n    br = random.randint(-RANDOM_BRIGHTNESS, RANDOM_BRIGHTNESS) / 100.\n    rgb_img = rgb_img + br\n    \n    # Random contrast\n    cr = 1.0 + random.randint(-RANDOM_CONTRAST, RANDOM_CONTRAST) / 100.\n    rgb_img = rgb_img * cr\n    \n    # clip values to 0-1 range\n    rgb_img = np.clip(rgb_img, 0, 1.0)\n    \n    return rgb_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e36792e155852f9bfcc9f564da0328ff872bf0d","_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(2,5, figsize=(20,8))\nfig.suptitle('Cropped histopathologic scans of lymph node sections',fontsize=20)\n# Negatives\nfor i, idx in enumerate(shuffled_data[shuffled_data['label'] == 0]['id'][:5]):\n    path = os.path.join(train_path, idx)\n    ax[0,i].imshow(readCroppedImage(path + '.tif'))\nax[0,0].set_ylabel('Negative samples', size='large')\n# Positives\nfor i, idx in enumerate(shuffled_data[shuffled_data['label'] == 1]['id'][:5]):\n    path = os.path.join(train_path, idx)\n    ax[1,i].imshow(readCroppedImage(path + '.tif'))\nax[1,0].set_ylabel('Tumor tissue samples', size='large')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9568cbee8f077da727aa84fd13f459e5d03c5a44"},"cell_type":"markdown","source":"**To see the effects of our augmentation, we can plot one image multiple times.**"},{"metadata":{"trusted":true,"_uuid":"dd77955fd1742b4d72562113beca5712011122db","_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,5, figsize=(20,4))\nfig.suptitle('Random augmentations to the same image',fontsize=20)\n# Negatives\nfor i, idx in enumerate(shuffled_data[shuffled_data['label'] == 0]['id'][:1]):\n    for j in range(5):\n        path = os.path.join(train_path, idx)\n        ax[j].imshow(readCroppedImage(path + '.tif'))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2df0d87a864207ce1260acad6926f6dcf8d5fd58"},"cell_type":"markdown","source":"# Baseline model\nIn ML production pipeline, it is a good idea to start with a relatively simple model, sort of a minimum viable product (MVP) or a baseline. With MVP, we can very quickly see if there are some unexpected problems like bad data quality that will make any further investments into the model tuning not worth it.\n\n### What kind of model architecture suits the problem best?\n\nHere we will be using a pretrained convnet model and transfer learning to adjust the weights to our data. Going for a deeper model architecture will start overfitting faster.\n\nFor differenet pretrained model architectures, check [Fast.ai conv_learner.py](https://github.com/fastai/fastai/blob/master/old/fastai/conv_learner.py). Note that some of the architectures require additional pretrained weight files downloaded. In that case, uncomment the **Download missing weight files** -cell."},{"metadata":{"trusted":true,"_uuid":"31fef9db25fe198c5698d90db93ac91ddb60886c","_kg_hide-output":true},"cell_type":"code","source":"from fastai.conv_learner import *\nfrom fastai.dataset import *\nfrom fastai.plots import ImageModelResults\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_curve, auc","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"c599f75232ba439bc3e645356db2010c74f00f2a"},"cell_type":"code","source":"# Download missing weight files for inceptionresnet pretrained model\n#!wget \"http://files.fast.ai/models/weights.tgz\"\n# unzip to the correct location\n#!mkdir /opt/conda/lib/python3.6/site-packages/fastai/weights/\n#!tar -xvzf weights.tgz -C /opt/conda/lib/python3.6/site-packages/fastai/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9d217ce198f2f23fc722f8606a722fb9b3e71d38"},"cell_type":"code","source":"nw = 3   #number of workers for data loader\narch = resnet50 #specify target architecture\nBATCH_SIZE = 128\nsz = CROP_SIZE\nMODEL_PATH = 'resnet50_1'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0dbd4bddb400e4f0b5aa7d94d8dda552f2e883a5"},"cell_type":"markdown","source":"### Prepare the data and split train\nSplit train data to 90% training and 10% validation parts. "},{"metadata":{"trusted":true,"_uuid":"c1b95582bc4bf32c8a3686325f165ad72b5a48ed"},"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/train_labels.csv').set_index('id')\ntrain_names = train_df.index.values\ntrain_labels = np.asarray(data['label'].values)\ntest_names = [f.replace(\".tif\",\"\") for f in os.listdir(test_path)]\ntr_n, val_n = train_test_split(train_names, test_size=0.1, random_state=123)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5d25df6f20f350b1e796fa4ed34e134a4f11a826","_kg_hide-input":true},"cell_type":"code","source":"class hcdDataset(FilesDataset):\n    def __init__(self, fnames, path, transform):\n        self.train_df = train_df\n        super().__init__(fnames, transform, path)\n\n    def get_x(self, i):\n        img = readCroppedImage(os.path.join(self.path, str(self.fnames[i]) + '.tif'))\n        return img\n\n    def get_y(self, i):\n        if (self.path == test_path): return 0\n        return self.train_df.loc[self.fnames[i]]['label']\n\n    def get_c(self):\n        return 2 #number of classes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e82294a7507e48b0481cfb67e3e7c93e9dfc1821","_kg_hide-input":true},"cell_type":"code","source":"# Fast.ai data loader\ndef get_data(sz, bs):\n    aug_tfms = [] # we are using our own augmentations with our image loader function so we leave this array empty\n    \n    # mean and std in of each channel in the train set\n    # these stats have been calculated for 48x48 crop size but it has hopefully captured the general variance\n    stats = A([0.70185, 0.54483, 0.69568], [0.22262, 0.26757, 0.1995 ])\n    \n    # Here we define the transforms that are performed for images when loaded. Only statistical regularization.\n    tfms = tfms_from_stats(stats, sz, crop_type=CropType.NO, tfm_y=TfmType.NO,\n                           aug_tfms=aug_tfms)\n    \n    ds = ImageData.get_ds(hcdDataset, (tr_n[:-(len(tr_n) % bs)], train_path),\n                          (val_n, train_path), tfms, test=(test_names, test_path))\n    md = ImageData(\"./\", ds, bs, num_workers=nw, classes=None)\n    return md","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0eda8faa0b5dfa5f38808dded2738a895dfec11f"},"cell_type":"markdown","source":"### Compute image statistics\nDoing these once is enough. **Do not use color/brightness/contrast augmentation here!**\nThis statistics function is copied and altered from [iafoss's kernel](https://www.kaggle.com/iafoss/pretrained-resnet34-with-rgby-0-460-public-lb)\n\nThis will give channel averages of [0.70185, 0.54483, 0.69568],\nand std's of [0.22262, 0.26757, 0.1995 ]."},{"metadata":{"trusted":true,"_uuid":"35a9788f26ff06c92fb6592e3c8ee67db2f7415a","_kg_hide-input":true},"cell_type":"code","source":"#md = get_data(sz, BATCH_SIZE)\n#x_tot = np.zeros(3)\n#x2_tot = np.zeros(3)\n#for x,y in iter(md.trn_dl):\n#    tmp =  md.trn_ds.denorm(x).reshape(BATCH_SIZE,-1)\n#    x = md.trn_ds.denorm(x).reshape(-1,3)\n#    x_tot += x.mean(axis=0)\n#    x2_tot += (x**2).mean(axis=0)\n#\n#channel_avr = x_tot/len(md.trn_dl)\n#channel_std = np.sqrt(x2_tot/len(md.trn_dl) - channel_avr**2)\n#channel_avr,channel_std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b03f5cf0fd15fe757e5eca23720c5cf39c8cf431"},"cell_type":"markdown","source":"### Training\nWe will use [Adam](https://arxiv.org/abs/1412.6980) optimizer"},{"metadata":{"trusted":true,"_uuid":"ebfd9acf7bec2c429dd8499514b9adcc5a235e44","_kg_hide-output":true},"cell_type":"code","source":"md = get_data(sz, BATCH_SIZE)\nlearn = ConvLearner.pretrained(arch, md) # Add dropout with ps=0.5 (dropout 50%) \nlearn.opt_fn = optim.Adam","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac6b9d503b9144b04c7edd8fe033721ac7269a54"},"cell_type":"markdown","source":"First, we find the optimal learning rate. The optimal lr is just before the base of the loss and before the start of divergence. It is important that the loss is still descending where we select the learning rate."},{"metadata":{"trusted":true,"_uuid":"49c4bb9a9d899c9cfbfb483e486ae3338a8373bb"},"cell_type":"code","source":"learn.lr_find()\nlearn.sched.plot()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0adfcfd595dc62d6cf4b8f77f4ba235e82e18980"},"cell_type":"markdown","source":"We can select the learning rate around 1e-3 where it is close to the bottom but still descending.\n\nNext, we train only the heads while keeping the rest of the model frozen. Otherwise the random initialization of the head weights could harm the relatively well performing pretrained weights of the model. After the heads have adjusted and the model somewhat works, we can continue to train all the weights."},{"metadata":{"trusted":true,"_uuid":"2f456a26a2e422ded89a6fdee489bec0c061d77f"},"cell_type":"code","source":"lr = 1e-3\nlearn.fit(lr,2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"47f7e7e9eb03a3b1f17f8474dcf7e214162f9d17"},"cell_type":"markdown","source":"Next, we unfreeze the model and train it with differential learning rates. This is because the lower levels that activate on low-level shapes and patterns are probably well suited for this image detection task as is and don't need much adjusting. The higher levels that activate on more detailed features however, will need more adjusting to this training set. \n\nSo we want to train the lower levels with very small adjustments (**lr/100**) and the middle (**lr/10**) and higher (**lr**) levels with larger adjustments."},{"metadata":{"trusted":true,"_uuid":"7f3a660d0be1146e1783aad4a7d6b0bf99bded33"},"cell_type":"code","source":"learn.unfreeze()\n# set different learning rate for low-mid-top layers\nlrs=np.array([lr/100,lr/10,lr])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39674cab2127108d2e2ceacbd829177747dba19e"},"cell_type":"markdown","source":"To avoid getting stuck to local minima, we train by lowering the learning rate as we train (cosine annealing) but periodically hop back up (restart). This is called a stochastic gradient descent with restarts (SGDR) and it has proven to be very effective. The idea behind it is when we train and get closer to points of minima, it makes sense to slow down the learning rate and take smaller adjustment steps to make sure we don't jump over the minima. The minima however, is never the global one where we want but a local minima. The learning rate restarts help us to gain momentarily the momentum to break out those local minimas and hopefully get closer to the global minima. When we do enough of these restarts, we can explore more of the loss landscape (pictured below) and typically get to a good stable minima.\n\n![LR cycles](https://cdn-images-1.medium.com/max/880/1*9Fca3kpx3pVW8SaYz2pjpw.png)\n<a href=\"https://openreview.net/pdf?id=BJYwwY9ll\"><p style=\"text-align: center;\"> Example of cyclic learning rates from the paper: Snapshot Ensembles </p></a>\n\nThe [authors of SGDR](https://arxiv.org/abs/1608.03983) also propose to lengthen the restart cycle as the training progresses."},{"metadata":{"trusted":true,"_uuid":"6e2326fe8a16469f9b120b6e7d19989f8f3affd9"},"cell_type":"code","source":"# breakdown of this train phase: 1.cycle len=1 epoch, 2.cycle len=2 epochs, 3.cycle len=4 epochs\nlearn.fit(lrs/4, 3, cycle_len=1, cycle_mult=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8857a7f3d1fc06119d416c32fd67bf684536d8f8"},"cell_type":"code","source":"# plot the learning rate to see the SGDR\nlearn.sched.plot_lr()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4b46c33f3338636560f1ffaca4fd1845e8a4b49a"},"cell_type":"markdown","source":"Finally, we finish by finetuning our model with small learning rates and one long cosine annealing cycle."},{"metadata":{"trusted":true,"_uuid":"68d5cd7dac6ee5fa1a41ef6a1029c18c64b709dd"},"cell_type":"code","source":"learn.fit(lrs/10,1,cycle_len=5,use_clr=(5,20))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2754b6c934e558b382e72e363438a6f819acc4fd"},"cell_type":"code","source":"#save model for later\nlearn.save(MODEL_PATH)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c4b0274418a67c87f5d9dd538a392d3d520cc70f"},"cell_type":"markdown","source":"-------------------------\n# Validation and analysis\nNow the training is done.\n\n### How good does the model perform technically?\n\nWe can only get metrics from our validation set, and the final test metrics will be most likely a bit different.\nLoss? Accuracy?"},{"metadata":{"trusted":true,"_uuid":"4891afd988159436ec176dbcca6ddcf7cfd962fa"},"cell_type":"code","source":"# plot the loss\nlearn.sched.plot_loss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0d83d060e345c0599f840bd12fbb4a56f6162e3a"},"cell_type":"code","source":"# Get metrics: accuracies\nlearn.sched.rec_metrics","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8b994cb6e711b0e0702ecdcbc86168a5d93258e"},"cell_type":"markdown","source":"### How good is the model in terms of project requirements?\nIt is a good idea to look at examples of images from:\n\n- The most correctly labeled (highest probability)\n- The most incorrectly labeled (highest probability but wrong label)\n- The most uncertain labels (probability closest to 0.5).\n\nThe visualization is a good way of understanding where our model performs well and what are the images it struggles with. It might also reveal something about the dataset such as bad quality data.\n"},{"metadata":{"trusted":true,"_uuid":"0399dfe70ae97a156c84034c8093413af61a6334"},"cell_type":"code","source":"# Predict the validation set\nlog_preds = learn.predict()\nlog_preds.shape\n# probs from log preds\nprobs = np.exp(log_preds[:,1])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"adcd722a33466b0a40a193775026d4f86d1b1129"},"cell_type":"code","source":"# this is not the fastest way of iterating through our training set - TODO optimize code\nval_labels = np.zeros((len(val_n)))\nval_indexes = np.zeros((len(val_n)))\nfrom tqdm import tqdm\nfor i in tqdm(range(len(val_n))):\n    val_labels[i] = train_df.loc[val_n[i]]['label']\n    val_indexes[i] = np.where(train_df.index == val_n[i])[0]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5c979b33048aacf8965dcc0d61d3c94584085bab"},"cell_type":"code","source":"# we create the dataset again for validation purposes, the previous dataset is not compatible with the following visualization methods\ntrain_names_tif = []\nfor name in train_names:\n    train_names_tif.append(name + '.tif')\n\nval_data = ImageClassifierData.from_names_and_array(\n    path=train_path, \n    fnames=train_names_tif,\n    val_idxs=val_indexes.astype(int),\n    y=train_labels, \n    classes=['negative', 'metastases'],  \n    tfms=tfms_from_model(arch, sz)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"49e4fe1a40de489b25b637dc7e8f3da8ca7675ad"},"cell_type":"code","source":"# a class that will help us plot our results\nresults = ImageModelResults(val_data.val_ds, log_preds)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c270a3545fd39bc6f7e02bfda971e59d97b4bb46"},"cell_type":"markdown","source":"### The most correct **Negatives**\nThese negative samples our model got right with very high prediction probability."},{"metadata":{"_kg_hide-input":false,"trusted":true,"_uuid":"34e57f42859587eb528c0c3d07c6f2ee88f58b52"},"cell_type":"code","source":"results.plot_most_correct(0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ce6e00fe93eedb1b5ea375a1d60fe2bfe28bd61c"},"cell_type":"markdown","source":"### The most correct **Metastases**\nThese metastase samples our model got right with very high prediction probability."},{"metadata":{"_kg_hide-input":false,"trusted":true,"_uuid":"26965128db3010f2e43dd0875fed848601411265"},"cell_type":"code","source":"results.plot_most_correct(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7aa5155837fd8b391afaa61bc5d655e890fa30a3"},"cell_type":"markdown","source":"### The most incorrect **Negatives**\nOur model predicted incorrectly that these were metastases with high probability."},{"metadata":{"trusted":true,"_uuid":"68bb4d8082558af8c0812718e07e75a3d65b5b3b"},"cell_type":"code","source":"results.plot_most_incorrect(0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"06281e16d8e738e3f925e1d4a41dd8af9073b1d4"},"cell_type":"markdown","source":"### The most incorrect **Metastases**\nOur model predicted incorrectly that these were negative samples with high probability."},{"metadata":{"trusted":true,"_uuid":"0aca68db15c36c5485f9ce9344d80f58ac6fcfcd"},"cell_type":"code","source":"results.plot_most_incorrect(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d64055423aa0bc3528fbb73a756290e6bc5eb8e7"},"cell_type":"markdown","source":"### The most uncertain **Negatives**\nThese were the most confusing negative samples to our model. It could not decide to what class these belonged to."},{"metadata":{"trusted":true,"_uuid":"abe8f0776cf239f317fd8fc23756ea64baab9101"},"cell_type":"code","source":"results.plot_most_uncertain(0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"11c88529f9944fc0b01b27c69469a8142a85fd32"},"cell_type":"markdown","source":"### The most uncertain **Metastases**\nThese were the most confusing metastase samples to our model. It could not decide to what class these belonged to."},{"metadata":{"trusted":true,"_uuid":"c2e688c3fa87b965eea8fa0cd43b6e9f98ae72e7"},"cell_type":"code","source":"results.plot_most_uncertain(1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88f7f40774a37b16202929e6e47c2d957411096c"},"cell_type":"markdown","source":"### Confusion matrix\nConfusion matrix can help us understand the ratio of false negatives and positives. It is a simple table that shows the counts in a way of **true label vs. predicted label**."},{"metadata":{"trusted":true,"_uuid":"c51268f2b6ae9375554cc9e52e22246db0a4bf36"},"cell_type":"code","source":"from sklearn.metrics import *\nfrom sklearn.metrics import confusion_matrix\nfrom fastai.plots import *\npreds_th = np.where(probs > 0.5, 1, 0)\ncm = confusion_matrix(val_labels, preds_th)\nplot_confusion_matrix(cm, ['negative','metastases'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1afca3014ca68714928c96c578b6bd1e4f820fc8"},"cell_type":"markdown","source":"### ROC curve and AUC\nRemember, AUC is the metric that is used for evaluating submissions. We can calculate it here for ou validation set but it will most likely differ from the final score."},{"metadata":{"trusted":true,"_uuid":"02b156c4b335575ac8b9e64e7c9ea8d2a1dead34"},"cell_type":"code","source":"# Compute ROC curve\nfpr, tpr, thresholds = metrics.roc_curve(val_labels, probs, pos_label=1)\n\n# Compute ROC area\nroc_auc = auc(fpr, tpr)\nprint('ROC area is {0}'.format(roc_auc))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c6f6818925d9356185761f12f12594f3decd8ded"},"cell_type":"code","source":"plt.figure()\nplt.plot(fpr, tpr, color='darkorange', label='ROC curve (area = %0.2f)' % roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', linestyle='--')\nplt.xlim([-0.01, 1.0])\nplt.ylim([0.0, 1.01])\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('Receiver operating characteristic')\nplt.legend(loc=\"lower right\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"54efc207cf58612e6c27ba69cf7fbcd987a28ce3"},"cell_type":"markdown","source":"----------------\n\n# Submit predictions\n### TTA\nTo evaluate the model, we run inference on all test images. As we have test time augmentation, our results will probably improve if we do predictions multiple times per image and average out the results. "},{"metadata":{"trusted":true,"_uuid":"5b9d4a6eee767bbed3f0ca889ea1ce4c4fffc607"},"cell_type":"code","source":"# Use a fair number of iterations to cover different combinations of flips and rotations.\n# The predictions are then averaged.\npreds_t,y_t = learn.TTA(n_aug=12, is_test=True)\npreds_t = np.stack(preds_t, axis=-1)\n# Fast.ai returns the log of the prediction. To get probabilities from log, we do exp()\npreds_t = np.exp(preds_t)\npreds_t = preds_t.mean(axis=-1)[:,1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eaa1a9651dff68793b4c66431d549e7eac2659a5"},"cell_type":"markdown","source":"### Submit the model for evaluation"},{"metadata":{"trusted":true,"_uuid":"453af28df5ec8db3bf01b51febe332af3622d37c"},"cell_type":"code","source":"SAMPLE_SUB = '/kaggle/input/sample_submission.csv'\nsample_df = pd.read_csv(SAMPLE_SUB)\nsample_list = list(sample_df.id)\npred_list = [p for p in preds_t]\npred_dic = dict((key, value) for (key, value) in zip(learn.data.test_ds.fnames,pred_list))\npred_list_cor = [pred_dic[id] for id in sample_list]\ndf = pd.DataFrame({'id':sample_list,'label':pred_list_cor})\ndf.to_csv('{0}_submission.csv'.format(MODEL_PATH), header=True, index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3769b99bcc4e7515ed7bd441db8c934354e88883"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}