OCT Reader has the goal to provide a reliable tool to analyse Optical coherence tomography (OCT) retinal images to determine whether the retinal structure is normal or displays signs of one of the following abnormal conditions: CNV, DME or Drusen. The model was developed using jupiter notebooks to complete different tasks, namely, i) fetch and clean data, 2) provide visual aid to understand the qualities of the dataset use in the project and 3) create a machine learning model (More especifically a deep learning Convolutional Neural Network) to make predictions on OCT retinal images. The project user interface consist of a streamlit dashboard that contains 5 pages to allow the user to understand the aim of the project, visualise the characteristics of the data use to train, validate and test the model, see the performance of the model, and make predictions in a fast and clear fashion.
You can visit the deployed site here
Hypothesis and How To Validate?
The Rationale to Map the Business Requirements to the Data Visualizations and ML Tasks
Main Data Analysis and Machine Learning Libraries
The database containes a total of 84495 Optical coherence tomography (OCT) retinal images organised in three folders: train, val and test. In each folder, images are split in four different sets pertaining to three different abnormal conditions, namely, CNV (Choroidal Neovascularization), DME (Diabetic macular edema), DRUNSEN, and healthy maculas (NORMAL). The data can be found here.
The initial split of data among train, val and test folders is different than the default 0.7, 0.2 and 0.1, respectively. Besides data is highly unbalanced and requires balancing.
The project aims to create a CNN machine learning model capable to use retinal OCT images to decide whether a patient's macula is healthy or displays one of the following abnormal conditions (labels): CNV, DME, DRUNSEN. Thus, key stakeholders for this projects are medical professionals and health care providers. Normally, OCT image analysis requires a team of highly trained technicians to provide an assessment; the model is aimed to provide stakeholders an alternative way to deliver quick feedback to patients in order to create a timely adequate plan of action to treat a potentially damaging condition to their sight.
-
The client requires to understand visual differences between the different macular conditions and healthy maculas. For this purpose, the client should be provided with visual aids such as image average, variability per label, as well as pair differences of average images.
-
The client requires an effective tool to determine the medical condition of a macula based on a OCT scan. The client needs to distinguish amongst CNV, DME, DRUNSEN, and healthy maculas. A high model occurracy is neccesary due to potential on patient's health due to possible false negatives.
-
Specialists dedicated to OCT analysis are capable to distinguish certain anomalities in non-healthy macula images when compare to healthy ones. Based on this, it is plaussible to assume that subtle difference that are not immediately perceived at first sight in OCT images might still allow for detection of macular anomalities by machine learning models.
-
A deep learning CNN multiclass classification model trained with the appropriate datasets might be capable to identify the medical condition of a macula from an OCT image.
-
A first aspect to consider to assess the model is the training history via accurracy and loss curves and later evaluation from data previously unseen by the model. These factor are paramount to determine if model has been under/over fitted or properly trained.
-
Additionally, the confussion matrix and classification reports will provide solid metrics (precision, recall and f1_score) to understand the model performance and validate our hypothesis.
- Business Requirement 1: Data Visualisation
-
As a client, I can viasualise arithmetic average and variability of images per label, pair differences between label's image average and sample images for each label in order to understand the characteristics of the dataset and similarities/differences among images.
Tasks:
-
Create a function to load a finite number of OCT image files storing them in arrays.
-
Create a function to use image arrays to calculate artihmetic average and standard deviations per label.
-
Create a function to calculate pair differences between image averages.
-
Create a page in a streamlit dashboard to visualise OCT image characteristics.
-
In the page, create a checkbox to display arithmetic average and variability images for each label and add corresponding backend functionality developed in step 2.
-
Create a second checkbox to display pair differences and add corresponding functionality developed in step 3.
-
-
As a client, I can see an image montage for a particular label so I can appreciate especific characteristics pertaining to each label.
Tasks:
-
Create a function do display read and display sample images in the form of a montage for a selected image.
-
Add a third checkbox to display a select box with the different labels and button to display the montage for the selected label.
-
Add backend functionality to display montage.
-
- Business Requirement 2: Classification
-
As a client, I can upload retinal OCT scan images so that I can use the ML model to predict the patients' macula condition, i.e., determine if patients' macula are healthy or present signs of an abnormal condition, namely, CNV, DME or Drusen.
Tasks:
-
Load the project dataset from kaggle using the API.
-
Prepare dataset from model training: cleaning, splitting (train, validation, test datasets) and balanced if needed.
-
Create and train deep learning multiclass classification ML model using the TensorFlow framework.
-
Evaluate the model using the test dataset.
-
Create a page in a streamlit dashboard to permit the user to upload and analyse and OCT image.
-
Add backend functionality to feed the image to the model and create a prediction.
-
Create a function to display prediction probabilities in a graphical representation (boxplot) and label prediction.
-
-
As a client I can download a dated csv file containing a report of predictions so that I can track all diagnosis performed using the model.
-
Develop function to create a pandas dataframe to store image name and corresponding prediction.
-
Create a back end function to transform the dataframe into a dated csv file.
-
Add a link to streamlit page to download the data in a csv file and connect to back end functionality.
-
Project presentation: The project must be presented to the client in an interactive dashboard with multiple pages to satisfy all business requirements and give information abou the project development and aims.
Model Accuracy: Providing information about the model training history and evaluation with data unseen by the model is paramount to understand the performance.
Precision, recall and f1_score: On the one hand, high precision, i.e., ensuring low number of false positives indicating a healthy macula is paramount for patient's health. High precision on determining a healthy macula is required. On the other hand, mislabeling abnormal conditions may lead to delays on patients treatment, a high f1_score is asked for any other labels.
-
The client is interested to create ML model to analyse OCT images and predict one of the following macular conditions: Normal, CNV, DME or DRUNSEN. Thus, the model will be a multiclass classification model.
-
Heuristics: Normally, macular condition is detected by highly trained medical professionals carefully inspecting the OCT images. On a patient consultation, the image is obtained in a non-invasive procedure using a Optical Coherence Tomography (OCT) equipment by a trainned technician. The current optical procedure in combination with high demand for analysis is known to produce considerable errors. This model should serve to reduce misinterpretation of the images and more reliable advice to patients.
-
The multiclass classification model will be created using a Convolutional Neuronal Network within the TensorFlow framework. The model training history and evaluation must show NO signs of over or underfitting.
-
The ideal outcome is to provide the means to interpret OCT macular images in the spot with higher reliability than conventional human visual inspection procedures. The OCT image will be obtained by a trained proffessionals and supplied to the model via a dashboard.
-
The model is expected to have high occuracy value of at least 0.95.
-
The multiclass classification model will determine from an OCT image if a macula is healthy (Normal) or suffers from one of three conditions: CNV, DME or DRUNSEN. Technicians will obtain the OCT images in a regular procedure and they will not require any additional pre-processing to be fed to the model. The automated analysis (prediction) can be performed immediately after obtaining the image.
-
The training data is accessible in Kaggle_dataset_directory with non-commercial license CC BY-NC-SA 4.0 DEED. In agreement to kaggle the dataset is a product of a multi-institutional effort and involved cohorts of adult patients wich OCT retina scans were taken between July 1, 2013 and March 1, 2017 in one of the following institutions: the Shiley Eye Institute of the University of California San Diego, the California Retinal Research Foundation, Medical Center Ophthalmology Associates, the Shanghai First People’s Hospital, and Beijing Tongren Eye Center. The data is properly labeled allowing the proper supervised training of the model.
-
Potential erroneous results from the model giving false positive outcomes identifying a macula as healthy (label=NORMAL) could be particularly detrimental for the patient's health since it may lead to no further investigation on the patients eye condition. Thus, the precision score for this particular label is set to 0.98.
-
Missidentification of an abnormal condition can be corrected on further investigation by more qualified health professionals after the model a problem in the OCT scan. However, to ensure that most patients can receive the appropriate treatment in a rapid fashion, the number of false negatives and false positives for labels identifying abnormal conditions must be minimize. Thus f1_score for labels other than NORMAL will be required to be 0.9 or higher.
The ML multiclass classification model was conceived using the Keras open source library. Due to the multiclass nature of the problem, the output layer required the use of a softmax activation function, 4 units. The model was compiled using the categorical_crossentropy for loss and the Adam optimizer. Three different versions were explored in order to get the best model, but all models have the same structure.
The average image size showed to be too large for model training purposes. An image size of 170 x 180 px was chosen helping to speed model training while preserving the important characteristics.
Underfitting: To prevent underfitting, data augmentation techniques will be used.
Overfitting: An early stop, limiting the number of epochs without improvement in loss on the validation with a patience of 3 is set to prevent overfitting.
This structure was chosen as a starting point since showed good performance in binary classification examples (e.g. code institute malaria detector)
-
Convolutional Layers: The model contains three pairs of convolution layer with kernel size (3 x 3) + max pooling layer (2 x 2). First convolution layer uses 32 units, while the rest uses 64 units. All convolution layers use a relu activation function.
-
Flattening: Before feeding the next (dense layer) the data was converted into one-dimensional array using a flatten layer.
-
First Dense Layer: The first dense layer uses a relu activation. Units vary with version.
-
Drop: A 0.5 drop is use to avoid over training.
-
Output Layer: The output is also a fully connected dense layer containing 4 units and softmax activation.
In this version, the model was trained using an early stop with patience of 1. The model was fitted using a maximum number of 20 epochs. The model show no underfitting or overfitting in agreement, however the occuracy levels were low and didn't meet the desired criteria as shown in the training history and the confussion matrix bellow.
Training accuracy history curve:
Training loss history curve:
Confussion matrix for test dataset:
The model accuracy was well below the 0.95 value required. The loss and ocurracy curves show that increasing number of epochs could still lead to higher accuracy.
In this version, the model was trained using an early stop with patience of 3 allowing more epochs to be performed during training. During the training a total of 20 epochs were conplited. The model show no underfitting or overfitting in agreement, however this model also failed to comply with the occuracy levels as shwon in the training history and the confussion matrix bellow.
Training accuracy history curve:
Training loss history curve:
Confussion matrix for test dataset:
The model accuracy was still well below the 0.95 value required by the client.
This model was optimized using the hyperband tunner class in KerasTuner. The hyperband tunner provides different classes designed to optimized hyperparameters of a given model estructure via the hp parameter. The hp parameter is passed to a function containing the neural network and permits to establish a range of values for the choice of tunnable hyperparameters. The Keras tunner hyperband class was used for tunning this model. Hyperband algorithm chooses values from the established hyperparameters ranges and performs a limited amount of epochs to test the model performance. Two hyperparameters were used to train the model, namely, number of units in first dense layer and training date of the adam optimizer. The objective was set to 'val_accuracy' and max_epochs=30. The hyperparameter optimization suggested optimal hyperparameters to be: number of units of 256 and learning rate of 0.0005. The hyperparameter optimization led to a considerable improvement of the model ,allowing it to reach the desired accurracy and precision as shown in the images below.
Training loss history curve:
Confussion matrix for test dataset:
A clear improvement was made thanks to the hyperband hyperparameter tuning technique, taking accurracy level slightly below the 0.9. However this value is still well below the expected 0.95.
At the start of the project it was hypothesised that a properly trained CNN model could be capable to use OCT digital images of a patients' retina to determine if the macula is healthy or presents an abnormal condition.
Image Visualisation: A dataset containing a sufficient number of images was used for training purposes in this project. A first step consisted in performaing a simple visual analysis of images, namely, image average and variability, as well as arithmetic difference between pair of dataset labels. The visual analysis did not show significant patterns to determine macular condition from simple visual inspection by non-trained professionals.
Multiclass ML Model: A CNN deep learning model using the Keras tunner API to optimize hyperparameters. The model performance was measured using different metrics: i) Accuracy during training and evaluation with test dataset, and ii) precision, recall and f1_score.
Despite showing potential, the model did NOT meet the high stadards for occuracy and precision needed for a medical application, as explained bellow:
i) Accuracy: The training hystory curves and test evaluation showed no evidence of overfitting or underfitting. Evaluation of model demonstrated an accuracy value of 0.88 lower than the required 0.95.
ii) Precision, f1_score: Precision values for label NORMAL was bellow the required value of 0.98 and f1_scores for other labels were also bellow the 0.9 minimum value required.
In order to improve the model performance and attempt to meet the metrics required in ML Business Case, the following steps might be followed in the future:
i) Implement new steps in the hyperparameter optimisation function in order to include hyperparameters such as number of convolution layers and their filters.
ii) Search for an alternative dataset to increase quality and number of training images.
-
Introduction
-
The macula is a small area that sits around the central area of the retina at the back of the eye. The macula plays an important role being responsible to process frontal vision. Some factors such as age and chronic diseases can be detrimental to the structural integrity of the macula decreasing vision quality. This project concerns with three well known conditions:
-
Drunsen: Age related celular degeneration of the macula causing yellow deposits in the retina. In the more serious cases Drunsen can cause loss of peripherial vision. Prevalence of Drunsen in a world population is estimated between 0.3 to 2.5% 1.
-
CNV: It is detected by the abnormal growth of vessels from the choroidal vasculature to the neurosensory retina through the Bruch's membrane 2. CNV can lead to age related macular degeneration which is linked to 8.7 % of all legal blindness worldwide 3.
-
DME: Causes thickening of the retinal tissue due to fluid accumulation and it is associated with diabetic retinopathy. Just in the USA the WHO estimates 15 million undiagnosed cases of DME 4, with 25-30% risk of vision loss.
-
-
All mention conditions cuase structural changes on the macular tissue. Thus, their detection is based on imaging techniques, most common is OCT. Normally, a qualified medical practisioner will inspect the images carefully to diagnose a condition.
-
-
Dataset
- The database containes a total of 84495 Optical coherence tomography (OCT) retinal images organised in three folders: train, val and test. In each folder, images are split in four different sets pertaining to three different abnormal conditions, namely, CNV (Choroidal Neovascularization), DME (Diabetic macular edema), DRUNSEN, and healthy maculas (NORMAL). The training data is accessible in Kaggle dataset directory with non-commercial license CC BY-NC-SA 4.0 DEED. In agreement to kaggle the dataset is a product of a multi-institutional effort and involved cohorts of adult patients wich OCT retina scans were taken between July 1, 2013 and March 1, 2017 in one of the following institutions: the Shiley Eye Institute of the University of California San Diego, the California Retinal Research Foundation, Medical Center Ophthalmology Associates, the Shanghai First People’s Hospital, and Beijing Tongren Eye Center.
-
Business requirements
-
The client requires to understand visual differences between the different macular conditions and healthy maculas.
-
The client requires an effective tool to determine the medical condition of a macula based on a OCT scan. The client needs to distinguish amongst CNV, DME, DRUNSEN, and healthy maculas.
This page is designed to answer bussiness requirement 1
- Checkbox 1: Image average (aritmethic mean) and variablility (standard deviation)
- Checkbox 2: Average pair difference between images for different macular conditions.
- Checkbox 3: Image Montage
This page is designed to answer bussiness requirement 2 and must contain:
-
Link to download OCT images for live prediction.
-
Widget to upload image file for live prediction.
-
Table with model prediction and image name.
-
Button to download prediction results.
- Display project hypothesis separately together with validation results.
-
Plot of dataset distribution.
-
Plot of model training history.
-
Show model evaluation results.
-
Show model performance metrics: Confusion matrix and classification report.
This is the default page on openning the website.
General information about the project and aims, as well as a description of the conditions included in the model development.
The page also gives informaiton about the datasets used to train, validate and test the model. A link to the readme file is provided for further information. Finally, the project requirements are clearly displayed to inform the user the expectations of the project.
Summary: All qualities desired for this page were achived on the live project, conveying a clear message to the user about project deliveries and expectations.
The streamlit dashboard provides a sidebar with a radio buttons to access the different pages. On clicking on the image visualiser button a page with information of business requirement 1 and three checkboxes to show i) image average and variability, ii) differences between label pair images and iii) a montage of images for a particular label.
On checking the first box, images for average and variability are displayed for al the labels. Unchecking the box retrieves the images.
On checking the second box, images for average pairs of labels are display together with their arithmetic difference. Unchecking the box retrieves the images.
On checking the third box, the user is conducted to a select box, where they can choose a label to display example OCT images.
On clicking the display montage button, example images are shown with a small time delay as advertised in the page.
Summary: The Bussiness requirement 1 is met by this page.
When the radio button for OCT Image Analiser is clicked the user is prompted to the OCT macular inspector page. This page is designed to answer the Business requirement 2 which is clearly estated in the page. The page also provides a link to the original kaggle to download example images to test the model.
On clicking the Browse Files button the user is allowed to load multiple image files in jpeg format. On loading the image the analysis is automatically displayed showing the image and name, predicted label and prediction probabilities plots. At the end of the page an analysis report for all images loaded is display together with a link to download it in csv format.
Summary: The Bussiness requirement 2 is met by this page.
When the radio button for Project Hypothesis is clicked a page showing the Project hypothesis and Validation assessments points appears. Additionally the validation results are clearly shown.
Summary: All qualities desired for this page were achived on the live project.
When the radio button for Model Performance is clicked the correspondant page appears. The model performance starts by showing the image count for the train, validation and test datasets. The datasets are well balanced before using them to create the models.
This is followed by showing the model training history displaying plots for accuracy and loss for the train and validation datasets during the model fitting. The curves and evaluation results clearly showed the model was not over fitted. However, the model acurracy is around 0.87 well bellow the desire value of 0.95.
The model performance is clearly represented in the form of confussion matrice and classification report. The reports show a model precision, recall and f1_score for all labels.
Summary: The information displayed on this page is enough to satisfy the curiosity of users interested in the model performance.
No unfixed bugs.
The project was deployed in heroku.
-
Login to Keroku.
-
Create a new app by clicking the new app button as shown below.
- When prompted to the create new app function, write the name of the app and click create app.
-
Create the runtime.txt and write the Python version supported by Heroku-20 stack.
-
Install heroku CLI in the IDE workspace by using the node package manager: npm install -g heroku, and login by using the command: heroku login.
-
Connect your workspace with your heroku app using the command heroku git:remote -a "YOUR_APP_NAME"
-
Set heroku stack to Heroku-20 using the command: heroku stack:set heroku-20.
-
Deploy the application from bash using: git push heroku main.
If the slug size exceeds Heroku allowance:
- In the root directory of the proyect create .slugignore file and include names off files and folders not required for Heroku.
By forking the GitHub Repository we make a copy of the original repository on our GitHub account to view and/or make changes without affecting the original repository by using the following steps...
- Log in to GitHub and locate the GitHub Repository
- At the top of the Repository (not top of page) just above the "Settings" Button on the menu, locate the "Fork" Button.
- You should now have a copy of the original repository in your GitHub account.
- Log in to GitHub and locate the GitHub Repository
- Under the repository name, click "Clone or download".
- To clone the repository using HTTPS, under "Clone with HTTPS", copy the link.
- Open Git Bash
- Change the current working directory to the location where you want the cloned directory to be made.
- Type
git clone, and then paste the URL you copied in Step 3.
-
random: Shuffle lists to create randomized datasets.
-
os: Folders navigation, setting correct working directory.
-
joblib: Save and reload model, evaluation, label names dictionary and other files needed to run application as pkl files.
-
Matplotlib: Handling and showing graphical representations. Loading images with a desired shape to create averages and variability image representation.
-
Seaborn: Create graphical representation of dataset characteristics, e.g., bar plot of label availability. Setting graphs style to "whitegrid".
-
NumPy: Other libraries used in this project depend on numpy library. Some image processing such as statistic analysis required tranformation into numpy arrays for easy manipulation.
-
Pandas: Categorical and numerical data was stored in pandas dataframes for easy processing, manipulation and later graphical representation.
-
Scikit-learn: Produce model performance metrics (Confussion matrix and classification report).
-
TensorFlow: ML CNN model development and image processing.
-
Keras Tuner: ML model hyperparameter optimisation.
-
Pillow: Manipulation of image for prediction uploaded by user through the dashboard.
-
Streamlit: Creation user interface (dashboard).
-
plotly Interactive graphical representation of prediction probabilities in dashboard.
Am I Responsive App: Creation of cover image in readme file.
Github: Source code storage and version control.
Gitpod: Cloud IDE service used to create the app.
Heroku: Used as deployment platform of project as web app.
CI Python Linter: PEP8 validation of python code.
All functions in jupiter notebook, streamlit and sorce code were validates using the CI Python Linter
- The Code Institute Malaria Detector walkthrough project served as guidance during the development of this project.
-
Hyperparameters optimisation using keras tuner API was developed following the tensor flow webpage and Medium.
-
Datasets were obtained from kaggle. Retinal OCT Images (optical coherence tomography), PAUL MOONEY. License CC BY-NC-SA 4.0. You can visit the data webpage here.
- I would like to thank my mentor Mo Shami for his guidance during this project development.
-
Auw-Haedrich C, Staubach F, Witschel H. Optic disk drusen. Surv Ophthalmol. 2002;47:515–32.
-
Turbert, D. (2020, March 18). What are Choroidal Neovascular membranes? Reviewed by Ninel Z Gregori, MD. American Academy of Ophthalmology. https://www.aao.org/eye-health/diseases/choroidal-neovascular-membranes.
-
Congdon N, O’Colmain B, Klaver CCW, et al. Causes and prevalence of visual impairment among adults in the United States. Arch Ophthalmol. 2004;122:477–85.
-
Albert and Jakobiec’s Principles and Practice of Ophthalmology. Third edition, second volume. Canada: SAUNDERS ELSEVIER; 2008. pp. 1793–1996.