Swimming pool detection and classification using deep learning
This project was presented at the Esri UC 2018 Plenary session. This repository contains code for creating a swimming pool detector. However, with minor modifications it can be trained to detect objects of different types from aerial, drone or satellite imagery. The following blog articles outline our approach.
An end-to-end object detection workflow
The field of Deep Learning has recently witnessed groundbreaking research with state of the art results, but taking this research to the field and solving real-world problems is still a challenge. Integration of the latest research in AI with ArcGIS opens up a world of opportunities. The notebooks in this folder show an end-to-end workflow that starts from extracting image chips to training a deep learning object detection model, deploying it over a large geographical area and creating information products like maps, layers and web apps to share the results, as well as creating assignments for field workers for verification and field assessment of results.
Representative results of the detected swimming pools. The parcels outlined in red are being assessed incorrectly.
Update assessor data
Tax assessors at local government agencies must often rely on expensive and infrequent surveys, leading to assessment inaccuracies. Swimming pools are an important part of assessment records because they impact the value of the property. Finding pools that are not on the assessment roll (such as those recently constructed) will be valuable to the assessor, and will ultimately mean additional revenue for the community. Doing this through GIS and AI would certainly reduce the expensive human labor involved in updating the records through field visits of each property.
Visualizing residential parcels in ArcGIS Pro
Identify neglected pools
If a swimming pool is neglected, as when the property is foreclosed, it often turns a green color and becomes inviting to mosquitoes – standing water with no inputs or outputs. The sheer volume of properties affected in warmer climates, even as the market rebounded, has made the detection of these risky pools challenging for many organizations.
Public health and mosquito control agencies are tasked with protecting the public from vector-borne diseases, including viruses carried by mosquitoes like West Nile and Chikungunya. These agencies need a simple solution that helps them locate neglected pools from imagery and then use this intelligence to drive field activity and mitigation efforts.
Green or Clean Pools
The project uses PyTorch and fast.ai deep learning libraries for training and deploying deep learning models. An easy and efficient way to set up the environment with the required libraries and the necessary GPU support is to create an EC2 instance using Amazon Web Services. Any AWS GPU compute instance, such as
p2.xlarge will do. The fastai-part1v2-p2 (ami-c6ac1cbc) AMI comes set up with all the required enviroments and libraries. To setup your own AWS instance follow this guide
After setting the AMI up you need to run the following commands to update the environment.
sudo apt update sudo apt upgrade cd fastai conda env update conda update --all
If you want to set up the environment locally, you need to have a CUDA supported NVIDIA GPU and install the following libraries and tools:
Anaconda PyTorch v0.3+ Fastai (https://github.com/fastai/fastai) CUDA V9.0 CuDNN
After setting up the AMI/ local development machine, install ArcGIS API for Python and its dependencies using the following command in the terminal:
conda install -c esri arcgis conda install -c conda-forge shapely
In your project home make a symlink using the following command
ln -s PATH_TO_CLONED_FASTAI_GITHUB_REPO/fastai fastai
Label object locations
ArcGIS Pro can be used to can manually label sample object locations and create a shapefile containing the labeled locations.
It provides access to a host of aerial, satellite and drone imagery from Esri and its partners and includes an easy to use interface to label data as well as advanced GIS functionality, including tools for reviewing data to manage its quality. Additionally, it includes geoprocessing tools to create buffers and bounding boxes around labeled pool locations.
Extract training data
Once you have the labeled object locations, you have the following two options to export the necessary training chips and training data:
Option A: Use Python code to export training samples
ArcGIS API for Python has methods for exporting images from Imagery (eg NAIP imagery layers) as well as Tile layers (such as the Esri World Imagery layer). The
Export Training Samples notebook uses that approach. Run the
Export Training Samples.ipynb after setting up the path to a shapefile containing the sample object locations in the
PATH variable. The images will be extracted from the specific locations in the shapefile and will be stored in the images folder. Additionally, a tilemapping file containing a mapping of each image chip to the center coordinates of each object within that image is created.
Option B: Use ArcGIS Pro's 'Export Training Data for Deep Learning’ tool
ArcGIS Pro includes the ‘Export Training Data for Deep Learning’ tool that can be used to create labelled image chips that are needed to train a deep learning model. For object detection workflows, the output data format is in the "Pascal VOC (Visual Object Challenge)" format.
Pascal VOC to Tile Mapping notebook reads in the labeled Pascal VOC format files, and creates the necessary tilemapping file needed by the training step below.
Train object detector
Once the training data is obtailed, we can begin training the deep learning network using the
Train a Swimming Pool detector notebook. Set the
PATH variable to point to the folder containing the training data:
PATH = Path('path/to/folder/which/contains/tilemapping')
Make sure the images are also stored in a subdirectory named
Note that you may need to train for longer depending upon your data. The visualization of results are at the end of the notebook. This project is for detecting objects of one class (type). With minor changes it can be made to work on multiple classes of objects.
Train 'clean or green' pools classifier
The detected pools as further classified as clean or green (i.e. neglected pools) using the
Clean or Green Pools Training notebook.
To train this Resnet-34 based classification model, some detected pools are downloaded and then mannually labeled as clean or green. The training notebook assumes that the root directory for classification will have
valid directories to contain the training and validation images. It also assumes that each directory will have subdirectories for each class (in this case,
Deploy trained model
The deployment code for the project is in the
Deploy model to find swimming pools notebook. This notebook is used to run the object detector on a large area whose predictions can be accumulated and then be used to create information products such as feature layers for visualization and further analysis. A webmap is then created using this feature layer. The locations of the detected swimming pools are used to identify parcels that were not being accessed correctly using analysis tools from ArcGS Online. The Spatial DataFrame in ArcGIS API for Python is used for generating reports of such parcels.
Later the the predictions from object detector are used to further classify the pools are clean or green, which also can be converted to a feature layer for visualization on a web map.
The deployment notebook also demonstrates creation of Workforce assignments for mosquito-control field workers based on the results of the neglected pool detection analysis.