Tracking vessel activity on the global oceans from space with AI and cloud computing
A project to develop the next-generation automated system for tracking, classifying and reporting vessel activities worldwide. The objective is two-fold: improve vessel detection accuracy and workflow efficiency. The approach leverages freely-available satellite radar and optical imagery, and state-of-the-art AI algorithms on cloud infrastructure for global-scale monitoring of ships on the oceans. The development consists of a two-step process: First, implement an artificial neural network framework for ship detection using freely available SAR amplitude data that can be scaled globally. Second, with a fully working vessel-detection system in place, extend the data capability and model sophistication to improve detection accuracy, assimilating SAR polarization data and optical imagery.
Illegal and unsustainable fishing practices can deplete marine resources and endanger food security. Illegal, unreported and unregulated (IUU) fishing affects legitimate commercial fishers, impacts the accuracy of stock estimates, and induce severe damage to non-target species and vulnerable marine ecosystems. It is estimated that IUU fishing impacts the global economy on a billion-dollar-scale annually. Most developing countries do not have sufficient infrastructure in place to monitor vessel activity at large scale.
In recent years, AI computer vision methods have dominated the analyses of natural, medical and satellite images. AI-driven approaches have been shown to outperform standard statistical methods on complex tasks such as classification, object detection and semantic segmentation of massive data streams from surveillance systems, mobile devices, and commercial satellites. Unlike standard statistical methods that are data-type specific, Convolutional Neural Networks (CNN) have been implemented on a wide range of image types and complex backgrounds. CNNs are a natural way forward to improve upon and extend the capabilities of current ship detection systems (moving beyond CFAR-based methods).
We propose to implement and test three state-of-the-art CNN architectures for (near) real-time object detection: You Only Look Once (YOLOv3), Faster Region-based CNN (Faster R-CNN), and Single Shot Multibox Detector (SSD), as some studies suggest these are among the best performing CNNs for detecting ships on satellite images (see refs below). Python implementations of these architectures on top of TensorFlow and/or PyTorch are also available. Typical outputs from these CNNs consist of the center coordinates, bounding box, and class probability of identified objects, a convenient way to report location, size and uncertainty of detected vessels.
Although the use of all-day/all-weather SAR amplitude images (by CFAR systems) constitutes a significant improvement over traditional optical methods (suffering from cloud coverage and light conditions), single-channel SAR images still suffer from inherent speckle noise, azimuth ambiguities, and low contrast on low backscattering background, characteristic of rough ocean environments. What's more, in-shore ships can be confounded with the infrastructure of harbors, with similar brightness and shapes.
We propose to use additional information such as SAR polarization and co-located detections from optical imagery. Numerous studies have shown the applicability of Polarimetric SAR data for the ship-detection problem (see refs below). The idea is that backscattering from a complex structure (a ship) consists of a mixture of single-bounced, double-bounced and depolarized scatterings, and only a strong single-bounce or double-bounce scatterer will produce (certain) ambiguities in azimuth, allowing the separation of the different scatterers (ship and sea). Different combination modes of polarization channels can be used to increase the ship-ocean contrast and train CNN models to better recognize vessel features. Because the same CNN architectures can be trained with optical images, we also plan to incorporate detections from optical sensors. This will allows us to better estimate uncertainties on co-located predictions (SAR + Optical), evaluate and adapt network architecture (why a detection is possible on one image type but not the other), and provide historical context for large vessels (e.g. from the Landsat archive). For ground truth, we will rely on the Automatic Identification System (AIS) carried by most medium-to-large vessels.
While the Google Earth Engine platform allows geospatial analysis at planetary scale by providing pre-processed satellite imagery and convenient access to analysis tools, it also poses significant limitations within the scope of this project. The capability of the analysis tools is limited, with little-to-no support for modern neural-net powered machine learning. Predictions are bottlenecked by exports and intermediate file formats. Scaling is also limited, and it is difficult (sometimes impossible) to implement custom image analysis operations such as filtering, transforming, and augmenting.
Recently, Google has integrated the Earth Engine with TensorFlow and the AI Platform. The AI Platform integrates other services such as Cloud Storage, the BigQuery data warehouse, and Google's powerful Cloud Compute Engine. Through the AI Platform we can also access external databases and, perhaps most importantly, we can implement different deep learning frameworks (such as PyTorch) and CPU/GPU parallel architectures (such as Ray).
We propose to move not only the processing-prediction workflow to Google's AI Platform, but also (given its convenient GUI/SSH interface) move the full development stack from code prototyping and hyperparameter tuning to large-scale data visualization. A further improvement in the efficiency of our system comes from the adoption of cloud-optimized parallelization and data formats, such as
▶︎ Ray - A fast and simple framework for building and running distributed applications. Ray is packaged with RLlib, a scalable reinforcement learning library, Tune, a scalable hyperparameter tuning library, and Modin, a scalable high-performance DataFrame.
▶︎ HDF5 and Zarr - Hierarchical open source data formats that support large, complex, heterogeneous, chunked and compressed N-dimensional data. These formats are optimal for fast synchronous I/O operations (i.e. parallelization) of numerical types.
▶︎ COG - A cloud optimized GeoTIFF file aimed at being hosted on a HTTP file server, with an internal organization that enables more efficient workflows on the cloud. It does this by leveraging the ability of clients issuing HTTP GET range requests to ask for just the parts of a file they need.
Some practical considerations to keep in mind. Overall, the vessel detection framework needs to be:
- Fast - aiming at near-real time detections in future
- Transparent - to facilitate implementation and modifications
- Scalable - identify and asses scalability bottlenecks early on
- Automated - with as minimal human intervention as possible
- Proven - technologies are mature and/or have been successfully applied
- Documented - throughout the full dev stack to be accessible by any team member
- Open - based on actively maintained open-source code and publicly-available data
NOTE.
Because we want to minimize the development time, it seems practical not to focus on the SAR phase information in the first implementation of the system. This is experimental and will likely require substantial research. This will also require additional development as the complex data is not easily available and the phase information requires sophisticated pre-processing. We aim at implementing, testing and adapting working methods first, and then investigating further improvements to the vessel-detection problem.
- Identify data sources
- Identify data formats (original -> pipeline input -> pipeline output)
- Identify ingestion mechanism (cloud-to-cloud, external-to-cloud)
- Identify task-specific parallelization (CPU vs. GPU)
- Architect cloud workflow (input -> transform -> develop -> predict -> test -> deploy)
- Identify task automation (some human intervention will be needed in the pipeline)
- Make shareable/editable documentation
- Implement a simplified/reduced version of the above pipeline
- Select a few (manageable) locations with data availability
- Device data labeling strategy (manual vs semi-automated)
- Implement CNN methods (YOLOv3, Faster R-CNN, and SSD)
- Workout optimal data transformations (filtering, cropping, etc.)
- Workout best data augmentation approach (key aspect, large effort!)
- Workout representative training/testing data sets (little info available for remote sensing)
- Define a baseline (every ML implementation needs a baseline! We have the CFAR implementation)
- Develop web-based visualization to inspect and track intermediate results
- Make shareable/editable documentation
- Workout with engineers the infrastructure (will need a lot of engineering!)
- Perform global analyses
- Synthesize and distribute data
- Publish (high-impact) paper
- Investigate better data augmentation
- Assimilate polarization and optical information
- Extend object classes (containers, navy, cargo, passenger, fishing, etc.)
- Incorporate historic information to delineate strategic areas (e.g. protected ecosystems)
- Device automated warning system? (when vessel type crosses pre-defined boundary)
- Investigate other AI methods to our problem (e.g. semantic segmentation)
- Investigate the use of SAR phase information
- Investigate better uncertainty estimation
Given the adoption of novel technologies and global scope of this project, significant challenges still remain. As the project develops, we will test and update our adopted strategies. Significant effort will be required to train and evaluate the CNN models at global scale, as well as generating optimal labeled training SAR data sets.
A word of caution.
There is no guarantee that a deep learning approach will outperform a (standard) working method. An optimal deep learning model for a specific problem relies on numerous trial-and-error tests (i.e. brute force), where the model is tuned for the specific data in question. Success heavily relies on a combination of creativity and domain expertise.
▶︎ The file example.ipynb is a Jupyter Notebook with a simple exercise to test setting up, compiling (C code) and training a basic CNN on a cloud GPU instance.
Figures have been modified from Google and the following references (this is just an exercise):
PolSAR for small ship detection
PolSAR method for ship detection
PolSAR use in ship detection
SAR dataset for deep learning
Status of vessel detection with SAR