This is an exploration on what Segment Anything Model (SAM) can do for remote sensing involving medium-resolution satellite imagery. Specifically, an automated sample generation approach based on SAM is proposed to solve sample scarcity problem in crop mapping.The workflow is shown in the picture. More details can be found in our ongoing paper. Notably, thanks to powerful ability of SAM, our approach can be extended to various types of imagery and areas. We hope our little contribution can offer some references on utlizing foundation models to solve remote sensing problems.
We assume you already have understandings of Anaconda and PyTorch. Only the instructions of some specific required components are provided below. We utilized shapely, geopandas to manipulate shapefile files; rasterio to manipulate raster files; scipy and fastdtw to perform sample cleaning.
-
Enter your environment.
-
Install required packages:
pip install -r requirements.txt
or use conda to install, but fastdtw may not be properly installed.
conda install --yes --file requirements.txt
- Download SAM checkpoint, we used ViT-H SAM model. As you finish download, put the checkpoint in the same folder as main.py.
We provided a simple demo of using Sentinel-2 time series and field-survey samples from AAFC to generate more samples.
You can use commands below to run the code. Considering not everyone can have resources to execute SAM on GPU, you can also try our demo by using CPU. The processing time depends on your equipments and data used.
GPU:
python main.py --optimalImage ./images/20190919.tif --patch_x 200 --patch_y 200 --device cuda
CPU:
python main.py --optimalImage ./images/20190919.tif --patch_x 200 --patch_y 200
If you want to fully automate the process, you can choose not to use the parameter --optimalImage. The code can automatically select the optimal image by our standard from the time series.
(1) (id)_BeforeSAM.png : indicates the patch extracted as the input for SAM, and where the reference sample located.
Example 1924_BeforeSAM.png:
(2) (id)_AfterSAM_mask_(num).png × 3: indicates the mask producted by SAM, and where the mask located in the patch. Because of our settings of SAM, it produces 3 masks with scores.
Example 1924_AfterSAM_mask_[1-3].png:
(3) (id)_AfterSAM_SITS.png : The NDVI time series curves of the reference sample and generated samples. The red line is the curve of the reference sample.
Example 1924_AfterSAM_SITS.png:
(4) (id)_AfterSAM_SITS_cleaned.png : The NDVI time series curves of the reference sample and generated samples after sample cleaning. The red line is the curve of the reference sample.
Example 1924_AfterSAM_SITS_cleaned.png:
After the last iteration, Generated_samples.shp would be generated to store all reference samples and generated samples. You can see it along with images in softwares like Arcmap.
The raster and vector data we used have been processed according to our requirements, and our code may only be applicable to these processed data. If you intend to use our code with your own data, please review our code and make modifications specific to your data format, such as how to read the temporal information of rasters and attribute tables of vector files. If you encounter any difficulties, feel free to ask the main author of the code through jialinsun4815162342@gmail.com. We are delighted to engage in any academic and code-related discussions.
Feel free to use our codes for any academic purposes, and academic purposes only.
Facebook Research Segment Anything — Apache-2.0 license
Still in progress.