Official Implementation of Adversarial Instance Augmentation for Building Change Detection in Remote Sensing Images.
We propose a novel data-level solution, namely Instance-level change Augmentation (IAug), to generate bi-temporal images that contain changes involving plenty and diverse buildings by leveraging generative adversarial training. The key of IAug is to blend synthesized building instances onto appropriate positions of one of the bi-temporal images. To achieve this, a building generator is employed to produce realistic building images that are consistent with the given layouts. Diverse styles are later transferred onto the generated images. We further propose context-aware blending for a realistic composite of the building and the background. We augment the existing CD datasets and also design a simple yet effective CD model - CDNet. Our method (CDNet + IAug) has achieved state-of-the-art results in two building CD datasets (LEVIR-CD and WHU-CD). Interestingly, we achieve comparable results with only 20% of the training data as the current state-of-the-art methods using 100% data. Extensive experiments have validated the effectiveness of the proposed IAug. Our augmented dataset has a lower risk of class imbalance than the original one. Conventional learning on the synthesized dataset outperforms several popular cost-sensitive algorithms on the original dataset.
See building generator for details.
Synthesized images (256 * 256) by the generator (trained on the AIRS building dataset).
Synthesized images (64 * 64) by the generator (trained on the Inria building dataset).
This code requires PyTorch 1.0 and python 3+. Please install dependencies by
pip install -r requirements.txtOnce the dataset is ready, the result images can be generated using pretrained models.
-
Download the tar of the pretrained models from the Google Drive
-
Generate images using the pretrained model.
python test.py --model pix2pix --name $pretrained_folder --results_dir $results_dir --dataset_mode custom --label_dir $label_dir --label_nc 2 --batchSize $batchSize --load_size $size --crop_size $size --no_instance --which_epoch lastest
pretrained_folderis the directory name of the checkpoint file downloaded in Step 1,results_diris the directory name to save the synthesized images,label_diris the directory name of the semantic labels,sizeis the size of the label map fed to the generator. -
The outputs images are stored at
results_dir. You can view them using the autogenerated HTML file in the directory.
For simplicity, we also provide the test script in scripts/run_test.sh, one can modify the label_dir and name and then run the script.
New models can be trained with the following commands.
-
Prepare the dataset. You can first prepare the building image patches and corresponding label maps in two folders (
image_dir,label_dir). -
Train the model.
# To train on your own custom dataset
python train.py --name [experiment_name] --dataset_mode custom --label_dir [label_dir] -- image_dir [image_dir] --label_nc 2There are many options you can specify. Please use python train.py --help. The specified options are printed to the console. To specify the number of GPUs to utilize, use --gpu_ids. If you want to use the second and third GPUs for example, use --gpu_ids 1,2.
To log training, use --tf_log for Tensorboard. The logs are stored at [checkpoints_dir]/[name]/logs.
This code borrows heavily from spade.
See Color Transfer for deteils.
We resort to a simple yet effective nonlearning approach to match the color distribution of the two image sets (GAN-generated images and original images in the change detection dataset).
- Matlab
We provide two demos to show the color transfer.
When you do not have the object mask. You can edit the file ColorTransferDemo.m, modify the file path of the Im_target and Im_source. After you run this file, the transfered image is saved as result_Image.jpg.
When you do have both the building image and the object mask. You can edit the file ColorTransferDemo_with_mask.m, modify the file path of the Im_target, Im_source, m_target and m_source. After you run this file, the transfered image is saved as result_Image.jpg.
This code borrows heavily from https://github.com/AissamDjahnine/ColorTransfer.
We show a simple shadow extraction method. The extracted shadow information can be used to make a more realistic image composite in the latter process.
We provide some examples for shadow extraction. The samples are in the folder samples\shadow_sample.
You can edit the file extract_shadow.py and modify the path of the image_folder, label_folder and out_folder. Make sure that the image files are in image_folder and the corresponding label files are in label_folder. Run the following script:
python extract_shadow.pyOnce you have successfully run the python file, the results can be found in the out folder.
Here, we provide the python implementation of instance augmentation.
We provide some examples for instance augmentation. The samples are in the folder samples\SYN_CD.
You can edit the file composite_CD_sample.py and modify the following values:
# first define the some paths
A_folder = r'samples\LEVIR\A'
B_folder = r'samples\LEVIR\B'
L_folder = r'samples\LEVIR\label'
ref_folder = r'samples\LEVIR\ref'
# instance path
src_folder = r'samples\SYN_CD\image' #test
label_folder = r'samples\SYN_CD\shadow' # test
out_folder = r'samples\SYN_CD\out_sample'
os.makedirs(out_folder, exist_ok=True)
# how many instance to paste per sample
M = 50
Then, run the following script:
python composite_CD_sample.pyOnce you have successfully run the python file, the results can be found in the out folder.
Coming soon~~~~
If you use this code for your research, please cite our paper:
@Article{chen2021,
title={Adversarial Instance Augmentation for Building Change Detection in Remote Sensing Images},
author={Hao Chen, Wenyuan Li and Zhenwei Shi},
year={2021},
journal={IEEE Transactions on Geoscience and Remote Sensing},
volume={},
number={},
pages={1-16},
doi={10.1109/TGRS.2021.3066802}
}