In this paper, we have developed a general-purpose architecture, Vit-Gan, capable of performing most of the image-to-image translation tasks from semantic image segmentation to single image depth perception. This paper is a follow-up paper, an extension of generator-based model [1] in which the obtained results were very promising. This opened the possibility of further improvements with adversarial architecture. We used a unique vision transformers-based generator architecture and Conditional GANs(cGANs) with a Markovian Discriminator (PatchGAN) (this https URL). In the present work, we use images as conditioning arguments. It is observed that the obtained results are more realistic than the commonly used architectures.
Clone the repo
git clone https://github.com/yigitgunduc/vit-gan/Install requirements
pip3 install -r requirements.txtFor GPU support setup
TensorFlow >= 2.4.0withCUDA v11.0 or above
- you can ignore this step if you are going to train on the CPU
Train the model
python3 src/train.pyWeights are saved after every epoch and can be found in ./weights/
After you have trained the model you can test it against 3 different criteria (FID, Structural similarity, Inceptoin score).
python3 src/evaluate.py path/to/weightsImplementation support 8 datasets for various tasks. 6 pix2pix datasets and two additional ones.
6 of the pix2pix dataset can be used by changing the DATASET variable on the src/train.py
for the additional datasets please see notebooks/object-segmentation.ipynb and
notebooks/depth.ipynb
Dataset available thought the src/train.py
cityscapes99 MBedges2handbags8.0 GBedges2shoes2.0 GBfacades29 MBmaps239 MBnight2day1.9 GB
Dataset available though the notebooks
Oxford-IIIT PetsRGB+D DATABASE
If you use this code for your research, please cite our paper link
@article{gunducc2021vit,
title={Vit-GAN: Image-to-image Translation with Vision Transformes and Conditional GANS},
author={G{\"u}nd{\"u}{\c{c}}, Yi{\u{g}}it},
journal={arXiv preprint arXiv:2110.09305},
year={2021}
}