Implementation of the ICCV 2021 paper "Multi-Class Multi-Instance Count Conditioned Adversarial Image Generation MCC-StyleGAN2". The paper can be found here. The code allows the users to reproduce and extend the results reported in the paper. Please cite the above paper when reporting, reproducing or extending the results.
This software is a research prototype, solely developed for and published as part of the publication. It will neither be maintained nor monitored in any way.
This repository is a PyTorch implementation of a simple convolution based architecture with the same concept for toy experiments.
Please add the project folder to PYTHONPATH and install the required dependencies:
- python 3.6.10
- pytorch 1.4.0
pip install -r requirements.txt
To train the model, run this command:
python main.py --input_images <path_to_image_array> --countvec_path <path_to_count_csvfile>
For MCC-StyleGAN2 repository we adapted the official tensorflow implementation of StyleGAN2 "https://github.com/NVlabs/stylegan2". For CityCount images, adaptive discriminator augmentation technique based on the implementation in "https://github.com/NVlabs/stylegan2-ada" is utilized while training. Please note that the network configuration and the loss functions however, remains the same for all datasets. The main differences are in the network architecture and the loss functions used in the training.
training/networks_stylegan2.py - The modified generator and discriminator for count conditioned image generation. To be precise we introduced count vector mapping to each layer in the mapping network. We also introduced dense connectivity where output from a layer is connected to all its following layers.
def denseblock(x, resolution):
dense = [None] * 5
if (resolution > 2 and resolution < 7):
dense[resolution - 3] = x
if resolution - 3 > 0:
#Denseskip0
dense[0] = conv2dlayer(dense[0])
if resolution - 4 > 0:
#Denseskip1
dense[1] = conv2dlayer(dense[1])
if resolution - 5 > 0:
#Denseskip2
dense[2] = conv2dlayer(dense[2])
if resolution - 6 > 0:
#Denseskip3
dense[3] = conv2dlayer(dense[3])
if resolution - 7 > 0:
#Denseskip4
dense[4] = conv2dlayer(dense[4])
if resolution > 3:
#Denseskipx
dense[resolution - 4] = conv2dlayer(dense[resolution - 4])
for iter in range(0, resolution - 3):
x = x + dense[iter]
x = x * (1 / np.sqrt(max(2, resolution - 3)))
return x
training/loss.py - An additional count loss functions used for training the network
countlossfake = tf.reduce_mean(tf.squared_difference(fake_count_out, count_labels))
countlossreal = tf.reduce_mean(tf.squared_difference(real_count_out, count_labels))
countloss = (countlossfake + countlossreal) / 2
Incorporate these additional files provided in the MCC-StyeGAN2 directory to the corresponding folders in the original repo.
-
To create MultiMNIST dataset, please refer to the repo here.
-
To create CLEVR dataset, please refer to the CLEVR directory adapted from the repo. Incorporate the additional files provided in the CLEVR directory to the corresponding folders in the original repo. The CLEVR directory includes image generation scripts specifically modified for CLEVR2 and CLEVR3 with multiprocessing enabled. Refer to the original repo README for instructions on running the image generation.
-
To create CityCount dataset, please refer to the Citycount directory. Please download the required dataset(leftImg8bit_trainvaltest.zip, gtBbox3d_trainvaltest.zip and gtBbox_cityPersons_trainval.zip) from here.
MCC-StyleGAN2 is open-sourced under the AGPL-3.0 license. See the LICENSE file for details. For a list of other open source components included in the MCC-StyleGAN2, see the file 3rd-party-licenses.txt. For further queries, please contact amrutha.saseendran21@gmail.com