Handloom-Design-Generation-using-Deep-Neural-Networks-

This is our 4th year Engineering final year project. It aims to use techniques like Conditional GAN , Image to Image translation , Texture and content transfer for aiding as a design tool for handloom weavers and also designers and industry as a whole.

NOTE: Paper Accepted at ICIP 2020

Contrib: Meghali Nandi , Amrit Jha

If you find this work exciting or have somehow inspired your work , feel free to cite us

@misc{handloomGeneration,
  author = {{Bhattacharjee}, R. K and  {Nandi}, M. and  {Jha}, A..},
  title = {Handloom  Design  Generation  using  Deep Neural Networks},
  year = {2018},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/rajatkb/Handloom-Design-Generation-using-Deep-Neural-Networks}},
  commit = {3a2e05caef787dbc34f473ccf1338e730dd1c76e}
}

Demo

How to use the data_builder

$ python data_builder.py -h
usage: data_builder.py [-h] --tfx TARGET_FOLDERX --tfy TARGET_FOLDERY
                       [--augx AUGMENTX] [--augy AUGMENTY]

data builder

optional arguments:
  -h, --help            show this help message and exit
  --tfx TARGET_FOLDERX
  --tfy TARGET_FOLDERY
  --augx AUGMENTX
  --augy AUGMENTY

Run the the script at any target folder to create a dataset out of all the images inside the folder. Augmentation of image includes current colour shifting and inverting of axis and both combined. Each of the script uses a similar data builder method. You can also use the batchwise data builder for the Perceptual and GAN scripts.

Dataset

We collected a complete orignal set of dataset for this purpose. Collected 500 around Saree where we randomly cropped to get 5 patches of design. These patches were then augmented to give us more number of dataset which is very much required for the GANs.

The dataset comprises of two collections of images - normal sarees and handloom sarees. The exact details about the collections are stated below.

Normal sarees

Handloom sarees

Normal Sarees

• Number of Images: 580
• Number of Images (after crop): 2900
• Number of Images (after augmentation): 26100
• Image Resolution: 256 X 256 X 3
• Image Format: RGB - JPG/JPEG
• Size of Collection: 200 MB

Handloom Sarees

• Number of Images: 33
• Number of Images (after crop): 165
• Number of Images (after augmentation): 1185
• Image Resolution: 256 X 256 X 3
• Image Format: RGB - JPG/JPEG
• Size of Collection: 18 MB

The dataset can be downloaded from here. New images are being included to the dataset.

Methodologies

1. Simple GAN

Attempt at achieving simple designs using very low amount of data of Meckala Sador dataset and also Saree Dataset.
Results : The DCGAN was not able to generate higher resolution images. We targeted for atleast 512x512 or 256x256 . The results were taking to long on a single P100 GPU. (GCP virtual machine GPU)

2. Variation AutoEncoder

A simple attempt at genrative model to using very low amount of data for generating design after learning from Saree dataset. To be quoted "Miserable results" is what we we got. more updates on the methods later on but the very early version is not working. Results: Similar to what the DCGAN gave.

3. Fast Neural Style Transfer using Perceptual loss

This will work as a image inpaniting method of various cases of probelm.

Credits: fast-neural-style-keras
The final models are from this particular scripts since, the model here was not showing any window artifact i.e having a kind of padded window to the right side of the image of the same pattern otherwise both the scripts from this and ours can be used for style transfer.

The notebooks inside the repo of fast-neural-style-transfer is for the inference and testing of single image. Try those notebook if you do not want to deploy the full application.

If you want to retrain running the train.py or the train.sh (in case of prelim work scripts) will do the job. The shell script is configurable.

When trying fast neural style transfer the network was trained using MS-COCO validation dataset which is smaller than the actual dataset. The results are in the location "preliminary work/Perceptual Loss Style transfer - rajat/Final Result from the 2nd script/"

4. Image to Image translation problem i.e Pix2Pix

We treat generating handloom design as an image to image translation problem where take normal saree dataset will be treated as the input image and the Mekhala dataset as the target distribution which the normal dataset must be converted to. This can be tackled by following

CycleGAN

Saree2Handloom

The saree2handloom model was trained for about 420 epochs on a batch size of one as recommended in the CycleGAN paper. The model quickly optimized upon the cycle consistency loss as the results showed impressive reconstruction of images back to the original domain from as early as the tenth epoch.

On further training, the model produced results showing that it was able to recognize patches of colour segments as well as the design patterns in both the domain as it was learning to fill the patches with colours from the target domain during the translations. The model was trained for more number of epochs in the hope of better translations leading to generation of unseen textures, but such targets were never met.

Coco2Handloom

Amazed by the results produced by CycleGAN when employed for translations between images of abruptly different domains, the CycleGAN technique was employed to train and build a "natural images to handloom" model which would have generated handloom cloth designs from natural image samples.

The model was trained with two domain of images - the MS-COCO validation set (2017) and our handloom image samples. We observed the results obtained from a traning of over 130 epochs on a batch size of one. The results highlighted the sparsity of data in the handloom images collection as most of the translations coloured the natural image to white (the dominant colour in handloom cloth). Clearly, for hoping any success with the model, we need to first get rid of the sparsity and other inherent flaws in our dataset.

DiscoGAN

The DiscoGAN gave more promising results when compared to other methods , The discoGAN was was used to convert hand drawn threasholded geomteric strcutures as base template which will then be colored in way that it looks like a traditional handloom or generic saree. The input to the model would be a standard binary mask of pattern and output will be a colored design. The model was trained on the entire Saree dataset.

App demo

This is the final application which shows how Neural Style tranfer can be employed into designing saree patches. Such patches can then be used as design suggestion for the rest of saree body.

link: neural-loom-app , style transfer demo

The original plan was though to deploy the models in the front end but the fast style transfer model computation took too much time on CPU so we refrained from something like that. Still if possible I will look for free alternative that will atleast allow us to host the app and the model together for free. Firebase ML toolkit or google cloud ml engine maybe options.

Installation & Deployment

You will need these preinstalled on your server for both Perceptual Style Transfer and DiscoGAN (script is similar but the server are sepparated , since DISCOGAN is too large to deploy on one single GPU)

• Python3
• Numpy
• Matplotlib
• OpenCV
• tensorflow with keras api (we are using the Keras api sepparately not the only available in tf)

for deploying in the server set your port in the script app.py located in neural-loom-server folder

python app.py

Should do the job. Things to consider

• If deploying on containers or virtual machine instances make sure Firewall is allowing the traffic through the particular port you are using. We were misconfuguring a lot of times , resulting in wastage of time.
• The native CORS for flask failed in our usecase for some reason unexplainable. So we resorted to manually appending the headers.
• The DISCOGAN and PERCEPTUAL NEURAL STYLE both are to be hosted in sepparate servers hence have sepparate server scripts

For the front end you need npm. So install

• node and npm (they come together for windows for linux may need to install separtely in some distributions)

In the neural-loom-app , go to the services and then to upload.service.ts , there edit the ip address according to your setup.

ng build --prod
Will give you the html , css and js files. Deploy them accordingly. Using flask or http-server (npm app for deploying html project quickly)

hardware: We deployed on Google Cloud platform VM with Nvidia P100 GPU 16GB VRAM, i5 4 core , 16GB RAM. The client can be any simple rustbucket.

Todo

• add option for resolution change
• Find proper paramters for Faste Neural Style transfer for global learning
• Use natural images and artistic image to train Network, Use the network to appy style on existing Saree (Normal and Mekhala data)
• Use saree data to train to apply style and color of Mekhala dataset on the normal saree dataset
• Use artistic dataset to train , and use masking technique on hand drawn image for Image-inpainting
• Use Mekhala sador dataset for image inpainting
• Test CycleGAN on normal dataset prepare a testing script for our experiments
• Test DiscoGAN on normal dataset prepare a testing script for our experiments
• Test both network for domain transfer
• make web based pipeline for all operation
• make python server for getting image and serving the processed image back
• make frontend Angular application
• have the python server interact with the model.predict to get output
• Finish the README.