Interior design is a multi-billion-dollar industry. Interior designers spend a significant amount of time during the initial consultation phase with their clients because clients often have difficulties visualizing the designers' sketches. One AI solution to this problem to this problem is to apply neural style transfer that transfer desired pattern onto an existing photo. Neural style transfer alone, however, does not solve interior designer's problem. Neural style transfer requires a style image that contains the desired pattern. For interior design, finding the right style images with clients will be also time-consuming.
DeepDeco generates customizable interior design images with sketches and text description. It uses neural style transfer and Word2Vec style search.
Resources: slides
Clone repository and update python path
git clone https://github.com/peter0083/DeepDeco.git
cd DeepDeco
In a terminal, execute the following command:
curl -F "file=@data/images/chair_sketch.jpeg" -F "text='ice cold patterned glass'" -X POST http://deepdeco.club/image --output flask_output.gifIt is recommended to run this inference script on AWS EC2 with GPU for optimal result.
- setup your
awsclicredentials (required to download the dataset) - run 'inference_aws.py'
python inference_aws.py --input "ice cold patterned glass" --content path/to/designer_sketch.jpg --speed "medium"
- Python 3.6
Deep Photo Style Transfer
Style Search Engine
Installation
To install the packages above, please run:
pip install -r requirements.txt
The VGG-19 model of tensorflow is adopted from VGG Tensorflow with few modifications on the class interface. The VGG-19 model weights is stored as .npy file and could be download from Google Drive or BaiduYun Pan. After downloading, copy the weight file to the /src/ftdeepphoto/deep-photo-styletransfer-tf/vgg19/ directory
Download DeepLab V3 weights from DeepLabV3 Tensorflow. After downloading the file, copy the weight file to src/ftdeepphoto/deeplab/models/ directory
Download GloVe 6B 300d weights from Stanford NLP group.
Unzip the zip file and move glove.6B.300d.txt to the src/stylesearch/pickles/glove.6B/ directory.
AWS
Deep Learning AMI (Ubuntu) Version 13.0 (ami-00499ff523cc859e6)
Comes with latest binaries of deep learning frameworks pre-installed in separate virtual environments:
MXNet, TensorFlow, Caffe, Caffe2, PyTorch, Keras, Chainer, Theano and CNTK.
Fully-configured with NVidia CUDA, cuDNN and NCCL as well as Intel MKL-DNN
Platform: Ubuntu
Image Size: 75GB
- to load different Word2Vec weights to the style search engine, run the following script:
python src/stylesearch/train.py --weight_path /path/to/your_word2vec_weight.txt- to train style transfer model with your own images, run the following script:
python src/ftdeepphoto/style_fpst.py # see arguments belowRemember to specify the following arguments when you run the train script.
--style /path/to/image_style_to_transfer.jpg \
--style-seg /path/to/style_image_segmentation_map.jpg \
--checkpoint-dir directory_to_checkpoint/ \
--train-path dir_to_training_images/ \
--resized-dir dir_to_resized_training_images/ \
--seg-dir dir_to_training_segmaps/ \
--vgg-path vgg/imagenet-vgg-verydeep-19.mat \
--content-weight 1.5e1 \
--photo-weight 0.005 \
--checkpoint-iterations 10 \
--batch-size 1 \
--epochs 20000 \
--deeplab-path ../deeplab/models/deeplabv3_pascal_train_aug_2018_01_04.tar.gz \
--matting-dir matting/- fast-deep-photo-style-transfer-tf by Handa Yang
- Ikea data set by Ivona Tautkute et al
@inproceedings{FedCSIS201756,
author={Ivona Tautkute and Aleksandra Możejko and Wojciech Stokowiec and Tomasz Trzciński and Łukasz Brocki and Krzysztof Marasek,},
pages={1275--1282},
title={What Looks Good with my Sofa: Multimodal Search Engine for Interior Design},
booktitle={Proceedings of the 2017 Federated Conference on Computer Science and Information Systems},
year={2017},
editor={M. Ganzha and L. Maciaszek and M. Paprzycki},
publisher={IEEE},
doi={10.15439/2017F56},
url={http://dx.doi.org/10.15439/2017F56},
volume={11},
series={Annals of Computer Science and Information Systems}
}
- "What Looks Good with my Sofa: Multimodal Search Engine for Interior Design" by Ivona Tautkute, Aleksandra Możejko, Wojciech Stokowiec, Tomasz Trzciński, Łukasz Brocki and Krzysztof Marasek https://arxiv.org/abs/1707.06907
@inproceedings{FedCSIS201756,
author={Ivona Tautkute and Aleksandra Możejko and Wojciech Stokowiec and Tomasz Trzciński and Łukasz Brocki and Krzysztof Marasek,},
pages={1275--1282},
title={What Looks Good with my Sofa: Multimodal Search Engine for Interior Design},
booktitle={Proceedings of the 2017 Federated Conference on Computer Science and Information Systems},
year={2017},
editor={M. Ganzha and L. Maciaszek and M. Paprzycki},
publisher={IEEE},
doi={10.15439/2017F56},
url={http://dx.doi.org/10.15439/2017F56},
volume={11},
series={Annals of Computer Science and Information Systems}
}
- Image icon reference: Icon made by Freepik from www.flaticon.com
The directory structure of this project is organized in the diagram below:
├── LICENSE
├── README.md <- The top-level README for developers using this project.
├── flask_output.gif <- Sample output
├── data
│ ├── coco_stuff <- COCO Data Set for DeepLab testing.
│ └── images <- IKEA Data Set for training and inference.
│
├── webapp <- Scripts for Flask API
│
├── output <- Sample output images
│
├── checkpoints <- Save checkpoints during training and testing.
│
├── requirements.txt <- The requirements file for reproducing the analysis environment
│
├── src <- Source code for use in this project.
│ ├── stylesearch <- Scripts for Style Search Engine
│ │
│ ├── ftdeepphotosegMaps<- Segmented images during training and inference
│ │
│ ├── ftdeepphoto <- Scripts for Deep Photo Style Transfer (TensorFlow)
│ │
│ ├── checkpoints <- Save checkpoints during training and testing.
│ │
│ └── tests <- Unit tests for pytest.
│
└── tox.ini <- tox file with settings for running tox; see tox.testrun.org
Peter Lin: peter.tingyao@gmail.com