Skip to content
The official Tensorflow implementation for CVPR'19 paper 'Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture'
Branch: master
Clone or download
Ning Yu
Ning Yu updates
Latest commit 58b2621 Jun 4, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
datasets updates May 30, 2019
examples updates May 29, 2019
fig updates Jun 4, 2019
hybridization_fig updates May 29, 2019
metrics updates Feb 15, 2019
nets updates Feb 15, 2019
stroke_fig updates May 29, 2019
LICENSE Update LICENSE May 30, 2019
README.md Update README.md May 30, 2019
config.py updates May 30, 2019
custom_vgg19.py updates Feb 15, 2019
dataset.py updates Feb 15, 2019
dataset_tool.py updates Feb 15, 2019
legacy.py updates Feb 15, 2019
loss.py updates May 30, 2019
misc.py updates Feb 15, 2019
networks.py updates Feb 15, 2019
run.py updates May 30, 2019
tfutil.py updates Feb 15, 2019
util_scripts.py updates May 29, 2019

README.md

Texture Mixer

  • Official Tensorflow implementation for our CVPR'19 paper on controllable texture interpolation and several applications.
  • Contact: Ning Yu (ningyu AT umd DOT edu)

Texture Interpolation 128x1024 (more results are shown in the paper)

Texture Dissolve 1024x1024

Texture Brush 512x2048

Animal hybridization

Prerequisites

  • Linux
  • NVIDIA GPU + CUDA + CuDNN
  • Python 3.6
  • tensorflow-gpu
  • Other Python dependencies: numpy, scipy, moviepy, Pillow, skimage, lmdb, opencv-python, cryptography, h5py, six
  • Clone the official VGG repository into the current direcotory.

Datasets: Animal Texture, Earth Texture, Plant Texture

  • Raw training and testing images of earth texture are downloaded from Flickr under Creative Commons or public domain license. They are saved at datasets/earth_texture/.
  • Raw training and testing images of animal texture or plant texture are copyrighted by Adobe and can be purchased from Adobe Stock. We list the searchable image IDs at:
    • datasets/animal_texture/train_AdobeStock_ID_list.txt
    • datasets/animal_texture/test_AdobeStock_ID_list.txt
    • datasets/plant_texture/train_AdobeStock_ID_list.txt
    • datasets/plant_texture/test_AdobeStock_ID_list.txt
  • Given raw images, run, e.g., the following command for data augmentation.
    python3 data_augmentation.py --iPath earth_texture/test_resize512/ --oPath earth_texture_test_aug/ --num_aug 10000
    
  • Then follow the official Progressive GAN repository "Preparing datasets for training" Section for dataset preparation. Use the create_from_images option in dataset_tool.py. The prepared data enables efficient streaming.
  • For convenience, the prepared testing dataset of earth texture can be downloaded here. Unzip and put under datasets/.

Pre-Trained Models

Training

After data preparation, run, e.g.,

python3 run.py \
--app train \
--train_dir earth_texture_train_aug_with_labels/ \
--val_dir earth_texture_test_aug_with_labels/ \
--out_dir models/earth_texture/ \
--num_gpus 8

where

  • train_dir: The prepared training dataset directory that can be efficiently called by the code.
  • val_dir: The prepared validation dataset directory that can be efficiently called by the code.
  • num_gpus: The number of GPUs for training. Options {1, 2, 4, 8}. Using 8 NVIDIA GeForce GTX 1080 Ti GPUs, we suggest training for 3 days.

Applications

Texture Interpolation

Run, e.g.,

python3 run.py \
--app interpolation \
--model_path models/animal_texture/network-final.pkl \
--imageL_path examples/animal_texture/1000_F_99107656_XvbvoVVRintE5tmuh1MkdXqs8rkzoahB_NW_aug00000094.png \
--imageR_path examples/animal_texture/1000_F_109464954_aBfyWSbdZt5PNpUo7hOqDRPWmmvQj3v9_NW_aug00000092.png \
--out_dir results/animal_texture/horizontal_interpolation/

where

  • imageL_path: The left-hand side image for horizontal interpolation.
  • imageR_path: The right-hand side image for horizontal interpolation.

Texture Dissolve

Run, e.g.,

python3 run.py \
--app dissolve \
--model_path models/animal_texture/network-final.pkl \
--imageStartUL_path examples/animal_texture/1000_F_23745067_w0GkcAIQG2C4hxOelI1aQYZglEXggGRS_NW_aug00000000.png \
--imageStartUR_path examples/animal_texture/1000_F_44327602_E6wl8FNihQON8c704fE5DEY2LOJPQQ1V_NW_aug00000018.png \
--imageStartBL_path examples/animal_texture/1000_F_66218716_rxcsXWQzYpIWVB8a09ZcuNzE7qAJ3HEk_NW_aug00000098.png \
--imageStartBR_path examples/animal_texture/1000_F_40846588_APKTS3BpiRvR1nvUx0FRa7qjjR788zt8_NW_aug00000001.png \
--imageEndUL_path examples/animal_texture/1000_F_40300952_3dgaCtcLCrhzU0r6HEfnwr7nujDWXbSQ_NW_aug00000029.png \
--imageEndUR_path examples/animal_texture/1000_F_44119648_3vskjRwVc4NVT1Pf0l3RlvIFUemo8TM1_NW_aug00000001.png \
--imageEndBL_path examples/animal_texture/1000_F_70708482_2N8lknTCJg2Q8JQeomFYaFttxId9rulj_NW_aug00000070.png \
--imageEndBR_path examples/animal_texture/1000_F_79496236_8mxTSHy5OilHnJaAxWcw2dwC9SoBLmDK_NW_aug00000057.png \
--out_dir results/animal_texture/dissolve/

where

  • imageStartUL_path: The upper-left corner image in the starting frame.
  • imageStartUR_path: The upper-right corner image in the starting frame.
  • imageStartBL_path: The bottom-left corner image in the starting frame.
  • imageStartBR_path: The bottom-right corner image in the starting frame.
  • imageEndUL_path: The upper-left corner image in the ending frame.
  • imageEndUR_path: The upper-right corner image in the ending frame.
  • imageEndBL_path: The bottom-left corner image in the ending frame.
  • imageEndBR_path: The bottom-right corner image in the ending frame.

Texture Brush

Run, e.g.,

python3 run.py \
--app brush \
--model_path models/animal_texture/network-final.pkl \
--imageBgUL_path examples/animal_texture/1000_F_23745067_w0GkcAIQG2C4hxOelI1aQYZglEXggGRS_NW_aug00000000.png \
--imageBgUR_path examples/animal_texture/1000_F_44327602_E6wl8FNihQON8c704fE5DEY2LOJPQQ1V_NW_aug00000018.png \
--imageBgBL_path examples/animal_texture/1000_F_66218716_rxcsXWQzYpIWVB8a09ZcuNzE7qAJ3HEk_NW_aug00000098.png \
--imageBgBR_path examples/animal_texture/1000_F_40846588_APKTS3BpiRvR1nvUx0FRa7qjjR788zt8_NW_aug00000001.png \
--imageFgUL_path examples/animal_texture/1000_F_40300952_3dgaCtcLCrhzU0r6HEfnwr7nujDWXbSQ_NW_aug00000029.png \
--imageFgUR_path examples/animal_texture/1000_F_44119648_3vskjRwVc4NVT1Pf0l3RlvIFUemo8TM1_NW_aug00000001.png \
--imageFgBL_path examples/animal_texture/1000_F_70708482_2N8lknTCJg2Q8JQeomFYaFttxId9rulj_NW_aug00000070.png \
--imageFgBR_path examples/animal_texture/1000_F_79496236_8mxTSHy5OilHnJaAxWcw2dwC9SoBLmDK_NW_aug00000057.png \
--stroke1_path stroke_fig/C_skeleton.png \
--stroke2_path stroke_fig/V_skeleton.png \
--stroke3_path stroke_fig/P_skeleton.png \
--stroke4_path stroke_fig/R_skeleton.png \
--out_dir results/animal_texture/brush/

where

  • imageBgUL_path: The upper-left corner image for the background canvas.
  • imageBgUR_path: The upper-right corner image for the background canvas.
  • imageBgBL_path: The bottom-left corner image for the background canvas.
  • imageBgBR_path: The bottom-right corner image for the background canvas.
  • imageFgUL_path: The upper-left corner image for the foreground palatte.
  • imageFgUR_path: The upper-right corner image for the foreground palatte.
  • imageFgBL_path: The bottom-left corner image for the foreground palatte.
  • imageFgBR_path: The bottom-right corner image for the foreground palatte.
  • stroke1_path: The trajectory image for the 1st stroke. The stroke pattern is sampled from the (3/8, 3/8) portion of the foreground palatte.
  • stroke2_path: The trajectory image for the 2nd stroke. The stroke pattern is sampled from the (3/8, 7/8) portion of the foreground palatte.
  • stroke3_path: The trajectory image for the 3rd stroke. The stroke pattern is sampled from the (7/8, 3/8) portion of the foreground palatte.
  • stroke4_path: The trajectory image for the 4th stroke. The stroke pattern is sampled from the (7/8, 7/8) portion of the foreground palatte.

Animal hybridization

Run, e.g.,

python3 run.py \
--app hybridization \
--model_path models/animal_texture/network-final.pkl \
--source_dir hybridization_fig/leoraffe/ \
--out_dir results/animal_texture/hybridization/leoraffe/

where

  • source_dir: The directory containing the hole region to be interpolated, two known source texture images adjacent to the hole, and their global Content-Aware Fill results from Photoshop.

After that, post-process the output hybridization image by Auto-Blend Layers with the original image (with hole) in Photoshop, so as to achieve the demo quality as shown above.

Citation

@inproceedings{yu2019texture,
    author = {Yu, Ning and Barnes, Connelly and Shechtman, Eli and Amirghodsi, Sohrab and Lukáč, Michal},
    title = {Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture},
    booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2019}
}

Acknowledgement

  • This research is supported by Adobe Research Funding.
  • We acknowledge the Maryland Advanced Research Computing Center for providing computing resources.
  • We thank to the photographers for licensing photos under Creative Commons or public domain.
  • We express gratitudes to the Progressive GAN repository as we benefit a lot from their code.
You can’t perform that action at this time.