Contents

• Style Transfer Description
• Model Architecture
• Dataset
• Pretrained model
• Environment Requirements
• Script Description
  • Script and Sample Code
  • Script Parameters
  • Training Process
• Model Description - Performance - Training Performance
  - Evaluation Performance
• Description of Random Situation
• Example
• ModelZoo Homepage

Style Transfer Description

This is the Implementation of the Mindspore code of paper Exploring the structure of a real-time, arbitrary neural artistic stylization network.

Authors: Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, Jonathon Shlens

In this paper, we present a method which combines the flexibility of the neural algorithm of artistic style with the speed of fast style transfer networks to allow real-time stylization using any content/style image pair. We build upon recent work leveraging conditional instance normalization for multi-style transfer networks by learning to predict the conditional instance normalization parameters directly from a style image. The model is successfully trained on a corpus of roughly 80,000 paintings and is able to generalize to paintings previously unobserved. We demonstrate that the learned embedding space is smooth and contains a rich structure and organizes semantic information associated with paintings in an entirely unsupervised manner.

Model Architecture

The style prediction network P predicts an embedding vector S from an input style image, which supplies a set of normalization constants for the style transfer network. The style transfer network transforms the photograph into a stylized representation. The content and style losses are derived from the distance in representational space of the VGG image classification network. The style transfer network largely follows and the style prediction network largely follows the Inception-v3 architecture.

Dataset

To train Style Transfer Networks, Two datasets are used:

• content dataset: MS COCO
• style dataset: Painter by Number dataset (PBN) and Describable Textures Dataset (DTD) PBN training PBN testing DTD dataset

Train dataset：

• content dataset: MS COCO
• style dataset PBN training DTD dataset, place the images in both datasets in the same folder

Test dataset:

• content dataset: arbitrarily collect images by yourself or Select some images from MS COCO as the data set.
• style dataset: arbitrarily collect style images or PBN testing

Pretrained model

The process of training Style Transfer Networks needs a pretrained VGG16 and Inception-v3. VGG16 pretrained model and Inception-v3 pretrained model

Modify the name of ckpt to inceptionv3.ckpt and vgg16.ckpt

After downloading the pre-training model, please place it in folder './pretrained_model'.

Environment Requirements

• Hardware Ascend
  • Prepare hardware environment with Ascend processor.
• Framework
  • MindSpore
• For more information, please check the resources below：
  • MindSpore Tutorials
  • MindSpore Python API

Script Description

Script and Sample Code

style transfer

├─ README.md # descriptions about style transfer
├── scripts  
 ├─ run_distribute_train.sh # launch Acsend training (8 Acsend)
 ├─ run_eval.sh # launch ascend eval
 └─ run_stranalone_train.sh # launch Acsend training(1 Acsend)
├─── dataset # dataset
 ├── train # train content data and train style data
  ├─ content
  └─ style
 └── test # test content data and test style data
  ├─ content
  └─ style
├── pretrained_model  
 ├─ vgg16.ckpt # VGG16 pretrained model
 └─ inceptionv3.ckpt # Inception-v3 pretrained model
├── src
 ├─ ckpt # save ckpt  
 ├─ testdataset.py # dataset for evaling  
 ├─ traindataset.py # dataset for training
 ├─ loss.py #style transfer losses function define
 ├─ model.py # style transfer model define  and training process
 ├─ networks.py # sub-networks define  
 ├─ vgg16.py # vgg16 network define  
 └─ inceptionv3py # inception-v3 network define  
├─ output #result
├─ test.py # generate style transfer images
└─ train.py # train script

Script Parameters

Training Script Parameters

# distributed Ascend training
bash ./scripts/run_distribute_train_ascend.sh [RANK_TABLE_FILE] [PLATFORM] [CONTENT_PATH] [STYLE_PATH] [CKPT_PATH]
# distributed GPU training
bash ./scripts/run_distribute_train_gpu.sh [CONTENT_PATH] [STYLE_PATH] [CKPT_PATH]

# standalone Ascend\GPU training
bash ./scripts/run_standalone_train.sh [PLATFORM] [DEVICE_ID] [CONTENT_PATH] [STYLE_PATH] [CKPT_PATH]

Training Result

Training result will be stored in './src/ckpt'. You can find checkpoint file.

Evaluation Script Parameters

• Run run_eval.sh for evaluation.

# evaling
bash ./scripts/run_eval.sh [PLATFORM] [DEVICE_ID] [CONTENT_PATH] [STYLE_PATH] [INCEPTION_CKPT] [CKPT_PATH]

Evaluation result

Evaluation result will be stored in the output. Under this, you can find style transfer pictures.

Model Description

Performance

Training Performance

Parameters
Model Version	v1
Resource	Ascend RAM：754G
MindSpore Version	1.3.0
Dataset	content: MS COCO. style: PBN training and DTD dataset
Training Parameters	epoch=100, batch_size = 16
Optimizer	Adam
Loss Function	content loss and style loss
outputs	style transfer pictures
Speed	1 Ascend: 392ms/step; 8 Ascend: 303ms/step;
Total time	1 Ascend: 56h20m21s; 8 Ascend: 6h15m40s;
Checkpoint for Fine tuning	71.27M (.ckpt file)

Evaluation Performance

Parameters	single Ascend
Model Version	v1
Resource	Ascend;
MindSpore Version	1.3.0
Dataset	content: MS COCO. style: PBN testing
batch_size	1
Speed
outputs	style transfer pictures

Example

Results

Results_interpolation

ModelZoo Homepage

Please check the official homepage.