TransformerVAE for Sketch Tutoring

This repository contains the code for the workshop paper "Variational-autoencoder-based Environment for Interactive Sketch Tutoring Aiming for Kids", accepted by WAIE 2020

Inspired by the generative model sketch-rnn proposed by David Ha et. al. We trained a VAE model using Google QuickDraw dataset. Besides using CNN-based encoder to encode space information, we further replaced the the auto-regressinve network by a transformer decoder to enhence the stability of the generated drawing in an interative environment. Python code is modified from this repository. The tfjs code and GUI is modified from the open source code in this repository

Requirements

Python >= 3.6
tensorflow >= 2.0

Weight Converting

1. Using Python code to train and save model weights:

python new_seq2seqVAE_train.py  

2. Converting the weights into tfjs format:

# bash

tensorflowjs_converter --input_format keras \
                       path/to/my_model.h5 \
                       path/to/tfjs_target_dir  

3. The model wieghts can then be initilized in tfjs as LayerModels in model.ts file:

  public encoder: tfl.LayersModel;
  public decoder: tfl.LayersModel;
  public input_to_embed_dense: tfl.LayersModel;

  public wq_dense: tfl.LayersModel[] = new Array(4);
  public wk_dense: tfl.LayersModel[] = new Array(4);
  public wv_dense: tfl.LayersModel[] = new Array(4);
  public concat_attention_to_decoder_output: tfl.LayersModel[] = new Array(4);