R_Unet

This project applies recurrent method upon U-net to perform pixel level video frame prediction.
Part of our result is published at IEEE GCCE 2020. pdf.

Brief introduction

Taking advantage of LSTM and U-net encode-decoder, we wish to be able in predicting next (n) frame(s).
Currently using a 2 layer LSTM network (V1) or convolution LSTM (V2) as RNN network applying on latent feature of U net
In our latest v4 model, we use convolutional LSTM in each level and take short cut in v2 out

On the other hand, we are now using v4_mask model to train mask, image input and mask, image prediction output
This model holds same structure as v4 but simply change output layer to output mask tensor.

Usage

• configuration: config.json
  
• parse configuration: class parse_arguement.py
  
• training file: train.py
  
• V1 model: R_Unet_v1.py
  
• V2 model: R_Unet_ver_2.py
  
• V4 model: R_Unet_ver_4.py
  

to train v1 model: python3 train.py config 
to train other model: python3 train_v2.py config 

Our Model Architecture

Current we are working on a better model using convolution lstm, name as runet_v2

• model v1:
  
  
  

• model v2:
  
  
  

• model v4:
  

Some result

prediction: Ground truth:
mask prediction
prediction: Ground truth:

References

[1] Stochastic Adversarial Video Prediction, CVPR 2018
[2] High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks, NeurIPS 2019
[3] convLSTM - The convolution lstm framework used

Hsu Mu Chien, Watanabe Lab, Department of Fundamental Science and Engineering, Waseda University.