next_frame_prediction is a tool dedicated to creating video by learning to predict a frame from the previous one, and apply it recursively to an initial frame.
Magenta working environement
Please setup the Magenta developement environement first, following the main documentation here.
This tools need some additional dependencies to run.
You'll need to first install
You will also need to copy a modified version of the
pix2pix-tensorflow library using the following commands:
curl -LO https://github.com/dh7/pix2pix-tensorflow/archive/0.1.tar.gz tar -xvf 0.1.tar.gz
The goal of this tool is to create video frame by frame. From a first arbitrary frame, the algorithm will predict a plausible next one. Then from this new frame, the algorithm will predict another one, and repeat that until it creates a full video.
Predicting a plausible next frame for a video is an open problem. Several publications have tried to solved it. The approach here is less scientific and was inpired by this tweet from Mario Klingemann: It predicts the next frame from the previous one only, and it tends to diverge into something unrealistic.
The algorithm in this folder implements a technique to prevent the algorithm from diverging as you can see in this video.
To learn how to predict the next frame, the algorithm needs a video as a source (the training set). The algorithm splits the video in frames, and asks pix2pix to learn to predict frame n+1 from frame n. The algorithm then generates some predictions (that diverge) and creates a new training set from it. The algorithm then refines the prediction by asking pix2pix to learn to predict a real frame from the predicted (divergent) ones.
To try you'll need a video.mp4 file as the source and two folders:
mkdir working_folder mkdir output_folder cp <your_video> working_folder/video.mp4
Then you'll need to launch the main script from this directory:
./create_video.sh \ $(pwd)/working_folder \ 20 \ 500 \ $(pwd)/output_folder
This script will extract frames from the video. Then, it will train pix2pix 20 times and generate a video made of 500 frames at each step.
The script will ask you questions, and the first time you launch it, you should answer yes to all of them. If you interupt the script and want to restart it, you can answer yes or no to decide which part of the process you want to keep or restart from scratch.
There are some constraints that you may want to change by editing the script if the default values don't give the optimal result.