This video and audio have been created with neural networks that have not seen any data. The video samples from a ProGAN and shows interpolations through the latent and parameter space of the generator. The audio samples from an LSTM to produce a midi and slowly interpolates through the parameter space to achieve variation.
Running the Code
This was created with PyTorch and is written assuming access to a CUDA device, but can be modified to be run without CUDA.
- Create frames with
- Create video with
ffmpeg -framerate 20 -i frames/%06d.png -c:v libx264 -pix_fmt yuv420p -crf 23 vid.mp4
- Create MIDI with
- Synthesize the MIDI with audio software like
GarageBandor with a Python library like pretty_midi.
- Add the audio to the video with
ffmpeg -i vid.mp4 -i audio.mp3 -c:v libx264 -c:a copy vid.audio.mkv
This portion uses code from warmspringwinds/pytorch-rnn-sequence-generation-classification and the midi library from here.
Thanks to Roger Iyengar for helpful suggestions.
Unless otherwise noted, the code in
in the public domain. The code in
midi remains under the original licensing.