Ngx - Neural network based visual generator and mixer
Switch branches/tags
Nothing to show
Clone or download
Latest commit e314cbf Oct 19, 2018
Permalink
Failed to load latest commit information.
Assets FPS display Oct 14, 2018
Packages Add UI Oct 14, 2018
ProjectSettings update models Oct 12, 2018
.gitattributes initial commit Oct 2, 2018
.gitignore initial commit Oct 2, 2018
.gitmodules Add UI Oct 14, 2018
README.md Update README.md Oct 19, 2018

README.md

Ngx - Neural network based visual generator and mixer

Ngx is an attempt at utilizing a neural network for VJing. It implements pix2pix (image-to-image translation with cGAN) as an ad-hoc next-frame prediction model that is trained with pairs of consecutive frames extracted from a video clip, so that it can generate an image sequence for an infinite duration just by repeatedly feeding frames back. It also has functionality for mixing (crossfading) two pix2pix models that gives unexpected variation and transition to generated video.

gif gif

(The gif on the right is from a tweet by @chaosgroove.)

Vimeo - Ngx demonstration

Pre-built executables

You can download a pre-built executable from Releases page. It contains pre-trained pix2pix models with Beeple's VJ clips.

Model Name Original clip Vimeo link
Beeple 1 FIBER OPTICAL https://vimeo.com/238083470
Beeple 2 redgate.v1 https://vimeo.com/106577855
Beeple 3 p-crawl https://vimeo.com/116330714
Beeple 4 exhaust https://vimeo.com/95513505
Beeple 5 quicksilver https://vimeo.com/153493904
Beeple 6 MOONVIRUS https://vimeo.com/155970396
Beeple 7 TENDRIL https://vimeo.com/158477477
Beeple 8 shifting pains https://vimeo.com/48818536

The original video clips are licensed under a Creative Commons Attribution license (CC-BY 3.0). These pre-trained models are also attributed to the original author.

How to control

Ngx can be controlled with the on-screen controller or a MIDI controller.

Name Description MIDI CC
Mix Crossfading between model 1 and 2 77
Noise 1 Noise injection (low density) 78
Noise 2 Noise injection (medium density) 79
Noise 3 Noise injection (high density) 80
Name Description MIDI CC
Feedback Feedback rate 81, 82
FColor False color effect 83
Hue Shift Amount of hue shift 55, 56
Invert Color inversion 84
Name MIDI notes
Model Select 1 41, 42, 43, 44, 57, 58, 59, 60
Model Select 2 73, 74, 75, 76, 89, 90, 91, 92

The default MIDI mapping is optimized for Novation Launch Control XL: You can change model with the track buttons and control parameters with the track faders. The 7th and 8th panning knobs are used to tweak the hue shift values.

How to open the project in Unity

This repository uses Git submodules to manage dependent packages. To check these submodules out, execute git submodule init && git submodule update from the command line, or specify --recursive option when initially cloning.

After cloning the repository, .pict (pix2pix weight data) files should be manuyally downloaded and copied into the Streaming Assets directory because these files are excluded from the repository to save bandwidth and storage usage. Please download the pict file package and extract it into Assets/StreamingAssets.

How to train pix2pix models

Prerequisites: Basic experience on a machine learning framework. It's recommended to have a look into the pix2pix tutorial from Machine Learning for Artists if you haven't tried pix2pix or any similar GAN model.

You can train your own pix2pix model using pix2pix-tensorflow. I personally recommend using Google Colaboratory for training. It's easy to set up, handy to use and cost effective (it's free!).

The followings are Colaboratory notebooks I used to train the default models. These notebooks are written to use Google Drive as a dataset storage.

Frequently asked questions

What are the hardware recommendations?

I used a Windows system with GeForce GTX 1070 for a live performance. It runs at around 21 fps; this might be the minimum practical spec for the system.