An experiment with learned image filters for small GLSL shaders. Based on the μNCA model.
This repository is split into two parts:
- renderer, a Windows C++ OpenGL renderer, and
- training, a Jupyter notebook for training the filters with PyTorch.
Unfortunately the first runs only on Windows and the second one on Linux. I've used Windows 10 with WSL myself to run everything.
Based on Alexander Mordvintsev and Eyvind Niklasson's research on neural cellular automata (NCA) published in these three papers:
- 2020 Growing Neural Cellular Automata
- 2021 Self-Organising Textures
- 2021 μNCA: Texture Generation with Ultra-Compact Neural Cellular Automata
Both the model and the training objective have been modified for this experiment. The model is now running two smaller CAs in a row and starts from a "feature" image instead of pure noise. See the included notebook for more details.
I used four channel images to make it easier to implement on the C++ side. With more channels more elaborate textures can be generated, as shown in papers linked above.
The loss function is a combination of a "content" and "style" losses just like in style transfer tasks. Increasing the content loss weight makes results sharper but with less contrast:
Download a short video demonstration of the final filter: processing.mp4.
It may be possible to get similar results with a small convolutional neural network or a hand-tweaked bilateral filter.
It's pretty arduous but all the pieces are there:
- Export a raw .data image and a matching PNG with renderer
- Crop the PNG to your liking and encode its x and y in the filename like
img_crop_x_y.png. - Stylize your PNG in whatever way you wish.
- Put the two files in training/images and add their paths to the notebook script.
- Run the notebook and wait until it prints the GLSL code at the end.
- Copy the GLSL snippets to
draw.fragandpresent.fragin the renderer/blossom directory. - Compile & run the renderer project