(HackPrinceton 2018) Game asset style transfer for easily extensible fan mods and creations.
Style Transfer is a relatively new subfield in machine learning that allows for the quantification of image "styles," typically thought of as "brush patterns" or other local geometry in an image. By training a neural network to quantify such styles, it's possible to "transfer" the style of one image onto another, creating a variety of interesting results, including "Deep Fakes' and other forms of content.
However, while these algorithms have been adapted to video content, including some working demos on live video feeds, such techniques often remain computationally infeasible, requiring anywhere from seconds to minutes to process a single frame on typical hardware. In this project, we offer true real-time style transfer through pre-computed "style transferred" modified game assets, rendering such assets in real time with the original game engine.
Base Style:
To run our modded first level of "Paper Mario: The Thousand Year Door":
- Install Dolphin Emulator
- Clone this repo, and copy "demo/modded_textures" to "Your Documents Folder/Dolphin Emulator/Load/Textures"
- Under Dolphin/Graphics Settings/Advanced, tick "Load Custom Textures"
- Run TTYD from Dolphin
(For recomputing results/changing paramters/textures, etc.) Running this project depends on the following packages (available through conda or pip)
- torch, torch.nn, numpy
- torch.optim
- PIL, PIL.Image,
- torchvision.transforms
- torchvision.models
- Tick "dump textures" under Dolphin/Graphics/Advanced. Then play through the desired level(s) that you wish to mod/style transfer onto.
- Run
python run_this.py
in your terminal of choice.
By default, this uses the textures listed in demo/dumped_textures to determine which textures to replace/style on, searching through all the files listed in demo/base_textures and replacing them in-place. The style to be used is computed from "demo/style.jpg."
After doing this, follow the instructions for running the demo.
- Style transferring 3D assets, using UV projection information for preservation of spatial features
- Application to other games/engines
- Other neural style transfer algorithms
Thanks to The Dolphin Community TTYD Texture pack, Alexis Jacq et. al, and the Dolphin Team.