This is the repository for our SDHacks project Re:Art, including the website design, the back-end server backup, and the algorithm used for automating image processing and style transfer using Convolutional Neural Network
- Devpost: https://devpost.com/software/re-art-f92i1b
- Media Exposure: https://cse.ucsd.edu/about/news/cse-sophomores-take-top-prize-sd-hacks-2016
Re: Art is an awards-winning web app that implemented convolutional neural network to allow transforming an uploaded video into animated artwork as if it was drawn by artist like Van Gogh. We built this app at SD Hacks 2016, making it the first video style transfer web app on the internet, and showered with prizes including Best of Show, Best Use of AWS, and Most Commercializable Hack. Check out our project on the Devpost to see how it works!
Source of Style:
Output of user-uploaded video:
Inspiration and Background
When we were developing this webapp, Prisma just had its debut and won a lot of popularity with its style transfer filter for images, but its functionalities are only limited to processing images instead of videos at that time (09/2016). We, therefore, wanted to make the video artistic as well, so we created this project Re:Art. Interestingly enough, couple weeks after our application was published in the SD Hacks, Prisma released the video style transfer feature for their application as well.
How does it work:
We set up a comprehensive back-end still image stylization service based on the AWS and G2 instance in the EC2 service. We achieve the non-linear video processing result through Open CV (Open Source Computer Vision), which enable the web app Re:Art allowing users to process a self-designed animated artwork.
Read user input: When a user uploads a video and chooses a picture style, we will read in the video and style they choose, and we will decide the frame number based on how long we want the output gif file to be.
Transform the video: We pick up the images based on the frame rate, and then crop the images into the size with the ratio 8.5:11, which is roughly 300px * 388px. Also, we uses advanced compression algorithm to make the image further smaller without losing much of its original quality. Due to the undesirable efficiency of the Neural Network Framework, we came up with several ways to increase our efficiency use multi-parallel subprocess to transform the image simultaneously, and use the resources of the server to its maximum. Because we depend on GPU rather than CPU to have a better performance, we chose the AWS instance that has more GPUs. Because the feature of the Open Source Torch Framework is based on Lua, we manipulate the iteration time on each image to achieve an effect of transition, giving user a feeling of more dynamic stylized images.
Output Gif: We store the image to an array and restructure them to form a gif, which is finally presented to the user.
- Wanze (Russell) Xie - (https://github.com/russellxie7)
- Jianhan Xu - (https://github.com/Jorx11)
- Chen Yang - (https://github.com/azureric)
- Zhuojun Chen
- Best of Show by SDHacks Committee
- Best Use of AWS by Amazon.com Inc.
- Most Commercializable Hack presented by the UCSD Office of Innovation and Commercialization