Can neural networks help bring a colourful life back to old black and white photos? Let's check it out!
Note: This is a living document, summarising my research on the topic of photo colorization. It is far from its final version, but I would be more than happy, if it helps other people’s work. Feel free to send me comments, suggestions, or pull request changes
Inspired, by Emil Wallner's Medium post titled "How to colorize black & white photos with just 100 lines of neural network code", I decided to delve into the field myself, and see what I might end up with.
Design and train a machine learning model, which by presenting it with a black-and-white photo, will result in a decently colorised version of the same. The goal here is not really to achieve a factual correctness (e.g the model decides that a red car should be blue), but a level of satisfactory fidelity. Moreover, from a purely artistic perspective, it might be interesting to see, even if wrongly coloured, if the end results add an unexpected aesthetic value.
As it was reported by [1], when trained with a large set of images, the model becomes so generic that, in fact, colorised results come out mostly brownish, or very desaturated, as if a sepia filter has been applied on the photo. I have reported the same in my first versions of my experiment, while I still used a simple generic CNN model. As it would turn out, this might have been caused by the choice of a loss function. I used a mean squared error (mse) loss. This is a good default loss function, when solving regression problems, where the input space is continuous. As research points out, however, the ab space is discrete. It can be confined to a number of 313 classes, which changes the nature of the problem from regression to classification. This requires a choice of a more appropriate loss function.
[1] Wallner, E.: "How to colorize black & white photos with just 100 lines of neural network code" Medium.com, Oct 29, 2017 - https://medium.freecodecamp.org/colorize-b-w-photos-with-a-100-line-neural-network-53d9b4449f8d
[2] Dahl, R.: "Automatic Colorization" Tinyclouds.org, Jan, 2016 http://tinyclouds.org/colorize/
Cheng, Z., Yang, Q., Sheng, B.: "Deep colorization". In: Proceedings of the IEEE International Conference on Computer Vision. (2015) 415–423
Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." European Conference on Computer Vision. Springer, Cham, 2016.
Zhang, Richard, et al. "Real-time user-guided image colorization with learned deep priors." arXiv preprint arXiv:1705.02999 (2017).