This is a package that wraps ResMem. This is a residual neural network that estimates the memorability of an image, on a scale of 0 to 1.
To install from PyPI:
pip install resmemThe package contains two objects, ResMem itself, and a preprocessing transformer function built on torchvision.
from resmem import ResMem, transformer
model = ResMem(pretrained=True)The transformer is designed to be used with pillow.
from PIL import Image
img = Image.open('./path/to/image') # This loads your image into memory
img = img.convert('RGB')
# This will convert your image into RGB, for instance if it's a PNG (RGBA) or if it's black and white.
model.eval()
# Set the model to inference mode.
image_x = transformer(img)
# Run the preprocessing function
prediction = model(image_x.view(-1, 3, 227, 227))
# For a single image, the image must be reshaped into a batch
# with size 1.
# Get your prediction!For more advanced usage, see the sample.py file in this repository.
Historically, the next big advancement in using neural networks after AlexNet, the basis for MemNet, was ResNet. This allowed for convolutional neural networks to be built deeper, with more layers, without the gradient exploding or vanishing. Knowing that, let's try to include this in our model. What we will do is take a pre-trained ResNet, that is the whole thing, not just the convolutional features, and add it as an input feature for our regression step. The code for this is here.
For the following discussion, while ResMem is initialized to a pre-trained optimum, I have allowed it to retrain for our problem. The thought is that given a larger set of parameters the final model should be more generalizable. Using weights and biases, we can run a hyperparameter sweep.
Here we can see much higher peaks, reaching into the range of 0.66-0.67! All of these runs were both trained and validated on a dataset that was constructed from both MemCat and LaMem databases.
If you want to fiddle around with the raw github sourcecode, go right ahead. But be aware that we use git lfs for the model storage. You'll have to install the package at https://git-lfs.github.com/ and then run:
git lfs install
git lfs pullto get access to the full model. Also note that the tests.py file that's included with this repository references a
database that we, at this time, do not have permission to distribute, so consider that test file to be a guide rather
than a hard-and-fast test.
@inproceedings{ResMem2021,
title = {Embracing New Techniques in Deep Learning for Predicting Image Memorability},
author = {Needell, Coen D. and Bainbridge, Wilma A.},
booktitle = {Proceedings of the Vision Sciences Society, 2021},
year = {2021},
publisher = {ARVO Journals},
url = {https://www.coeneedell.com/publication/vss2021/vss2021.pdf}
}