Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model. IEEE Transactions on Image Processing (2018)
Python
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
figs Initial commit Mar 17, 2017
predictions Initial commit Mar 17, 2017
sample_images Initial commit Mar 17, 2017
LICENSE.md Update LICENSE.md Feb 28, 2017
README.md Update README.md Jul 9, 2018
attentive_convlstm.py Initial commit Mar 17, 2017
config.py Initial commit Mar 17, 2017
dcn_resnet.py Initial commit Mar 17, 2017
dcn_vgg.py Initial commit Mar 17, 2017
gaussian_prior.py Initial commit Mar 17, 2017
main.py Update main.py Aug 3, 2017
models.py Update models.py Aug 3, 2017
utilities.py Initial commit Mar 17, 2017

README.md

Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model

This repository contains the reference code for computing SAM (Saliency Attentive Model) saliency maps based on the following paper:

Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, Rita Cucchiara
Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model
IEEE Transactions on Image Processing, 2018

Please cite with the following Bibtex code:

@article{cornia2018predicting,
  author = {Cornia, Marcella and Baraldi, Lorenzo and Serra, Giuseppe and Cucchiara, Rita},
  title = {{Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model}},
  journal = {IEEE Transactions on Image Processing},
  year = {2018}
}

The PDF of the article is available at this link.

Abstract

Data-driven saliency has recently gained a lot of attention thanks to the use of Convolutional Neural Networks for predicting gaze fixations. In this paper we go beyond standard approaches to saliency prediction, in which gaze maps are computed with a feed-forward network, and we present a novel model which can predict accurate saliency maps by incorporating neural attentive mechanisms. The core of our solution is a Convolutional LSTM that focuses on the most salient regions of the input image to iteratively refine the predicted saliency map. Additionally, to tackle the center bias present in human eye fixations, our model can learn a set of prior maps generated with Gaussian functions. We show, through an extensive evaluation, that the proposed architecture overcomes the current state of the art on two public saliency prediction datasets. We further study the contribution of each key components to demonstrate their robustness on different scenarios.

sam-fig

Requirements

  • Theano 0.9.0
  • Keras 1.1.0, configured for using Theano as backend
  • OpenCV 3.0.0

Note: Be sure to have "image_dim_ordering": "th" and "backend": "theano" in your keras.json file.

Usage

We built two different versions of our model: one based on the VGG-16 (SAM-VGG) and the other based on the ResNet-50 (SAM-ResNet). It is possible use both versions of SAM by changing the version variable in the config.py file (set version = 0 for SAM-VGG or version = 1 for SAM-ResNet).

To compute saliency maps using our pre-trained model:

python main.py test path/to/images/folder/

where "path/to/images/folder/" is the path of a folder containing the images for which you want to calculate the saliency maps.

To train our model from scratch:

python main.py train

It is also necessary to set parameters and paths in the config.py file.

Note: To train our model, both binary fixation maps and groundtruth density maps are required. The current code for loading binary fixation maps supports the format used in SALICON (.mat files). If you want to train our model with other datasets, be sure to appropriately change the loading functions.

Pretrained Models

Download one of the following pretrained models and save it in the code folder:

Precomputed Saliency Maps

We provide saliency maps predicted by SAM-VGG and SAM-ResNet for three standard datasets (SALICON, MIT1003 and CAT2000):

Contact

For more datails about our research please visit our page.

If you have any general doubt about our work, please use the public issues section on this github repo. Alternatively, drop us an e-mail at marcella.cornia@unimore.it or lorenzo.baraldi@unimore.it.