Inferring Attention Shift Ranks of Objects for Image Saliency [CVPR 2020]
Authors:
Avishek Siris, Jianbo Jiao, Gary K.L. Tam, Xianghua Xie, Rynson W.H. Lau
- PDF: Paper
- Supplemental: Supplementary Material
We propose a new large-scale saliency rank dataset from attention shift, motivated by psychological and behavioural studies, and justified by our user study.
Abstract
Psychology studies and behavioural observation show that humans shift their attention from one location to another when viewing an image of a complex scene. This is due to the limited capacity of the human visual system in simultaneously processing multiple visual inputs. The sequential shifting of attention on objects in a non-task oriented viewing can be seen as a form of saliency ranking. Although there are methods proposed for predicting saliency rank, they are not able to model this human attention shift well, as they are primarily based on ranking saliency values from binary prediction. Following psychological studies, in this paper, we propose to predict the saliency rank by inferring human attention shift. Due to the lack of such data, we first construct a large-scale salient object ranking dataset. The saliency rank of objects is defined by the order that an observer attends to these objects based on attention shift. The final saliency rank is an average across the saliency ranks of multiple observers. We then propose a learning-based CNN to leverage both bottom-up and top-down attention mechanisms to predict the saliency rank. Experimental results show that the proposed network achieves state-of-the-art performances on salient object rank prediction.
Installation
The code is based on the Mask-RCNN implementation by matterport, https://github.com/matterport/Mask_RCNN. Please follow and install the requirements they list. Additionally, install pycocotools.
Dataset
Download our dataset from google drive.
Training
The current implementation and results are based on pre-computing backbone and object features, then training the rest of the saliency rank model seperately.
- Pre-train backbone for salient object detection (binary, no ranking). Download pre-trained COCO weights (mask_rcnn_coco.h5) from matterport, https://github.com/matterport/Mask_RCNN/releases. Create a new "weights/" folder in the root directory, then put the weight file inside it. Set data paths and run:
python obj_sal_seg_branch/train.py
- Pre-compute backbone and object features of GT objects for "train" and "val" datasplits. Set data paths and run twice for "train" and "val":
python pre_process/pre_process_obj_feat_GT.py
- Finally train the saliency rank model. Set data paths and run:
python train.py
Testing
Similarly to training, we need to pre-compute features.
- Perform object detection. You can download the pre-trained weights from google drive. Set data paths and run:
python pre_process/object_detection.py
- Pre-compute corresponding object features. Set data paths and run:
python pre_process/pre_process_obj_feat.py
- Test the saliency rank model. You can download the weights of the trained model from google drive. Set data paths and run:
python evaluation/predict.py
This will generate predictions and save into files.
- Generate predicted Saliency Rank Maps (rank based on grayscale value). Set data paths and run:
python evaluation/generate_saliency_map.py
Evaluate
To evaluate MAE and Salient Object Ranking (SOR) scores, set data paths and run:
python evaluation/evaluate.py
For further mAP evaluation, set data paths and run:
python evaluation/evaluate_map.py
Download Pre-computed Features
You can download pre-computed features for training and testing the model from google drive.
Results
You can download predicted Saliency Rank Maps by our trained model from google drive.
User Study Data
You can find results from our user study at google drive.
Citation
@InProceedings{Siris_2020_CVPR,
author = {Siris, Avishek and Jiao, Jianbo and Tam, Gary K.L. and Xie, Xianghua and Lau, Rynson W.H.},
title = {Inferring Attention Shift Ranks of Objects for Image Saliency},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}