PathGan: Visual Scan-path Prediction with Generative Adversarial Networks
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
figs
src
weights
.DS_Store
.gitignore
README.md
requirements-no-gpu.txt
requirements.txt

README.md

PathGan: Visual Scan-path Prediction with Generative Adversarial Networks

Marc Assens Kevin McGuinness Xavier Giro-i-Nieto Noel O'Connor
Marc Assens Kevin McGuinness Xavier Giro-i-Nieto Noel O'Connor

A joint collaboration between:

logo-insight logo-dcu logo-gpi
Insight Centre for Data Analytics Dublin City University (DCU) UPC Image Processing Group

Abstract

We introduce PathGAN, a deep neural network for visual scanpath prediction trained on adversarial examples. A visual scanpath is defined as the sequence of fixation points over an image defined by a human observer with its gaze. PathGAN is composed of two parts, the generator and the discriminator. Both parts extract features from images using off-the-shelf networks, and train recurrent layers to generate or discriminate scanpaths accordingly. In scanpath prediction, the stochastic nature of the data makes it very difficult to generate realistic predictions using supervised learning strategies, but we adopt adversarial training as a suitable alternative. Our experiments prove how PathGAN improves the state of the art of visual scanpath prediction on the iSUN and Salient360! datasets.

Publication

Find the pre-print version of our work on arXiv.

Image of the paper

Please cite with the following Bibtex code:

@inproceedings{Assens2018pathgan,
title={PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks},
author={Marc Assens, Xavier Giro-i-Nieto, Kevin McGuinness, Noel E. O'Connor},
journal={ECCV Workshop on Egocentric Perception, Interaction and Computing (EPIC)},
year={2018}
}

You may also want to refer to our publication with the more human-friendly Chicago style:

Marc Assens, Xavier Giro-i-Nieto, Kevin McGuinness, Noel E. O’Connor. “PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks”, ECCV Workshop on Egocentric Perception, Interaction and Computing (EPIC), 2018.

Models

The model is composed by two deep neural networks, the generator and the discriminator, whose combined efforts aim at predicting a realistic scanpath from a given image.

Model Architecture: architecture-fig

Examples

We provide examples of predicted object sequences for two datasets.

1. iSun

ex-isun

2. Salient360!

ex-360

The big dot indicates the first fixation of the scanpath.

Software frameworks: Keras

The model is implemented in Keras, which at its time is developed over Tensorflow.

Acknowledgements

We especially want to thank our technical support team:

AlbertGil-photo
Albert Gil
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GeForce GTX Titan Z and Titan X used in this work. logo-nvidia
The Image Processing Group at the UPC is a SGR14 Consolidated Research Group recognized and sponsored by the Catalan Government (Generalitat de Catalunya) through its AGAUR office. logo-catalonia
This work has been developed in the framework of projects TEC2013-43935-R and TEC2016-75976-R, financed by the Spanish Ministerio de Economía y Competitividad and the European Regional Development Fund (ERDF). logo-spain