Skip to content
Switch branches/tags
Go to file

Latest commit


Git stats


Failed to load latest commit information.

MindreaderNet (Mr. Net)

This is the repository for our NeurIPS19 paper:

A Self Validation Network for Object-Level Human Attention Estimation

If you find this repository helpful, consider citing our paper:

  title={A Self Validation Network for Object-Level Human Attention Estimation},
  author={Zhang, Zehua and Yu, Chen and Crandall, David},
  booktitle={Advances in Neural Information Processing Systems},

If any other mentioned work here also helps, please do cite them as well. Thanks!

To begin with...

cd your_working_directory
git clone

Environments and Dependencies

Our model is implemented base on Keras v2.1.5 and Tensorflow v1.10.0

The CUDA version is 9.2

It runs on Linux and we haven't tested it on other OS.

It also uses functions from ssd_keras

git clone

Make sure ssd_keras is put in the same parent directory as MindreaderNet-Mr.-Net-

Data Preparation

In our paper two datasets are used: ATT and EPIC-Kitchens

In this repository we give an example on running our code on EPIC-Kitchens.

ATT cannot be made directly available to public due to human ethic and privacy issues. Please contact the authors (also us!) for the access to it.

To download EPIC-Kitchens, use the script provided by the authors:

sh download_rgb your_working_directory/MindreaderNet-Mr.-Net-

It will download all the rgb frames to your_working_directory/MindreaderNet-Mr.-Net-/EPIC_KITCHENS_2018. Uncompress each .tar file by:

python --option extractepic

Download the annotations of EPIC_train_object_labels, EPIC_train_object_action_correspondence and EPIC_train_action_labels to your_working_directory/MindreaderNet-Mr.-Net- (We've already included them in the repository)

Split the data into two sets for training and testing (This will generated two .txt files --- epic_train.txt and epic_test.txt --- listing sample information. Note that since we choose 90% of the samples randomly, you will get different training and testing lists if you generate them yourself. We've already included the list files we used in the repository):

cd your_working_directory/MindreaderNet-Mr.-Net-
python --option processepic

Prepare the labels (Four files --- epic_train_onehot.h5, epic_train_labels.h5, epic_test_onehot.h5, epic_test_labels.h5 -- will be generated):

python --option encodeepic

Load the samples into a big binary file (formatted in .h5) for faster reading during training and testing (Two files will be generated respectively for the training and testing set. By default, both will be put in your_working_directory/MindreaderNet-Mr.-Net-. They take several TBs so make sure you have enough disk space.):

python --option preloadepic

To generate the optical flow maps, we use PWC. Particularly, we use this implementation. Flow maps should be generated corresponding to frames stored in the .h5 file and also stored as a .h5 file in the directory of your_working_directory/MindreaderNet-Mr.-Net-. For example, the dataset of frames in the training set has shape of (N, H, W, 3), then the dataset of flows in the training set should correspondingly have shape of (N, H, W, 2).

You have the option to download the flow maps generated by the EPIC-Kitchen team with this script. We haven't tested with these flows and cannot gurantee the performance.

So far we've done with the data preparation.

For more details about how we process our data, please refer to Section 3.5 of our paper.


Download the pretrained weights (I3D is pretrained on ImageNet and Kinetics, VGG16 is pretrained on ImageNet): I3D rgb stream, I3D flow stream, VGG16

We also provide the weight we obtained by pretraining the spatial branch: pretrained_spatial_weight. Simply download it and put it in your_working_directory/MindreaderNet-Mr.-Net-.

To train the model

python --option trainepic --batch_size B --steps_per_epoch S --where_help_inside --what_help_inside --softArgmax

We train our model on a single Titan Xp with batch size set to 4. You can specify your own batch size by setting --batch_size your_batch_size. Also, the code supports multi-gpu training by setting your own --num_gpu (we haven't tested it thoroughly).

When training the model, make sure you specify --softArgmax so that the soft version of what→where validation is used in order to have the gradients backpropagate properly. More detains can be found in Section 3.2 of our paper.

--where_help_inside and --what_help_inside tell the model whether to use the where→what validation or what→where validation respectively.

By default, a weight file epic_mindreader.h5 will be saved to your_working_directory/MindreaderNet-Mr.-Net-.

Other arguments are also available and please check the code for more details.


python --option testepic --where_help_inside --what_help_inside --load_exist_model --load_from path_to_your_your_weight

Make sure batch_size is set to 1 (or just leave it with its default value of 1). Only single GPU testing is supported. We also provided the weight file we reported in our paper. Download it and put it in your_working_directory/MindreaderNet-Mr.-Net-, then

python --option testepic --where_help_inside --what_help_inside --load_exist_model --load_from ./epic_acc312.h5


This is the repository for our NeurIPS19 paper of Mindreader Net (Mr. Net)




No releases published


No packages published