GFIE: A Dataset and Baseline for Gaze-Following from 2D to 3D in Indoor Environments

This official repository is the pytorch implementation of our CVPR 2023 work: GFIE: A Dataset and Baseline for Gaze-Following from 2D to 3D in Indoor Environments

Prerequisite

1. Clone our repo

git clone https://github.com/nkuhzx/GFIE

2. (optional) Use the .yaml file to re-create the environment

conda env create -f gfie.yaml
conda activate gfie.yaml

3. Set the available GPU in inference.py and main.py

os.environ['CUDA_VISIBLE_DEVICES'] = ID

ID is available gpu id.

Dataset Preparation

Download from Google Drive

1. Please download the GFIE dataset and CAD120 dataset from the Google Drive.
2. Unzip rgb.zip and depth.zip into corresponding folders.

Download via shell script

1. If the Google Drive is not available, you can run a script to download the GFIE dataset

2. run the download_gfie.sh/ download_cad120.sh script

   sh download_gfie.sh
   sh download_cad120.sh

3. The file structure should be as follows:

   ├── GFIE_dataset/
   |   ├── rgb/
   |   |   ├── train/
   |   |   |   ├── scene1/ # scene id
   |   |   |   |   └── '*.jpg' # n frame jpg`
   |   |   |   └── ...
   |   |   ├── valid/
   |   |   |   └── ...
   |   |   ├── test/
   |   |   |   └── ...
   |   ├── depth/
   |   |   ├── train/
   |   |   |   ├── scene1/ # scene id
   |   |   |   |   └── '*.npy' # n frame npy`
   |   |   ├── valid/
   |   |   |   └── ...
   |   |   ├── test/
   |   |   |   └── ...
   |   ├── CameraKinect.npy
   |   ├── train_annotation.txt
   |   ├── valid_annotation.txt
   |   └── test_annotation.txt
   ├── CAD120_dataset/
   |   ├── rgb/
   |   |   ├── D1S1A001/
   |   |   |   └── 'RGB_*.png' # n frame png`
   |   |   └── ...
   |   ├── depth/
   |   |   ├── D1S1A001/
   |   |   |   └── 'Depth_*.png' # n frame png`
   |   |   └── ...
   |   ├── CameraKinect.npy
   └── └── annotation.txt

   Note: The decompressed file is about 350 GB, please check the capacity of your hard disk to ensrure that the dataset can be stored.

4. Then you need to modify the address of the configuration (cad120evaluation.yaml | gfiebenchmark.yaml)

   gfiebenchmark.yaml

   DATASET:
     root_dir: "YOUR_PATH/GFIE_dataset"

   cad120evaluation.yaml

   DATASET:
     root_dir: "YOUR_PATH/CAD120_dataset"

YOUR_PATH is the root path of GFIE_dataset and CAD120_dataset.

Getting start

Training

After all the prerequisites are met, you can train the GFIE baseline method we proposed in the paper.

1. Set the path STORE_PATH to save the model file in the gfiebenchmark.yaml

   TRAIN:
     store: "STORE_PATH"

2. Download the pre-trained model weights to PATH, and then set the path of pre-trained weights in gfiebenchmark.yaml

   MODEL:
     backboneptpath: "PATH/ptbackbone.pt"

3. Then run the training procedure

   python main.py

Evaluation

1. Set the absolute path of the model weight cpkt_PATH in the cad120evaluation.yaml and gfiebenchmark.yaml

   OTHER:
     cpkt: "cpkt_PATH"

2. Run the inference program and the evaluation results will be displayed in the termainal.

   # evaluation on GFIE dataset
   python inference.py --mode gfie
   
   # evaluation on CAD120 dataset
   python inference.py --mode cad120

Model weights

We also provide the model weights for evaluation.

gfiemodel.pt.tar

Citation

If you fine our dataset/code useful for your research, please cite our paper

@InProceedings{Hu_2023_CVPR,
    author    = {Hu, Zhengxi and Yang, Yuxue and Zhai, Xiaolin and Yang, Dingye and Zhou, Bohan and Liu, Jingtai},
    title     = {GFIE: A Dataset and Baseline for Gaze-Following From 2D to 3D in Indoor Environments},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {8907-8916}
}

Acknowledgements

We would like to thank Eunji Chong for her work publised on CVPR 2020 and others that have contributed to gaze-following.