Neural Architecture Search of SPD Manifold Networks

Accepted to IJCAI 2021

Authors: Rhea Sukthanker, Zhiwu Huang, Suryansh Kumar, Erik Goron Endsjo, Yan Wu and Luc Van Gool

Paper: https://www.ijcai.org/proceedings/2021/0413.pdf

Abstract

In this paper, we propose a new neural architecture search (NAS) problem of Symmetric Positive Definite (SPD) manifold networks, aiming to automate the design of SPD neural architectures. To address this problem, we first introduce a geometrically rich and diverse SPD neural architecture search space for an efficient SPD cell design. Further, we model our new NAS problem with a one-shot training process of a single supernet. Based on the supernet modeling, we exploit a differentiable NAS algorithm on our relaxed continuous search space for SPD neural architecture search. Statistical evaluation of our method on drone, action, and emotion recognition tasks mostly provides better results than the state-of-the-art SPD networks and traditional NAS algorithms. Empirical results show that our algorithm excels in discovering better performing SPD network design and provides models that are more than three times lighter than searched by the state-of-the-art NAS algorithms.

Overview

1. Installation & Dependencies
2. Prepration
   1. Directories
   2. Data
3. Training
   1. Searching + Training and Evaluation
4. Results
5. Contact
6. How to Cite

1. Installation & Dependencies

The code has been tested with Python 3.6 and Pytorch 1.0.1.

To install the dependencies:

conda create --name py37 python=3.7
pip install -r requirements.txt

2. Preparation

2.1. Directories

Our code is based upon a DARTS style implementation. The code for darts is available here darts. Further we follow SPDNetBN for implementation of the batch normalization layers,

The root directory contains the models and the training files.

The root folder contains a folder called data where all the datasets are saved. The datasets can be downloaded from the links below.

The Code folder contains the Emotion-FAN model's code which is the pretrained model used in our experiments for emotion-recognition (AFEW). Download the pretrained Emotion-FAN model from its repository and place it in the root directory.

2.2. Data

Data preparation is done based on the setup you are using. The code is adapted to three setups:

1. Radar(Drone Recognition): 3000 SPD datapoints distributed among 3 classes (size 20 × 20)
2. HDMO5(Action Recognition): 2083 SPD datapoints distributed among 117 classes (size 93×93)
3. AFEW(Emotion Recognition): 1345 SPD datapoints distibuted among 7 classes (Image size 100x100)

We use the Radar and HDM05 datasets directly from the SPDNetBN. While for the AFEW2014 we use the raw images below. Below you can find and download the datasets used in our experiments:

• RADAR: This dataset’s synthetic setting is composed of radar signals distributed into 3 classes, where each signal is split into windows of length 20, resulting in a 20x20 covariance matrix for each window.
• HDM05: The dataset has 2083 SPD matrices distributed amongst 117 action classes
• AFEW:This dataset has 1345 videos of facial expressions classified into 7 distinct classes i.e. angry, fear, disgust, surprise, neutral, happy and sad. You can get the AFEW dataset by ask the official organizer Shreya Ghosh and Emotiw2014 . Unzip the dataset and place it in the data/ folder

Note the data/ folder contains all the datsets and is placed in the root folder.

3. Training and Evaluation

To launch the search followed by training and evaluation for the Radar datset:

python -u train_spd_search_radar_sparsemax.py --unrolled --epochs 100  "$@"
python -u train_spd_radar.py --arch radar_sparsemax "$@"

To launch the search followed by training and evaluation for the HDM05 datset:

python -u train_spd_search_hdm05_sparsemax.py --unrolled --epochs 300 "$@"
python -u train_spd_hdm05.py --arch hdm05_sparsemax "$@"

For the afew dataset we directly transfer the searched architectures from Radar and HDMO5 datasets. Launch the training and evaluation using the command below:

python -u train_spd_afew.py --arch radar_sparsemax "$@"
python -u train_spd_afew.py --arch hdm05_sparsemax "$@"

4. Results

Edge weight distribution and derived sparsemax architectures:

Performance comparison of our SPDNetNAS against existing SPDNets and traditional NAS on drone, action and emotion recognition

Ablation studies:

Convergence Analysis:

5. Contact

For any questions, suggestions, or issues with the code, please contact rheasukthanker@gmail.com and zzhiwu.huang@gmail.com.

7. How to Cite

If you find this project helpful, please consider citing us as follows:

@inproceedings{ijcai2021-413,
  title     = {Neural Architecture Search of SPD Manifold Networks},
  author    = {Sukthanker, Rhea Sanjay and Huang, Zhiwu and Kumar, Suryansh and Goron Endsjo, Erik and Wu, Yan and Van Gool, Luc},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Zhi-Hua Zhou},
  pages     = {3002--3009},
  year      = {2021},
  month     = {8},
  note      = {Main Track}
  doi       = {10.24963/ijcai.2021/413},
  url       = {https://doi.org/10.24963/ijcai.2021/413},
}